© 2026 Medoya LLC
Powered by voilamia
Subscribe to receive new insights
Medoya Labs
jev-medhallu-benchmark: an open harness that scores Jev 1.13, a model that answers with a probability, against fast LLMs on the same medical hallucination test, for accuracy, latency and cost. Python, September 2026.
A benchmark harness for Jev 1.13 from TypeSafe, a "System One" model that answers typed questions about text with a probability instead of generated text. It runs Jev and any model on OpenRouter on MedHallu, where each item is a PubMed abstract, a question and a proposed answer to judge as faithful or hallucinated. Stanford's MedHELM publishes the per-item answers of the 13 LLMs it tested, so every new model lands as another column in the same table.
The results are in the article A fast hallucination check that knows when it's unsure.
One runner, one gateway. run_bench.py calls every model, Jev included, through OpenRouter, so one key pays for everything and every latency includes the same network hop. LLMs answer through a strict JSON schema, so like Jev they can only return one of the task's labels.
One scorer. score.py reports accuracy with 95% intervals, paired significance tests against a reference model, median and 95th-percentile latency, and billed cost per check, as Markdown, CSV and a spreadsheet.
Plans written first. Every test run and follow-up check has a written plan committed before it ran, and every run file is kept, including the ones that went against Jev.
Checks on the benchmark itself. Follow-up checks test the test: removing the abstract to see how much a model can get right on wording alone, sending only Jev's uncertain cases to an LLM, and timing open models on fast inference hardware.
More than one dataset. A second, small dataset (29 synthetic clinical notes coded to ICD-10) checks that the harness works beyond MedHallu, and the repo ships a Claude Code skill for building with Jev: calling it, writing questions, and gating on its probability.
Re-scoring the published runs needs no API key. With uv installed, from the bench folder:
uv run datasets/medhallu/prepare.pyuv run score.py runs/medhallu/test-v2 --threshold 0.65 --ref jev-1.13 --xlsxTo run models yourself, copy bench/.env.example to bench/.env, add an OpenRouter key and follow bench/README.md. A full Jev pass over the 1,000 test items costs about 3 cents.
A check that runs before every answer is a speed and cost decision as much as an accuracy one, and a published leaderboard rarely answers it for your workload. This harness is how I approach that choice: write the plan before the run, measure speed and cost beside accuracy, and publish the results that go against the model under test.
Python, September 2026. Open source on GitHub, MIT license. Medoya has no relationship with TypeSafe.
Book a free 30-minute call. Tell me which AI tools your team uses today and one workflow you'd like an agent to take on. You'll leave knowing the three biggest things in your way and a first step for each.