A 3-cent model topped 13 LLMs on Stanford's medical hallucination test

What I tested

Every team shipping a healthcare chatbot runs into the same problem: the model sometimes states things the source never said. The usual fix is a second LLM call that checks the first one. It works, but it doubles your latency and your bill, on every single message.

So I tested a different kind of model on that exact job, against numbers I did not produce myself. Stanford's MedHELM project evaluates LLMs on medical tasks and publishes everything, including each model's answer to each question. One of its tasks is MedHallu: given a PubMed abstract, a question and a proposed answer, decide whether the answer is faithful or hallucinated. The test set has 1,000 items, 520 faithful and 480 hallucinated, and 13 LLMs have been scored on it, from GPT-4o mini to GPT-5 and Claude 3.5 Sonnet.

Because Stanford publishes per-item predictions, I did not need to rerun any of those models. I downloaded their answers, checked that I could reproduce all 13 published accuracies exactly, and added one row. The table is mine, built from Stanford's published predictions: Stanford did not run or review the new row.

That row is Jev 1.13 from TypeSafe. Jev is not a chat model. You send it text plus a typed question, and it returns a probability instead of prose. There is no prompt in the usual sense: the request is a small JSON object. TypeSafe calls this a "System One" model, built for fast judgments rather than reasoning.

What the results actually show

Accuracy, median time per check and cost per 1,000 checks, on the same 1,000 items:

• Jev 1.13: 92.9%, 204 ms, USD 0.03
• Claude 3.5 Sonnet: 92.6%, 2.7 s, about USD 2.16
• Gemini 2.0 Flash: 90.8%, 0.3 s, about USD 0.11
• GPT-5: 90.5%, 3.1 s, at least USD 0.87
• o3-mini: 89.6%, 4.0 s, at least USD 0.78
• GPT-4o: 84.6%, 0.3 s, about USD 3.52
• GPT-4o mini: 80.1%, 0.4 s, about USD 0.11

The full table has 14 rows. Compared item by item (exact McNemar test, Holm-corrected), Jev is statistically level with Claude 3.5 Sonnet, borderline ahead of Gemini 2.0 Flash, and significantly ahead of GPT-5 and the ten models below it. The whole Jev run, all 1,000 checks, cost 3 cents.

Two more things an LLM judge does not give you. When Jev flags an answer it is right 93.8% of the time, and it catches 91.2% of the hallucinations. And because it returns a probability (AUROC 0.974), you choose the operating point: block above one threshold, send to human review in the middle, pass below.

What I learned about prompting it

My first attempt translated Stanford's LLM prompt into Jev's format field by field: the role, the four hallucination types, the definitions. That scored 90.3%, already level with GPT-5.

Then I tried a dozen alternatives on separate development data: detailed criteria, lists of warning signs, splitting the judgment into sub-questions and averaging them. All of them lost to a single plain sentence:

"Would the authors of the study say that this answer misrepresents their findings?"

That question scored 92.9%. The instinct we have built with LLMs, to add more instructions, was the wrong instinct here. The model wanted a direct, literal question.

What it means for an engineering team

A large share of what we pay frontier models to do in production is not generation at all. It is judgment. Is this answer supported? Is this message urgent? Does this reply give medical advice? For that kind of call, on this benchmark, a specialized model matched the best published LLM at around one seventieth of the cost and returned in a fifth of a second.

At that price the check stops being a budget decision. You can run it on every message, not a sample, and keep the large model for the work only it can do.

A probability also turns the guardrail into a policy you can write down: which scores block, which go to a person, which pass. Pick those thresholds on your own data, and measure each judgment step in your pipeline separately before you decide which model does it.

What is still unsettled

I want these numbers to survive scrutiny, so here are their limits.

• The LLMs were not tuned. Stanford ran one generic zero-shot prompt across all 13 models. Jev's question and threshold were chosen on 1,000 separate development items, then run once on the test set. That is why I report both Jev results: 90.3% with the LLM-style prompt, 92.9% with the question written for it. Choices were written down before each run.
• Time is not apples to apples. Stanford's timings come from their own model deployments. Jev's were measured from my own machine through OpenRouter. The gap is large enough to be meaningful, but do not read it as a race.
• LLM costs are estimates: Stanford's published token counts multiplied by LiteLLM list prices. For reasoning models it is a floor, since hidden reasoning tokens are billed but not reported.
• The 13 models date from 2024 to mid 2025. Newer models would likely score higher.
• The test data is synthetic and narrow. The hallucinated answers in MedHallu were generated by a model, not collected from real failures. And this is one task. Jev cannot explain its verdict, write text, do arithmetic or reason through several steps. On tasks like those, the LLMs win.

The harness, the run files and the run plans written before each run are reproducible end to end. If you want to check the numbers or add your own model as another row, get in touch.

Sources

• Stanford CRFM. MedHELM leaderboard. MedHELM v4.0.0 public results, per-item predictions. Retrieved September 21, 2026.
• Bedi, S., Cui, H., Fuentes, M. et al. MedHELM: Holistic Evaluation of Large Language Models for Medical Tasks. arXiv:2505.23802, May 26, 2025.
• Pandit, S., Xu, J., Hong, J. et al. MedHallu: A Comprehensive Benchmark for Detecting Medical Hallucinations in Large Language Models. arXiv:2502.14302, February 20, 2025.
• UT Austin AI Health. MedHallu dataset. Hugging Face, MIT license.
• BerriAI. LiteLLM model prices and context window. Retrieved September 21, 2026.
• TypeSafe AI. TypeSafe AI. Company site, Jev model.
• OpenRouter. Jev 1.13: API pricing and providers. Model typesafe/jev-1.13-20260917.