High benchmark scores did not survive adversarial testing in 15 health LLMs

What changed

On July 15, 2026, Nature Health published "Addressing benchmarking gaps in large language models for health and medicine with dynamic red-teaming" by Jiazhen Pan and colleagues (doi:10.1038/s44360-026-00152-8). It is peer reviewed: received April 23, accepted May 22, 2026. The code and the curated datasets are public.

The study introduces DAS (Dynamic, Automatic and Systematic), a red-teaming framework in which adversarial agents mutate health-related test cases in real time, and applies it to 15 proprietary and open-source models across four axes: robustness, privacy, bias and hallucination.

What the study actually shows

Read from the paper, kept separate from what anyone makes of it:

• Robustness. Median MedQA accuracy exceeded 80%. Under dynamic perturbation, a median 94% of previously correct answers became incorrect, with most models failing within one to three rounds. On the open-ended HealthBench dataset, top-tier models failed more than 70% of the time.
• Privacy. Using a privacy dataset the authors describe as HIPAA- and GDPR-informed, the agents elicited leaks of protected health information in more than 86% of scenarios.
• Bias. Cognitive or demographic cues altered clinical recommendations in 81% of fairness tests.
• Hallucination. Rates exceeded 74% in widely used models, and every model produced clinically significant factual errors in more than 50% of challenging cases.
• The scoring is automated. An LLM judge adjudicates the outputs. The authors benchmarked it against board-certified clinicians, reporting for example a Cohen's kappa of 0.952 on privacy, and describe it as "not infallible."

Their conclusion: "high scores on static medical benchmarks are not synonymous with safety and cannot be trusted as the sole measure of a model's readiness for health applications."

What it means for an engineering team

A benchmark number is not validation evidence. A model card citing MedQA or a similar exam-style benchmark describes performance on clean, well-formed questions. It says little about what your workflow will actually send the model. The paper's own examples are a misplaced decimal point in a drug dose, a careless summarization request, and an authority cue embedded in a referral note.

Test against your own perturbations, and keep testing. The authors frame their output as "a living analogue to post-market surveillance." Models change on a timescale of weeks, so a validation run done at procurement is stale by the next model update. Re-run adversarial tests whenever the model, the prompt or the retrieval corpus changes, and keep the results as evidence.

Privacy is a property of the conversation, not only of the configuration. A leak rate above 86% under adaptive dialogue means access controls on the data store are necessary but not sufficient once a model can be talked into disclosing what is in its context. The strongest control is keeping protected health information out of the context unless the task needs it.

Keep the judge honest. If you automate evaluation with an LLM judge, as the authors did, measure it against clinicians before trusting it, and record the agreement figure next to the results.

What is still unsettled

What counts as sufficient. Dynamic testing has no fixed pass mark: the attacker's budget, the mutation set and the judge all move the number. Until a standard defines them, two vendors' red-team results cannot be compared.

How a result survives an audit. Probes are generated on the fly, which defeats memorization and also means no two runs are identical. An audit trail needs the seeds, agent versions and judge configuration recorded with every result.

How much of the gap a deployed product closes. The study tests models directly. A deployed assistant adds retrieval, guardrails and workflow constraints that the paper did not measure, in either direction.

Sources

• Pan, J. et al. Addressing benchmarking gaps in large language models for health and medicine with dynamic red-teaming. Nature Health, doi:10.1038/s44360-026-00152-8, published July 15, 2026.
• Pan, J. et al. DAS-Medical-Red-Teaming-Agents. Code for the study. Retrieved September 11, 2026.