acceptodds
Under review as a conference paper at ICLR 2027

MedEviBench: Evaluating Large Language Models for Evidence-Based Medicine

Abstract

Medical question answering benchmarks measure what language models know, but provide a limited view of how well they use evidence to answer questions from clinical practice. Evaluating this ability requires questions that reflect physicians' information needs and measures that assess the evidence supporting a clinical answer. We introduce MedEviBench, a Chinese-language benchmark of 240 physician questions spanning 12 medical specialties and ten clinical intents. Given a clinical question and reference passages, a model must select relevant evidence and produce an answer with citations supporting its conclusions. We evaluate these answers using nine criteria grouped into evidence quality, clinical content, and answer presentation. Blinded physician comparisons on questions sampled from the benchmark yield a Spearman correlation of 0.80 between automatic and physician system rankings. We evaluate 18 LLMs, including 13 general-purpose and five medical models. Our results show that producing reliable clinical answers remains difficult even when reference evidence is provided. Among the general-purpose models, clinical content scores range from 47.31 to 82.96 out of 100, falling below evidence quality for 12 of the 13 models and presentation for all models. These differences are not consistently reflected in medical knowledge scores: an exploratory comparison across 12 models finds a moderate correlation between AA-Omniscience Health and clinical content, but almost no linear correlation with evidence quality. To examine where models fail, we analyze judge explanations for zero-scored criteria and find recurring errors in clinical conclusions and inferences that exceed the available evidence. MedEviBench tests whether models can turn reference evidence into sound clinical answers, making these failures part of the assessment of medical language models.

Primary area:
datasets and benchmarks
Keywords:
Large Language Models, Evidence-Based Medicine, Medical Question Answering, Evidence Grounding, Benchmarking and Evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_FscwOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents the Chinese evidence-based medicine benchmark MedEviBench, which includes 240 doctor questions covering 12 medical specialties and ten clinical intents. Given clinical questions and reference paragraphs, the model needs to select relevant evidence and provide cited answers. The benchmark is scored using nine weighted criteria, which are divided into three parts: evidence quality, clinical content, and answer presentation. The authors evaluated 18 models, used blind doctors to compare and verify the consistency between automatic scoring and doctor ranking, and analyzed the main failure modes of the models.

Critical strengths
  1. The direction of the paper is a real but rarely explored topic – whether a model can answer with evidence rather than merely provide the correct answer, which is of great significance to evidence-based medicine.

  2. There is a blind review by doctors for verification. The automatic ranking and the doctor ranking are generally consistent, indicating that this benchmark has a certain connection with clinical judgment.

Critical weaknesses
  1. Although the overall ranking was consistent, when it came to each specific question, the differences among doctors were substantial, with a Krippendorff's alpha of 0.0402. Moreover, 38.16% of the comparison items had opposite votes. The paper also says that agreement of the overall ranking and disagreement on single comparisons exist at the same time. Model scores are added up from single question scores, so if single comparisons are this noisy, the small gaps between models may not be reliable.

  2. The robustness check only covers two systems, but there are 18 main experiments. Moreover, there is overlap between the GPT series models used for scoring and the GPT series being evaluated, which may introduce self-preference bias.

Questions for authors
  1. How do you plan to deal with the large disagreement on single comparisons?

  2. Does the judge prefer GPT models, and how do control for it?

  3. Can the robustness checks covers more systems, especially non-GPT and medical models?

Reviewer_YoQ2OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The authors propose MedEviBench, a Chinese-language benchmark for clinical question answering consisting of 240 questions. The novelty is that each question is paired with supporting evidence, and models must generate answers with references to this evidence. An LLM judge evaluates nine criteria covering evidence quality, clinical content, and presentation, with expert-assigned aggregation weights. The authors evaluate 18 models on this benchmark and analyze errors made by the models.

Critical strengths
  • The paper tackles a useful problem.
  • The authors test a wide array of open source and API-based models.
  • The authors conducted a physician evaluation of their LLM-judge system.
Critical weaknesses
  1. The dataset does not contain any ground truths, which is concerning for the quality of the evaluation. The only source of external supervision appears to be the checklist for completeness (one of the nine scores). The authors provide very little detail on how these checklists are generated. Everything else is evaluated with an LLM judge. This is particularly concerning for evaluations of accuracy and evidence quality, since it is unclear why evaluation of an answer is any easier than generating an answer in the first place.

  2. The authors state that relevance is only evaluated with respect to the title of the references, which does not seem sufficient to assess that a reference sufficiently justifies a claim.

  3. I am not convinced that the physician study sufficiently validates the evaluator. The reported correlation of 0.80 is over only four systems, which are separate from the 18 evaluated models. Further, the appendix reports an inter-rater agreement (Krippendorff's alpha) of only 0.0402, and so it seems like even physicians agree slightly more than chance of this task.

  4. For the evidence that is supplied with each question, the authors state that passages are drawn from an “internal medical knowledge base” via an “internal retrieval pipeline. However, there is no verification of this pipeline. How do we know that there is adequate evidence to answer each question at all? This might be responsible for some of the "unsupported inference" errors the authors observe.

  5. The authors should provide additional details on some basic descriptive statistics of the dataset, as well as some additional details on dataset construction. For example, the length of the questions, how many references per question, the distribution of reference source types, and more details on the checklists.

  6. The dataset only contains 240 examples, which is rather small compared to other clinical diagnosis datasets. The authors are able to establish small confidence intervals in Figure 5, but I would be concerned about whether there is enough statistical power to distinguish between models that are close in capability.

Questions for authors

Please address the weaknesses above.

Reviewer_ktFYOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces MedEviBench, a benchmark of 240 Chinese clinician questions across 12 specialties. The benchmark include nine criteria assessing evidence quality, clinical content, and presentation through LLM judgments and predefined rubrics. The authors evaluate 18 models and report gaps in clinical accuracy and completeness. The results show that producing reliable clinical answers remains difficult, even when the reference evidence is provided. Meanwhile, these differences are not consistently reflected by medical knowledge scores, demonstrated by an additional external evaluation on the AA Omniscience Health Benchmark.

Critical strengths
  1. The benchmark addresses a useful setting for evidence-based medicine. Providing the same passages to all models and also introduce a benchmark with physician reference is useful for the community.
  2. Reporting evidence, clinical content, and presentation separately helps distinguish shortcomings that an overall score would obscure. The physician study and judge robustness checks provide useful initial validation.
Critical weaknesses
  1. I am not yet convinced the benchmark adds much beyond the existing evaluations discussed in the paper. HealthBench already examines clinical content and communication, but Table 1’s “EBM eval.” column does not make this overlap clear. With 240 questions and 20 per specialty, the paper also offers limited evidence for broader conclusions.
  • The evidence scores do not directly check whether cited passages support the answer: relevance uses titles and metadata, and traceability checks whether citations are present. The physician study validates overall preferences for four separate systems, rather than individual scoring decisions. The multi-judge check also covers only two GPT generators, leaving potential bias in comparisons with non-GPT models unresolved.
Questions for authors
  1. Have clinicians checked a sample of the main experiment’s evidence-use judgments? Do you have results from different judges on non-GPT model answers?
  2. Could you share the filtering prompt, retrieval details, data-release plan, and a complete example from input passages through to the final score?
  3. Could differences in the scoring rules explain some of the gap between clinical-content and presentation scores? How should readers interpret comparisons across these dimensions?
Reviewer_NjRDOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces MedEviBench, a Chinese-language benchmark comprising 240 real-world clinical questions collected from practicing physicians to evaluate large language models (LLMs) in evidence-based medicine. The authors develop a comprehensive evaluation framework consisting of nine automated criteria across three dimensions: evidence quality, clinical content quality, and answer presentation. The study systematically evaluates 18 LLMs, including general-purpose and medical-domain models, revealing important limitations in clinical reasoning, evidence utilization, and the reliability of evidence-supported conclusions.

Critical strengths
  1. Strong clinical relevance. The benchmark is grounded in real-world clinical needs, with questions collected from practicing physicians across 12 medical specialties and 10 clinical intents. This design enhances its practical relevance compared with benchmarks primarily based on examination-style questions.
  2. Extensive empirical analysis and actionable insights. The evaluation of 18 LLMs, together with a detailed analysis of failure modes such as unsupported clinical inferences, provides valuable insights into current model limitations and identifies concrete directions for improving evidence-grounded medical AI systems.
Critical weaknesses
  1. Limited benchmark scale. The benchmark contains only 240 clinical questions. Although these questions are derived from real-world practice, the relatively small sample size may limit its coverage of complex, heterogeneous, and long-tail clinical scenarios.
  2. Limited linguistic diversity. MedEviBench focuses exclusively on Chinese-language clinical questions, which may constrain the generalizability of its findings to multilingual and international healthcare settings. Further cross-lingual validation would strengthen its broader applicability.
Questions for authors

Do the authors plan to extend MedEviBench to English or other languages? If so, what challenges do they anticipate in aligning clinical guidelines, evidence standards, and medical terminology across different linguistic and healthcare contexts?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.