acceptodds
Withdrawn from ICLR 2027

HealthCoreBench: Measuring Evaluation Artifacts for Reliable Medical LLM Evaluation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Medical evaluation of large language models (LLMs) increasingly relies on heterogeneous benchmarks, yet benchmark failure is not always a reliable indicator of limited medical intelligence: unsolved samples may reflect genuine capability gaps or evaluation artifacts such as ambiguity, incorrect references, or outdated knowledge. We formulate medical benchmark construction as a **failure attribution** problem: identifying samples where failures can be meaningfully attributed to capability limitations. We introduce **HealthCoreBench**, a unified medical evaluation benchmark distilled from **47** language benchmark sources spanning **165,120** samples, of which only **13,987** (**8.47%**) are retained. HealthCoreBench uses progressive model-based filtering to identify frontier-challenging samples, followed by clinician validation to flag and correct unreliable failures. Auditing large-scale medical benchmark collections reveals that: 1) unreliable failures are prevalent, and difficulty filtering alone can even enrich evaluation artifacts, since selecting samples that strong models fail on preferentially retains defective rather than genuinely hard questions; 2) these artifacts create false successes and shift model scores unevenly, with physician validation alone changing overall scores by up to **5.18** points across models and distorting apparent medical capability; and 3) retaining only clinician-validated failures preserves model rankings while cutting average token consumption by **77.17%** (from **82.82M** to **18.91M** tokens). Overall, unreliable failures undermine evaluation validity, and failure attribution offers a path forward. We release HealthCoreBench to shift medical evaluation from “unsolved as failure” toward “failure attributable to capability limitations.”

Primary area:
datasets and benchmarks
Keywords:
Large Language Models, Benchmarks, Medical Evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_oHgCOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper propose a benchmark construction pipeline through progressive difficulty filtering followed by physician validation. Applying the pipeline on 47 medical benchmarks results in HealthCoreBench, which contains 13987 samples. Several LLMs are then evaluated on HealthCoreBench.

Critical strengths
  • The paper focus on an important problem: the quality of benchmark is crucial for reliable LLM evaluation.
Critical weaknesses
  • The novelty of the paper is limited. Difficulty filtering and expert validation are common strategies for constructing benchmarks, e.g. MMLU-Pro [1]. Though the authors try to formulate the pipeline as failure attribution, it is actually achieved through manual review. The authors should clarify the contribution beyond existing filtering and validation practices.
  • The filtering process may introduce bias to the resulting benchmark. As only the most difficulty questions are preserved, the pipeline may change the distribution of the benchmark. The performance on HealthCoreBench could not reflect LLM's mastery of other clinical scenario. Moreover, questions that are filtered out by frontier proprietary models may still be difficult for less capable models. Therefore, more detailed analysis on the coverage would be valuable.

[1] MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark, NeurIPS 2024

Questions for authors
  • The purpose of progressive filtering could be better explained. Why not just filtering with most strong models in tier 5? How many samples are solved by smaller models but are difficult for frontier closed-source models?
  • The system or model used for automated pre-screening is not specified.
  • Combining temperature=0 with reasoning mode may lead to performance decrease. Would using each model's suggested sampling parameters change the result?
Reviewer_cPkbOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether failures on medical benchmarks reflect genuine model capability limitations or evaluation artifacts. The authors aggregate 165,120 samples from 47 existing medical benchmarks, apply five stages of progressively stronger model-based filtering, and then use automated auditing and physician review to construct HealthCoreBench with 13,987 retained samples. The paper reports that difficulty filtering increases the estimated defect rate from 1.36% to 4.20%, physician validation changes model scores by up to 5.18 points, and the final subset reduces average token consumption by 77.17% while preserving the ranking of three evaluated models.

Critical strengths

1.The paper addresses an important evaluation problem: a difficult medical question is not necessarily a valid question, and model errors caused by ambiguous questions, incorrect references, missing evidence, or outdated information should not be interpreted as medical capability failures.

2.The benchmark construction effort is substantial. The paper combines a large collection of 47 source benchmarks with multi-tier model filtering, an explicit taxonomy of evaluation defects, automated pre-screening, and physician review. The representative error cases also make the motivation concrete and easy to understand.

3.The observation that model-based difficulty filtering may enrich defective items is potentially useful for future benchmark construction, since it challenges the common practice of retaining samples simply because strong models fail on them.

Critical weaknesses

1.The benchmark contribution is not sufficiently differentiated from prior validity-aware hard-subset curation. Although the paper frames benchmark construction as “failure attribution,” its operational pipeline still consists of selecting items that remain difficult for strong models, auditing potential defects, and validating them with experts. The paper does not yet establish what new evaluation construct or benchmark capability is introduced beyond applying this workflow at substantial scale in the medical domain.

2.The central finding that difficulty filtering enriches evaluation artifacts is not sufficiently isolated from changes in benchmark composition. Because the mixture of source benchmarks, languages, and task formats changes across filtering tiers, the increasing defect rate may partly reflect dataset-composition shifts rather than a general relationship between difficulty and evaluation invalidity.

3.The claimed efficiency–fidelity advantage is under-validated. Ranking preservation is demonstrated for only three models and is not compared against random, source-stratified, difficulty-only, or informative subset-selection baselines. The current experiments therefore do not establish that the proposed pipeline preserves model comparisons more faithfully than simpler alternatives.

4.The full construction pipeline is not currently reproducible because the submission does not provide executable code or sufficient intermediate artifacts for the filtering, auditing, correction, and re-evaluation stages.

Questions for authors

What new evaluation construct or scientific conclusion does the failure-attribution formulation enable beyond standard hard-example selection followed by expert validity review?

Does artifact enrichment persist within source benchmarks and after controlling for source, language, task format, modality, and capability category?

At a matched sample size or inference budget, how does HealthCoreBench compare with random, source-stratified, IRT-based, and other informative subset-selection baselines in rank fidelity and model discrimination?

Will the authors release the complete construction and evaluation code, together with the intermediate annotations and provenance required to reproduce the final benchmark?

Reviewer_ob3bOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper looks at developing a healthcare benchmark based on model failures. However, the authors correctly identify that model failure can be the result of genuine capability failure or an artifact failure. They develop a filtering method to identify which questions uniformly fail across models then use clinicians to validate the type of failure. They present 13,987physician-validated samples drawn from 47 source benchmarks.

Critical strengths

It is interesting to look at the edge cases where models are getting answers wrong and identify WHY in order to have a better sense of how models perform at the edge. Looking at this in a deeper way is useful for the community at large.

Critical weaknesses

-Q and A datasets don't provide strong signal for performance on real world medical tasks. This hurts the perceived benefit of this benchmark because Q&A doesn't have ecological validity for medicine.

-The background of physician raters is important. Can you please describe what kinds of physicians were used in evaluating the questions? What was their experience, where do they practice?

-Another problem is that these come from public benchmarks, and as we know, there has been a lot of saturation around public benchmarks. There is a big concern that models have memorized much of the data in these datasets. It would have been more valuable to develop a novel dataset with these same issues and to test that out.

Questions for authors

-How did you evaluate across languages with physicians? How did you recruit physicians?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.