- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates the limitations of rubric-based evaluation in detecting clinically meaningful hallucinations in medical LLM responses. Overall, this paper is clearly motivated and addressed an important problem. The authors develop a clinician-validated hallucination taxonomy and a controlled error-injection pipeline to examine rubric sensitivity across several medical benchmarks. The results reveal substantial blind spots in existing rubrics and explore targeted criteria and retrieval-based factuality checking as potential improvements.
Critical strengths
The paper addresses an important and timely problem in medical AI evaluation. The controlled error-injection framework and clinician validation provide meaningful evidence of the limitations of existing rubric-based benchmarks. The experiments cover multiple datasets and hallucination types, and the findings offer useful insights into the relationship between rubric specificity and error detection. Overall, the paper is clearly written and the experimental analysis is relatively comprehensive.
Critical weaknesses
My main concern is the novelty relative to closely related work, particularly RubricRobustness (ICML 2026), which has already studied rubric sensitivity to controlled response perturbations on HealthBench. Although this work extends the analysis to clinically meaningful errors, a more thorough comparison would help establish its additional contributions. Authors mentioned this work in the paper briefly, but I think more detailed comparisons are needed.
Another concern is the scalability and reliability of the proposed evaluation approach. Clinical expert availability, annotation costs, and inter-reviewer disagreement remain important challenges. Furthermore, the proposed hallucination taxonomy relies on predefined error categories, which may introduce coverage limitations similar to those of existing rubrics, particularly when considering previously unanticipated errors. The experiment results also show that experts cannot agree with each other on some instances.
Questions for authors
Could the authors provide a more detailed comparison with RubricRobustness and clarify what new insights the proposed framework reveals beyond existing perturbation-based evaluation methods?
How could the proposed approach scale given the cost, limited availability, and potential disagreement of clinical experts?
Since the hallucination taxonomy also relies on predefined error categories, how would the framework identify clinically important errors that fall outside the existing taxonomy?
Paraphrase Control Experiment
Paraphrase Control Experiment
We again thank the reviewer for suggesting a paraphrase control, which measures how much a meaning-preserving rewording changes the score. Injected errors lower the score significantly more often than a meaning-preserving rewording of the same answer (30.7% vs 20.0% of pairs), and change the score by about twice as much on average. The rubric is therefore responding to the error itself, not just to the change in wording, though the win rates in the paper are likely an upper bound on detection. Injected answers sometimes score higher than the original, but no more often than a paraphrase does (8.6% vs 10.7%), and most of these cases disappear when the error is reworded. They therefore mostly reflect the rubric’s sensitivity to wording rather than a preference for incorrect content. Rewording the injected errors themselves gives results consistent with the main experiment (57.9% of reworded injected answers tie with the original, against 59.9% in the main o3 results), so the paper’s central finding, that most injected errors leave the score unchanged, does not depend on how each error was worded.
Experimental Detail
For each of 140 o3 pairs from our main experiment, we took the passage of the original answer that the injection had edited and reworded it without changing its meaning. This gives a version of the original answer that is changed in the same place as the injected answer but contains no error. Rewordings stayed within 15% of the original length, and were rejected by a separate LLM judge if they changed clinical content, certainty, urgency or detail. All the other text was left unchanged. We also took each injected answer and reworded the passage containing the error, keeping the same error with the same certainty and detail. This tests whether detection depends on how the error is worded. All answers were graded with the paper's original pipeline (GPT-4.1, temperature 0).
The table compares each answer's score with the original answer's, on the same 140 pairs (counts in brackets):
An injected error lowers the score significantly more often than a meaning-preserving paraphrase. Rewording without changing the meaning (first row) changes the score in 30.7% of pairs (20.0% lower, 10.7% higher), by 0.043 on average. This is more often than re-grading the same text (10%, Appendix E.6), but less often than injecting an error (39.3%). Injected answers (second row) score lower than the original in 30.7% of pairs, against 20.0% for paraphrases of the same answers (paired difference 10.7 percentage points, 95% CI [2.9, 18.6]), and change the score about twice as much on average (paired difference in mean absolute change 0.048, 95% CI [0.017, 0.083]). The injections themselves changed answer length by a median of 1 word (IQR 0 to 12), and length change was not significantly associated with score change (Spearman's ρ = 0.11, 95% CI [-0.06, 0.27]).
"Losses", where the error-injected answer scores higher than the original, are no more frequent than paraphrases scoring higher (8.6% vs 10.7%). When the error was reworded, most wins survived (34 of 43), but only 5 of 12 losses did, and most of the rest became ties (6 of the remaining 7). Losses are therefore stable for a given text but sensitive to its wording, which is why most replicated on re-grading. Because the rewording kept the error's certainty and detail, those side effects of the injection do not explain the losses. We read the losses as mostly reflecting sensitivity to wording, not a preference for incorrect content.
Rewording the error (third row) produced win/tie/loss rates of 34.3/57.9/7.9%, consistent with the main o3 results (30.8/59.9/9.3%), and no detectable change in aggregate rates or AUROC against the original injections. The reported aggregate rates in the paper therefore do not depend on how each error was worded. Pooling o3 and Kimi-K3 answers gives the same pattern.
We will add this control to the paper, using the additional page in the camera-ready version. The main win and loss rates will be reported alongside the paraphrase rates, so readers can compare them directly.