- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper introduces MIRAGE (medical inference representation analysis of geometry and errors), a framework that the authors propose for testing what constitutes a correct and an incorrect LLM answer, and what differences between correct and incorrect answers can reveal about a medical language model’s errors. The authors highlight three claims as signs that can be used to tell when a model "knows when it is wrong": (i) geometry: different embeddings of correct and incorrect answers; (ii) prediction: classifiers trained on the model's internal states may be able to predict if a model will be wrong; and (iii) intervention: input/ internal state alterations can change the model's answer. The MIRAGE framework tests each claim separately on the same set of questions and responses to show what each type of evidence can and cannot support. It is applied on five small open medical models and four benchmarks (two text MCQ and two radiology VQA datasets); each dataset contributes 400 evaluation questions and 8 calibration questions. Each model answers each question 32 times, producing about 209,000 responses in total, and a response is correct only if its parsed answer matches the benchmark's answer key.
In the application of the framework, for the geometry claim, the authors embed biomedical and a general sentence encoder and compare how spread out the correct and incorrect responses are within each question. Only 12.6% of question–model pairs (808 of 6,400) have enough of both to be analyzed. Within this subset, incorrect responses are more spread out in about two-thirds of cases, though some incorrect answers cluster tightly. As for prediction, the authors train simple linear classifiers on the models’ internal states using only the 8 calibration questions. These predict errors reasonably well on the visual datasets but mostly at chance on text, and on the visual datasets a baseline that only knows whether a question is open-ended or yes/no performs nearly as well. Lastly, for interventions, they corrupt the input, paste the internal state from the uncorrupted run back in, and measure the probability of the correct answer against sham and random controls.
The authors conclude that interpreting these signals is limited by three obstacles: geometric comparisons cover only a small subset of questions, the signals depend heavily on answer content and question format, and changing a model’s internal scores does not mean its answers are repaired. They present MIRAGE as a reproducible framework for testing such claims while keeping benchmark agreement separate from clinical validity.
Critical strengths
-
The paper's methodology is well-reported, with each of the 3 claims tested with a thorough set of controls (geometry reported with its coverage, interventions compared against both a sham control and a random change of the same size). The authors also disclose a bug in their embedding setup that gave identical texts different embeddings, which increases trust in findings and makes the study easier to reproduce.
-
The study produces a large 209,000 set of responses with their parsed answers, and embeddings. This opens the possibility for future research on error-detection methods.
Critical weaknesses
-
The paper's novelty is limited, with MIRAGE simply combining existing tools and adding controls to each (APORIA geometry analysis, linear probes, activation patching). This is further exacerbated by the fact that the main conclusions of the paper confirm limitations already established in earlier work. In other words, the paper proposes no new method for detecting or correcting errors and offers no positive result that future work can build on.
-
Some analyses are too small: (i) probes are trained on only 8 calibration questions per dataset; and (ii) geometry covers only 12.6% of question–model pairs. As such, lots of the negative findings could potentially be fue to a lack of data. Although the authors acknowledge this, it still is a weakness since those negative findings are a core contribution of the paper.
-
There is no expert review, so “correct” and “incorrect” only mean agreement with the benchmark’s answer key. This is especially problematic for open-ended visual questions, which are scored by exact match after light text normalization.
Questions for authors
-
What does MIRAGE show that was not already known from prior work on probing and activation patching?
-
How do you think the probe and intervention conclusions would change with substantially more data?
-
How many of the responses marked incorrect on open-ended visual questions are actually clinically equivalent to the reference answer? Having a clinician review a random sample, or rerunning the visual analyses with more lenient answer matching, would show whether the question-type effect and the error labels are driven by exact-match scoring.
5 comments
Response to Reviewer CJrq (Part 2)
... (continued) ...
We agree that a reader should leave with results to build on, and several MIRAGE results are positive in that sense. They are listed below with the use we see for each.
Table R2. Positive results and how they can be used.
| Result | Measurement | Use |
|---|---|---|
| A stable dispersion asymmetry on questions with mixed answers | 545 of 808 eligible pairs (67.5%); 0.630 to 0.692 across two encoders, three metrics, and two views; unchanged by answer masking; 67.2% in six general models | A measured prior for consistency-based detectors, with known coverage |
| Probe signal beyond answer type | Gain over answer type on SLAKE of 0.058 to 0.097 with held-out intervals above zero; within-open AUC 0.596 to 0.731 under clinician-validated labels | A target and a stratified protocol for prompt-time detectors |
| Output-consistency features as the baseline to beat | Add 0.155 AUC beyond answer type and probe scores; up to 0.958 AUC in recent general models | A low-cost baseline that internal-state detectors should be required to exceed |
| Localized clinical evidence that changes behavior | Shallow evidence-token patches switch sampled answers (MedGemma-4B laterality 1.00, negation 0.50; BioMistral laterality +0.354 over random); no effect at deeper layers or the final prompt; dose arithmetic not localized | Where and when to intervene, and which relations are not carried by a single token |
| Exact matching misses most clinically equivalent open answers | Exact match recovers 29.6% of answers clinicians judge equivalent; a validated lenient matcher agrees with clinicians on 84.5% | Relabeling protocol for open-ended medical VQA evaluation |
| An encoder defect and its fix | Identical texts differed by up to 1.323 in L2 distance under dynamic-padding attention; the corrected setting agrees within 2.2e-6 | A reproducibility check for embedding-geometry studies |
| A linked resource | 208,896 responses with parsed answers, corrected embeddings, internal states, intervention records, clinician ratings, seeds, and hashes | Training and evaluating error detectors with predeclared splits |
W2. Whether the analyses are large enough
Response: Thank you for your concern that some negative findings may reflect too little data. This needed to be tested rather than argued, and we tested the two analyses named. For the probes, the question is whether the weak results with eight calibration questions persist when more labels are available. We refit every probe on 20 group-disjoint subsets at budgets up to 240 questions for text and up to 96 and 32 image groups for VQA-RAD and SLAKE, and evaluated each fit on a fixed held-out set that never enters fitting.
Table R3. Prompt-only probe AUC on a fixed held-out set.
| Model | Dataset | 8 questions | Largest budget | Gain over answer type at largest budget [held-out CI] |
|---|---|---|---|---|
| BioMistral-7B | MedMCQA | 0.587 | 0.678 | 0.178 [0.086, 0.268] |
| BioMistral-7B | MedQA | 0.547 | 0.647 | 0.147 [0.067, 0.224] |
| OpenBioLLM-8B | MedMCQA | 0.549 | 0.578 | 0.078 [-0.013, 0.176] |
| OpenBioLLM-8B | MedQA | 0.543 | 0.692 | 0.192 [0.104, 0.277] |
| MedGemma-4B | MedMCQA | 0.507 | 0.586 | 0.086 [-0.005, 0.176] |
| MedGemma-4B | MedQA | 0.516 | 0.548 | 0.048 [-0.040, 0.131] |
| MedGemma-1.5-4B | MedMCQA | 0.566 | 0.695 | 0.195 [0.116, 0.276] |
| MedGemma-1.5-4B | MedQA | 0.500 | 0.591 | 0.091 [-0.004, 0.182] |
| Lingshu-7B | MedMCQA | 0.528 | 0.615 | 0.115 [0.032, 0.207] |
| Lingshu-7B | MedQA | 0.509 | 0.602 | 0.102 [0.010, 0.187] |
| MedGemma-4B | SLAKE | 0.786 | 0.804 | 0.097 [0.020, 0.178] |
| MedGemma-1.5-4B | SLAKE | 0.768 | 0.807 | 0.058 [0.009, 0.112] |
| Lingshu-7B | SLAKE | 0.835 | 0.871 | 0.080 [0.021, 0.147] |
| MedGemma-4B | VQA-RAD | 0.715 | 0.805 | 0.033 [-0.031, 0.096] |
| MedGemma-1.5-4B | VQA-RAD | 0.708 | 0.778 | 0.045 [-0.018, 0.112] |
| Lingshu-7B | VQA-RAD | 0.770 | 0.818 | -0.011 [-0.057, 0.040] |
The review's intuition holds for text, and the result is informative. The text signal is weak with 8 questions but grows with labels, and with 240 questions it exceeds chance with held-out intervals above zero in 6 of 10 cells (for text, answer type is a single category, so the gain equals AUC minus 0.5). The text result is therefore label-limited rather than absent, and output-consistency features reach comparable accuracy with far fewer labels (0.656 with 8 questions for BioMistral MedMCQA, against 0.678 for the probe with 240). On SLAKE, probes exceed answer type with held-out intervals above zero for all three checkpoints. Low-rank probes, shrinkage discriminants, and stronger regularization change 8-question AUCs by a few points without altering any conclusion. Section 6.2 now presents the 8-question results as the low-label end of these curves.
Response to Reviewer CJrq (Part 3)
For geometry, the question is whether the 12.6% coverage would rise with more data. A pair is eligible when at least five of its 32 responses are correct and at least five incorrect, and most pairs fail this because the model answers consistently: 42.1% of pairs are answered incorrectly in all 32 samples and 36.6% correctly in all 32. Adding questions does not change the rate (0.157 with 10 and with 400 MedMCQA question groups), and lowering the threshold to two raises coverage to 18.7% with the dispersion asymmetry unchanged (0.670 to 0.679). Adding samples per question does raise coverage for models with varied outputs (BioMistral MedMCQA: 0.56, 0.75, and 0.83 of questions eligible at 32, 64, and 128 samples) while the asymmetry stays between 0.760 and 0.843. In auditing decoding for this response, we also found that our pipeline set only temperature and top-p, and that Lingshu-7B's shipped configuration sets top_k = 1, which made its samples effectively greedy and explains its low coverage. Its base model with the same setting reproduces that coverage, and with top_k = 50 the base reaches 28.7% to 52.0%. Lingshu's prompt-only probe inputs are computed before sampling and are unaffected, and the revised paper states each checkpoint's effective decoding settings. Coverage is therefore set by the response distribution, and the confidently wrong 42.1% is a deployment result in its own right: no within-question geometry or self-consistency method can flag these questions.
W3. Expert review and exact-match scoring of open-ended answers
Response: Thank you for this concern. The paper treats benchmark agreement as distinct from clinical correctness, and benchmark agreement remains the appropriate target for auditing claims that are themselves evaluated against benchmark keys. The review's point is nonetheless important, because exact matching could distort the open-ended visual labels on which part of the analysis rests. Text multiple-choice items are scored by option letter and cover 62.5% of evaluation pairs and 74.0% of eligible geometry pairs; closed yes/no items are also unambiguous. Only open-ended visual labels can change, so we measured how much they change and whether the conclusions depend on them.
Three clinicians rated a stratified sample of 239 open-ended answers, blinded to the model and to the exact-match label: 199 that exact matching scored as incorrect and 40 that it scored as correct, balanced across MedGemma-4B, MedGemma-1.5-4B, Lingshu-7B, and both datasets. Labels were assigned by majority of the three raters, and agreement was very high (Fleiss kappa 0.976; pairwise Cohen kappa 0.964 to 0.986). (One sample was left unrated by all raters and is excluded ().)
Table R4. Clinician review of open-ended answers.
| Exact-match label | n | Clinically equivalent | Partially equivalent | Not equivalent | Unanswerable | Share equivalent [Wilson 95% CI] |
|---|---|---|---|---|---|---|
| Incorrect | 199 | 95 | 37 | 64 | 3 | 0.477 [0.409, 0.547] |
| Correct | 40 | 40 | 0 | 0 | 0 | 1.000 [0.912, 1.000] |
Nearly half of the exact-match errors are clinically equivalent to the reference, and exact matching produced no false-correct labels. We then compared automatic matchers with the clinician consensus. Exact matching agrees on 60.3% of answers and recovers only 29.6% of clinically equivalent ones. A lenient matcher that accepts an answer when it matches after normalization or when Qwen3.5-9B, given a fixed rubric, judges it equivalent agrees with the clinicians on 84.5% of answers (precision 0.86, recall 0.87). We re-ran every visual analysis with these clinician-validated labels.
Response to Reviewer CJrq (Part 1)
We thank Reviewer CJrq for a careful and accurate summary of MIRAGE, and for highlighting the reporting choices we consider central to the work: geometry reported together with its coverage, interventions compared against both a sham and a norm-matched random control, the disclosure of the embedding defect that gave identical texts different vectors, and the 208,896-response resource.
The review raises three questions: what MIRAGE adds and what positive results it offers, whether some analyses are too small to support their conclusions, and how much the open-ended visual labels depend on exact-match scoring. We answered the last two with new experiments, including a blinded review of open-ended answers by three clinicians and calibration budgets up to 30 times larger, and we revised the paper's presentation. Intervals are 95% bootstrap intervals over question or image groups with 1,000 replicates. We have also highlighted all changes in the revised manuscript in blue.
Response to Weaknesses
W1. Novelty and positive results
Response: Thank you for the concern. We acknowledge that some of the tools in MIRAGE are established, and we do not claim otherwise. What MIRAGE adds is a single frozen cohort in which geometric, predictive, and interventional evidence are recorded for the same questions, so that the evidence types can be tested against one another rather than reported separately. This matters because each type of claim is currently made in isolation: APORIA (ICML 2026) uses response geometry to analyze hallucinations in small language models [Ricco et al. 2026], and medical systems published in 2026 train hidden-state probes for hallucination or ambiguity detection [Wielinga et al. 2026; Liu et al. 2026]. The linked cohort produces findings that none of these single-tool analyses can state.
Table R1. Findings that require linked evidence.
The same patterns hold in six general-purpose models run through the identical pipeline (incorrect responses are more dispersed in 67.2% of eligible pairs, compared with 67.5% for the medical checkpoints; details in our response to Reviewer 8CuG), so they describe the evidence types rather than one model family. This type of contribution has precedent at ICLR: Zhang and Nanda describe their study of activation patching as "the first such systematic study" of an existing technique, with no new method [Zhang and Nanda 2024].