acceptodds
Withdrawn from ICLR 2027

Characterizing Errors in Small Medical Language Models: Geometry, Prediction, and Causal Sensitivity

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

What do differences between correct and incorrect medical answers reveal about a language model's errors? Geometric separation, predictive internal signals, and intervention effects offer different evidence, yet none alone establishes that a model recognizes or can correct its mistakes. We introduce ***Mirage***, a framework that tests these claims separately on shared questions and responses, using coverage checks, predictive baselines, and controlled interventions. Across five medical checkpoints and four datasets, we collect 208,896 responses across calibration and evaluation. Incorrect responses are more dispersed in 545 of 808 eligible evaluation pairs, but this geometric comparison covers only 12.6% of the 6,400 evaluated pairs. On 278 matched pairs, masking explicit answers reduces biomedical Fisher label propagation macro-F1 from 0.807 to 0.670, revealing a substantial contribution from answer content. Internal states predict errors on unseen questions, while gains beyond answer type vary across visual tasks and fitting partitions. Interventions produce protocol dependent effects, and a supplementary restoration study finds no observed benefit in sampled answer recovery. Together, these results identify three obstacles to interpreting correctness signals: selective coverage, dependence on answer content and task structure, and a gap between output sensitivity and demonstrated repair. By turning these obstacles into explicit testbeds, we provide a reproducible framework for testing what geometric, predictive, and intervention evidence can establish about model errors, while keeping benchmark agreement separate from clinical explanation.

Primary area:
interpretability and explainable AI
Keywords:
Medical language models, Response geometry, Error prediction, Activation probing, Causal sensitivity

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_CJrqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces MIRAGE (medical inference representation analysis of geometry and errors), a framework that the authors propose for testing what constitutes a correct and an incorrect LLM answer, and what differences between correct and incorrect answers can reveal about a medical language model’s errors. The authors highlight three claims as signs that can be used to tell when a model "knows when it is wrong": (i) geometry: different embeddings of correct and incorrect answers; (ii) prediction: classifiers trained on the model's internal states may be able to predict if a model will be wrong; and (iii) intervention: input/ internal state alterations can change the model's answer. The MIRAGE framework tests each claim separately on the same set of questions and responses to show what each type of evidence can and cannot support. It is applied on five small open medical models and four benchmarks (two text MCQ and two radiology VQA datasets); each dataset contributes 400 evaluation questions and 8 calibration questions. Each model answers each question 32 times, producing about 209,000 responses in total, and a response is correct only if its parsed answer matches the benchmark's answer key.

In the application of the framework, for the geometry claim, the authors embed biomedical and a general sentence encoder and compare how spread out the correct and incorrect responses are within each question. Only 12.6% of question–model pairs (808 of 6,400) have enough of both to be analyzed. Within this subset, incorrect responses are more spread out in about two-thirds of cases, though some incorrect answers cluster tightly. As for prediction, the authors train simple linear classifiers on the models’ internal states using only the 8 calibration questions. These predict errors reasonably well on the visual datasets but mostly at chance on text, and on the visual datasets a baseline that only knows whether a question is open-ended or yes/no performs nearly as well. Lastly, for interventions, they corrupt the input, paste the internal state from the uncorrupted run back in, and measure the probability of the correct answer against sham and random controls.

The authors conclude that interpreting these signals is limited by three obstacles: geometric comparisons cover only a small subset of questions, the signals depend heavily on answer content and question format, and changing a model’s internal scores does not mean its answers are repaired. They present MIRAGE as a reproducible framework for testing such claims while keeping benchmark agreement separate from clinical validity.

Critical strengths
  1. The paper's methodology is well-reported, with each of the 3 claims tested with a thorough set of controls (geometry reported with its coverage, interventions compared against both a sham control and a random change of the same size). The authors also disclose a bug in their embedding setup that gave identical texts different embeddings, which increases trust in findings and makes the study easier to reproduce.

  2. The study produces a large 209,000 set of responses with their parsed answers, and embeddings. This opens the possibility for future research on error-detection methods.

Critical weaknesses
  1. The paper's novelty is limited, with MIRAGE simply combining existing tools and adding controls to each (APORIA geometry analysis, linear probes, activation patching). This is further exacerbated by the fact that the main conclusions of the paper confirm limitations already established in earlier work. In other words, the paper proposes no new method for detecting or correcting errors and offers no positive result that future work can build on.

  2. Some analyses are too small: (i) probes are trained on only 8 calibration questions per dataset; and (ii) geometry covers only 12.6% of question–model pairs. As such, lots of the negative findings could potentially be fue to a lack of data. Although the authors acknowledge this, it still is a weakness since those negative findings are a core contribution of the paper.

  3. There is no expert review, so “correct” and “incorrect” only mean agreement with the benchmark’s answer key. This is especially problematic for open-ended visual questions, which are scored by exact match after light text normalization.

Questions for authors
  1. What does MIRAGE show that was not already known from prior work on probing and activation patching?

  2. How do you think the probe and intervention conclusions would change with substantially more data?

  3. How many of the responses marked incorrect on open-ended visual questions are actually clinically equivalent to the reference answer? Having a clinician review a random sample, or rerunning the visual analyses with more lenient answer matching, would show whether the question-type effect and the error labels are driven by exact-match scoring.

5 comments
AuthorsOpenReview ↗

Response to Reviewer CJrq (Part 1)

We thank Reviewer CJrq for a careful and accurate summary of MIRAGE, and for highlighting the reporting choices we consider central to the work: geometry reported together with its coverage, interventions compared against both a sham and a norm-matched random control, the disclosure of the embedding defect that gave identical texts different vectors, and the 208,896-response resource.

The review raises three questions: what MIRAGE adds and what positive results it offers, whether some analyses are too small to support their conclusions, and how much the open-ended visual labels depend on exact-match scoring. We answered the last two with new experiments, including a blinded review of open-ended answers by three clinicians and calibration budgets up to 30 times larger, and we revised the paper's presentation. Intervals are 95% bootstrap intervals over question or image groups with 1,000 replicates. We have also highlighted all changes in the revised manuscript in blue.


Response to Weaknesses

W1. Novelty and positive results

Response: Thank you for the concern. We acknowledge that some of the tools in MIRAGE are established, and we do not claim otherwise. What MIRAGE adds is a single frozen cohort in which geometric, predictive, and interventional evidence are recorded for the same questions, so that the evidence types can be tested against one another rather than reported separately. This matters because each type of claim is currently made in isolation: APORIA (ICML 2026) uses response geometry to analyze hallucinations in small language models [Ricco et al. 2026], and medical systems published in 2026 train hidden-state probes for hallucination or ambiguity detection [Wielinga et al. 2026; Liu et al. 2026]. The linked cohort produces findings that none of these single-tool analyses can state.

Table R1. Findings that require linked evidence.

FindingEvidence
Separability depends on answer content; the dispersion asymmetry does notMasking the answer line lowers within-question separability from 0.807 to 0.670 macro-F1, while the share of pairs with more dispersed incorrect responses is unchanged (0.675 vs 0.676)
Geometry adds no predictive information beyond output featuresPredicting majority error on eligible pairs: answer type 0.528 AUC, adding probe scores 0.567, adding output-consistency features 0.722, adding geometric distance 0.721
The evidence levels are largely distinct on the same questionsAll question-level associations between geometry, probe scores, and intervention effects are at most 0.104 in absolute value after controls, except a link between dispersion and probe score on visual items (0.242 [0.106, 0.375])
Sampling diversity controls coverage and output-based detection, not prompt-time signalsOn identical Qwen2.5-VL weights, top_k 1 instead of 50 lowers coverage from 28.7% to 52.0% of pairs down to 1.3% to 2.8% and output-feature AUC by 0.229 to 0.362, while probe AUC changes by at most 0.078
Likelihood gains overstate behavioral recoveryOpenBioLLM restoration raises reference log-probability by 4.6 to 5.2 nats but sampled accuracy by 0.031 to 0.037 over a random patch
Clinical evidence acts at a specific site and depthPatching the edited evidence token at a shallow layer switches every sampled MedGemma-4B laterality answer and half the negation answers, with no switches under random, nuisance, or sham controls and none at the final prompt position

The same patterns hold in six general-purpose models run through the identical pipeline (incorrect responses are more dispersed in 67.2% of eligible pairs, compared with 67.5% for the medical checkpoints; details in our response to Reviewer 8CuG), so they describe the evidence types rather than one model family. This type of contribution has precedent at ICLR: Zhang and Nanda describe their study of activation patching as "the first such systematic study" of an existing technique, with no new method [Zhang and Nanda 2024].

AuthorsOpenReview ↗

Response to Reviewer CJrq (Part 2)

... (continued) ...

We agree that a reader should leave with results to build on, and several MIRAGE results are positive in that sense. They are listed below with the use we see for each.

Table R2. Positive results and how they can be used.

ResultMeasurementUse
A stable dispersion asymmetry on questions with mixed answers545 of 808 eligible pairs (67.5%); 0.630 to 0.692 across two encoders, three metrics, and two views; unchanged by answer masking; 67.2% in six general modelsA measured prior for consistency-based detectors, with known coverage
Probe signal beyond answer typeGain over answer type on SLAKE of 0.058 to 0.097 with held-out intervals above zero; within-open AUC 0.596 to 0.731 under clinician-validated labelsA target and a stratified protocol for prompt-time detectors
Output-consistency features as the baseline to beatAdd 0.155 AUC beyond answer type and probe scores; up to 0.958 AUC in recent general modelsA low-cost baseline that internal-state detectors should be required to exceed
Localized clinical evidence that changes behaviorShallow evidence-token patches switch sampled answers (MedGemma-4B laterality 1.00, negation 0.50; BioMistral laterality +0.354 over random); no effect at deeper layers or the final prompt; dose arithmetic not localizedWhere and when to intervene, and which relations are not carried by a single token
Exact matching misses most clinically equivalent open answersExact match recovers 29.6% of answers clinicians judge equivalent; a validated lenient matcher agrees with clinicians on 84.5%Relabeling protocol for open-ended medical VQA evaluation
An encoder defect and its fixIdentical texts differed by up to 1.323 in L2 distance under dynamic-padding attention; the corrected setting agrees within 2.2e-6A reproducibility check for embedding-geometry studies
A linked resource208,896 responses with parsed answers, corrected embeddings, internal states, intervention records, clinician ratings, seeds, and hashesTraining and evaluating error detectors with predeclared splits

W2. Whether the analyses are large enough

Response: Thank you for your concern that some negative findings may reflect too little data. This needed to be tested rather than argued, and we tested the two analyses named. For the probes, the question is whether the weak results with eight calibration questions persist when more labels are available. We refit every probe on 20 group-disjoint subsets at budgets up to 240 questions for text and up to 96 and 32 image groups for VQA-RAD and SLAKE, and evaluated each fit on a fixed held-out set that never enters fitting.

Table R3. Prompt-only probe AUC on a fixed held-out set.

ModelDataset8 questionsLargest budgetGain over answer type at largest budget [held-out CI]
BioMistral-7BMedMCQA0.5870.6780.178 [0.086, 0.268]
BioMistral-7BMedQA0.5470.6470.147 [0.067, 0.224]
OpenBioLLM-8BMedMCQA0.5490.5780.078 [-0.013, 0.176]
OpenBioLLM-8BMedQA0.5430.6920.192 [0.104, 0.277]
MedGemma-4BMedMCQA0.5070.5860.086 [-0.005, 0.176]
MedGemma-4BMedQA0.5160.5480.048 [-0.040, 0.131]
MedGemma-1.5-4BMedMCQA0.5660.6950.195 [0.116, 0.276]
MedGemma-1.5-4BMedQA0.5000.5910.091 [-0.004, 0.182]
Lingshu-7BMedMCQA0.5280.6150.115 [0.032, 0.207]
Lingshu-7BMedQA0.5090.6020.102 [0.010, 0.187]
MedGemma-4BSLAKE0.7860.8040.097 [0.020, 0.178]
MedGemma-1.5-4BSLAKE0.7680.8070.058 [0.009, 0.112]
Lingshu-7BSLAKE0.8350.8710.080 [0.021, 0.147]
MedGemma-4BVQA-RAD0.7150.8050.033 [-0.031, 0.096]
MedGemma-1.5-4BVQA-RAD0.7080.7780.045 [-0.018, 0.112]
Lingshu-7BVQA-RAD0.7700.818-0.011 [-0.057, 0.040]

The review's intuition holds for text, and the result is informative. The text signal is weak with 8 questions but grows with labels, and with 240 questions it exceeds chance with held-out intervals above zero in 6 of 10 cells (for text, answer type is a single category, so the gain equals AUC minus 0.5). The text result is therefore label-limited rather than absent, and output-consistency features reach comparable accuracy with far fewer labels (0.656 with 8 questions for BioMistral MedMCQA, against 0.678 for the probe with 240). On SLAKE, probes exceed answer type with held-out intervals above zero for all three checkpoints. Low-rank probes, shrinkage discriminants, and stronger regularization change 8-question AUCs by a few points without altering any conclusion. Section 6.2 now presents the 8-question results as the low-label end of these curves.

AuthorsOpenReview ↗

Response to Reviewer CJrq (Part 3)

For geometry, the question is whether the 12.6% coverage would rise with more data. A pair is eligible when at least five of its 32 responses are correct and at least five incorrect, and most pairs fail this because the model answers consistently: 42.1% of pairs are answered incorrectly in all 32 samples and 36.6% correctly in all 32. Adding questions does not change the rate (0.157 with 10 and with 400 MedMCQA question groups), and lowering the threshold to two raises coverage to 18.7% with the dispersion asymmetry unchanged (0.670 to 0.679). Adding samples per question does raise coverage for models with varied outputs (BioMistral MedMCQA: 0.56, 0.75, and 0.83 of questions eligible at 32, 64, and 128 samples) while the asymmetry stays between 0.760 and 0.843. In auditing decoding for this response, we also found that our pipeline set only temperature and top-p, and that Lingshu-7B's shipped configuration sets top_k = 1, which made its samples effectively greedy and explains its low coverage. Its base model with the same setting reproduces that coverage, and with top_k = 50 the base reaches 28.7% to 52.0%. Lingshu's prompt-only probe inputs are computed before sampling and are unaffected, and the revised paper states each checkpoint's effective decoding settings. Coverage is therefore set by the response distribution, and the confidently wrong 42.1% is a deployment result in its own right: no within-question geometry or self-consistency method can flag these questions.

W3. Expert review and exact-match scoring of open-ended answers

Response: Thank you for this concern. The paper treats benchmark agreement as distinct from clinical correctness, and benchmark agreement remains the appropriate target for auditing claims that are themselves evaluated against benchmark keys. The review's point is nonetheless important, because exact matching could distort the open-ended visual labels on which part of the analysis rests. Text multiple-choice items are scored by option letter and cover 62.5% of evaluation pairs and 74.0% of eligible geometry pairs; closed yes/no items are also unambiguous. Only open-ended visual labels can change, so we measured how much they change and whether the conclusions depend on them.

Three clinicians rated a stratified sample of 239 open-ended answers, blinded to the model and to the exact-match label: 199 that exact matching scored as incorrect and 40 that it scored as correct, balanced across MedGemma-4B, MedGemma-1.5-4B, Lingshu-7B, and both datasets. Labels were assigned by majority of the three raters, and agreement was very high (Fleiss kappa 0.976; pairwise Cohen kappa 0.964 to 0.986). (One sample was left unrated by all raters and is excluded (n=239n=239).)

Table R4. Clinician review of open-ended answers.

Exact-match labelnClinically equivalentPartially equivalentNot equivalentUnanswerableShare equivalent [Wilson 95% CI]
Incorrect19995376430.477 [0.409, 0.547]
Correct40400001.000 [0.912, 1.000]

Nearly half of the exact-match errors are clinically equivalent to the reference, and exact matching produced no false-correct labels. We then compared automatic matchers with the clinician consensus. Exact matching agrees on 60.3% of answers and recovers only 29.6% of clinically equivalent ones. A lenient matcher that accepts an answer when it matches after normalization or when Qwen3.5-9B, given a fixed rubric, judges it equivalent agrees with the clinicians on 84.5% of answers (precision 0.86, recall 0.87). We re-ran every visual analysis with these clinician-validated labels.

Reviewer_8CuGOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces MIRAGE, a framework for studying errors in small medical language models. It examines differences between correct and incorrect responses, whether internal states can predict errors, and how interventions affect model outputs. The authors evaluate five medical models on four datasets. They find that geometric differences depend partly on answer content, that internal-state probes do not consistently outperform simple baselines, and that the tested interventions show no clear benefit in correcting errors.

Critical strengths

The paper addresses an interesting problem: understanding what model outputs and internal representations can tell us about errors in medical language models. I appreciate that the authors distinguish between finding patterns in incorrect answers, predicting errors, and testing causal sensitivity. The paper also includes useful controls, such as masking answer text, comparing against answer-type baselines, and using sham and random interventions.

Critical weaknesses

1. The contribution is limited, and the main findings provide relatively little new insight.

Although the paper combines several types of analysis, I am not convinced that it contributes enough beyond what is already known about probing and model representations. The main conclusions are that geometric separation can depend on answer content, that information extracted from internal states does not necessarily reflect what the model actually uses, and that interventions can change outputs without correcting errors. These are already well-known limitations in the probing and activation patching literature [1–3]. MIRAGE mainly brings together existing methods and confirms these limitations in a medical setting.

The results themselves have important limitations. The geometric analysis covers only 808 of 6,400 question-model pairs (12.6%), and classification performance drops substantially when explicit answers are masked (Section 6.1 and Appendix Table 15). The probes are trained on only eight calibration questions (Section 4.3), which raises concerns about their reliability. On text tasks, the pre-generation probes perform mostly around chance level (Table 16). On visual tasks, a simple answer-type baseline already achieves comparable results (Table 17), while Table 2 shows no consistent improvement from adding residual features. The gains are positive on SLAKE but negative on VQA-RAD, and the reported intervals across calibration splits all include zero. The answer-type baseline was also introduced post hoc (Appendix G.1).

Negative results can certainly be valuable, and I appreciate that the authors report these limitations. However, I find it difficult to identify what new understanding of medical model errors emerges from these experiments beyond confirming known issues.

2. The causal analysis provides limited insight, and the choice of models restricts the conclusions.

I also found the intervention experiments rather unconvincing. Figure 6 shows that the effects vary considerably across models and experimental settings. The largest effects come from OpenBioLLM, which uses a fallback prompting format, while the other models show smaller or mixed results.

Moreover, the text experiments use a broad corruption of the prompt embeddings, so observing changes after restoring an internal state does not tell us much about the specific mechanisms behind medical errors. Most of the analysis measures changes in reference-answer likelihood rather than whether the model actually corrects its answers. The sampled follow-up in Table 3 shows no recovery benefit, although this experiment is itself quite limited in scope, covering one model and two visual datasets with 40 questions each.

I understand that the authors do not claim to develop an error-correction method. Still, given that causal sensitivity is one of the three main components of the paper, I would have expected stronger evidence of what these interventions actually tell us about the causes of errors.

The choice of models is another concern. I understand the motivation for focusing on smaller medical models, especially when studying internal representations. However, the evaluation includes some relatively old baselines and no general-purpose models for comparison.

Medical specialization does not necessarily imply better medical knowledge or reasoning. For example, ReMedQA [4] finds that medical specialization does not consistently improve robustness on medical question-answering tasks. Similarly, Dada et al. [5] directly compare biomedical models with their general-purpose counterparts and find performance declines in 11 of 12 specialized models on at least one clinical task. Their evaluation also includes BioMistral and OpenBioLLM, two of the models used in this paper.

Without comparisons against general-purpose models, particularly more recent models with strong reasoning capabilities, it is difficult to understand whether the observed patterns are specific to medical specialization or reflect more general properties of language models. This limits how broadly the results can be interpreted.

References

[1] Designing and Interpreting Probes with Control Tasks

[2] Probing Classifiers: Promises, Shortcomings, and Advances

[3] Towards Best Practices of Activation Patching in Language Models: Metrics and Methods

[4] ReMedQA: Are We Done With Medical Multiple-Choice Benchmarks?

[5] Does Biomedical Training Lead to Better Medical Performance?

Questions for authors
  • Do the internal-state probes clearly outperform the answer-type baseline when trained on more calibration questions? The current experiments use only eight questions for fitting, which seems too limited to draw strong conclusions. An evaluation with larger calibration sets and separate held-out questions would help establish how reliable these signals are.

  • Is there evidence that the changes in reference-answer likelihood observed in Figure 6 are associated with actual answer correction? In particular, are there settings where residual restoration performs better than matched-random interventions in terms of generated-answer correctness? This would help clarify what the intervention results actually tell us about model errors.

  • As a general suggestion, I would encourage the authors to include comparisons with general-purpose models, ideally the more recent reasoning models of similar sizes where feasible (e.g. Qwen3.5, Gemma.4, GPT-OSS, etc.). This would help establish whether the findings are specific to the selected medical checkpoints or apply more broadly.

  • I would also suggest making the distinction between the smaller cohort shown in Figure 3 (640 question-model pairs) and the main evaluation (6,400 pairs) clearer in the paper.

6 comments
AuthorsOpenReview ↗

Response to Reviewer 8CuG (Part 1)

We thank Reviewer 8CuG for a detailed and well-referenced review, and for recognizing the value of separating pattern-finding, error prediction, and causal sensitivity, together with the controls built for that purpose: answer masking, answer-type baselines, and sham and random interventions.

The review asks whether MIRAGE says more than the known limitations of probing and patching, and whether its intervention and model choices support general conclusions. To answer, we ran each analysis the review proposed: six general-purpose models, including the base model of every medical checkpoint; calibration budgets up to 30 times larger with a fixed held-out set; and tests of sampled behavior under matched interventions. The results make the paper's conclusions more specific and show that they hold beyond medical checkpoints. Intervals are 95% bootstrap intervals over question or image groups with 1,000 replicates. We have also highlighted all changes in the revised manuscript in blue.


Response to Weaknesses

W1. Contribution beyond the known limitations of probing and patching

Response: Thank you for this concern. We agree that the caveats in [1, 2, 3] are established in principle, and the paper cites them for that reason. Our position is that a caveat known in principle differs from a measured property of medical correctness signals, and that several MIRAGE results are not predicted by those caveats. The table below sets out what each prior line of work established and what MIRAGE adds; none of [1, 2, 3] studies correctness signals in generated medical answers, output-embedding geometry, prompt-time error prediction, or the link between intervention scores and sampled behavior.

Table R1. Prior results and what MIRAGE adds.

Prior work and settingEstablishedWhat MIRAGE measures
Hewitt and Liang [1]; POS and dependency probesProbe accuracy can reflect probe capacityProbes add to the answer-type baseline on SLAKE with held-out intervals above zero, and predict errors within open-ended questions, where answer type carries no information
Belinkov [2]; surveyDecodable information need not be usedOn 640 questions, probe score and restoration effect are uncorrelated (0.008 [-0.065, 0.086])
Zhang and Nanda [3]; GPT-2 circuits, logit metricsCorruption and metric choices change localizationA 4.6 to 5.2-nat restoration gain yields 10% to 13% behavioral recovery, while a shallow evidence-token patch flips every sampled laterality answer
Ricco et al. (APORIA), ICML 2026; small LMsResponse geometry separates hallucinationsCoverage is 12.6% and is set by decoding; separability is answer-specific and does not generalize to unseen wrong answers
Orgad et al., ICLR 2025; hidden statesTruthfulness concentrates at exact answer tokensIn output-embedding space, masking the answer lowers separability but leaves the dispersion asymmetry unchanged

Five findings go beyond confirming the caveats. First, incorrect responses are more dispersed than correct ones in 67.5% of eligible pairs (pairs with at least five correct and five incorrect responses among 32), and this asymmetry is unchanged when the explicit answer is masked (0.675 vs 0.676), even though masking lowers within-question separability from 0.807 to 0.670; the same asymmetry appears in six general-purpose models (67.2%). Second, prompt-only probes carry error information beyond answer type, and more clearly so when open-ended answers are scored with clinician-validated labels (shown below). Third, clinical evidence acts at a specific site and depth, and patching it there changes sampled answers. Fourth, sampling diversity, a decoding property, determines geometric coverage and whether output-based detection works, but not whether prompt-time states carry error information. Fifth, within-question separability measures which answer was given: it lies between a majority baseline (0.382) and an answer-identity ceiling (0.961) and does not flag wrong answers absent from the labeled set, which is a direct test of APORIA-style claims that had not been run before.

AuthorsOpenReview ↗

Response to Reviewer 8CuG (Part 2)

.... (continued) ...

This type of contribution has precedent at ICLR, including in the review's own references. Zhang and Nanda [3] describe their work as "the first such systematic study" of an existing technique, with no new method; Hewitt and Liang [1] add controls to existing probes; and Makelov et al. expose an interpretability illusion with established patching tools [Makelov et al. 2024]. The claims MIRAGE tests are also being made now, in isolation: APORIA proposes geometry-based hallucination analysis, and medical systems published in 2026 train hidden-state probes for hallucination or ambiguity detection [Wielinga et al. 2026; Liu et al. 2026]. A concurrent preprint reports a gap between decodability and controllability at the neuron level in medical models [Vankadaru et al. 2026], which confirms that the question is open. The first contribution in the introduction now reads:

"A linked audit that tests evidence types against one another. MIRAGE collects 208,896 responses with internal states and intervention records on one frozen cohort, so geometric, predictive, and interventional claims can be tested on the same questions. On this cohort, separability depends on answer content while dispersion does not, geometry adds no predictive information beyond output features, sampling diversity controls coverage and output-based detection, and decodability, causal sensitivity, and sampled behavior diverge. The same patterns appear in six general-purpose models."

W1-2nd: Coverage, calibration size, and the answer-type baseline

The review lists four limitations of the individual results. We take them in turn.

Geometry covers 808 of 6,400 pairs because models usually answer consistently. Of all pairs, 42.1% are answered incorrectly in every sample, 36.6% correctly in every sample, and 8.7% have fewer than five responses in the minority class. Drawing more samples raises coverage for models with varied outputs (BioMistral MedMCQA: 0.56, 0.75, and 0.83 of questions eligible at 32, 64, and 128 samples) while the dispersion asymmetry stays between 0.760 and 0.843. While auditing decoding for this response, we found that our pipeline set only temperature and top-p, so each checkpoint's shipped generation configuration supplied the rest. Lingshu-7B ships top_k = 1, which made its samples effectively greedy and explains its low coverage. Its base model with the same setting reproduces that coverage (difference in eligibility between -0.005 and +0.015, all intervals including zero), and the same base with top_k = 50 reaches 28.7% to 52.0%. Lingshu's prompt-only probe inputs are computed before sampling and are unaffected. The revised Section 4 and Appendix C.3 state each checkpoint's effective decoding settings. The 42.1% of confidently wrong pairs is itself a deployment result: no within-question geometry or self-consistency method can flag them.

The drop in separability under answer masking is the quantity the mask was designed to measure, and the answer-lookup ceiling now places it on a scale from majority to pure answer identity.

The effect of using only eight calibration questions is measured by the budget study in Q1. With more labels, text probes improve beyond their weak 8-question values, so the text result is label-limited rather than absent, and on SLAKE probes exceed the answer-type baseline with held-out intervals above zero.

The answer-type baseline raises a fair question: if answer type alone predicts most errors on visual data, what is left for the probe? Two analyses separate the two. Within a single answer type, type carries no information, so any remaining AUC comes from the probe. And because open-ended answers were scored by exact match, part of the answer-type effect could be a scoring artifact; we therefore re-scored them with lenient labels validated by three clinicians (described in our response to Reviewer CJrq; the lenient matcher agrees with the clinician consensus on 84.5% of 239 rated answers).

Table R2. Probe AUC within answer type, and gain over answer type under exact and clinician-validated labels.

ModelSLAKE open, exact / validatedVQA-RAD open, exact / validatedClosed questions, exactGain over type on VQA-RAD, exact / validatedGain over type on SLAKE, exact / validated
MedGemma-4B0.731 / 0.7190.631 / 0.6190.507 to 0.578-0.030 / +0.044+0.082 / +0.116
MedGemma-1.5-4B0.658 / 0.6570.620 / 0.6960.508 to 0.540-0.042 / +0.061+0.021 / +0.061
Lingshu-7B0.698 / 0.7310.532 / 0.5960.502 to 0.588-0.080 / -0.001+0.032 / +0.179
AuthorsOpenReview ↗

Response to Reviewer 8CuG (Part 3)

.... (continued) ...

Within open-ended questions the probe predicts errors on both datasets, and under clinician-validated labels the AUC of answer type alone falls from 0.692 to 0.811 down to 0.530 to 0.604, while the probe's within-open AUC is unchanged or higher. The negative VQA-RAD gain in the paper was therefore partly an artifact of exact-match scoring, and with validated labels the gain is positive for both MedGemma checkpoints. The answer-type baseline was added after the main analysis to make the probe result harder to pass; a control of this kind can only reduce the apparent effect, it is fitted on calibration labels alone, and its timing is disclosed in Appendix G.1. In the budget study it is predeclared alongside every other comparator.

W2. Intervention design and the choice of models

Response: Thank you for your follow-up on the three aspects of the interventions we tested: OpenBioLLM's prompt format, the use of broad corruption, and the reliance on reference likelihood. We address each and then turn to the models.

OpenBioLLM uses a raw prompt because the model-card recipe breaks answer parsing. With the model's system prompt, chat template, and end-of-turn token, only 6.2% (MedMCQA) and 5.0% (MedQA) of answers parse with our parser, and 36.2% and 34.7% with a parser extended on calibration outputs, compared with 94.2% and 96.7% under the raw prompt. Under the model-card recipe, restoration shifts the answer by -0.143 [-0.340, 0.048] and 0.316 [0.137, 0.500] nats, and the base model Meta-Llama-3-8B-Instruct gives -0.257 and -0.326 under the same corruption. OpenBioLLM's large effect is therefore tied to its raw serialization, not inherited from its base, which is why the paper requires intervention claims to state their serialization.

Broad corruption cannot show where a specific piece of evidence acts, so MIRAGE also includes minimal pairs that change a single clinical relation (laterality, negation, or dose arithmetic) and patch the internal state from the source prompt into the edited one. The question these pairs answer is whether the evidence is carried at its own token, at which depth, and whether patching it changes what the model generates. We now report sampled answers under these patches, alongside matched random, nuisance-donor, and sham controls.

Table R3. Share of sampled answers that switch to the source-consistent answer after patching (12 pairs per feature, 16 samples each).

ModelFeatureUnpatched sourceEvidence token, shallow layer: patch / random / nuisance / shamEvidence token, deeper layersFinal prompt position, any layer
MedGemma-4BLaterality1.0001.000 / 0.000 / 0.000 / 0.0000.0000.000
MedGemma-4BNegation1.0000.500 / 0.000 / 0.000 / 0.0000.0000.000
MedGemma-4BDosage0.6250.469 / 0.458 / 0.396 / 0.4060.000 over randomat most 0.000 over random
BioMistral-7BLaterality0.9640.396 / 0.042 / 0.062 / 0.062at most 0.010 over randomat most 0.000 over random
BioMistral-7BNegation0.5360.391 / 0.312 / 0.307 / 0.307at most 0.021 over randomat most 0.026 over random

The patch at the evidence token reproduces the source behavior completely for MedGemma-4B laterality and halfway for negation, and raises BioMistral's laterality switch rate by 0.354 [0.234, 0.516] over the random patch. The same patch has no effect at deeper layers or at the final prompt position, and dose arithmetic is not carried at a single token in either model. OpenBioLLM parses only 10% to 20% of minimal-pair generations under the raw prompt, which limits what its switch rates can show (laterality +0.016, negation +0.026 over random). We also applied targeted corruption to real benchmark items. Replacing only the key clinical span in text questions and restoring one position did not produce a consistent effect (53 of 60 intervals include zero). For answer-changing image swaps on SLAKE, restoring the final prompt position helps only at middle and deep layers (MedGemma-4B: 0.001, 0.233, and 0.297 at layers 8, 16, and 24) and recovers at most 2.9% of the corruption effect, and horizontal image flips are not restored at that position. Textual clinical evidence thus acts at its own token early in the network, while visual evidence reaches the final position later and only partially.

Reviewer_LEz3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The study analyses differences between correct and incorrect answers in small medical LMs. It introduced MIRAGE as a new framework that tests three components (descriptive geometry, predictive accessibility, causal sensitivity). The authors claim three contributions: a coordinated audit that keeps these claims distinct, evidence that geometric patterns depend on coverage and answer content, and limits on predictive and causal interpretations. The evaluation covers five medical checkpoints (MedGemma-4B, MedGemma-1.5-4B, Lingshu-7B, BioMistral-7B, OpenBioLLM-8B) on four datasets (MedMCQA, MedQA, VQA-RAD and SLAKE).

Critical strengths

Important problem and string experimental setup. Findings are presented with caveats rather than a clearly stated, generalizable lesson.

Critical weaknesses
  1. Limited novelty, and the combination of methods is not exploited. The three evidence levels rely on established methods. The paper itself describes its contribution as “a medical triangulation study” that “connects established tools through a common protocol”. The combination could add value if it revealed something the separate analyses do not, but, it is my understanding that the levels are run on diff erent subsets (808 eligible pairs, 400 questions, a 40-question intervention subset, 80 visual questions) and are never related on the same questions. This is problematic because the claimed contribution is the framework. However, without joint insight, the paper offers little beyond applying known tools to medical benchmarks.
  2. The fi ndings are mixed, and their generality is unclear. For example, Geometry covers only 12.6% of question–model pairs, and, fi sher label propagation, which is never explained in the main text, is a within-question task that requires labeled responses from the same question. This is problematic because it is impossible to tell whether the negative results reflect the models or the study protocol.
Questions for authors
  1. What does the joint framework reveal that the three analyses would not reveal separately? Using the existing linked records, does geometric separation predict probe accuracy or intervention sensitivity on the same questions? Which signals should a practitioner trust for detecting medical errors, and under which conditions?
  2. Please explain Fisher label propagation in the main text. As I understand, it labels some responses to a question and predicts the correctness of that question’s remaining responses. In what practical scenario would labeled responses from the same question be available, and how much of the 0.807 macro-F1 reflects identifying which answer was given?
6 comments
AuthorsOpenReview ↗

Response to Reviewer LEz3 (Part 1)

We thank Reviewer LEz3 for recognizing the importance of the problem and the strength of the experimental setup. The review raises two questions that we consider central to the paper: whether combining the three kinds of evidence reveals anything the separate analyses do not, and what the Fisher label-propagation score actually measures.

We answer both below with analyses on the linked records, including new tests that relate the evidence levels question by question, and we have revised the paper so that its structure and its practical lessons are stated explicitly. Intervals are 95% bootstrap intervals over question or image groups with 1,000 replicates. We have also highlighted all changes in the revised manuscript in blue.


Response to Weaknesses

W1. Novelty and the relation between the three evidence levels

Response: The review reads the three analyses as running on different subsets that are never related. We see why: the paper did not explain how the subsets fit together. All analyses draw on one frozen cohort of 400 evaluation questions per dataset (1,600 questions and 6,400 question-model pairs), linked by hashed question, image, response, and seed identifiers. The geometry analysis uses the 808 eligible pairs, meaning pairs with at least five correct and five incorrect parsed responses among the 32 samples, because a within-question comparison needs both classes. The probes are evaluated on all 6,400 pairs. The 40-question intervention subset and the 80-question sampled follow-up are subsets of the same questions, fixed before correctness was observed. Because the subsets are nested, their overlaps are known exactly: 80 of the 640 intervention pairs are also eligible for geometry, and 1,364 pairs have a defined within-question probe AUC. Section 4 now opens with this map.

Table R1. One cohort, three claims.

ClaimUnit of analysisSubset of the frozen cohortMain controls
Descriptive geometryResponses within a question808 eligible pairs of 6,400Permutation null, answer masking, answer-lookup ceiling, two encoders, three metrics
Predictive accessibilityQuestion (prompt-only) or response (post-answer)All 6,400 pairsAnswer-type baseline, surface baseline, full-refit nulls, output-consistency features
Causal sensitivityQuestion by layer640 pairs; 80-question sampled follow-up; 18 minimal pairsExact sham, equal-norm random patch, nuisance donor, serialization control

Linked records allow the question the review asks: does evidence at one level predict evidence at another on the same questions? If geometric separation reflected the same error information that probes decode or that interventions act on, then questions with stronger separation should also have more accurate probes and larger intervention effects, and geometry should add predictive value once the other signals are known. Table R2 tests the first prediction with Spearman correlations, controlling for answer type and error rate, and additionally for model-dataset cell.

Table R2. Associations between evidence levels on the same questions.

Relation testedPairsSpearman, controlled for type, error rate, and cell [95% CI]
Geometric distance (W1) vs prompt-only probe score8080.054 [-0.018, 0.121]
Dispersion gap vs prompt-only probe score, all items8080.104 [0.040, 0.171]
Dispersion gap vs prompt-only probe score, visual items2100.242 [0.106, 0.375]
Label-propagation F1 vs within-question post-answer probe AUC7730.016 [-0.056, 0.078]
Restoration effect vs prompt-only probe score6400.008 [-0.065, 0.086]
Restoration effect vs geometric distance (W1)80-0.062 [-0.340, 0.193]

On the same questions, the three levels are largely distinct: every controlled association is at most 0.104 in absolute value, with one exception. On visual items, questions whose wrong answers scatter more widely are also the questions that the prompt-only probe flags (0.242). The second prediction fails as well. Predicting whether a question's majority answer is wrong, a model with answer type alone reaches AUC 0.528; adding the prompt-only probe score raises it to 0.567; adding output-consistency features (answer entropy, variation ratio, embedding dispersion) raises it to 0.722; adding the per-question geometric distance leaves it at 0.721 (change -0.001 [-0.003, 0.001]). Geometry is therefore a descriptive property of a question's response set rather than an independent detector, which is the role the paper assigns it.

AuthorsOpenReview ↗

Response to Reviewer LEz3 (Part 2)

The linked records support several further findings that no single-level study could state. Masking the explicit answer line lowers within-question separability (label-propagation macro-F1 from 0.807 to 0.670, a drop of 0.137 [0.117, 0.158]), yet on the same pairs the share in which incorrect responses are more dispersed than correct ones is unchanged (0.675 vs 0.676). Separability depends on answer content; the dispersion asymmetry does not. Decodability and causal sensitivity also come apart at the model level: OpenBioLLM's prompt-only probe is below chance on MedMCQA (AUC 0.418) while restoring its clean state shifts the answer by 5.529 nats, and Lingshu's probe reaches 0.830 on VQA-RAD while restoration has no reliable effect (0.110 [-0.002, 0.232]). Score changes overstate behavioral changes: OpenBioLLM restoration raises reference log-probability by 4.6 to 5.2 nats but sampled accuracy by only 0.031 to 0.037 over the random patch, about 10% to 13% of the gap between clean and corrupted inputs. Finally, evidence acts at specific sites and depths: patching the edited evidence token at a shallow layer flips every sampled MedGemma-4B laterality answer and half the negation answers, while the same patch at the final prompt position, or at deeper layers, flips none.

One of these findings also required a disclosure. Auditing effective decoding settings, we found that our pipeline set temperature 1.0 and top-p 0.95 while each checkpoint's shipped generation configuration supplied the remaining fields. Lingshu-7B ships top_k = 1, so its samples were effectively greedy, which explains why 88% to 95% of its responses duplicate one another. Its prompt-only probe inputs are computed before any sampling and are unaffected. We turned this into a controlled test on identical weights: with Qwen2.5-VL-7B, changing top_k from 50 to 1 lowers geometric coverage from 28.7% to 52.0% of pairs down to 1.3% to 2.8%, lowers the AUC of output-consistency features by 0.229 to 0.362, and changes prompt-only probe AUC by at most 0.078. Sampling diversity, a decoding property, determines geometric coverage and whether output-based detection works, but not whether prompt-time states carry error information. The revised Section 4 and Appendix C.3 report each checkpoint's effective decoding settings.

These results define the contribution more precisely than our original framing did. The tools are established; what is new is a linked, coverage-accounted cohort of 208,896 responses with internal states and intervention records, in which geometric, predictive, and causal claims are tested against one another. Such claims are currently made in isolation. APORIA (ICML 2026) uses response geometry to analyze hallucinations in small language models [Ricco et al. 2026], medical systems published in 2026 train hidden-state probes for hallucination or ambiguity detection [Wielinga et al. 2026; Liu et al. 2026], and "Truth is Universal" argues for a general truth direction [Bürger et al. 2024]. The same patterns also appear in six general-purpose models under the identical pipeline (67.2% of eligible pairs with more dispersed incorrect responses, compared with 67.5% for the medical checkpoints; details in our response to Reviewer 8CuG). This type of contribution has precedent at ICLR: Zhang and Nanda describe their study of activation patching as "the first such systematic study" of an existing technique [Zhang and Nanda 2024], and Makelov et al. expose an illusion using established patching tools [Makelov et al. 2024].

W2. Generality, and whether the negative results reflect the models or the protocol

Response: The review asks whether low coverage and weak signals are properties of the models or artifacts of the study design. We tested each component that could produce such an artifact.

Coverage first. Of the 6,400 pairs, 2,340 (36.6%) are answered correctly in all 32 samples, 2,696 (42.1%) are answered incorrectly in all 32, 556 (8.7%) have fewer than five responses in the minority class, and 808 (12.6%) are eligible. Pairs are excluded because models answer consistently, not because of the threshold: lowering it to two responses per class raises eligibility to 18.7% while the dispersion asymmetry stays at 0.670 to 0.679 across thresholds of two to six. The eligibility rate does not depend on cohort size (0.157 with 10 and with 400 MedMCQA question groups). It does depend on sampling depth for models with varied outputs: drawing 128 instead of 32 samples raises BioMistral's MedMCQA eligibility from 0.56 to 0.83, with the dispersion asymmetry between 0.760 and 0.843 and label-propagation F1 between 0.771 and 0.803. Together with the identical-weights decoding test above, this shows that coverage is set by the response distribution, not by the protocol. It also yields a practical result: 42.1% of pairs are confidently wrong, and no within-question geometry or self-consistency method can flag them.

AuthorsOpenReview ↗

Response to Reviewer LEz3 (Part 3)

Fisher label propagation is a held-out measure of separability within a question, used to test how much geometric separation depends on answer content. As stated in Section 6.1, it does not predict errors on unseen questions; that is the role of the probes. Q2 below shows what the score measures.

Predictive results were the remaining concern. We refit every probe at calibration budgets up to 30 times larger and evaluated them on a fixed held-out set that never enters fitting (full table in our response to Reviewer 8CuG). For text, the prompt-only signal is weak with 8 training questions (AUC 0.500 to 0.587) but grows with labels, reaching 0.548 to 0.695 with 240 questions, with held-out intervals above chance in 6 of 10 model-dataset cells. Output-consistency features reach comparable values with far fewer labels: on BioMistral MedMCQA they give 0.656 with 8 questions, against 0.678 for the probe with 240. On SLAKE, the probe adds to the answer-type baseline for all three vision-language checkpoints (gains of 0.058 to 0.097, held-out intervals above zero). The text result is therefore label-limited rather than absent, and Section 6.2 now presents the 8-question results as the low-label end of these curves.

These findings translate into three lessons, which Section 7 now states as testable rules. First, a geometric correctness signal exists only for questions with both correct and incorrect answers, and its coverage depends on sampling diversity, so coverage and the confidently wrong share should be reported with it. Second, a correctness signal should be compared with task-structure and output baselines within strata; in our data, probes pass this test on SLAKE and within open-ended questions, while geometry adds nothing beyond output features. Third, an intervention claim should be evaluated on sampled behavior at a stated site and depth, because a 5-nat score change can correspond to a 3-point accuracy change.


Response to Questions

Q1. What the combination reveals, whether geometry predicts probe accuracy or intervention effects, and which signals to trust

Response: The first two parts are answered in W1: on the same questions the levels are largely dissociated (Table R2), geometry adds no predictive value beyond output features, and the joint view reveals content-dependent separability with content-independent dispersion, decoding-controlled coverage, and site-specific causal effects. For the third part, the linked measurements support a concrete guide, now added to Section 7.

Table R3. Which signal to use, under which measurable condition.

ConditionSignal to useSupporting measurement
Varied sampled responses and about 32 samples per question affordableOutput-consistency featuresAdd 0.155 AUC beyond answer type and probe scores; geometry adds -0.001
Greedy or near-greedy decodingPrompt-only probesOn identical weights, top_k 1 lowers output-feature AUC by 0.229 to 0.362; probe AUC changes by at most 0.078
Text multiple choice, about 8 labeled questionsOutput-consistency featuresOutput 0.481 to 0.656; probe 0.500 to 0.587
Text multiple choice, about 240 labeled questionsPrompt-only probes become usableProbe 0.548 to 0.695; held-out intervals above chance in 6 of 10 cells
Open-ended visual questionsProbes evaluated within answer typeWithin-open AUC 0.658 to 0.731 on SLAKE; 0.596 to 0.731 under clinician-validated labels
Closed (yes/no) visual questionsAnswer-type priorProbe AUC within closed questions 0.502 to 0.588
Open-ended answers scored by exact matchClinician-validated lenient matchingThree clinicians judged 47.7% [40.9, 54.7] of exact-match errors clinically equivalent
Any interventionSampled behavior at a stated site and layer4.6 to 5.2-nat score gain with 10% to 13% behavioral recovery

Q2. Fisher label propagation: definition, practical setting, and what 0.807 measures

Response: We agree that the main text explained label propagation too briefly. Section 4.2 now reads:

"Fisher label propagation (LP). LP asks how well a few labeled responses to a question identify the correctness of that question's remaining responses. For an eligible question, we label a small subset of its 32 responses (5, 10, or 20, with at least two per class), fit the APORIA Fisher direction (a ridge-regularized linear discriminant) on the labeled embeddings only, and project all responses onto it. Each unlabeled response is assigned to the class whose labeled distance distribution it matches more closely in 1-Wasserstein distance. Identical response texts never cross labeled and held-out sets. LP is therefore a held-out, within-question measure of separability. It does not predict errors on unseen questions, which Section 6.2 tests with probes."

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of the Initial Rebuttal and Revision

We thank the Area Chair and all reviewers for their careful reading, thoughtful feedback, and constructive suggestions. We appreciate that the reviewers recognized the importance of the question MIRAGE addresses: how reliably correctness signals in medical models reflect actual clinical correctness. Their comments also helped us identify areas where the evidence and scope needed to be clearer, particularly regarding coverage, answer dependence, label quality, and causal claims.

In this initial rebuttal and revision, we have added the following analyses, all conducted on the same frozen cohort:

  • Coverage: All 6,400 question model pairs are now classified, and Figure 3 has been redrawn using the full cohort. A decoding control on identical weights confirms that coverage depends on sampling diversity.
  • Answer dependence: An answer lookup ceiling and an unseen answer test help separate answer identity from geometry. Masking explicit answers reduces separability, while the dispersion asymmetry remains unchanged.
  • Predictive signals: Using up to 240 calibration questions and a fixed held out set, we find that text signals emerge and that SLAKE's gains over answer type remain positive.
  • Label validity: Three clinicians (Fleiss κ = 0.976) assessed the clinical equivalence of exact match errors, finding that 47.7% were clinically equivalent. Using labels validated against their judgments makes the residual signal clearer.
  • Causal claims: We now measure sampled behavior directly. Patching the edited evidence token switches answers, while broad restoration recovers 10 to 13% of the behavioral gap.
  • Generality: Six general purpose models, including the base model of each checkpoint, reproduce these patterns. Question level linkage further confirms that the three evidence levels carry distinct information.

These revisions have helped us clarify what the results establish, acknowledge the remaining limitations, and better connect our conclusions to the supporting evidence. We have also revised the manuscript throughout to improve readability, strengthen the empirical grounding of our claims, and clarify the presentation and interpretation of our findings. All changes in the revised manuscript are highlighted in blue. We are grateful for the reviewers' feedback and the opportunity to improve the work.

We hope these revisions adequately address the reviewers' concerns and strengthen the paper. We would be grateful if the reviewers could consider the additional analyses and clarifications in their updated assessments and, consider increasing their scores accordingly. Thank you again for your time, thoughtful feedback, and consideration. We look forward to the continued discussion.

AuthorsOpenReview ↗

Our Work Was Not Fairly Evaluated: Final Statement Upon Withdrawal

We are withdrawing this submission due to serious concerns about the fairness of the review process. Despite our substantial efforts to address the reviewers' concerns through additional experiments requiring considerable time, computational resources, and financial investment, the resulting evidence received little engagement and was not adequately reflected in the evaluation, leaving us feeling that our efforts in good faith were not given due consideration. While we appreciate the Area Chair's helpful engagement, the limited engagement with our rebuttal and the overall evaluation have left us without confidence in the fairness of this review cycle.


1. We delivered the evidence the reviewers asked for

All three reviews rated the paper's soundness as "minor or fixable concerns", yet gave ratings of 1, 2, and 2. Each review named the evidence that would resolve its concerns. We supplied all of it:

  • Small calibration set. Reviewers 8CuG and CJrq argued that probes trained on eight questions are unreliable and that our negative findings "could potentially be due to a lack of data". We refit every probe on up to 240 questions with a fixed held-out set. Text signals emerge, and visual gains beyond answer type exceed zero on SLAKE for all three models.
  • No expert review. Reviewer CJrq asked for a clinician review of open-ended visual answers. Three clinicians rated 239 answers (Fleiss' κ = 0.976) and judged 47.7% of exact-match errors clinically equivalent. Under validated labels, our visual results become stronger, not weaker.
  • No general-purpose models. Reviewer 8CuG suggested Qwen3.5 and Gemma-4. We ran both, plus the general base model of every medical checkpoint, through the full pipeline. They reproduce the same patterns.
  • Likelihood versus correction. Reviewer 8CuG asked whether intervention effects lead to corrected answers. Patching the edited evidence token switches every sampled laterality answer in one model, whereas broad restoration recovers only 10–13% of the behavioral gap.
  • Unlinked analyses. Reviewer LEz3 stated that the three analyses "are never related on the same questions". They share question identifiers by design, and we tested their relationships directly.

Reviewer CJrq had also written that the paper "offers no positive result that future work can build on", but out results include several.


2. Limited engagement with our rebuttal

We addressed every weakness and question with additional analysis and experiments, and uploaded a revised PDF with all changes marked in blue. Reviewers 8CuG and CJrq have not responded to our rebuttal or revised manuscript. Reviewer LEz3 acknowledged that one clarification "fully answers" a main question, that the coverage analysis was "largely convincing", and that the cross-level analysis answered the core question.

We accept that reviewers are not obliged to change their scores. Our concern is that substantial new evidence received no engagement from two reviewers, while the reviewer who engaged acknowledged that several central concerns had been addressed but did not act accordingly.


3. Objections that did not account for the revised contribution

A recurring objection was that the paper uses established tools rather than introducing new methods. However, empirical and analytical contributions can provide new knowledge through controlled analyses and findings without proposing a new algorithm. We addressed concerns about connecting the three evidence levels by linking geometry, probe scores, and intervention effects on shared question identifiers and testing their relationships. The results show that these signals capture distinct aspects of model behavior, while geometry adds little predictive value beyond output consistency.

The new analyses were requested by the reviewers or directly address their questions, and they extend rather than overturn our core findings. We therefore found it difficult to understand why the revision was considered to require another review cycle, as Reviewer LEz3 suggested, particularly since ICLR allows revised PDF uploads and extended discussion so that reviewer concerns can be addressed within the same review cycle.


We recognize that reviewing is voluntary and demanding, and that disagreement is part of peer review. However, limited rebuttal engagement and insufficient consideration of the evidence the reviewers themselves requested have left us without confidence in the fairness of this evaluation.

We have therefore decided to withdraw the submission. We hope these concerns will be taken seriously and that future review cycles of this conference will provide authors with the careful, independent, and evidence-based assessment the process is intended to ensure.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.