acceptodds
Under review as a conference paper at ICLR 2027

Verification Mirage: Mapping and Repairing the Reliability Boundary of Self-Verification in Medical VQA

Abstract

Self-verification is increasingly used to improve the reliability of vision-language models (VLMs) and provide feedback for iterative self-correction, yet it is useful only when checking an answer provides information that discriminates correct from incorrect predictions rather than simply re-solving the same visual problem. We introduce VeriMap, an auditing framework for mapping this reliability boundary in medical visual question answering, where many answers hinge on fine-grained but auditable visual distinctions. Across six VLMs, five datasets, and seven task types, VeriMap reveals a verification mirage: vanilla verifiers accept 71.0% of incorrect candidates on average, and all six models are more likely to accept their own errors than errors from other models. When reused as feedback, these false acceptances can persist across self-correction rounds. We find that stronger access to the correct image or candidate-hidden re-solving is insufficient. Instead, verification becomes more selective when the candidate is reduced to a falsifiable visual distinction and the resulting observation governs the verdict. Guided by this finding, we introduce MirageCheck, which separates test construction, candidate-hidden visual measurement, and evidence-to-verdict binding through CONTRAST, MEASURE, and BIND. Across models and datasets, MirageCheck reduces mean false acceptance by 14.6 percentage points relative to the strongest candidate-independent baseline while improving balanced accuracy, and substantially reduces false acceptance during iterative correction. Together, our results show that reliable visual self-verification requires turning acceptance to discriminative evidence from the current image rather than another judgment of the same problem.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Vision-Language Models, Self-Verification, Multimodal Reliability, Self-Improving Systems, Medical Visual Question Answering

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_R9hBOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the reliability of self-verification in medical VQA. Through VeriMap, the authors analyze verification behavior across six VLMs and five datasets, finding that models frequently accept incorrect answers and show a stronger tendency to accept their own errors. Based on these observations, the authors propose MirageCheck, which separates verification into CONTRAST, MEASURE, and BIND. The method constructs targeted visual checks and uses the resulting evidence to decide whether a candidate answer should be accepted. Experiments show improvements in false acceptance and balanced accuracy.

Critical strengths
  1. Self-verification is widely used in VLMs, but its reliability is not well understood. The observation that models preferentially accept their own incorrect answers is interesting, especially when verification results are reused as feedback for self-correction.

  2. The authors evaluate six models across five medical VQA datasets and conduct cross-generator verification experiments. The statistical analysis also attempts to control for differences between generators and verifiers, making the findings more convincing than simply reporting verification accuracy.

  3. MirageCheck consistently reduces false acceptance across most settings. The ablation studies and multi-turn experiments provide additional evidence that the proposed verification strategy can reduce the propagation of incorrect feedback.

Critical weaknesses
  1. MirageCheck builds on the idea of decomposing verification into multiple steps and collecting evidence independently of the candidate answer. The proposed information separation is interesting, but its relationship to existing structured verification approaches, particularly SGV and Chain-of-Verification, could be discussed more clearly.

  2. The ablations support the overall three-stage design, but it is less clear how much each information barrier contributes independently. For example, the role-visible BIND ablation shows a relatively small effect. More targeted comparisons would help establish whether the improvement comes specifically from evidence-to-verdict binding or from the additional structured reasoning process.

  3. Although false acceptance decreases considerably, the final rate is still relatively high, and balanced accuracy improves by a smaller margin. The method also requires multiple model calls, which may limit practical use. The authors provide some compute-matched controls, but broader efficiency comparisons would strengthen the evaluation.

Questions for authors
  1. Can the authors clarify the key methodological distinction between MirageCheck and existing multi-step or candidate-independent verification approaches?

  2. Which part of the information separation is most important for improving verification? Could the authors further isolate the effects of candidate-hidden measurement and role-blind binding?

  3. How does the method behave when the independently generated rival answer agrees with the original incorrect candidate? How often can a useful contrast be constructed in practice?

  4. Can the authors provide a more comprehensive analysis of the trade-off between verification accuracy, false rejection, and inference cost, especially on the more challenging datasets?

Reviewer_8hq1OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks whether visual self-verification in medical VQA provides a discriminative check or merely re-solves the same problem. VeriMap audits three questions: how often verifiers accept incorrect candidates (Verification Mirage); whether a verifier preferentially accepts errors it produced itself (generator–verifier coupling); and what happens when such verdicts are reused as feedback (lock-in). Across 6 open-weight VLMs, 5 datasets and 7 task types, vanilla verifiers accept 71.0% of incorrect candidates, and a complete 6×6 generator–verifier audit with mixed-effects modelling shows all six verifiers over-accept their own errors (adjusted OR 2.32–8.53, Holm p < .001).

Critical strengths
  1. Important, well-scoped problem. Treating false acceptance — not aggregate verifier accuracy — as the safety-critical error in medical VQA is the right framing.

  2. The mechanism diagnosis is the paper's best contribution. The authors actively falsify the two obvious stories instead of assuming them. The coordinate-fact probe (Table D.4: discriminative vs. orthogonal-axis fact) is a clean, well-controlled design.

  3. MirageCheck is simple and adoptable. No training, no external critic, one frozen checkpoint. "The rival defines the test, not the vote" is a genuine design principle that cleanly separates the method from a second opinion or self-consistency.

  4. Thoughtful controls (matched random rejection, re-solve gate, matched request count, information-hiding ablations, contemporaneous replay) and a system-level consequence analysis (multi-turn lock-in) at the right granularity.

Critical weaknesses
  1. W1 — Unexplained coverage gap in the core mechanism. CONTRAST needs a rival r that disagrees with the candidate a. But the paper's own diagnostic shows candidate-hidden re-solving reproduces the original error in 79.0% of wrong cases. By that accounting, most false acceptances should yield r ≡ a and no testable distinction — yet FPR falls by 30+ points. The paper never reports the disagreement rate among accepted-but-wrong candidates, the behaviour when r ≡ a, or a decomposition of revocations into disagreement-driven vs. not. Without it, the gains may come from incidental re-asking rather than the proposed mechanism. This is the analysis I most want added.
  2. W2 — The sufficiency evidence is oracle-based. Table D.4 supplies privileged bounding-box coordinates on synthetic questions over 40 images; the paper concedes it does not establish that MEASURE recovers such evidence in practice. Its primary outcome was also selected after inspecting output availability. Real-pipeline support rests on small ablation effects (revealing hypotheses during MEASURE: FPR +3.04 pp [0.61, 5.55], Acc −1.32 pp [−2.87, 0.19]) and on modest selectivity (23.9% of false acceptances caught vs. 12.1% of true ones harmed; fails in 5/30 settings).
  3. W3 — Two essentially free baselines are missing. (i) The verifier prompt already elicits a Confidence score on every call, yet it is never used — a calibrated threshold / selective-prediction baseline is the standard comparator for any FPR-reduction claim. (ii) The 6×6 matrix is already built; reporting off-diagonal (cross-model) FPR would directly test the remedy implied by the own-error finding and give MirageCheck a much stronger baseline than SGV. Also absent: uncertainty/entropy-based detectors (e.g. UniVRSE, cited but untested), a CoVe-style VQA adaptation, and any frontier-scale verifier (all six models are ≤8B, so scale-dependence is unknown).
  4. W4 — Absolute performance and the false-rejection cost are understated. The best method still accepts ~45% of wrong answers; BAcc ~59.9% against a 50% always-reject floor is incremental. FNR rises 14.81% → 31.41% on PMC-VQA (Table F.3); BAcc is worse than SGV in 6/30 settings (MedXpert-QA for 5/6 models) and loop accuracy worse in 7/30. The paper argues FP/FN costs are asymmetric in medicine yet uses cost-symmetric BAcc as its primary metric — a cost-weighted operating point or full confusion tables are needed. MirageCheck is also strictly one-sided (only initial acceptances are revisited, so FNR can only increase); state this up front.
Questions for authors
  1. Among accepted-but-incorrect candidates, how often does r ≠ a, and what happens when they agree? Please decompose revocations by whether a disagreement existed.
  2. Please report off-diagonal FPR from the existing 6×6 matrix — as a baseline and as a direct test of the proposed remedy.
  3. Please add a confidence-threshold baseline (the scores are already collected) and report AUC or a cost-weighted operating point.
Reviewer_yQCQOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies when visual self-verification in medical VQA provides a genuinely discriminative check rather than a re-solution of the same visual problem. The authors introduce VeriMap, an auditing framework applied over 6 open-weight VLMs × 5 medical VQA datasets (30 settings, 7 task types).

Critical strengths
  1. The complete cross-generator matrix with mixed-effects controls cleanly separates own-error dependence from general permissiveness and error acceptability; "candidate independence and stronger visual grounding are insufficient, answer-relevant evidence is what matters" is a substantive, transferable conclusion, not an incremental method result.

  2. Equal-request, matched-random-rejection, re-solve-gate, and information-hiding controls support the specific MirageCheck design; the method is training-free, same-model, and requires no reference answer, making it practical.

Critical weaknesses
  1. The Figures are likely generated by GenAI, especually for Figures 1 and 5.

  2. The paper itself frames FPR as "safety-critical," yet MirageCheck raises FNR substantially (PMC-VQA 14.8% → 31.4%; probe harms 12.1% of true acceptances) and regresses BAcc on MedXpert-QA and in 6/30 settings. Whether the intervention is net-favorable depends on the relative cost of endorsing a wrong medical answer versus withholding endorsement from a correct one — a ratio never stated.

  3. The main table does not state how parse failures or no-disagreement cases are handled or their frequency; MEASURE's observation accuracy is never compared against ground truth (Appendix D.4 relies on privileged annotations); and since re-solving repeats errors ~79% of the time, disagreement is often absent — the fraction of accepted candidates with a usable disagreement, and the method's behavior otherwise, are unquantified. These determine how much of the FPR gain is genuine selective revocation versus pipeline mechanics.

Questions for authors
  1. In Table 1/Figure 6, how are candidates with failed CONTRAST/MEASURE/BIND outputs or candidate–rival agreement treated — kept as accepted, excluded, or otherwise? What are the per-setting frequencies, and how do FPR/BAcc change under extreme assumptions

  2. NR rises ~17 pp on PMC-VQA and BAcc regresses in 6/30 settings. Under what false-acceptance/false-rejection cost ratio is MirageCheck preferable to SGV or vanilla? Could you report an FPR–FNR operating curve (or BAcc conditioned on the initial verdict) rather than a single average?

  3. On the SLAKE native-bbox subset, what is the agreement between MEASURE's recorded observation and the ground-truth annotation? This directly bounds the achievable selectivity of the method.

1 comment
Reviewer_yQCQOpenReview ↗

Complex figures

Figure1 still has some issues to be revised. For example, the icons in Part C do not have any information and look like generated by AI, and the texts below are complex and difficult to understand.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal and Key Revisions

Dear PC, SAC, AC, and Reviewers:

We sincerely thank all reviewers for their constructive feedback. Following the discussion, all three reviewers currently rate the paper as Weak Accept with score of 3. We have also revised the manuscript and conducted additional experiments addressing the main concerns:

1. Scientific contribution. We clarified our central finding: candidate independence and stronger visual grounding alone are insufficient for reliable self-verification. Answer-discriminative visual evidence is essential. We strengthened the distinction between MirageCheck and existing verification approaches.

2. Mechanism validation. New experiments show that CONTRAST identifies disagreements in 28.7% of incorrect vs. 14.7% of correct initial acceptances. Every revocation follows a disagreement, and 72.5% of revocations remove incorrect answers, compared with 60.0% under matched random revocation. MirageCheck further reduces FPR by 9.20 points over a plain re-answer rule. We also quantified disagreement coverage and pipeline fallbacks.

3. Additional baselines and scalability. MirageCheck outperforms tuned confidence thresholding by 2.49 BAcc points and reduces FPR to 44.76%, compared with 54.51% for the average cross-model verifier. Experiments with four additional 27-32B VLMs confirm that verification mirage persists in the larger models tested.

4. Performance and trade-offs. Across all settings, MirageCheck reduces FPR from 59.39% to 44.76% and improves BAcc from 57.28% to 59.85% over SGV. We added cost-sensitive analyses, explicitly quantified FNR increases and limitations, and showed that cumulative false acceptance in iterative self-correction decreases from 87.67% to 42.72%.

We also redesigned Figures 1 and 5 and expanded the methodological clarifications and limitations. Together, these revisions strengthen the evidence that reliable visual self-verification requires answer-discriminative evidence, not merely another judgment of the same visual problem.

We sincerely thank the reviewers and Area Chair again for their consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.