- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates the reliability of self-verification in medical VQA. Through VeriMap, the authors analyze verification behavior across six VLMs and five datasets, finding that models frequently accept incorrect answers and show a stronger tendency to accept their own errors. Based on these observations, the authors propose MirageCheck, which separates verification into CONTRAST, MEASURE, and BIND. The method constructs targeted visual checks and uses the resulting evidence to decide whether a candidate answer should be accepted. Experiments show improvements in false acceptance and balanced accuracy.
Critical strengths
-
Self-verification is widely used in VLMs, but its reliability is not well understood. The observation that models preferentially accept their own incorrect answers is interesting, especially when verification results are reused as feedback for self-correction.
-
The authors evaluate six models across five medical VQA datasets and conduct cross-generator verification experiments. The statistical analysis also attempts to control for differences between generators and verifiers, making the findings more convincing than simply reporting verification accuracy.
-
MirageCheck consistently reduces false acceptance across most settings. The ablation studies and multi-turn experiments provide additional evidence that the proposed verification strategy can reduce the propagation of incorrect feedback.
Critical weaknesses
-
MirageCheck builds on the idea of decomposing verification into multiple steps and collecting evidence independently of the candidate answer. The proposed information separation is interesting, but its relationship to existing structured verification approaches, particularly SGV and Chain-of-Verification, could be discussed more clearly.
-
The ablations support the overall three-stage design, but it is less clear how much each information barrier contributes independently. For example, the role-visible BIND ablation shows a relatively small effect. More targeted comparisons would help establish whether the improvement comes specifically from evidence-to-verdict binding or from the additional structured reasoning process.
-
Although false acceptance decreases considerably, the final rate is still relatively high, and balanced accuracy improves by a smaller margin. The method also requires multiple model calls, which may limit practical use. The authors provide some compute-matched controls, but broader efficiency comparisons would strengthen the evaluation.
Questions for authors
-
Can the authors clarify the key methodological distinction between MirageCheck and existing multi-step or candidate-independent verification approaches?
-
Which part of the information separation is most important for improving verification? Could the authors further isolate the effects of candidate-hidden measurement and role-blind binding?
-
How does the method behave when the independently generated rival answer agrees with the original incorrect candidate? How often can a useful contrast be constructed in practice?
-
Can the authors provide a more comprehensive analysis of the trade-off between verification accuracy, false rejection, and inference cost, especially on the more challenging datasets?
Complex figures
Figure1 still has some issues to be revised. For example, the icons in Part C do not have any information and look like generated by AI, and the texts below are complex and difficult to understand.