- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper research on the topic that whether VLMs can correct their own incorrect visual premises when the original image remain avalilable. The authors propose false-premise replay. Through controlled experiments across different models, the paper show the effectiveness of the method.
Critical strengths
-
This paper addresses an important problem in VLMs. The proposed method, false-premise replay protocol, provides a useful way to investigate whether VLMs can correct false visual assumption;
-
The paper provides a relatively comprehensive diagnostic experiments. The authors evaluate different VLMs and conduct some controlled comparisons involving different settings.
Critical weaknesses
-
The reliability and quality of the constructed premise pairs remains a concern. The dataset largely relies on model-based verification, while human review covers only 120 of the 217 pairs;
-
The evaluation is conditional on a relatively small and selected subset of failures. Only 4.9–6.7% of candidate questions yield qualifying false premises, and the shared evaluation set contains 217 pairs. It remains unclear how broadly the findings generalize;
-
Limited Model Coverage: the authors do not evaluate frontier proprietary models such as GPT or Gemini, or stronger open-weight models such as Qwen3.5 and Kimi K3.
Questions for authors
Please see weakness section.
9 comments
Response to Reviewer Dmwm (Part 2/3)
W1. Reliability of the premise pairs; human review covers 120 of 217.
Human review now covers all 217 pairs, and on the 143 pairs it supports, accuracy under corrected-premise replay still exceeds that under false-premise replay for every configuration. The replay contrast needs the false premise to be false and the corrected premise to be true, so following your suggestion two human reviewers judged each remaining pair in a second round, without seeing the labels (Section 2.2, "Human review"; Appendix A.1–A.3). The construction label is the model-review label that built the pair: false for the false premise and true for the corrected premise. Agreement over the premises of all pairs (Appendix A.3, Table 8):
| Agreement on premises, all 217 pairs | Agreement rate [95% interval] |
|---|---|
| Construction label vs each human reviewer | 84.1% [80.4, 87.5] |
| Human reviewer vs human reviewer | 82.9% [78.5, 87.1] |
| Construction label vs both reviewers, where they agree | 91.1% [87.4, 94.3] |
The construction label agrees with the human reviewers about as often as the two reviewers agree with each other.
Supported pairs. Two further human reviewers, who saw neither the earlier judgments nor the labels, adjudicated the 34 first-round pairs on which the original two disagreed. A pair is supported when both of its premises resolve to the construction label under two vote rules fixed before any second-round judgment or adjudication was read: under statement-wise majority, a premise takes the value with strictly more votes, and a tie leaves it unresolved; under adjudication priority, the adjudicators decide when they agree with each other, and the premise is unresolved otherwise. 143 pairs are supported. The other 74 are 21 on which both original reviewers reject a label, 30 disputed in the second round, which have no adjudication, and 23 that the adjudication does not support.
Results on the supported pairs. Corrected − false is the accuracy under corrected-premise replay (only the false premise replaced by its corrected version) minus that under false-premise replay (the false premise in an earlier assistant turn), with 95% intervals (Section 4.1, "Label uncertainty"; Appendix E.1, Table 21). The two source models, whose errors built the pairs, and the three models with the smallest values:
| Model, 143 supported pairs | Corrected − false |
|---|---|
| InternVL3.5-8B | 46.2 [36.2, 56.3] |
| Qwen3-VL-30B-A3B | 49.7 [39.9, 59.3] |
| GPT-6-Astra | 5.6 [1.4, 10.5] |
| Claude Opus 5.5 | 6.3 [2.0, 11.3] |
| Gemini 3.8 Flash | 12.6 [7.0, 18.8] |
Corrected − false holds without the rejected and disputed pairs: it excludes zero for all 19 configurations, and for the other 14 it is 22.4 to 53.8 points. On these pairs, the point estimates of all 19 configurations are also lower under false-premise replay than under fresh query (the question with the image and no earlier turn).
Response to Reviewer Dmwm (Part 3/3)
W2. A small, selected subset of failures, and generalization.
The drop under false-premise replay holds in four further tests: on models that produced no premise, on pairs built from another model's errors, with the domains weighted equally, and on a new test set built after submission. A low rate raises the question whether the effect comes from the selection; to address it, we report these four tests (Section 4.1; Appendix E.1, E.2 and E.5). False-premise replay places a false visual premise from a failed reasoning trace in an earlier assistant turn. Fresh − false is the accuracy under fresh query (no earlier turn) minus that under false-premise replay; corrected − false uses corrected-premise replay, which replaces only the false premise with its corrected version, in place of fresh query.
- Models that produced no premise: the drop appears on each of the eleven added models, which produced no premise.
- Another model's errors: on 87 pairs built from Qwen3-VL-30B-A3B errors, InternVL3.5-8B answers 40 correctly under fresh query and 13 under false-premise replay (Appendix E, Table 15).
- Equal domain weights: fresh − false excludes zero for all 19 configurations (Appendix E.2, Table 22).
- New test set: 72 chart and document pairs, built after submission from 53 images that none of the 217 pairs uses, with image review by gpt-6.1-terra, the successor of gpt-5.6-terra. Eight open-weight models answer it (Appendix E.5, Table 25).
| Model, new test set (72 pairs) | Fresh − false | Corrected − false |
|---|---|---|
| InternVL3.5-8B | 27.8 [14.7, 41.8] | 63.9 [49.4, 76.7] |
| Qwen3-VL-8B | 22.2 [9.6, 35.7] | 54.2 [39.4, 68.1] |
| Qwen3-VL-30B-A3B | 37.5 [25.0, 50.0] | 66.7 [54.2, 78.3] |
| Qwen3.5-35B-A3B | 37.5 [25.0, 50.7] | 70.8 [58.3, 81.5] |
| Qwen3.5-27B | 27.8 [14.5, 40.8] | 61.1 [48.6, 72.4] |
| Qwen3.5-9B | 44.4 [30.1, 59.2] | 72.2 [59.7, 83.8] |
| Qwen3-VL-32B | 45.8 [32.9, 58.6] | 75.0 [63.9, 84.9] |
| InternVL3.5-38B | 45.8 [31.4, 60.6] | 77.8 [66.2, 87.7] |
Every interval on the new test set excludes zero, so the drop also holds on images outside the 217 pairs.
Why the 4.9–6.7% rate is low. The rate is the false-premise selection rate: accepted premises per candidate question, for each source model. An answer-free reference check removed 288 of the 1,070 candidate questions; each source model then sampled four reasoning traces for each remaining question (Section 2.2; Appendix A.4, Table 10):
| Of the 782 mined questions | InternVL3.5-8B | Qwen3-VL-30B-A3B |
|---|---|---|
| All four traces correct | 558 (71.4%) | 575 (73.5%) |
| Failed, no accepted premise | 152 | 155 |
| of these, no single answer-relevant visual claim, or the error lies in the reasoning | 127 | 118 |
| Accepted premise | 72 | 52 |
The rate is low mainly because the source models answer most questions correctly. Most other failures fail through reasoning rather than a visual claim, and the protocol excludes them by design because it tests visual premises. The selection uses no replay outcome.
We hope these additions address your concerns, and we would be grateful if you would consider raising your score.
Response to Reviewer Dmwm (Part 1/3)
Thank you for recognizing that false-premise replay gives a useful way to test whether a VLM can correct a false visual assumption. We address model coverage (W3), the reliability of the premise pairs (W1) and generalization beyond the selected failures (W2). Revisions are highlighted in blue.
W3. Model coverage: GPT, Gemini, Qwen3.5, Kimi K3.
Yes: false-premise replay lowers accuracy on all eleven models we added, including GPT-6-Astra, Gemini 3.8 Flash, five Qwen3.5 models and Kimi-K3.1-Pro. A diagnostic finding should hold beyond the models that produced the premises, so following your suggestion each added model answers the 217 premise pairs of the shared evaluation set with the paper's prompts (Section 4.1 and Table 2; Appendix E.1, Table 20). Fresh − false is the accuracy under fresh query (the question with the image and no earlier turn) minus the accuracy under false-premise replay (a false visual premise from a failed reasoning trace, placed in an earlier assistant turn). Corrected − false uses corrected-premise replay, which replaces only the false premise with its corrected version, in place of fresh query. Percentage points; brackets give 95% intervals from the paired image-cluster bootstrap.
The drop does not depend on replaying a model's own errors: none of the added models produced any premise. GPT-6-Astra and Claude Opus 5.5 sample at the provider's default temperature; the table gives the first of three runs, and every contrast keeps its sign in the other two. With the six submitted models and the two Inkling models also run with reasoning, the revision covers 19 model configurations.