acceptodds
Under review as a conference paper at ICLR 2027

Premise over Pixels: Diagnosing Visual Self-Correction in Vision-Language Models

15 views

Abstract

Multi-step reasoning requires a model to overturn its own earlier errors before they propagate. In vision-language models (VLMs), the same image remains available even after the model has stated an incorrect visual premise. We introduce false-premise replay to test whether VLMs can use that image to revise their own errors. From a failed reasoning trace, we extract one answer-relevant visual claim contradicted by the image, discard the remaining reasoning and final answer, and replay the claim with the original image and question. Across three model-reviewed evaluation sets, replay lowers accuracy relative to a fresh query by 25.8–31.0 percentage points, while substituting a corrected premise raises it. Image interventions show continued visual influence, but models do not reliably identify the image-supported premise. On InternVL3.5-8B, a false premise also reduces how often sampled answers include a correct one and impairs selection from fixed candidate pools. We consolidate 217 premise pairs into a shared evaluation set and compare correction baselines on five distinct models. On this set, false-premise replay lowers accuracy by 16.6 points even for a 975B-parameter model with reasoning enabled. On all five models, same-model visual-verification feedback improves accuracy over self-reflection with false premises, but lowers it when the earlier premise is correct. The findings distinguish obtaining corrective information from using it, and show why measuring repair under false-premise replay alone can overstate correction quality.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Vision-language models, visual premise verification, self-correction, premise selection, false-premise replay, multimodal reasoning, evaluation methodology

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_DmwmOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper research on the topic that whether VLMs can correct their own incorrect visual premises when the original image remain avalilable. The authors propose false-premise replay. Through controlled experiments across different models, the paper show the effectiveness of the method.

Critical strengths
  • This paper addresses an important problem in VLMs. The proposed method, false-premise replay protocol, provides a useful way to investigate whether VLMs can correct false visual assumption;

  • The paper provides a relatively comprehensive diagnostic experiments. The authors evaluate different VLMs and conduct some controlled comparisons involving different settings.

Critical weaknesses
  • The reliability and quality of the constructed premise pairs remains a concern. The dataset largely relies on model-based verification, while human review covers only 120 of the 217 pairs;

  • The evaluation is conditional on a relatively small and selected subset of failures. Only 4.9–6.7% of candidate questions yield qualifying false premises, and the shared evaluation set contains 217 pairs. It remains unclear how broadly the findings generalize;

  • Limited Model Coverage: the authors do not evaluate frontier proprietary models such as GPT or Gemini, or stronger open-weight models such as Qwen3.5 and Kimi K3.

Questions for authors

Please see weakness section.

9 comments
AuthorsOpenReview ↗

Response to Reviewer Dmwm (Part 1/3)

Thank you for recognizing that false-premise replay gives a useful way to test whether a VLM can correct a false visual assumption. We address model coverage (W3), the reliability of the premise pairs (W1) and generalization beyond the selected failures (W2). Revisions are highlighted in blue.

W3. Model coverage: GPT, Gemini, Qwen3.5, Kimi K3.

Yes: false-premise replay lowers accuracy on all eleven models we added, including GPT-6-Astra, Gemini 3.8 Flash, five Qwen3.5 models and Kimi-K3.1-Pro. A diagnostic finding should hold beyond the models that produced the premises, so following your suggestion each added model answers the 217 premise pairs of the shared evaluation set with the paper's prompts (Section 4.1 and Table 2; Appendix E.1, Table 20). Fresh − false is the accuracy under fresh query (the question with the image and no earlier turn) minus the accuracy under false-premise replay (a false visual premise from a failed reasoning trace, placed in an earlier assistant turn). Corrected − false uses corrected-premise replay, which replaces only the false premise with its corrected version, in place of fresh query. Percentage points; brackets give 95% intervals from the paired image-cluster bootstrap.

ModelFresh − falseCorrected − false
Proprietary
GPT-6-Astra6.0 [2.7, 9.8]11.5 [6.7, 16.8]
Claude Opus 5.55.1 [1.9, 8.3]11.1 [6.3, 16.3]
Gemini 3.8 Flash10.1 [5.6, 15.1]18.0 [12.6, 23.9]
Open-weight
Kimi-K3.1-Pro26.3 [19.1, 33.8]34.1 [27.3, 41.1]
Qwen3.5-397B-A17B34.6 [26.3, 42.8]57.1 [49.8, 64.6]
Qwen3.5-122B-A10B38.2 [30.3, 46.1]57.6 [49.8, 65.0]
Qwen3.5-35B-A3B31.3 [23.6, 39.1]57.6 [49.8, 65.1]
Qwen3.5-27B34.1 [26.3, 41.9]53.5 [45.6, 61.1]
Qwen3.5-9B37.3 [30.0, 44.9]43.3 [34.7, 51.6]
Qwen3-VL-32B31.8 [24.4, 39.6]59.4 [51.4, 67.1]
InternVL3.5-38B27.2 [19.4, 35.4]59.4 [51.7, 67.1]

The drop does not depend on replaying a model's own errors: none of the added models produced any premise. GPT-6-Astra and Claude Opus 5.5 sample at the provider's default temperature; the table gives the first of three runs, and every contrast keeps its sign in the other two. With the six submitted models and the two Inkling models also run with reasoning, the revision covers 19 model configurations.

AuthorsOpenReview ↗

Response to Reviewer Dmwm (Part 2/3)

W1. Reliability of the premise pairs; human review covers 120 of 217.

Human review now covers all 217 pairs, and on the 143 pairs it supports, accuracy under corrected-premise replay still exceeds that under false-premise replay for every configuration. The replay contrast needs the false premise to be false and the corrected premise to be true, so following your suggestion two human reviewers judged each remaining pair in a second round, without seeing the labels (Section 2.2, "Human review"; Appendix A.1–A.3). The construction label is the model-review label that built the pair: false for the false premise and true for the corrected premise. Agreement over the premises of all pairs (Appendix A.3, Table 8):

Agreement on premises, all 217 pairsAgreement rate [95% interval]
Construction label vs each human reviewer84.1% [80.4, 87.5]
Human reviewer vs human reviewer82.9% [78.5, 87.1]
Construction label vs both reviewers, where they agree91.1% [87.4, 94.3]

The construction label agrees with the human reviewers about as often as the two reviewers agree with each other.

Supported pairs. Two further human reviewers, who saw neither the earlier judgments nor the labels, adjudicated the 34 first-round pairs on which the original two disagreed. A pair is supported when both of its premises resolve to the construction label under two vote rules fixed before any second-round judgment or adjudication was read: under statement-wise majority, a premise takes the value with strictly more votes, and a tie leaves it unresolved; under adjudication priority, the adjudicators decide when they agree with each other, and the premise is unresolved otherwise. 143 pairs are supported. The other 74 are 21 on which both original reviewers reject a label, 30 disputed in the second round, which have no adjudication, and 23 that the adjudication does not support.

Results on the supported pairs. Corrected − false is the accuracy under corrected-premise replay (only the false premise replaced by its corrected version) minus that under false-premise replay (the false premise in an earlier assistant turn), with 95% intervals (Section 4.1, "Label uncertainty"; Appendix E.1, Table 21). The two source models, whose errors built the pairs, and the three models with the smallest values:

Model, 143 supported pairsCorrected − false
InternVL3.5-8B46.2 [36.2, 56.3]
Qwen3-VL-30B-A3B49.7 [39.9, 59.3]
GPT-6-Astra5.6 [1.4, 10.5]
Claude Opus 5.56.3 [2.0, 11.3]
Gemini 3.8 Flash12.6 [7.0, 18.8]

Corrected − false holds without the rejected and disputed pairs: it excludes zero for all 19 configurations, and for the other 14 it is 22.4 to 53.8 points. On these pairs, the point estimates of all 19 configurations are also lower under false-premise replay than under fresh query (the question with the image and no earlier turn).

AuthorsOpenReview ↗

Response to Reviewer Dmwm (Part 3/3)

W2. A small, selected subset of failures, and generalization.

The drop under false-premise replay holds in four further tests: on models that produced no premise, on pairs built from another model's errors, with the domains weighted equally, and on a new test set built after submission. A low rate raises the question whether the effect comes from the selection; to address it, we report these four tests (Section 4.1; Appendix E.1, E.2 and E.5). False-premise replay places a false visual premise from a failed reasoning trace in an earlier assistant turn. Fresh − false is the accuracy under fresh query (no earlier turn) minus that under false-premise replay; corrected − false uses corrected-premise replay, which replaces only the false premise with its corrected version, in place of fresh query.

  • Models that produced no premise: the drop appears on each of the eleven added models, which produced no premise.
  • Another model's errors: on 87 pairs built from Qwen3-VL-30B-A3B errors, InternVL3.5-8B answers 40 correctly under fresh query and 13 under false-premise replay (Appendix E, Table 15).
  • Equal domain weights: fresh − false excludes zero for all 19 configurations (Appendix E.2, Table 22).
  • New test set: 72 chart and document pairs, built after submission from 53 images that none of the 217 pairs uses, with image review by gpt-6.1-terra, the successor of gpt-5.6-terra. Eight open-weight models answer it (Appendix E.5, Table 25).
Model, new test set (72 pairs)Fresh − falseCorrected − false
InternVL3.5-8B27.8 [14.7, 41.8]63.9 [49.4, 76.7]
Qwen3-VL-8B22.2 [9.6, 35.7]54.2 [39.4, 68.1]
Qwen3-VL-30B-A3B37.5 [25.0, 50.0]66.7 [54.2, 78.3]
Qwen3.5-35B-A3B37.5 [25.0, 50.7]70.8 [58.3, 81.5]
Qwen3.5-27B27.8 [14.5, 40.8]61.1 [48.6, 72.4]
Qwen3.5-9B44.4 [30.1, 59.2]72.2 [59.7, 83.8]
Qwen3-VL-32B45.8 [32.9, 58.6]75.0 [63.9, 84.9]
InternVL3.5-38B45.8 [31.4, 60.6]77.8 [66.2, 87.7]

Every interval on the new test set excludes zero, so the drop also holds on images outside the 217 pairs.

Why the 4.9–6.7% rate is low. The rate is the false-premise selection rate: accepted premises per candidate question, for each source model. An answer-free reference check removed 288 of the 1,070 candidate questions; each source model then sampled four reasoning traces for each remaining question (Section 2.2; Appendix A.4, Table 10):

Of the 782 mined questionsInternVL3.5-8BQwen3-VL-30B-A3B
All four traces correct558 (71.4%)575 (73.5%)
Failed, no accepted premise152155
of these, no single answer-relevant visual claim, or the error lies in the reasoning127118
Accepted premise7252

The rate is low mainly because the source models answer most questions correctly. Most other failures fail through reasoning rather than a visual claim, and the protocol excludes them by design because it tests visual premises. The selection uses no replay outcome.

We hope these additions address your concerns, and we would be grateful if you would consider raising your score.

Reviewer_qaEnOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduce false-premise replay, a diagnostic protocol that extracts a single answer-relevant incorrect visual claim from a model's failed reasoning trace, removes the remaining reasoning and final answer, and reinserts that claim as an earlier assistant message to test whether VLMs can use that image to revise prior errors.

The proposed evaluation compares fresh querying, false-premise replay, correct-premise replay, and several control conditions to isolate the impact of visual information. The authors construct a shared benchmark of 217 premise pairs spanning chart, document, geometry, and natural-image reasoning tasks constructed from 4 datasets/benchmarks. Evaluation are conduced across multiple VLMs, with models of varying parameter scales and reasoning configurations.

The experiments show that incorrect premises degrades performance despite continued access to the image. Additional analyses suggest that false premises impair both candidate generation and answer selection. The authors further demonstrate that visual-verification feedback improves correction of false premises but can harm performance when the original premises are correct, highlighting a tradeoff between error repair and answer retention.

Critical strengths
  • The paper addresses an important problem in whether a model can reconsider its own previously generated interpretation of visual evidence. This is relevant for multi-step reasoning in VLMs, where a mistake or hallucination introduced in the reasoning process may influence subsequent and final predictions.
  • The false-premise replay protocol provides a methodical evaluation by isolating a single erroneous visual claim and comparing it with a corrected version while keeping the image and question fixed. Inclusion of neutral and task irrelevant controls helps isolate the influence of incorrect premise content from simply the presence of having additional conversational context history.
  • The paper evaluates the proposed phenomenon across several VLMs, including InternVL, Qwen-VL, Inkling, and Kimi models. The shared benchmark is evaluated not only on models that generated the original errors but also on models that did not participate in premise construction to prevent circular bias in the models.
Critical weaknesses
  • The construction of the evaluation set raises concerns regarding reproducibility and generalizability. The paper lists four source datasets and reports three source-disjoint construction batches of 395, 575, and 100 questions. However, I could not find details explaining how the original questions were sampled from each benchmark. Were the selection random, or difficulty and question-type distributions were controlled. Furthermore, only a small fraction of candidate questions yield accepted false-premise examples (4.9–6.7%), resulting in a shared evaluation set of just 217 premise pairs across four domains.
    The dataset domain distribution is also uneven with 27 chart, 28 document, 97 geometry, and 65 natural-image pairs.

  • Another concern for me is whether the paper provides a sufficiently significant advance over existing studies on visual hallucination propagation and VLM self-correction. Prior work, including MMHalSnowball (ACL 2024), VISCO (CVPR 2025), and VisualSwap (ICML 2026), has already established that VLMs may propagate erroneous visual information, fail to reconsider visual evidence, and introduce errors during self-correction.
    For example, the proposed false-premise replay protocol offers a controlled evaluation by isolating a single erroneous premise and pairing it with a corrected counterpart while keeping the image fixed. However, several of the main empirical findings appear qualitatively consistent with observations established in prior work i.e the reported repair-versus-retention tradeoff substantially overlaps with VISCO's findings that self-correction can turn initially correct answers into incorrect ones.

Questions for authors
  • How were the initial 1,070 questions sampled from the four benchmarks? Were they randomly selected, and what were the sampling proportions and selection criteria for each domain?

  • The work describe the criteria for yielding a qualifying false premise, but i think some analysis such as systematic characterization of the selection process, including the rejection reasons, characteristics of accepted versus rejected examples would have been useful.
    The yield was low (4.9–6.7%) as acknowledge by the authors. Why does the construction pipeline yield so few accepted premise pairs? Could the authors report a breakdown of rejection reasons and compare accepted versus rejected questions? These analysis would be helpful in assessing if there are selection bias introduced by the pipeline.

  • Could the authors provide confidence intervals for the main replay effects within each domain? Additionally, would the main conclusions remain consistent under an equally weighted domain-level aggregation, given the uneven domain composition of the shared evaluation set?

  • Clarify which new insights or failure modes are uniquely revealed by false-premise replay that could not be established using existing evaluation protocols? In particular, which conclusions depend on the paired false/corrected-premise design rather than simply providing a more controlled confirmation of previously observed phenomena?

8 comments
AuthorsOpenReview ↗

Response to Reviewer qaEn (Part 1/4)

Thank you for the detailed questions, and for noting that the protocol isolates a single erroneous visual claim. We address the sampling of the questions (Q1), the construction funnel and selection bias (Q2), per-domain effects (Q3) and what the paired design adds (Q4). Revisions are highlighted in blue.

Q1. How were the 1,070 questions sampled?

Within each domain, questions were taken in a fixed seeded order from three candidate pools, each frozen before either source model answered its questions. The sampling determines what the evaluation set represents, so following your suggestion the revision documents it (Section 2.2; Appendix A.4, Table 9). The source models, whose errors built the premise pairs, are InternVL3.5-8B (InternVL) and Qwen3-VL-30B-A3B (Qwen-30B).

PoolChart (ChartQA)Document (InfographicVQA)Geometry (Geometry3K)Natural image (GQA)
395 questions2004035120
575 questions250750250
100 questions25252525
All three pools (1,070)47514060395
Fourth pool, 400 questions (model-specific sets)14018122120
  • Quotas. The 395 and 100 pools fixed their domain quotas in advance; the 575 pool set its quotas from their yields and added no geometry questions, because geometry already had enough pairs.
  • Selection criteria. The 575 and 100 pools reject questions answerable from a single reading: a chart question must need an explicit chart operation after reading the values, and a document question must carry arithmetic or comparison metadata.
  • Fourth pool. It was drawn per domain in a seeded order and frozen in the same way. Its two model-specific sets and the 124 pairs of the primary evaluation set, built from the three pools, form the 217 pairs of the shared evaluation set, which every model answers.

Within a domain, no model outcome chose the questions.

Q2. Why so few accepted pairs, and the rejection reasons.

The yield is low mainly because the source models answer most questions correctly in all four sampled traces. Following your suggestion, the revision reports the construction funnel with its rejection reasons (Section 2.2; Appendix A.4, Table 10). An answer-free reference check first removed 288 of the 1,070 questions, whose dataset answer could not be confirmed from the image; each source model then sampled four reasoning traces for each remaining question, and image review by gpt-5.6-terra judged each extracted claim.

Outcome of the 782 mined questionsInternVLQwen-30B
All four traces correct558575
Failed trace, but no single answer-relevant visual claim7371
Claim rejected: the error lies in the reasoning5447
Claim rejected: the image supports the claim1921
Claim rejected: problem with the reference answer514
Ambiguous, not accepted12
Accepted: one premise pair7252

Most failed questions that yield no premise fail through reasoning rather than a visual claim, as the two rows after the first show. The protocol excludes them by design because it tests visual premises.

AuthorsOpenReview ↗

Response to Reviewer qaEn (Part 2/4)

Q2. Accepted versus rejected questions, and selection bias.

The selection rate differs strongly by domain, accepted and rejected questions have similar answer types within a domain, and the selection uses no replay outcome. Selection could bias the evaluation set if it depended on how a model reacts to a replayed premise, so following your suggestion we compared the two groups (Section 2.2; Appendix A.4, the rates by domain). The false-premise selection rate is the share of candidate questions that yield an accepted premise for a source model, InternVL3.5-8B (InternVL) or Qwen3-VL-30B-A3B (Qwen-30B):

False-premise selection rateInternVLQwen-30B
Chart (475 candidates)1.9%2.1%
Document (140)8.6%7.9%
Geometry (60)40.0%23.3%
Natural image (395)6.8%4.3%

Within a domain, the two groups have similar answer types:

Mined questions, InternVLWith an accepted premiseWithout
Chart, numeric answer6 of 9217 of 337
Natural image, yes/no answer25 of 27243 of 257

Selection uses only a failed sampled trace and its review, before any replay. The replay effect also appears on a model whose errors did not build the pairs: on the 87 pairs built from Qwen-30B errors, InternVL answers 40 correctly under fresh query (the question with the image and no earlier turn) and 13 under false-premise replay (the false premise placed in an earlier assistant turn) (Appendix E, Table 15).

AuthorsOpenReview ↗

Response to Reviewer qaEn (Part 3/4)

Q3. Per-domain effects and an equal-weight domain aggregate.

Yes, the conclusions remain consistent under equal domain weights: both replay contrasts exclude zero for all 19 model configurations. With 97 of the 217 pairs from geometry, a pooled estimate could rest on one domain, so following your suggestion Appendix E.2 now gives per-domain intervals (Table 23) and the equal-weight aggregate (Table 22) for all 19 configurations: the six submitted models, the two submitted Inkling models also run with reasoning, and the eleven added models. Fresh − false is the accuracy under fresh query (no earlier turn) minus that under false-premise replay (the false premise placed in an earlier assistant turn), in points. Corrected − false uses corrected-premise replay, which replaces only the false premise with its corrected version, in place of fresh query.

Within each domain. Table 23 gives normal-approximation 95% intervals. No per-domain estimate is negative.

Domain (pairs)Configurations whose fresh − false interval excludes zero
Natural image (65)19 of 19
Chart (27)14 of 19
Geometry (97)12 of 19
Document (28)11 of 19

Most intervals that include zero (14 of 20) belong to the five configurations with the smallest drops over all 217 pairs: the three proprietary models and the two Inkling models with reasoning.

Equal domain weights. The aggregate is the mean of the four per-domain contrasts, with 95% bootstrap intervals. The two source models and the three models with the smallest drops:

Model, four domains weighted equallyFresh − falseCorrected − false
InternVL3.5-8B29.2 [21.4, 37.4]60.8 [52.2, 68.3]
Qwen3-VL-30B-A3B31.0 [21.8, 40.2]58.9 [50.6, 66.5]
GPT-6-Astra6.6 [2.1, 11.4]10.2 [5.7, 14.6]
Claude Opus 5.55.7 [2.3, 9.4]10.3 [6.2, 14.4]
Gemini 3.8 Flash9.2 [4.5, 14.1]17.3 [12.2, 22.7]

For the other 14 configurations, the equal-weight fresh − false lies between 16.1 and 40.0 points.

The result does not rest on geometry: for every configuration, the equal-weight fresh − false differs by at most 5.6 points from its value over all 217 pairs.

Reviewer_nhLnOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

By introducing a new dataset of premise pairs, where the same image and question are paired with either a false or a corrected visual statement, this paper investigates how VLMs self correct after making a false statement and explore under what conditions they can do this. Each false premise is taken from a model’s failed reasoning and paired with a corrected version of the same claim under the same image and question. This lets the authors measure both whether the model overturns an error and whether it keeps a correct statement. They find that when inserting a single false premise accuracy lowers by around 26–31pp compared to just asking the question fresh. They also compared different self-correction methods across six models and found that some methods that fix false premises also lose accuracy on correct ones.

Critical strengths
  • By evaluating self-correction mechanisms on both corrected and incorrect premises the paper finds that in many cases these methods lose accuracy on the correct premises as well as improving the incorrect ones. For example, on InternVL, feedback improves accuracy compared with self-reflection by 8.76 points when the statement is wrong but loses 8.76 points when it is correct, so overall it gives no benefit. The effect on already correct answers has received less attention so this is a non-intuitive and useful finding that would be helpful for anyone studying or implementing self-correction mechanisms.
  • The experimental design is well constructed and contains a good number of controls. By including the neutral and task-irrelevant replays the experiments show that the results are not just due to an extra turn being added to the input, and the corrected-premise replay demonstrates that the results are due to the false premise itself.
Critical weaknesses
  • The dataset is quite small. Only 4.9–6.7% of candidate questions are accepted, and the shared set only contains 217-pairs. Geometry makes up 97 of the 217 pairs, while chart and document questions add only 27 and 28 examples respectively. This size and composition make it hard to judge how well the findings generalise across other tasks.
  • Human reviewers disagreed on 34 of 120 reviewed pairs and agreed that the label was wrong on a further 7, which raises concerns about label reliability. Additionally, human review covers only 120 of the 217 shared pairs and only checks whether each premise is true. It is not clear why only 120 of the pairs were reviewed this way. For the finding that inserting a false premise lowers accuracy compared with asking the question fresh, the authors show the result still holds with wider uncertainty. This is discussed in the appendix but I think that the main text would benefit from an overarching discussion as to how this label uncertainty affects how strongly the results should be interpreted and be clear about which results have been verified on the human supported subset of the data.
Questions for authors

Can you elaborate on the human review choices that I detail in the second point of the weaknesses?

7 comments
AuthorsOpenReview ↗

Response to Reviewer nhLn (Part 1/3)

Thank you for highlighting the repair-versus-retention finding and for noting that the controls isolate the effect of the false premise. We address the human review and label reliability (W2 and your question), the results on the human-supported pairs, and the size and composition of the evaluation set (W1). Revisions are highlighted in blue.

W2 / Question. Human review choices and label reliability.

Human review now covers all 217 pairs: the 120-pair limit is removed, and the 34 disputed first-round pairs have a blind adjudication. The 120 pairs of the first round were a seeded sample, stratified by domain and by the set each pair was built in, without looking at any replay outcome. Following your suggestion, no pair is now left without human review: the revision reviews the rest and states which results hold on the human-supported pairs (Section 2.2, "Human review"; Appendix A.1 and A.3).

  • Review rounds. In the second round, two human reviewers judged each of the remaining pairs with the same protocol. Two further human reviewers, who saw neither the earlier judgments nor the labels, adjudicated the 34 first-round pairs on which the original two disagreed.
  • Truth only. Replaying a false premise and its corrected version needs the false premise to be false and the corrected premise to be true, so human review checks truth. Relevance to the question is checked by the model review (gpt-5.6-terra).

What the disagreements mean. The construction label is the model-review label that built the pair: false for the false premise and true for the corrected premise. Agreement over the 434 premises (Appendix A.3, Table 8):

Agreement on premises, all 217 pairsAgreement rate [95% interval]; Cohen's κ
Human reviewer vs human reviewer82.9% [78.5, 87.1]; κ 0.66
Construction label vs each human reviewer84.1% [80.4, 87.5]; κ 0.68
Construction label vs both reviewers, where they agree91.1% [87.4, 94.3]

A disagreement between human reviewers does not by itself mark a wrong label: the construction label agrees with the human reviewers about as often as they agree with each other.

Supported pairs. A pair is supported when both of its premises resolve to the construction label. Two vote rules, fixed before any second-round judgment or adjudication was read, combine the votes. Under statement-wise majority, a premise takes the value with strictly more votes, and a tie leaves it unresolved; under adjudication priority, the adjudicators decide when they agree with each other, and the premise is unresolved otherwise. 143 pairs are supported under both rules, and 144 under statement-wise majority. The other 74 are the 21 on which both reviewers reject a label, the 30 disputed in the second round, which have no adjudication, and the 23 adjudicated pairs that the adjudication does not support.

AuthorsOpenReview ↗

Response to Reviewer nhLn (Part 2/3)

W2 / Question. Which results hold on the human-supported pairs.

All four results below keep their direction on the 143 human-supported pairs; corrected − false on all 19 configurations, and fresh − false and corrected − false on the primary evaluation set, also keep 95% intervals that exclude zero. Following your suggestion, the new paragraph "Label uncertainty" in Section 4.1 and Appendix A.2 state them (Section 4.1; Appendix A.2, Tables 6 and 7; Appendix E.1, Table 21; Appendix E.3). False-premise replay places a false visual premise from a failed reasoning trace in an earlier assistant turn. Corrected-premise replay replaces only the false premise with its corrected version. Fresh query is the question with the image and no earlier turn. Fresh − false and corrected − false are the differences in accuracy between these conditions, in points.

Direct replay. For all 19 configurations (the six submitted models, the two submitted Inkling models also run with reasoning, and the eleven added models), the point estimates are lower under false-premise replay and higher under corrected-premise replay than under fresh query. Corrected − false excludes zero for every configuration: 5.6 to 53.8 points (Table 21).

The analyses of Section 3, recomputed on the supported pairs (Table 7). The primary evaluation set is the 124 pairs built from the own errors of the two source models (InternVL3.5-8B and Qwen3-VL-30B-A3B), each answered by its source model. Neutral replay puts a generic statement in place of the premise. The unary test asks whether the image supports a premise, and the pairwise test which of the two premises it supports. False premise stated is the share of responses that state the false premise. The last two rows use the 61 supported pairs among 82 InternVL3.5-8B pairs not used to develop that analysis: coverage is the share of pairs with a correct answer among four sampled ones, and selection accuracy the share of correct choices from a fixed set of four candidates; both compare neutral text of the premise's length with the false premise.

Analysis, supported pairsPairsEstimate [95% interval]
Fresh − false, primary evaluation set8318.1 [9.4, 27.6]
Corrected − false, primary evaluation set8359.0 [47.1, 70.7]
Neutral − fresh, primary evaluation set830.0 [−10.2, 9.9]
Unary test accuracy, primary set (chance 50%)8349.4% [43.2, 55.6]
Pairwise test accuracy, primary set (chance 50%)8350.6% [41.9, 59.0]
False premise stated, replay − fresh14328.7 [20.0, 37.4]
Coverage, neutral − false6123.0 [9.8, 36.1]
Selection accuracy, neutral − false618.2 [0.8, 16.4]

The feedback trade-off. Visual-verification feedback has the model answer atomic questions about the premise from the image and then critique its answer; self-reflection asks the same model to check and revise its answer against the image. The gain is feedback minus self-reflection under false-premise replay, and the loss is self-reflection minus feedback under corrected-premise replay, in points (point estimates, Table 6):

Model, 143 supported pairsGain under false-premise replayLoss under corrected-premise replay
InternVL3.5-8B7.010.5
Qwen3-VL-8B11.913.3
Inkling-Small24.56.3
Inkling17.59.1
Kimi-K2.611.93.5

Correction procedures on 13 models. Each model gives six comparisons: five correction procedures (self-reflection, visual-verification feedback, four-answer vote, history reset, and reset with vote) against the unchanged replay answer, and feedback against self-reflection. History reset returns the answer under fresh query; the vote takes the most frequent of four sampled answers. Scored on false-premise replay alone, 54 of the 78 show a gain whose 95% interval excludes zero. Scored on balanced accuracy, the mean over both premise conditions, 25 of these 54 are net gains (interval above zero) and none is a net loss (Appendix A.2).

Repair and retention stay distinct in the point estimates on the human-supported pairs: each of the five models still gains under false premises and loses under corrected premises.

AuthorsOpenReview ↗

Response to Reviewer nhLn (Part 3/3)

W1. Dataset size and composition (geometry 97 of 217).

The results do not rest on geometry: they hold with the four domains weighted equally and on a new test set of chart and document pairs built after submission. With 97 of the 217 pairs from geometry, a pooled result could rest on one domain; to address it, we ran both checks (Section 4.1; Appendix E.2 and E.5). Fresh − false is the accuracy under fresh query (no earlier turn) minus that under false-premise replay (the false premise placed in an earlier assistant turn). Corrected − false uses corrected-premise replay, which replaces only the false premise with its corrected version, in place of fresh query.

Equal domain weights. Fresh − false and corrected − false exclude zero for all 19 configurations (Appendix E.2, Table 22).

New test set. 72 chart and document pairs, built after submission from 53 ChartQA and InfographicVQA images that none of the 217 pairs uses, with image review by gpt-6.1-terra, the successor of gpt-5.6-terra. Over the whole set, fresh − false and corrected − false exclude zero for all eight open-weight models run on it (Table 25). Corrected − false within each domain, with 95% intervals (Appendix E.5, Table 26):

Model, new test setChart (45 pairs)Document (27 pairs)
InternVL3.5-8B64.4 [48.8, 78.4]63.0 [34.6, 88.0]
Qwen3-VL-8B44.4 [25.6, 61.9]70.4 [46.7, 89.7]
Qwen3-VL-30B-A3B60.0 [42.9, 75.6]77.8 [60.7, 92.9]
Qwen3.5-35B-A3B66.7 [50.0, 81.4]77.8 [61.5, 92.3]
Qwen3.5-27B55.6 [40.9, 69.2]70.4 [48.1, 88.5]
Qwen3.5-9B73.3 [56.8, 87.5]70.4 [51.7, 88.0]
Qwen3-VL-32B68.9 [54.5, 81.8]85.2 [67.7, 100.0]
InternVL3.5-38B68.9 [52.4, 83.3]92.6 [80.0, 100.0]

Within each domain of the new test set, every interval excludes zero, on images that none of the 217 pairs uses.

We hope these additions address your concerns, and we would be grateful if you would consider raising your score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the final revision

We thank the reviewers and the AC for their time. Reviewers found the problem important (Dmwm), the experimental design well constructed with a good number of controls (nhLn), and the work technically sound (qaEn). This post lists the changes in the revised PDF, marked in blue, by concern. Fresh − false and corrected − false are the accuracy under fresh query and under corrected-premise replay, each minus that under false-premise replay, in points with 95% intervals.

Concern (raised by)What we addedWhere
Model coverage (Dmwm)Eleven more models, the frontier proprietary models GPT-6-Astra, Claude Opus 5.5 and Gemini 3.8 Flash and eight open-weight models, among them five Qwen3.5 models and Kimi-K3.1-Pro. Both contrasts exclude zero for all 19 configurationsSection 4.1, Table 2, App. E.1
Reasoning effort (Dmwm)Nine models rerun at other reasoning settings, with seven effort levels for the two Inkling models and all five levels for Claude Opus 5.5. The effect stays significant at every setting of every model, with both contrasts excluding zeroApp. E.7
Label reliability (Dmwm, nhLn)Human review of all 217 pairs. The construction label, the model-review label that built each pair, agrees with the human reviewers on 84.1% of 868 judgments, and the two reviewers agree with each other on 82.9%. Corrected − false excludes zero for all 19 configurations on the 143 pairs that human review supportsSection 2.2, Section 4.1, App. A.1 to A.3, E.1
Sampling and generalization (Dmwm, qaEn, nhLn)A set built from 1,200 questions drawn at random from four datasets, frozen before any model output, with no mining step. Both contrasts exclude zero for all 14 models. App. A.4 now gives how the original questions were sampled, in seeded order from candidate pools frozen before the source models answered them, and the construction funnel with the reason for each rejection. Selection uses no replay outcome, and the drop under false-premise replay also appears on the eleven added models, which produced no premiseApp. A.4, E.6
Domain composition (qaEn, nhLn)Results with the four domains weighted equally, with per-domain intervals, and a new test set of 72 chart and document pairs. Both contrasts exclude zero for every model in bothApp. E.2, E.5
What the paired design adds (qaEn)A matched false and corrected premise on the same item separates the repair of a false premise from the loss of a correct one. Five correction procedures against the unchanged answer, and feedback against self-reflection, on 13 models. Of 78 comparisons, 65 gain on false premises alone, and 35 of these remain gains on balanced accuracy, the mean over both premise conditions, and none is a net loss. Section 6 compares the design point by point with MMHalSnowball, VISCO and VisualSwapSection 6, App. E.3
Premises from model reasoning (qaEn, 17 November)On 165 of the 217 pairs and averaged over ten open-weight models, premises from failed traces turn a share of the fresh-correct answers wrong that is 10.3 [5.0, 15.9] points larger than constructed premises in the same position. On the randomly drawn set, visual-verification feedback raises expected accuracy only when the share of false earlier premises exceeds a break-even share of 47% to 63%, depending on the modelApp. E.6
Presentation (nhLn)Section 2.2 now opens with an overview table of all evaluation setsSection 2.2

Two further changes. Results on the 217 pairs now remove units and LaTeX that hosted models add to an answer, for example "8.73 ft" for 8.73, before the answer matcher, with the same rule for every model (Section 2.3, App. E.4). Section 2.2 and App. A give the number of traces the two source models sampled as 6,256, not 8,560 as the submission stated.

We thank the reviewers again for comments that improved the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.