acceptodds
Withdrawn from ICLR 2027

Recompute or Reuse? Analyzing Stale Reasoning in Vision-Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision-language models can answer a question about an image correctly in isolation yet change their answer when given reasoning from an earlier image. We study which parts of this reasoning affect the answer and whether their influence continues after the model answers correctly. We call the tendency of prior reasoning to favor the old answer despite changed visual evidence a textual shortcut. Across 16 VLMs, we construct contexts containing the current image and an earlier response. We compare deleting sentences selected by fixed evidence rules, deleting non-evidence sentences, and masking explicit answer cues. Deleting the selected sentences shifts answer preference toward the current answer in all 16 models relative to non-evidence deletion; changing sentence order has a smaller and more variable effect. After an initially correct response, deleting textual evidence for the current answer causes a larger increase in old answers when the earlier reasoning is retained. When the evidence is kept, initially correct responses show little old-answer repetition, although accuracy falls on follow-up calculations. Responses that initially retain the old answer show much more repetition and more follow-up results matching calculations with the old value. These findings show which prior text affects visual updating and how this influence varies with the initial answer and the follow-up question.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Vision-Language Models, Self-Reflection, Chain-of-Thought Reasoning, Textual Shortcuts, Visual Grounding

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_veDgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how prior reasoning affects visual updating and follow-up answers in VLMs. Using the VisualSwap dataset, the authors evaluate 16 VLMs by removing different types of reasoning sentences, changing sentence order, and asking follow-up questions. The results show that removing relevant information from prior reasoning reduces the influence of old answers. The appendix also introduces FSAF, which improves visual updating by blocking attention to prior reasoning.

Critical strengths

The research problem is meaningful. The paper studies how prior reasoning interferes with visual updating in VLMs, which is a practical issue.

Critical weaknesses
  1. The experimental controls are insufficient. Appendix D.3 shows that evidence deletion removes 75.67% of the text, while non-evidence deletion removes only 8.93%. This large difference makes it unclear whether the results come from removing important evidence or simply removing more text. In addition, evidence sentences are identified using broad keyword rules without sufficient validation.
  2. The research contribution is limited. VisualSwap has already studied how prior CoT affects visual updating. This paper mainly extends that work with additional experiments, but most findings are intuitive and lack deeper explanations. The paper also lacks comparisons with text-only LLMs, making it unclear whether the problem is specific to VLMs. Although FSAF improves performance over ordinary reflection, it is not compared with simpler methods such as directly removing old CoT.
Questions for authors
  1. How do the authors ensure that the improvement comes from removing important evidence rather than simply removing more text, given the large difference in deletion ratios?
  2. Please clarify whether similar problems would occur in text-only LLMs when image information is replaced with text descriptions, and what makes this problem specific to VLMs.
  3. Please discuss the advantages of FSAF over simply removing old CoT, given that both approaches prevent the model from relying on prior reasoning.
Reviewer_9NXeOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper attempts to understand which factors cause vision-language models to be distracted by stale reasoning and fail to update their answer given a new image and a prompt to reflect. The authors refer to this tendency to favour the answer from the stale reasoning as a textual shortcut. Building on the VisualSwap dataset, they vary the stale context by deleting evidence or non-evidence sentences and masking answer cues. Additionally they vary the sentence order of this context. The main findings here are that removing evidence sentences moves the answer more towards the updated answer than non evidence sentences and that sentence order has a smaller and less consistent effect. They also investigate how, after reflection, the presence of this stale reasoning affects follow up questions. Here they find if that reflection corrected the answer, follow-up questions rarely bring back the old answer and if the reflection keeps this answer it persists even after re-asking.

Critical strengths
  • This is an interesting problem and understanding the factors that cause models to unsuccessfully re-evaluate new visual evidence could help inform new training techniques or context engineering approaches to improve a models ability to do this kind of reflective reasoning.

  • The evaluation is broad and the main effects are consistent across models. Observing the effect of old answer and correct answer reflections in follow up questions is an interesting extension of prior work and the neutral history control is a useful methodological choice that separates the effect of stale content from that of additional conversation turns.

Critical weaknesses
  • I do not think the experiments provide convincing evidence for one of the main findings, namely, that removing evidence sentences has a larger effect than removing non-evidence sentences. The main text states that the non-evidence condition (C) targets the amount of text removed by the evidence condition (E) and notes that it typically removes less, however this difference is large. According to App. D.3, E removes on average 75.67% of the words, while C removes only 8.93%. This confounds the finding that removing evidence sentences improves VUR, since the effect could be driven by the amount of text removed rather than its content. Additionally, in 27.53% of cases the non evidence setting removes no text at all, making it exactly the same as the baseline. Since Tab. 12 reports C over all 4,159 paired instances, these cases appear to be included, so in over a quarter of instances C is identical to the baseline (N). Overall, I think this confound limits the findings as they are currently stated in the paper.
Questions for authors

Could you report the E–C comparison restricted to instances where C removes a similar amount of text to E, or add an amount-matched control (e.g. random deletion regardless of label)?

Reviewer_qtxfOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies how reasoning generated for an earlier image affects answers after the visual evidence changes. Using VS-Bench, it retains model–example pairs solvable on both images and evaluates edits to the earlier response across 16 VLMs. It also examines support removal after an initially correct answer and subsequent repeated or derived questions. An appendix investigates an attention firewall. The distinction between old-answer recurrence and other errors is useful.

Critical strengths
  1. I agree with the author’s approach of first verifying that the model correctly answers questions based solely on either the old or the new image, and then comparing results while retaining differing textual elements.

  2. The use of 16 models and multiple controls strengthens the credibility of the findings. Additionally, the author distinguishes between “reiterating the old answer” and other types of errors, avoiding the conflation of all subsequent performance degradation with the influence of prior reasoning.

Critical weaknesses
  1. My primary concern is that deleting text supporting the old conclusion appears more effective primarily because it removes significantly more content. Appendix D.3 shows that this operation deletes, on average, 75.67% of the words, whereas deleting irrelevant text removes only 8.93%. This substantial disparity makes it impossible for readers to determine whether the improvement stems from the content of the deleted text or merely from the drastic shortening of the old reasoning—such shortening has already been observed to improve performance in prior work [1].

  2. Subsequent computational errors may simply reflect the model reusing its most recent incorrect answer. Section 5.2 retains all prior responses: if the model just gave an incorrect numerical answer and is then asked to double that number, using the erroneous value alone suffices to produce an incorrect result; no re-reading of earlier reasoning is required. Conversely, after the model first corrects its answer, computation results consistent with the old (incorrect) value increase accuracy by only 0.07 percentage points.

  3. The direct re-prompting results in Table 14 require explanation. In the “first correction correct” subset, the model had previously answered correctly without old reasoning, and that correct answer remains in the context; yet upon re-prompting with the same question, accuracy drops to 54.8%. This does not necessarily indicate data error, but it does imply that many errors occur even in the absence of old reasoning.

[1] Are VLMs Seeing or Just Saying? Uncovering the Illusion of Visual Re-examination

Questions for authors

Please see weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.