- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how prior reasoning affects visual updating and follow-up answers in VLMs. Using the VisualSwap dataset, the authors evaluate 16 VLMs by removing different types of reasoning sentences, changing sentence order, and asking follow-up questions. The results show that removing relevant information from prior reasoning reduces the influence of old answers. The appendix also introduces FSAF, which improves visual updating by blocking attention to prior reasoning.
Critical strengths
The research problem is meaningful. The paper studies how prior reasoning interferes with visual updating in VLMs, which is a practical issue.
Critical weaknesses
- The experimental controls are insufficient. Appendix D.3 shows that evidence deletion removes 75.67% of the text, while non-evidence deletion removes only 8.93%. This large difference makes it unclear whether the results come from removing important evidence or simply removing more text. In addition, evidence sentences are identified using broad keyword rules without sufficient validation.
- The research contribution is limited. VisualSwap has already studied how prior CoT affects visual updating. This paper mainly extends that work with additional experiments, but most findings are intuitive and lack deeper explanations. The paper also lacks comparisons with text-only LLMs, making it unclear whether the problem is specific to VLMs. Although FSAF improves performance over ordinary reflection, it is not compared with simpler methods such as directly removing old CoT.
Questions for authors
- How do the authors ensure that the improvement comes from removing important evidence rather than simply removing more text, given the large difference in deletion ratios?
- Please clarify whether similar problems would occur in text-only LLMs when image information is replaced with text descriptions, and what makes this problem specific to VLMs.
- Please discuss the advantages of FSAF over simply removing old CoT, given that both approaches prevent the model from relying on prior reasoning.