- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies whether vision-language models can retrieve and use information from a specified source after conflicting evidence has been integrated into an intermediate representation and direct access to the original evidence has been removed. The authors vary whether an inserted summary omits source identities, lists source names or arbitrary handles without explicitly pairing them with values, or records explicit source–value associations. They distinguish named-source retrieval from direct source selection and reliability-conditioned selection using controlled tasks involving counts, colors, and chart values. In Qwen3-VL-8B and Pixtral-12B, source-neutral summaries lead to poor retrieval of non-default sources, while carrying source handles through integration substantially improves retrieval.
Critical strengths
- The paper introduces a setting that uses an intermediate representation and removes access to the original evidence before the source-conditioned question arrives.
- The paper reports prerequisite failures in Qwen3-VL-8B and Pixtral-12B.
Critical weaknesses
- I strongly suspect that the paper was generated using AI. The appendix appears to consist entirely of raw LLM output without human review.
- Why does the paper use only eight pages for the main text but more than 20 pages for the appendix?
- The benchmarks cover only simple questions about counts, colors, and chart values, and the experiments evaluate only two vision-language models.
- The experimental setting, protocol, and dataset are not clearly described in the current text.
- Some parts of the paper and appendix are unreadable.
- The paper’s contribution is entirely unclear.
Questions for authors
- Can the authors understand their own appendix?
The contribution. When a vision-language model has integrated conflicting sources into a summary and direct access to the evidence is then removed, it keeps a default answer but can no longer retrieve what a named non-default source reported. For Qwen3-VL-8B with a source-neutral summary, retrieval is 0.16, below a 0.50 shortcut ceiling. A failure to follow a later "that source is unreliable" can therefore be an access failure rather than a trust failure. Arbitrary source handles carried in the summary, with no value attached to them, restore most retrieval (0.80). Blocking the queried handle, but not an unqueried one, removes it in Qwen3-VL-8B and Gemma-3-12B-it (specificity +0.55 and +0.64 for retrieval).
What is new relative to prior work. Source-monitoring and binding studies examine attribution and label binding while the evidence remains readable. Other work shows that image information persists after image-attention knockout, without asking whether a named source can still be addressed. Late-correction studies after compression are text-only. Our design combines four elements:
This combination is what exposes the access failure and the dependence on carried handles. We do not claim a new mechanism.
The protocol, in one example.
Corrections. Re-checking the submission against the raw records found errors, all corrected in the revision:
We report unfavourable pre-set results exactly as frozen. Pixtral's ablation prerequisite failed, its layout re-test is indeterminate, we make no claim under the combined reliability cue, and model-written handle summaries failed their generation gate.
The appendix and AI use. You are right that the submitted appendix contained raw program output and editing damage. LLMs were used extensively for planning, code, analysis and drafting. We have revised the AI-use statement accordingly and take responsibility for the content. In the revised appendix, raw output is replaced with prose and captioned tables, corrupted experiment names are restored, and secondary controls move out of the main text.
Scope and length. The tasks are deliberately simple and single-attribute, with one natural-image check (100 VQAv2 photographs, colour, one model). The failure and the rescue hold in two models; two further models failed the prerequisite even with the evidence visible. The specificity result holds in Qwen and Gemma. The main text now presents one sequence, with secondary controls in the appendix: