Seeing Is Not Persuading: Authority Cues Drive VLM Belief Shifts
Abstract
Reliable evidence evaluation should depend on what evidence says and whether its provenance can be verified, rather than on how credible it appears. Yet vision-language models (VLMs) increasingly assess information presented as screenshots, reports, and institutional documents, where visual presentation can imitate provenance without authenticating it. We investigate whether VLM judgments remain stable when substantive evidence content and interaction history are fixed but its presentation is altered. To this end, we introduce Apparent Provenance Manipulation (APM), a paired evaluation framework that preserves substantive evidence and compares a nested sequence of presentation bundles spanning visual modality, document embodiment, and graphical authority styling. Across three benchmarks and four representative VLMs, simply rendering identical text as an image does not increase its influence. In contrast, increasingly document-like and authority-styled renderings receive greater weight, although the adjacent conditions do not isolate single visual cues. This effect is conditional on semantic relevance: official-looking but irrelevant artifacts remain largely ineffective, indicating that apparent provenance amplifies relevant evidence rather than acting as an unconditional visual trigger. The induced judgment changes persist under subsequent paraphrased questioning and generalize across languages, attacker configurations, and document formats. Our findings are consistent with an unauthenticated-provenance shortcut: model judgments shift across easily spoofed presentation bundles despite unchanged evidence text and no verified source. This reframes visual misinformation as a failure of epistemic robustness and motivates provenance-aware evaluation and model design. Our code is available at https://anonymous.4open.science/r/APM
Then back it, or bet against it.
Related papers
Open the market on this paper to see 7 more related papers.