acceptodds
Under review as a conference paper at ICLR 2027

Seeing Is Not Persuading: Authority Cues Drive VLM Belief Shifts

Abstract

Reliable evidence evaluation should depend on what evidence says and whether its provenance can be verified, rather than on how credible it appears. Yet vision-language models (VLMs) increasingly assess information presented as screenshots, reports, and institutional documents, where visual presentation can imitate provenance without authenticating it. We investigate whether VLM judgments remain stable when substantive evidence content and interaction history are fixed but its presentation is altered. To this end, we introduce Apparent Provenance Manipulation (APM), a paired evaluation framework that preserves substantive evidence and compares a nested sequence of presentation bundles spanning visual modality, document embodiment, and graphical authority styling. Across three benchmarks and four representative VLMs, simply rendering identical text as an image does not increase its influence. In contrast, increasingly document-like and authority-styled renderings receive greater weight, although the adjacent conditions do not isolate single visual cues. This effect is conditional on semantic relevance: official-looking but irrelevant artifacts remain largely ineffective, indicating that apparent provenance amplifies relevant evidence rather than acting as an unconditional visual trigger. The induced judgment changes persist under subsequent paraphrased questioning and generalize across languages, attacker configurations, and document formats. Our findings are consistent with an unauthenticated-provenance shortcut: model judgments shift across easily spoofed presentation bundles despite unchanged evidence text and no verified source. This reframes visual misinformation as a failure of epistemic robustness and motivates provenance-aware evaluation and model design. Our code is available at https://anonymous.4open.science/r/APM

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Vision-Language Models, Multimodal Robustness, Visual Persuasion, Apparent Provenance, Multimodal Misinformation, Adversarial Evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_M1hpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  • Investigates how VLM judgments change with the visual presentation and apparent authority of substantively identical evidence.
  • Proposes Apparent Provenance Manipulation (APM), a paired evaluation framework comparing Plain Text, Text-in-Image, Neutral Document, and Authoritative Document under fixed evidence content and conversation history.
  • Finds that official-looking documents are more persuasive than simple text-in-image renderings, particularly when the evidence is semantically relevant to the target claim.
Critical strengths
  • Provides a systematic evaluation framework that controls evidence content and conversation history to examine the influence of visual presentation on VLM judgments.
  • Evaluates multiple VLMs and datasets, with additional analyses of semantic relevance, document formats, and belief persistence.
  • Investigates a practical reliability concern regarding the influence of unverified visual authority cues on VLM judgments.
Critical weaknesses
  • Prior studies have already investigated authority bias and multimodal persuasion, limiting the methodological novelty beyond extending these findings to visual document presentation.
    • Authority Bias in Conversational Search Engines for Academic Paper Recommendation (EMNLP 2026)
    • LLMs Trust Humans More, That's a Problem! Unveiling and Mitigating the Authority Bias in Retrieval-Augmented Generation (ACL 2025)
    • .
  • Authoritative and Neutral Documents differ in layout, typography, and potentially legibility, making it difficult to isolate the effects of authority cues.
  • Reliance on self-reported confidence makes it difficult to distinguish persistent belief changes from temporary response compliance.
  • ASR includes successes from earlier textual persuasion rounds, making it difficult to assess the independent contribution of the final visual evidence.
Questions for authors

I would appreciate it if the authors could address the concerns raised in the weaknesses above.

2 comments

We thank Reviewer M1hp for the comments on positioning and measurement. We answer the three concerns below.

Question-1: Novelty relative to prior authority-bias work. Answer-1: We agree that authority bias and multimodal persuasion are established topics. The paper does not claim to discover authority bias. Prior work varies author and venue metadata in academic recommendation or textual source identity in retrieval-augmented generation. Neither holds substantive evidence and interaction history fixed while comparing rendered document presentations. The contribution is the formulation of evidence-presentation robustness and a paired, branch-isolated test of apparent visual provenance, including the distinction between apparent and verified provenance. The paper therefore presents an operational and empirical advance rather than a new psychological phenomenon. Both cited papers are discussed in Related Work.

Question-2: Layout, typography, and legibility. Answer-2: The paper's claim is at the bundle level, not an atomic authority-cue effect. Neutral and authoritative documents may differ in legibility, typography, and layout; the paper says so in Section 5.3 and the Limitations. The OCR(text) control, cue ablation, and occlusion analysis are the evidence the paper provides. The ablation is non-monotonic and does not support independent additive cue effects. A layout-matched OCR-like comparison would isolate legibility, but that is a different question. The paper's claim is that document-like presentation bundles change judgments, not that one graphical cue causes the entire difference.

Question-3: Measurement validity: confidence, persistence, and ASR. Answer-3: Confidence is an expressed behavioral signal, not a calibrated belief probability. The paper supplements it with Persist@1, S(5)S(5), half-life, and Wrong Frac under paraphrased follow-up questions with the history fixed and no new persuasive evidence. This is a direct behavioral test of persistence rather than a second persuasion attempt. The authoritative document has the highest short-term retention and longest half-life. The paper treats this as conditional retention, not proof of an internal belief state, and notes that manipulated subsets can differ across conditions.

ASR is cumulative by construction, and the paper does not use it alone to infer the image-stage contribution. ISR measures final-stage threshold crossing, IPR measures paired confidence decrease after image submission relative to the preceding textual stage, and the controlled ladder uses final confidence under a shared interaction history. The Legal Document case illustrates why ASR alone is insufficient: it has the highest ASR (0.73) but a low IPR (0.31). The paper uses ASR for end-to-end success and ISR/IPR for the image-stage analysis.

Thank you for the detailed rebuttal and clarifications.

While the responses have clarified several points, my main concerns remain regarding the limited methodological novelty and the difficulty of isolating authority effects from other presentation factors.

I will therefore maintain my original rating.

Reviewer_sMC7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Seeing Is Not Persuading: Authority Cues Drive VLM Belief Shifts essentially looks at how the content being fed in affects how it may change the VLMs final belief on what the answer. The paper observes that feeding in authoritative documents is able to sway VLMs decision the most.

Critical strengths
  • Provides a well-structured experimental framework that controls for underlying evidence content and conversation history, allowing for more direct comparisons of how different presentation formats influence VLM beliefs.
Critical weaknesses
  1. Insufficient statistical validation: The central comparison between neutral and authoritative documents yields a relatively small average confidence difference (2.69 vs. 2.47). However, the paper does not report paired uncertainty estimates or statistical significance tests, making it difficult to establish the reliability of this difference.

  2. Limited causal isolation of authority cues: Although the underlying evidence is held constant, neutral and authoritative documents differ in multiple presentation characteristics, including layout, typography, and potentially readability. The authors acknowledge these confounds, and their cumulative ablations do not independently isolate individual cues. Consequently, the experiments do not establish how much of the observed effect is specifically attributable to perceived authority.

Questions for authors
  1. Can the authors provide paired confidence intervals or statistical significance tests for the differences between neutral and authoritative documents? How consistent are these differences across models and datasets?

  2. Given that neutral and authoritative documents may differ in readability, typography, and layout, what evidence supports attributing the observed differences specifically to authority cues? Were any systematic quality assessments conducted to establish that the generated documents are comparable across conditions?

2 comments

We thank Reviewer sMC7 for the focused questions on statistical validation and confound control. The key clarification is that the paper's claims are descriptive and bundle-level; the requested tests would support a stronger inferential claim than the paper makes.

Question-1: Paired uncertainty and consistency across models and datasets. Answer-1: Yes, the design is paired at the instance level: each dataset--model cell reuses the same 100 instances across presentation conditions. The paper reports single-run descriptive means and explicitly does not claim paired significance or a statistically established 0.22 neutral-to-authoritative increment. The central pattern is the larger document-bundle shift: 3.00 for Plain Text and 3.35 for Text-in-Image versus 2.69 for Neutral Document. The authoritative mean of 2.47 is a further descriptive shift. The paper's conclusion remains even if the 0.22 adjacent increment is disregarded.

The paper does not infer cross-model or cross-dataset consistency from the pooled means. It reports the pooled descriptive ladder and keeps the per-cell evidence descriptive; the relevance analysis is the setting in which official relevant artifacts have the highest ISR in every reported cell. The error bars in the figure are descriptive and are not used as confidence intervals. A formal paired test of the 0.22 increment would support a stronger claim than the paper makes, and its absence does not undermine the larger descriptive bundle contrast.

The title uses ``authority cues'' as a concise label for the authority-styled document bundle. The abstract and Method define the treatment at the bundle level; the paper does not claim that a single cue produces the full shift.

Question-2: Confounds and systematic quality assessment. Answer-2: The paper does not attribute the neutral-to-authoritative difference specifically to authority cues; it defines that contrast as a presentation-bundle effect. The reviewer's concern is valid for an atomic-cue claim, which the paper does not make. The manuscript reports generation-time checks for semantic consistency, content completeness, legibility, chronological consistency, and required presentation elements, together with the OCR(text) control, the progressive cue ablation, and the occlusion analysis. The cue ablation is non-monotonic, and the paper explicitly uses that result to reject independent additive effects.

The remaining differences in layout and typography are acknowledged as part of the bundle, not treated as controlled equivalence. The evidence therefore supports a presentation-bundle effect while remaining explicit that it does not isolate a single authority cue. A layout-matched OCR-like comparison would test legibility rather than the bundle-level question the paper asks.

Thank you for your review, this answers most of my questions, but I will be keeping my score.

Reviewer_V9iSOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper examines a gap between visual and textual information, by presenting textual information as an image to test whether an image version distorts authoritative credibility. The authors find that in increasing a formative style of a document image increases its weight on the decision to carry it as an authority or not, but the authors are not able to disentangle specific visual queues that lend weight to this judgment.

Critical strengths
  • The problem of provenance and trust is important, and VLMs are a current and strong source of attack. Such a study is welcome to open up and examine this area.
  • The incremental framework and mathematical notation is crisp and strongly helps to frame the set of related problems in a useful and embodied framework for understanding.
Critical weaknesses
  • A key problem is that the work may measure the gap (I don't think it really does, see below) but doesn't present a clear analysis of how the gap is diagnosed or addressed.
  • The authors choose many different forms of analyses but as the analyses are superficial, I cannot say I learned much from the work performed.
  • The argumentation that a shift to an image representation heavily weights and shifts the confidence of incorrect judgements seems correct; however, it is counteracted by the author’s own conclusions that there is no persuasive gain from the text-in-image rendering (L087).
  • Quite a few of the citations are not embedded appropriately inside of the body text and contexts: if not part of the prose text proper, they should be parenthesised (e.g., L068, 071, but pretty much anywhere in the introduction)
  • If synthetic evidence is taken as correct (L090), and falsely, so, it would be important to say why and how a model would be able to conclude that it is not correct (misinformation), and to contrast that against textual knowledge.
  • The content-preserving paired intervention is supposed to test for authority, but in adding some specific hallmarks, the system must create additional information not available in the text modality. Said another way, you don’t have a comparison only of (0, *, 1). I understand that you say that “a” is meant specifically as graphical authority style, but we can also convey authority in a textual style too. How can you address this concern?
  • Similarly the details of what constitutes a controlled document is not very clear. The example in Figure 2 (when scrutinised under heavy zoom) introduced extra headers and footers with words that are not carried in the text modality, so a comparison may not be fair. I would suggest a version of the neutral document that simulates OCR output as a fair comparator instead.
  • Your work reports a fair number of metrics but don’t carefully define what these measures are in a self-contained way. Appendicular definitions are by definitions, not self-contained. Page limits should be maintained and you need to ensure that the key findings are distilled into the page limit.
  • I’m unsure why the authors need to go through rounds of textual persuasion since the hypothesis is on the manipulation of image evidence. I feel this weakens the contribution by presenting distracting variables.
  • Please do not fiddle with the spacing to compact more text into your submission. Use your editorial and writing power to give a clear storyline about your work. (L339 some form of \vspace manipulation was done).
  • I find a lot of phrases that seem to be the product of LLM-based writing. This is not a problem by itself, but when it interferes with reading and understanding for readers, it is. The authors are strongly encouraged to write from their direct perspective, rather than verify generated work. I’m sorry if I misattribute the team’s writing “should not be interpreted as”, “X but not Y”
  • While there is a lot of related work in this area, you may find it instructive to cite an earlier work that also examines adversarial that appears as precursor to works that you cite.
  • Guanzhen Li, Yuxi Xie, and Min-Yen Kan. 2024. MVP-Bench: Can Large Vision-Language Models Conduct Multi-level Visual Perception Like Humans?. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 13505–13527, Miami, Florida, USA. Association for Computational Linguistics.
Questions for authors

These are not so much questions as specific concerns on specific lines of the submission. I didn't find a field in the review for this, so I'm placing them here.

  • 054 Figure 1 text is too small, although helpful in reinterpreting the abstract. The authors should consider how the document looks like on the printed page as well.
  • 141 Vision–Language is a noun–noun compound and should use an endash as a separator, not a hyphen.
  • 384 the statement here says that the current conditions may also differ in legibility and type of graphic regularity, but I believe that earlier the authors did stated these are controlled. So I don't understand this statement.
7 comments

We thank Reviewer V9iS for the detailed reading. The comments concern one central issue---the scope of the presentation treatment---and several presentation details. We answer in five groups.

Question-1: Scope, apparent contradiction, and textual rounds. Answer-1: The apparent contradiction comes from using ``image'' at two different levels. In the controlled ladder, mean final confidence is 3.00 for Plain Text and 3.35 for Text-in-Image; modality transfer alone therefore shows no observed gain. It is 2.69 for Neutral Document and 2.47 for Authoritative Document; the shift appears when the same evidence is rendered as a document-like presentation bundle. L087 and the later results are therefore consistent. The paper does not claim that any image is persuasive, and it does not treat the 0.66 and 0.22 adjacent differences as atomic causal effects. The paper is a diagnostic evaluation, not a defense proposal: its contribution is to characterize the vulnerability and its boundary conditions, not to introduce a mitigation algorithm.

The analyses form one diagnostic chain. The ladder tests whether modality alone is sufficient. The relevance test asks whether official styling can operate without relevant content. The persistence test asks whether the effect survives rephrasing without new evidence. The cross-lingual and attacker comparisons ask whether it recurs beyond one setting. The cue ablation and occlusion analysis ask whether it reduces to a single visual element. The answers are consistent: modality alone is insufficient; official styling does not overcome irrelevant content; the authoritative bundle has the highest short-term retention and longest half-life; and the cue ablation is non-monotonic. The last result is why the paper treats the treatment as a bundle rather than an additive cue set. The estimates are explicitly descriptive; the paper does not claim statistical significance or a universal causal mechanism.

The textual rounds are not the presentation treatment. They construct the shared state Bi=(xi,Hi,ei,zi⋆)B_i=(x_i,H_i,e_i,z_i^\star), and all four presentation branches start from the same claim, evidence, and interaction history. The presentation contrast is branch-isolated and does not depend on textual-stage success. The five-versus-ten-round comparison shows that additional text does not consistently lower confidence, whereas the presentation step is associated with a further shift. The multi-round stage is a realistic prior, not a confounder for the controlled comparison.

Question-2: Content preservation, textual authority, and renderer differences. Answer-2: We agree that the R2--R3 step is not an atomic graphical-authority cue. The paper does not claim that it is: its estimand is the presentation bundle p=(m,d,a)p=(m,d,a), and the adjacent contrasts are explicitly path-specific. The R2--R3 bundle contains both graphical styling and source-identifying text, such as an institution name, document number, or signature. Those strings are part of apparent provenance, which is the object of study, not an uncontrolled addition. The reviewer's observation that authority can also be conveyed textually supports this framing: the paper studies a document-like bundle, not a logo in isolation.

The paper already contains three relevant controls. The OCR(text) condition removes the visual presentation while retaining the extracted text. The progressive authority-cue ablation adds document layout, organization name, logo, reference number, signature, and seal, and reports a non-monotonic trajectory (3.64, 3.46, 3.08, 3.94, 3.20, 2.76, 2.64). That non-monotonicity is direct evidence against an independent additive-cue interpretation. The occlusion analysis localizes image regions associated with the judgment. These analyses do not isolate a single graphical cue, and the paper does not claim that they do.

The headers and footers in Figure 2 are document-format and source-identifying elements, not additional propositional evidence. To the extent that they change apparent provenance, they are part of the treatment. The evidence body is held fixed across conditions, and the OCR(text) control provides the extracted-text counterpart. A layout-matched OCR-like rendering would be useful for isolating legibility, but that is a different estimand; the paper's claim is the bundle-level effect. The L384 statement refers to residual differences in legibility and typographic regularity, not to content completeness or date consistency. The paper does not claim strict equivalence of all rendering properties.

Question-3: Synthetic evidence and factuality. Answer-3:The paper does not assume that the model treats synthetic evidence as correct. The model may accept, reject, or qualify a claim on the basis of parametric knowledge, internal consistency, and plausibility. The black-box threat model intentionally provides no provenance-verification channel; the paper's question is whether presentation changes the weight assigned to the same substantive content, not whether the model can authenticate the source. The relevant contrast is therefore between plausible but false evidence and evidence that the model can evaluate against prior knowledge.

The existing relevance analysis provides direct evidence on this point. If official styling acted independently of content, officially styled artifacts would shift judgments even when their content was irrelevant. They do not: in the reported cells, official styling does not overcome semantic irrelevance. For example, on BBC with Claude, the three irrelevant conditions have ISR 0.13, 0.06, and 0.04, and the officially styled irrelevant condition is not stronger than its non-official counterpart. Relevant official artifacts, by contrast, have the highest ISR in every reported cell, reaching 0.49 for Claude and 1.00 for GPT and Gemini. This shows that content is a necessary condition and that the model is not simply accepting any official-looking image. The paper does not claim that the model can know a synthetic artifact is false; it measures how the model behaves when the artifact is plausible and its provenance cannot be verified.

Question-4: Citation style, metric definitions, and presentation. Answer-4: We appreciate the editorial points. The citations at L068 and L071 are integral citations; elsewhere the paper uses parenthetical citations where the citation is not part of the prose. The main text gives operational definitions of ASR and MinC, and Appendix B.5 gives the formal definitions. For convenience, the remaining metrics are: ISR, the final image-stage threshold-crossing rate; IPR, the paired rate at which confidence decreases after image submission relative to the preceding textual stage; IPS, ASR⋅IPR\sqrt{\mathrm{ASR}\cdot\mathrm{IPR}}; Persist@1 and S(5)S(5), retention at the first and fifth follow-up; half-life, the first follow-up at which retention falls to half its initial value; and Wrong Frac, the average fraction of follow-ups inconsistent with the original belief.

The paper uses the appendix for full formulas, prompt details, and artifact specifications, while the main text retains the controlled ladder, relevance test, persistence results, and cross-setting comparison. The spacing adjustment at L339 is a local typesetting choice and does not carry the argument. Figure 1 is vector graphics; the comment on print scale is noted. These presentation issues do not change the experimental evidence or the scope of the claims.

Question-5: Ethics and BBC content. Answer-5: The manuscript uses BBC News as a public benchmark and does not reproduce article text in the paper. The responsible-use discussion is in Appendix G because of the page limit; it covers the synthetic nature of the artifacts, the offline evaluation setting, the absence of human participants, and the defensive implications of the work. The code release is separate from the manuscript and is intended to contain only the experimental harness and document identifiers and metadata, not BBC article text or derived question--answer pairs.

I thank the author team for their rebuttal, which helps to clarify some of the doubts that I have with respect to the submission. I feel that some of the points in your rebuttal and clarifications (especially for Group 1 and the additional clarity in your Group 2 that it is a bundle effect and not ablation efforts) can make it into your writing for a subsequent submission; but respectfully, I still feel that your points require a full round of additional review in a subsequent cycle --- I would not be able to endorse your writing edits as of now.

Part of your rebuttal also reminds me of AI writing style, and I apologise if your writing was genuine grey-matter authorship, but please do engage directly with your reviewers rather than use an LLM to write on your behalf.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.