acceptodds
Under review as a conference paper at ICLR 2027

Self-Anchoring and Behavioral Dissociation in Closed-Loop Multimodal Verification

Abstract

Unified multimodal models (UMMs) increasingly verify their own visual generations in closed-loop self-critique and refinement. We show that such verification is not attribution-neutral. When the same image is framed as self-generated rather than generated by another model, UMMs systematically shift verification toward their generation-time intent, a phenomenon we call attribution-conditioned self-anchoring. Across nine UMMs spanning six LLM backbone families, self-anchoring exhibits distinct behavioral regimes and persists across tasks and capability-controlled stratification. We localize this effect to compact, task-stable sets of attention-shift coordinates and establish their causal role through matched ablations. Removing the top-ranked set reduces anchoring by 43% in Bagel while nearly eliminating attribution discrimination under an AUROC-based reliability measure, far exceeding random controls. The same intervention attenuates both behavioral anchoring and confirmation amplification, linking them to a shared internal pathway. A dependency-structure analysis further predicts when verification produces confirmation amplification rather than corruption, explaining the reversal from prior text-RAG findings. Together, our results identify self-attribution as a hidden control variable in multimodal verification, challenging the assumption that a model can neutrally verify its own generations.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Models, Multimodal Verification, Self-Verification, Self-Anchoring, Multimodal Evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_2z77OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies models that can both generate and understand images, and asks whether such a model is a reliable checker of generated images. The question is whether the model reports what it actually sees or is swayed by what was requested and by who it is told made the image. The paper tests this with the Bagel model on hand images generated by Janus-Pro, asking it to count fingers under four prompt versions: without the requested count, with the requested count, with the requested count plus a note that the image was made by “another model,” or by “you”. They find that showing the requested count roughly halves counting accuracy, while the note about who made the image has little effect. They also show that turning off selected attention heads recovers part of the lost accuracy.

Critical strengths

Clear experiment design:
The paper changes only one thing at a time in the prompt: whether the target count is shown, and who is said to have made the image. The image and the question stay the same, so each effect is easy to read. The main finding is clear: when the model sees the requested count, its accuracy drops from 77.5% to 37.5%.

Several ways of measuring:
The paper does not rely on a single number. It compares scores, final answers, and image selection, showing that these can disagree. For example, the condition with the best ranking score has the worst accuracy.

Honest intervention results:
Turning off selected attention heads raises accuracy by 27.5 points and works better than turning off random heads. The paper also openly reports that the fix is incomplete and has a small cost on normal counting.

Critical weaknesses

The title and abstract of the PDF version of the paper do not match those on OpenReview.

Too narrow for the general claim.
The study uses only one model, one image generator, one task (counting fingers), and 120 test images. It is unclear whether the results hold for other models, tasks, or objects. Also, the model never checks its own images, so the “generated by you” condition is only a change in wording, and the setting that motivates the paper is never tested.

Little new beyond prior work:
The large effect, in which the model follows the prompt rather than the image, is already known from Rudman et al., and the head-removal method follows that work. The paper’s new question, whether saying who made the image matters, gives almost no effect: both conditions reach exactly the same accuracy (35.83%). Simply hiding the target from the model performs better than the proposed fix (77.5% vs. 65.0%), and no simple prompt-based fix is included for comparison. In addition, Rudman et al. is cited as an arXiv preprint, although it was published at ACL. Since it is the closest prior work, it should be cited correctly.

Weak evidence for image selection, and hard to read:
Only 8 of 24 image groups contain a correct image, so the selection results tell us very little, yet they are listed as a main contribution. The heads are pre-selected using a simple activation measure; no correction for multiple comparisons is applied, and the image labels come from a single annotation column. The text also contains many caveats, making the main message hard to find.

Questions for authors

Do the results hold for images that Bagel itself generated?

How sensitive are the attribution effects to other wordings of the source statement?

Does turning off the heads make the model reject correctly generated images more often?

How does the head removal compare with a simple instruction to count independently of the request?

How were the labels produced, and how well do annotators agree?

Reviewer_xXj4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper investigates two compounding factors in multimodal verification: disclosing intention and explicit source attribution. The experiments use Janus-Pro to generate hand images and Bagel as the verifier. Four conditions are compared (two factors in a 2 × 2 design) while keeping the visual content fixed. The results show that disclosing intention substantially reduces counting accuracy, while the other factor does not significantly change the metric. A simple solution based on attention heads is also provided, indicating that selecting heads based on their correction effect improves counting accuracy under intention-disclosure settings.

Critical strengths
  1. This paper provides specific controlled comparisons that help distinguish errors compounded by different factors.

  2. The results of head selection are of some interest, showing robustness in counting accuracy under intent-conditioned settings.

Critical weaknesses
  1. (Major) It seems to me that this is a diagnostic paper that mainly focuses on finding the key factors behind a specific problem, with a relatively simple method to address it. The strongest finding is that disclosing the target can significantly affect the model’s understanding of visual content, which is a case of prompt-induced hallucinations. Although the finding is informative, the current study does not establish a truly broad or substantial new understanding of prompt-induced hallucinations. It mainly just finds that there is one factor that introduces hallucinations, and little else can be learned beyond this. This level of contribution is, to me, not sufficient to convince me to accept the paper at ICLR.

  2. For the evaluated counting task, another straightforward method is to ask the verifier to count independently and then compare its answer with the requested count. The paper reports a visual-only baseline accuracy of 77.5%. Disclosing the target reduces it to 37.5%, and the proposed intervention improves it to 65.0%, which is still below the visual-only baseline. It is then very unclear to me why we need to reveal such intent to the verifier for this task. Even if conditions or settings that require this information indeed exist, it only further narrows scope of the study, especially given the already narrow scope of the current study. The study also does not show a clear benefit of the intervention for downstream candidate selection.

Questions for authors

Most of my questions are illustrated in the weaknesses. An additional one is about the experimental setting. It seems to me that the introduction tries to suggest that the verifier and the image generator come from the same model, but in the actual experiments (correct me if I am wrong), the generator is Janus-Pro, while the verifier is BAGEL. Why not use the same model?

Btw, why the paper title is not aligned with the title presented in openreview?

Reviewer_LpU6OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper runs a controlled multimodal verification by studying disclosing a requested object count vs. explicitly attributing an image to the verifier. It shows that, on generated hand images, intent disclosure substantially reduces a small open model, Bagel’s, counting accuracy, while source attribution produces smaller, metric-dependent artifacts, with attention-head ablation improving intent-conditioned accuracy. However, the experiments remain limited in breadth, with the paper missing discussion of several closely related previous works.

Critical strengths
  1. The core comparison separating target disclosure and mentioning the explicit source wording is done carefully, with major experimental details disclosed in the main body of the paper.
  2. The paper conducts grouped bootstrapping, which strengthens its claims.
  3. It measures a suite of metrics, including probabilities, complete answers, candidate selection, and retained counting ability.
Critical weaknesses
  1. The paper’s main controlled experiments are quite limited: it tests only one verifier (Bagel) on a single generator (Janus Pro 7B) with one prompt. It also tests on just 120 held-out hand images with no discussion of whether the findings replicate to other use cases. While the appendix discusses a few extra models, they don’t extend the core experiment the main claims are based on.
  2. The related work overstates the novelty slightly. Some previous work [1] has already evaluated Bagel’s understanding of its generated images. There has also been some work on activation interventions to reduce self-preference [2] and isolating authorship labels while keeping the context fixed [3]. The paper does not discuss what its analysis adds to this.
  3. Across all experiments, the images are generated by a separate model (Janus-Pro). The experiments differ only in wording for the self/other conditions. The paper never tests the impact of sending an image generated by the same model, while the abstract promises it (Lines 12-14).
  4. The candidate pools remain unbalanced too: among the eight of 24 that contain a correct image, none are available at counts of six or seven. This means the reliability test has missing values. The authors should look into extending the experimental corpus, especially as it’s synthetic.
  5. Further experimental breadth remains missing. For instance, Lines 132-133 note that all tests were on “manually supplied visible-count labels,” yet no discussion is seen on the number of annotators used, their agreement, or the hiring process.

[1] Jin et al, 2025, SRUM: Fine-Grained Self-Rewarding for Unified Multimodal Models

[2] Roytburg et al, 2025, Breaking the Mirror: Activation-Based Mitigation of Self-Preference in LLM Evaluators

[3] Saraf et al, 2025, Quantifying Label-Induced Bias in Large Language Model Self- and Cross-Evaluations

Questions for authors
  1. Can you prove that the paper’s claimed findings generalize to other models noted in the appendix?
  2. How does the paper differ from SRUM [1], Breaking the Mirror [2], and Saraf et al [3] besides being a combination of their experiments?
  3. Do the findings generalize to other tasks outside of counting figures?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.