acceptodds
Under review as a conference paper at ICLR 2027

Looking but Not Updating: Visual Belief Inertia in Embodied Multimodal Models

Abstract

An embodied assistant must judge a scene when its action history describes an outcome that the image contradicts. We study visual belief inertia: state answers that follow an incorrect action-conditioned history in independent single-image queries. VBI-Bench comprises 2,285 robot-state questions and two complementary controls: vary history while fixing the image, or vary the image while fixing all text. These controls reveal that strong performance under both correct and incorrect histories can coexist with poor discrimination between images. On Qwen3-VL-8B, revision-invariant fine-tuning (RI-FT) raises full-conflict accuracy from 14.4% to 98.3% on a 713-item history grid, yet answers both members of only 6.5% of 552 fixed-text image pairs correctly. RI-FT-CF adds image pairs with shared history, question, and options but different targets to the same answer-supervised objective. Across three seeds, both-correct pair accuracy reaches 77.5%, alongside 94.6% full-conflict and 98.3% correct-history accuracy. Component comparisons link the paired gain to the added examples and show how history variation changes the joint performance across conditions. The resulting diagnosis and training recipe make image changes an explicit part of evaluating history-resistant state judgments.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Vision-language models, Visual belief inertia, Visual grounding, Embodied AI, Counterfactual evaluation, Supervised fine-tuning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_6z3qOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies visual belief inertia (VBI), a failure mode where embodied vision-language models answer robot-state questions according to an incorrect action history even when the image contradicts that history. The authors introduce VBI-Bench, a benchmark containing 2,285 robot-state QA examples together with two complementary evaluation settings: varying history while keeping the image fixed, and varying the image while keeping all text fixed through visual counterfactual pairs.

The paper shows that strong performance under conflicting histories does not necessarily imply image-dependent reasoning. In particular, RI-FT improves conflicting-history accuracy dramatically but performs poorly on paired image discrimination. To address this, the authors introduce RI-FT-CF, which adds fixed-text image pairs during supervision and substantially improves pair accuracy while maintaining strong history robustness. Extensive experiments, ablations, and intervention studies are presented primarily on Qwen3-VL-8B.

Critical strengths
  1. The paper identifies a useful evaluation blind spot. RI-FT reaches 98.3% full-conflict accuracy but answers both images correctly on only 6.5% of the fixed-text pairs. I found this result compelling: doing well under misleading histories does not necessarily mean the model is using the image correctly.

  2. I like the two complementary controls: changing the history while keeping the image fixed, and changing the image while keeping all text fixed. Together they give a more informative picture than either test alone. The component comparisons are also useful, showing that pair-derived supervision improves image discrimination while the history-training mixture helps preserve correct-history performance.

Critical weaknesses
  1. The paper does provide useful evidence about why answers change: outcome reports cause particularly strong interference, wording matters, and RI-FT develops a strong preference for the later-state answer even without images. What remains unclear is how much of the low pair accuracy reflects poor visual discrimination versus interpretation of the question. Appendix A.6 notes that 308/552 pair questions contain "after" or "final", and that the verifier receives an explicit current-state instruction which the evaluated models do not receive. I would like to see the paired evaluation repeated with clearly current-state wording. This seems important for interpreting the size of the image-dependence gap.

  2. The model-assisted annotation pipeline is described clearly, but independent human agreement on the final labels and VCF pairs is not measured. The 20-participant questionnaire provides a useful human performance reference, but does not resolve this annotation-quality concern. A human audit of a representative subset would help, especially for earlier frames and cases marked weak or invisible.

Questions for authors
  1. Could you repeat the paired evaluation with questions explicitly asking about the state visible now, using the same interpretation given to the annotation verifier? How much of RI-FT’s earlier-frame error remains under that wording? This would help separate visual-state errors from temporal interpretation.

  2. Have the authors performed any independent human validation study on a representative subset of benchmark annotations and VCF pairs?

  3. Table 1 shows history interference across nine models, but the RI-FT and RI-FT-CF training results are limited to Qwen3-VL-8B. Do you have evidence from another model or size that the same image-dependence gap appears after RI-FT, and that pair-derived supervision addresses it?

It would be valuable to extend VBI-Bench with richer failure categorization and interpretability analyses so that the benchmark can provide stronger guidance on how future embodied models should be improved.

Reviewer_dyKAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper studies a new task called visual belief inertia, where VLMs follow incorrect action histories despite given contradictory visual observations. It proposes VBI-Bench with complementary controls that vary either history or images. The central finding is that history-resistant fine-tuning can achieve high conflict accuracy while retaining poor image discrimination. Their experiments also find that adding fixed-text image pairs substantially improves paired accuracy while preserving strong performance under correct and conflicting histories.

Strengths

  • The overall idea of studying the distinction between resistance to misleading history and genuine image dependence is interesting. The paired evaluation exposes shortcuts that might be useful for future research.
  • The paper includes several different diagnostics for comparison.
  • The proposed data augmentation is simple and effective.

Weaknesses

  • The evaluated models are mainly around 7B–9B parameters. Larger models and more recent strong VLMs are missing. Moreover, training and mechanistic analyses focus on Qwen3-VL-8B, limiting the generality of the conclusions.
  • Many paired questions retain “after the action” or “final state” wording when applied to earlier frames. The annotator receives explicit instructions to judge the currently visible state, whereas evaluated models do not receive equally explicit clarification. Errors may therefore reflect temporal interpretation rather than visual belief inertia.
  • The method uses ordinary SFT, and simpler pair-derived baselines recover most of the paired improvement. Without equally diverse unpaired-data controls, the benefits of pairing cannot be separated from increased state and label coverage.

Questions

  • What if all questions explicitly ask about the currently visible state and annotation/evaluation instructions are aligned?
  • Does paired construction outperform equally diverse unpaired supervision under matched data and training budgets?
Critical strengths

See above

Critical weaknesses

See above

Questions for authors

See above

Reviewer_CSphOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates visual belief inertia in embodied multimodal models, where incorrect action-conditioned histories can override visual observations during state prediction. The authors introduce VBI-Bench, which evaluates models using two complementary controls: varying textual histories while keeping the image fixed, and varying images while keeping the text fixed. The paper shows that history-resistant fine-tuning can achieve high conflict accuracy without reliably distinguishing different visual states. To address this, the authors propose RI-FT-CF, combining history augmentation with counterfactual image-pair supervision. Experiments report substantial improvements in paired visual-state accuracy. However, concerns remain regarding evaluation reliability, temporal ambiguity, and whether the experimental evidence sufficiently supports the broader claims about embodied visual belief updating.

Critical strengths
  1. Interesting problem and evaluation perspective. The paper highlights an important distinction between resisting incorrect textual histories and genuinely relying on visual observations. The complementary evaluation controls provide a useful way to diagnose this issue.
  2. Strong empirical diagnosis. The experiments demonstrate that high conflict accuracy can coexist with poor visual discrimination. The analysis covers nine models and includes history-component interventions, image-free evaluations, activation patching, and extensive ablations.
  3. Promising paired evaluation results. Counterfactual-pair supervision substantially improves paired accuracy while maintaining relatively strong performance under both correct and conflicting histories. The paper also reports results across multiple training seeds and includes a stricter training-union-unseen evaluation.
Critical weaknesses
  1. Insufficient experimental evidence to support the broader claims. Although the paper includes extensive analyses, several experiments fail to provide convincing evidence for the claimed mechanisms and practical implications. The activation patching and probing results do not establish a belief-specific mechanism or demonstrate intact visual perception, while the downstream decision replay experiments fail to show improvements in robotic decision-making, as also acknowledged by the authors (e.g., "does not establish the mechanism of the training gain or improved robot task success," "do not establish accurate visual-state representation or use," and "make no claim that RI-FT-CF improves robot task success").

  2. Concerns about evaluation reliability and data independence. Among the 552 counterfactual evaluation pairs, 359 share recorded scene keys with the history-training data, and only 193 pairs have scene keys unseen by both training components. Although the authors provide results on this stricter subset, the full evaluation is not entirely independent of the training data. Furthermore, benchmark annotations and verification rely primarily on Gemini without independent human agreement measurements, raising concerns about the reliability of the evaluation labels.

Questions for authors
  1. Evaluation reliability: Can the authors provide independent human validation of benchmark labels and additional evaluation on fully disjoint tasks or environments, beyond the recorded scene-key-unseen subset?

  2. Temporal ambiguity: How does PairAcc change when the evaluated models receive the same explicit current-visible-state instruction used by the annotation verifier? In particular, can the authors separately evaluate questions with and without temporal expressions such as "after the action" or "final state"?

  3. Experimental evidence: Given that several mechanistic and downstream analyses do not support stronger conclusions, what are the main findings beyond the behavioral diagnosis and standard counterfactual-pair supervision? Can the authors provide more direct evidence that the proposed approach improves visual belief updating rather than merely learning different answer preferences?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.