- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies visual belief inertia (VBI), a failure mode where embodied vision-language models answer robot-state questions according to an incorrect action history even when the image contradicts that history. The authors introduce VBI-Bench, a benchmark containing 2,285 robot-state QA examples together with two complementary evaluation settings: varying history while keeping the image fixed, and varying the image while keeping all text fixed through visual counterfactual pairs.
The paper shows that strong performance under conflicting histories does not necessarily imply image-dependent reasoning. In particular, RI-FT improves conflicting-history accuracy dramatically but performs poorly on paired image discrimination. To address this, the authors introduce RI-FT-CF, which adds fixed-text image pairs during supervision and substantially improves pair accuracy while maintaining strong history robustness. Extensive experiments, ablations, and intervention studies are presented primarily on Qwen3-VL-8B.
Critical strengths
-
The paper identifies a useful evaluation blind spot. RI-FT reaches 98.3% full-conflict accuracy but answers both images correctly on only 6.5% of the fixed-text pairs. I found this result compelling: doing well under misleading histories does not necessarily mean the model is using the image correctly.
-
I like the two complementary controls: changing the history while keeping the image fixed, and changing the image while keeping all text fixed. Together they give a more informative picture than either test alone. The component comparisons are also useful, showing that pair-derived supervision improves image discrimination while the history-training mixture helps preserve correct-history performance.
Critical weaknesses
-
The paper does provide useful evidence about why answers change: outcome reports cause particularly strong interference, wording matters, and RI-FT develops a strong preference for the later-state answer even without images. What remains unclear is how much of the low pair accuracy reflects poor visual discrimination versus interpretation of the question. Appendix A.6 notes that 308/552 pair questions contain "after" or "final", and that the verifier receives an explicit current-state instruction which the evaluated models do not receive. I would like to see the paired evaluation repeated with clearly current-state wording. This seems important for interpreting the size of the image-dependence gap.
-
The model-assisted annotation pipeline is described clearly, but independent human agreement on the final labels and VCF pairs is not measured. The 20-participant questionnaire provides a useful human performance reference, but does not resolve this annotation-quality concern. A human audit of a representative subset would help, especially for earlier frames and cases marked weak or invisible.
Questions for authors
-
Could you repeat the paired evaluation with questions explicitly asking about the state visible now, using the same interpretation given to the annotation verifier? How much of RI-FT’s earlier-frame error remains under that wording? This would help separate visual-state errors from temporal interpretation.
-
Have the authors performed any independent human validation study on a representative subset of benchmark annotations and VCF pairs?
-
Table 1 shows history interference across nine models, but the RI-FT and RI-FT-CF training results are limited to Qwen3-VL-8B. Do you have evidence from another model or size that the same image-dependence gap appears after RI-FT, and that pair-derived supervision addresses it?
It would be valuable to extend VBI-Bench with richer failure categorization and interpretability analyses so that the benchmark can provide stronger guidance on how future embodied models should be improved.