acceptodds
Withdrawn from ICLR 2027

Vision-Language Models Cannot Tell What Moves Without Knowing How They Move

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision-language models for driving are asked whether a nearby vehicle is moving or stationary, and the usual remedy for their errors is to show them more frames. We find that more frames do not help. A BLIP-2 model finetuned on nuScenes scores 62.1% from one frame and 62.9% with two past frames fused through gated cross-attention, and three open vision-language models change by−3.8 to +2.7 points when given three frames instead of one. We prove that this failure concerns information and not architecture. The motion state of an object is not identifiable from its positions in the camera frame, because that frame mixes the motion of the object with the apparent motion that the observer causes, and it becomes identifiable once the relative pose of the camera between frames is known. On nuScenes, object displacement separates the two classes with an AUC of 0.557 in the ego frame and 0.935 after ego-motion compensation, and one token that carries the compensated displacement raises the finetuned model to 88.2 ±0.3%, or 88.1 ±0.4% with the image blacked out. How the quantity enters the model matters as much as whether it is present: the same twelve numbers give 66.2% through a subtractive structure that cannot represent rotation and 76.2% with auxiliary supervision of that structure, whereas a module that builds the exact group action in reaches 88.1 ±0.2%, and stating the compensated displacement in the prompt of an open model without training adds between 0 and 10 points. A model on a moving platform needs its own motion as an input and the compensation as a structure, not a longer clip.

Primary area:
applications to robotics, autonomy, planning
Keywords:
vision-language models, ego-motion, identifiability, object motion, autonomous driving, temporal reasoning, visual question answering, nuScenes

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Hh9bOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies whether driving VLMs can distinguish moving from stationary vehicles when the camera is also moving. It finds little benefit from additional frames in the tested settings, then uses target positions and ego poses to compensate for camera motion. Experiments on DriveLM-nuScenes show that providing this information improves motion-state question answering.

Critical strengths
  1. The paper takes an interesting angle on how driving VLMs judge object motion when the camera itself is moving. It highlights a failure case and shows that providing explicit geometric information can improve motion-state judgments.
Critical weaknesses
  1. The explanation for the failure is not established. In the theoretical analysis, the observation is limited to a single object’s relative position track. In the experiments, however, the models receive full images with background cues that can help estimate ego-motion. Poor performance and limited gains from extra frames may therefore reflect weak temporal learning or insufficient training, rather than missing information. The experiments do not separate these explanations.
  2. The improvement depends on additional inputs whose practical cost and reliability are not evaluated. The method uses annotated target positions, cross-frame matches, and ego poses. Obtaining comparable inputs in practice requires detection, position estimation, tracking, and synchronization, which add cost and introduce errors. The paper neither measures this cost nor evaluates actual detection and tracking outputs. Testing ego-pose noise alone does not establish whether the gains would remain in practice.
  3. Most importantly, it is unclear whether this weakness actually limits driving. Poor moving/stationary question answering does not by itself show that a model makes worse driving decisions. Relative-motion cues may already support the required driving decisions, but the paper does not test this possibility or show that adding motion relative to the ground improves planning or driving. The claimed practical importance of the diagnosed weakness therefore remains insufficiently supported.
Questions for authors
  1. What evidence separates missing information from insufficient temporal learning? How would a stronger temporal model, or a pipeline that estimates ego-motion from the same images, perform without annotated side inputs?
  2. With estimated target positions, tracks, and ego poses, how much improvement remains, at what additional cost, and does it lead to better planning or driving performance?
Reviewer_467BOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates whether temporal context improves moving-versus-stationary vehicle classification, evaluating a finetuned BLIP-2 and three prompted VLMs on nuScenes/DriveLM. It formalizes the ambiguity of object motion from ego-frame tracks without ego-motion and introduces an explicitly compensated displacement token. On 372 test questions, the token achieves 88.2% accuracy, compared with 62.1% for image-only BLIP-2. Blacking out images barely changes performance, and a simple displacement threshold achieves 88.4%.

Critical strengths

The compensated/uncompensated comparison, image-blackout ablation, and threshold baseline clearly identify the source of improvement. These controls make the empirical findings informative beyond the proposed model. The rotation-dependent error analysis provides a concrete explanation for the subtractive module’s limitations, supported by yaw-stratified results.

Critical weaknesses
  1. The central claims exceed the demonstrated scope. Non-identifiability applies to isolated object tracks, whereas full images contain background cues for estimating ego-motion. Similarly, non-significant differences between one and three frames do not establish that longer temporal context is ineffective.
  2. The methodological and practical contribution remains limited. Improvements rely on annotated object positions and ego poses, and a threshold matches the VLM. The main subtractive comparator cannot represent rotation; a competitive raw-input baseline on the corrected data and validation with actual perception outputs are missing.
Questions for authors
  1. What evidence distinguishes missing information from the tested models’ failure to infer ego-motion from the background? How should the abstract’s information-versus-architecture claim be interpreted?
  2. Are results available for a suitably tuned MLP receiving all twelve raw inputs on the corrected camera setup, with comparable supervision?
  3. Are auxiliary displacement targets expressed in the same coordinate frame as the recovered displacement? What specifically causes the reported 0.045 m residual when compensation is constructed from annotations?
Reviewer_gxtnOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies a specific problem, i.e., how VLMs distinguish moving from stationary vehicles while the ego vehicle is also moving. It shows that ego-motion helps distinguish the target vehicle’s actual motion. The authors compare additional frames, subtraction, and explicit SE(2) compensation on motion-state questions of DriveLM-nuScenes. The experiments focus on a binary classification task and use annotated object positions. The experimental results support the statement of paper.

Critical strengths
  1. The research question is clear, and the geometric analysis provides an interpretable explanation for the observed failure.

  2. The experiments include ablations, such as removing image content, comparing compensation structures, and testing a simple threshold baseline. These controls help identify the source of the improvement and support the importance of correctly compensated motion information.

Critical weaknesses
  1. Limited evaluation scope. The experiments focus on binary motion-state questions from DriveLM-nuScenes, with only 372 test questions, which supports a narrow finding about moving-stationary classification. Based on authors’ statements, such approach can also improve broader motion understanding, like direction, displacement, or speed estimation.

  2. Limited validation across VLMs. The proposed compensation structures are tested only on fine-tuned BLIP2 and T5 models. Evaluating the method on modern and stronger VLMs with general knowledge would help determine whether its benefits generalize and whether the observed limitations reflect broader challenges or the particular models and training settings used.

Questions for authors
  1. Additional evidence: Does the proposed compensation method provide similar improvements on a stronger VLMs like Qwen3, and InternVL3.5? Such evidence would help assess whether the findings generalize across backbones.

  2. Unfair comparison in Table 1, where author compare compensated displacement as text in open VLMs and as a learned token in finetuned BLIP-2. How do these approaches compare using the same backbone and training conditions? This would clarify whether a simple textual representation of coordinates or displacement could achieve comparable gains.

I would like to raise my rating if authors can address above problems.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.