- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper identifies a recurring failure mode in current VLMs: difficulty attributing observed visual changes to camera motion versus object motion. Its empirical analysis shows that models can detect motion cues yet fail to attribute them to the correct source and answer accordingly. To address this limitation, the paper proposes MoSDeR, which binds motion evidence to the queried entity, decomposes that evidence into camera and object factors, and incorporates both into the model’s reasoning. Experiments on curated and held-out benchmarks show consistent performance improvements.
Critical strengths
-
The problem is well motivated and practically important. Distinguishing camera motion from object motion is essential in applications such as robotic manipulation with arm-mounted cameras and autonomous driving. In both settings, accurate scene understanding requires separating motion induced by the agent from motion of other entities. Failures in this attribution can undermine downstream reasoning and decision-making.
-
The improvements are consistent across multiple backbones. Table 1 shows that MoSDeR improves overall accuracy across NVILA, Qwen3, and Molmo2. This consistency supports the method’s applicability beyond a single model family.
Critical weaknesses
It is not clear to me why standard fine-tuning or RLVR, neither of which is included in the comparisons, would be insufficient to address the attribution and reasoning problem identified in this paper. It is therefore unclear whether the proposed architectural changes are necessary to achieve the reported improvements. The fine-tuning procedure also requires clarification. The paper describes a frozen VLM backbone with small trainable adapters, but given the goal of improving visual understanding, comparisons that unfreeze the vision encoder or fine-tune the full model are warranted. Even with this computational disadvantage, fine-tuning achieves performance comparable to MoSDeR, although it does not outperform it. This raises the question of whether a stronger fine-tuning setup alone could achieve the same gains. Finally, training scale may play an important role in acquiring this capability. Evaluating performance as a function of training-data volume would help clarify whether the observed limitations stem from the architecture or from insufficient training.
Questions for authors
-
Could you clarify the fine-tuning procedure, including which components are trainable, and provide comparisons against standard fine-tuning under a matched training-data budget? In particular, do the gains persist when the vision encoder or the full backbone is unfrozen? A matched RLVR comparison would also help assess whether the proposed architecture is necessary.
-
How do frontier multimodal models, like GPT/Muse, and more recent open models, such as Qwen3.5, perform on these tasks relative to the evaluated backbones? Stronger performance from these models would help contextualize the claimed limitation and assess whether it may be mitigated by model or training scale.
5 comments
Response to Reviewer H7AK (2/2): comparison with fine-tuning, training data scale and frontier models
Fine-tuning vs. MoSDeR
Table H1 collects the comparison from Tabs. 1, 2 and A12. Fine-tuning trains slightly more parameters than MoSDeR on every backbone. On MoSA, MoSDeR leads by 11.08 to 16.25% Overall and by 26.00 to 48.00% on concurrent motion. The two are close only on OmniVCHall with Qwen3 (73.75% vs. 73.86%), while MoSDeR leads by 10.04 and 12.27% on NVILA and Molmo2.
Table H1. Fine-tuning vs. MoSDeR. Each cell gives fine-tuning / MoSDeR, accuracy in %.
| Backbone | Trainable | MoSA Overall | MoSA Both | OmniVCHall Overall |
|---|---|---|---|---|
| NVILA | 2.88M / 2.57M | 65.67 / 76.75 | 36.33 / 84.33 | 62.97 / 73.01 |
| Qwen3 | 3.41M / 2.98M | 73.08 / 84.83 | 58.33 / 84.33 | 73.75 / 73.86 |
| Molmo2 | 3.15M / 3.02M | 71.83 / 88.08 | 52.00 / 88.67 | 63.07 / 75.34 |
Training scale
We train fine-tuning and MoSDeR on Molmo2 with 25%, 50%, 75% and 100% of the state-balanced training set and evaluate on the MoSA test set (Table H3, Fig. 7(b) and App. E.3 of the revised PDF).
Table H3. Overall accuracy (%) vs. training data (Molmo2).
| Training data | 25% | 50% | 75% | 100% |
|---|---|---|---|---|
| Fine-tuning | 70.17 | 70.50 | 71.00 | 71.83 |
| MoSDeR | 71.08 | 80.75 | 85.75 | 88.08 |
| Gap (%) | 0.92 | 10.25 | 14.75 | 16.25 |
From 25% to 100% of the data, fine-tuning gains only 1.67%, whereas MoSDeR gains 17.00% and its lead grows from 0.92% to 16.25%. Within the available data, the gap therefore does not come from insufficient training, and more data widens it rather than closing it.
Frontier and newer models
Without MoSA training, Qwen3.5-27B, which you suggested, reaches 64.50% Overall and GPT-6.1 Sol 78.33% (Table H4 and Tab. 1). We use GPT-6.1 Sol as the representative closed frontier model and evaluate it under exactly the same protocol as our backbones. Both exceed the raw 7B and 8B models, yet each still fails on one state. Qwen3.5-27B reaches only 34.33% on camera-only motion, and GPT-6.1 Sol only 53.33% when neither source moves, so it reports motion in static scenes. Model scale therefore shifts the failing state across models but does not remove it. Training on MoSA does not remove it either, since LoRA and full fine-tuning still fail on Both (52.00% on Molmo2, Table H2), whereas MoSDeR keeps every state at 72.00% or higher on all three backbones (Tab. 1).
Table H4. MoSA test accuracy (%) without MoSA training, by motion state, with MoSDeR Overall for reference. The weakest state of each model is in bold.
| Model | Neither | Camera | Object | Both | Overall | MoSDeR |
|---|---|---|---|---|---|---|
| NVILA (8B) | 1.67 | 32.67 | 63.67 | 85.33 | 45.83 | 76.75 |
| Qwen3 (8B) | 54.67 | 43.67 | 87.00 | 48.67 | 58.50 | 84.83 |
| Molmo2 (7B) | 58.67 | 37.67 | 62.00 | 30.00 | 47.08 | 88.08 |
| Qwen3.5-27B | 50.33 | 34.33 | 94.00 | 79.33 | 64.50 | n/a |
| GPT-6.1 Sol | 53.33 | 90.67 | 77.33 | 92.00 | 78.33 | n/a |
Follow-up on our response
Dear Reviewer H7AK,
Thank you again for your review. We understand this is a busy period, and we just wanted to briefly follow up on our response.
Regarding whether the architecture is necessary, we added full fine-tuning (7.76B trainable parameters) and GRPO controls trained on the same data (Fig. 7(a), App. E.1). These improve the single-source states, but on concurrent motion they reach at most 62.00%, compared with 88.67% for MoSDeR, and their errors mostly attribute the motion to the camera alone. We also added a data-scale study (Fig. 7(b)) and results for Qwen3.5-27B, as you suggested (Tab. 1).
We would be grateful to know whether these address your concerns, and we would be happy to clarify anything further.
Best regards,
Authors
Response to Reviewer H7AK (1/2): fine-tuning setup, full fine-tuning and GRPO controls
Thank you for the review. Changes are marked in blue in the revised PDF.
Revisions at a glance
Fine-tuning setup
"Fine-tuning" applies LoRA (rank 32, α=32, no dropout) to the attention projections of four late decoder layers of the same VLM. These are the query, key, value and output projections of layers 23 to 26 for NVILA and 31 to 34 for Qwen3, and the fused input and output attention projections of blocks 27 to 30 for Molmo2 (zero-based). The vision encoder, visual projector and language model stay frozen. It uses the same MoSA training set, prompts, answer format and number of updates as MoSDeR. MoSDeR also keeps the backbone frozen and trains only its source routes and shared LoRA.
Is the architecture necessary?
Three controls in the paper separate the architecture from labels and capacity. Adding the same camera and object labels to a shared LoRA as auxiliary supervision gives 77.83% (Both 56.33%) vs. 89.17% (Both 90.00%) (Tab. 3, validation set). Collapsing the two routes into one shared factor with the same target binding gives 84.17% (Tab. 3, validation set). 4DP-QA trains 35.6 to 44.2 times more parameters and remains 3.08 to 4.25% below MoSDeR (Tabs. 1, A12).
Following your suggestion, we add RLVR (Lambert et al., 2025) with GRPO (Shao et al., 2024) at two budgets on Molmo2, with exactly the same training data as MoSDeR (Table H2, Fig. 7(a) and App. E.1 of the revised PDF). Each version starts from the corresponding supervised checkpoint. The matched version continues LoRA fine-tuning on the same 3.15M parameters, and the full version continues full fine-tuning on all 7.76B parameters, including the vision encoder. The reward is the evaluation rule itself, exact match on the assigned question, and the checkpoint is selected by early stopping on the validation set. Full fine-tuning (7.76B) unfreezes the vision encoder and the language model.
Table H2. Molmo2 on the MoSA test set, accuracy (%).
GRPO helps at both budgets, raising Overall from 71.83% to 78.58% and from 79.33% to 82.83%, mostly on the single-source states. Concurrent motion improves much less, to 58.33% and 62.00%, and stays far below 88.67% for MoSDeR, although the full version trains over 2,500 times more parameters. Table H2b, also at the bottom of Tab. A15, shows why.
Table H2b. Motion state implied by each answer on the 300 concurrent-motion (Both) test samples, from the inference results above.
Almost every wrong answer of the four baselines implies a single source, mostly camera-only motion. When both sources move, they attribute the whole motion to the camera, consistent with camera motion changing the entire frame and absorbing the motion of the target. MoSDeR, whose object route compares the target with its context, recovers 266 of 300 cases, and its few errors are of the same kind. Neither more capacity nor reward-based training closes the gap on source composition, which is the failure that the separate source routes address.