acceptodds
Under review as a conference paper at ICLR 2027

Whose Motion Is It? Learning Motion-Source Attribution in Vision-Language Models

Abstract

Video-language reasoning requires understanding how visual content evolves over time. Yet such changes may arise from camera motion, target-object dynamics, or both, making their physical source ambiguous. We identify a recurring failure mode in current video-language models (VLMs), termed motion-source misattribution, where observed changes are attributed to the wrong physical source. Our empirical analysis shows that motion judgments are not reliably bound to the queried target, source information is recoverable but not reliably source-selective, and camera and object evidence interact non-additively when both motions co-occur. These findings reveal a clear gap: encoding motion cues does not guarantee organizing them by physical source. To address this gap, we propose MoSDeR (Motion-Source Decomposition and Reasoning), which structures temporal evidence into query-conditioned camera and object factors. MoSDeR first binds motion evidence to the queried entity, then routes the two factors into the native reasoning pathway, and finally composes them non-exclusively under concurrent motion. To evaluate motion-source attribution across diverse conditions, we curate a unified target-level benchmark spanning egocentric and allocentric settings, in-door interactions, autonomous-driving scenes, and diverse camera–object motion configurations. Across three VLMs, MoSDeR exceeds the runner-up in Overall accuracy by 3.64% on our benchmark and 6.75% on OmniVCHall on average, indicating consistent attribution gains and broader downstream benefits.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Vision-Language Models, Video Understanding, Motion-Source Attribution, Motion Decomposition

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_H7AKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper identifies a recurring failure mode in current VLMs: difficulty attributing observed visual changes to camera motion versus object motion. Its empirical analysis shows that models can detect motion cues yet fail to attribute them to the correct source and answer accordingly. To address this limitation, the paper proposes MoSDeR, which binds motion evidence to the queried entity, decomposes that evidence into camera and object factors, and incorporates both into the model’s reasoning. Experiments on curated and held-out benchmarks show consistent performance improvements.

Critical strengths
  1. The problem is well motivated and practically important. Distinguishing camera motion from object motion is essential in applications such as robotic manipulation with arm-mounted cameras and autonomous driving. In both settings, accurate scene understanding requires separating motion induced by the agent from motion of other entities. Failures in this attribution can undermine downstream reasoning and decision-making.

  2. The improvements are consistent across multiple backbones. Table 1 shows that MoSDeR improves overall accuracy across NVILA, Qwen3, and Molmo2. This consistency supports the method’s applicability beyond a single model family.

Critical weaknesses

It is not clear to me why standard fine-tuning or RLVR, neither of which is included in the comparisons, would be insufficient to address the attribution and reasoning problem identified in this paper. It is therefore unclear whether the proposed architectural changes are necessary to achieve the reported improvements. The fine-tuning procedure also requires clarification. The paper describes a frozen VLM backbone with small trainable adapters, but given the goal of improving visual understanding, comparisons that unfreeze the vision encoder or fine-tune the full model are warranted. Even with this computational disadvantage, fine-tuning achieves performance comparable to MoSDeR, although it does not outperform it. This raises the question of whether a stronger fine-tuning setup alone could achieve the same gains. Finally, training scale may play an important role in acquiring this capability. Evaluating performance as a function of training-data volume would help clarify whether the observed limitations stem from the architecture or from insufficient training.

Questions for authors
  1. Could you clarify the fine-tuning procedure, including which components are trainable, and provide comparisons against standard fine-tuning under a matched training-data budget? In particular, do the gains persist when the vision encoder or the full backbone is unfrozen? A matched RLVR comparison would also help assess whether the proposed architecture is necessary.

  2. How do frontier multimodal models, like GPT/Muse, and more recent open models, such as Qwen3.5, perform on these tasks relative to the evaluated backbones? Stronger performance from these models would help contextualize the claimed limitation and assess whether it may be mitigated by model or training scale.

5 comments
AuthorsOpenReview ↗

Response to Reviewer H7AK (1/2): fine-tuning setup, full fine-tuning and GRPO controls

Thank you for the review. Changes are marked in blue in the revised PDF.

Revisions at a glance

PointChangeSectionPage, lines
SetupLoRA modules and frozen partsSec. 5.1p. 7, L363-365
SetupLayer indicesApp. C.4p. 24, L1273-1275
ArchitectureFull FT and GRPO, Fig. 7(a)Sec. 5.2pp. 9-10, L484-513
ArchitectureSetup and Both errors, Tab. A15App. E.1p. 27, L1417-1451
Data scaleData scale, Fig. 7(b)Sec. 5.2p. 10, L511-513
Data scaleExact values, Tab. A17App. E.3p. 28, L1466-1477
Frontier modelsReference rowsTab. 1p. 8, top rows
Frontier modelsFailure at larger scaleSec. 3p. 4, L197-199

Fine-tuning setup

"Fine-tuning" applies LoRA (rank 32, α=32\alpha = 32, no dropout) to the attention projections of four late decoder layers of the same VLM. These are the query, key, value and output projections of layers 23 to 26 for NVILA and 31 to 34 for Qwen3, and the fused input and output attention projections of blocks 27 to 30 for Molmo2 (zero-based). The vision encoder, visual projector and language model stay frozen. It uses the same MoSA training set, prompts, answer format and number of updates as MoSDeR. MoSDeR also keeps the backbone frozen and trains only its source routes and shared LoRA.

Is the architecture necessary?

Three controls in the paper separate the architecture from labels and capacity. Adding the same camera and object labels to a shared LoRA as auxiliary supervision gives 77.83% (Both 56.33%) vs. 89.17% (Both 90.00%) (Tab. 3, validation set). Collapsing the two routes into one shared factor with the same target binding gives 84.17% (Tab. 3, validation set). 4DP-QA trains 35.6 to 44.2 times more parameters and remains 3.08 to 4.25% below MoSDeR (Tabs. 1, A12).

Following your suggestion, we add RLVR (Lambert et al., 2025) with GRPO (Shao et al., 2024) at two budgets on Molmo2, with exactly the same training data as MoSDeR (Table H2, Fig. 7(a) and App. E.1 of the revised PDF). Each version starts from the corresponding supervised checkpoint. The matched version continues LoRA fine-tuning on the same 3.15M parameters, and the full version continues full fine-tuning on all 7.76B parameters, including the vision encoder. The reward is the evaluation rule itself, exact match on the assigned question, and the checkpoint is selected by early stopping on the validation set. Full fine-tuning (7.76B) unfreezes the vision encoder and the language model.

Table H2. Molmo2 on the MoSA test set, accuracy (%).

MethodTrainableNeitherCameraObjectBothOverall
LoRA fine-tuning (Tab. 1)3.15M87.0075.0073.3352.0071.83
GRPO, matched3.15M89.6789.0077.3358.3378.58
Full fine-tuning7.76B92.0090.6782.6752.0079.33
GRPO, full7.76B92.3394.3382.6762.0082.83
MoSDeR (Tab. 1)3.02M89.3382.3392.0088.6788.08

GRPO helps at both budgets, raising Overall from 71.83% to 78.58% and from 79.33% to 82.83%, mostly on the single-source states. Concurrent motion improves much less, to 58.33% and 62.00%, and stays far below 88.67% for MoSDeR, although the full version trains over 2,500 times more parameters. Table H2b, also at the bottom of Tab. A15, shows why.

Table H2b. Motion state implied by each answer on the 300 concurrent-motion (Both) test samples, from the inference results above.

Implied stateLoRA FTGRPO, matchedFull FTGRPO, fullMoSDeR
Both (correct)156175156186266
Camera-only13410814411030
Object-only317043
Neither70001

Almost every wrong answer of the four baselines implies a single source, mostly camera-only motion. When both sources move, they attribute the whole motion to the camera, consistent with camera motion changing the entire frame and absorbing the motion of the target. MoSDeR, whose object route compares the target with its context, recovers 266 of 300 cases, and its few errors are of the same kind. Neither more capacity nor reward-based training closes the gap on source composition, which is the failure that the separate source routes address.

AuthorsOpenReview ↗

Response to Reviewer H7AK (2/2): comparison with fine-tuning, training data scale and frontier models

Fine-tuning vs. MoSDeR

Table H1 collects the comparison from Tabs. 1, 2 and A12. Fine-tuning trains slightly more parameters than MoSDeR on every backbone. On MoSA, MoSDeR leads by 11.08 to 16.25% Overall and by 26.00 to 48.00% on concurrent motion. The two are close only on OmniVCHall with Qwen3 (73.75% vs. 73.86%), while MoSDeR leads by 10.04 and 12.27% on NVILA and Molmo2.

Table H1. Fine-tuning vs. MoSDeR. Each cell gives fine-tuning / MoSDeR, accuracy in %.

BackboneTrainableMoSA OverallMoSA BothOmniVCHall Overall
NVILA2.88M / 2.57M65.67 / 76.7536.33 / 84.3362.97 / 73.01
Qwen33.41M / 2.98M73.08 / 84.8358.33 / 84.3373.75 / 73.86
Molmo23.15M / 3.02M71.83 / 88.0852.00 / 88.6763.07 / 75.34

Training scale

We train fine-tuning and MoSDeR on Molmo2 with 25%, 50%, 75% and 100% of the state-balanced training set and evaluate on the MoSA test set (Table H3, Fig. 7(b) and App. E.3 of the revised PDF).

Table H3. Overall accuracy (%) vs. training data (Molmo2).

Training data25%50%75%100%
Fine-tuning70.1770.5071.0071.83
MoSDeR71.0880.7585.7588.08
Gap (%)0.9210.2514.7516.25

From 25% to 100% of the data, fine-tuning gains only 1.67%, whereas MoSDeR gains 17.00% and its lead grows from 0.92% to 16.25%. Within the available data, the gap therefore does not come from insufficient training, and more data widens it rather than closing it.

Frontier and newer models

Without MoSA training, Qwen3.5-27B, which you suggested, reaches 64.50% Overall and GPT-6.1 Sol 78.33% (Table H4 and Tab. 1). We use GPT-6.1 Sol as the representative closed frontier model and evaluate it under exactly the same protocol as our backbones. Both exceed the raw 7B and 8B models, yet each still fails on one state. Qwen3.5-27B reaches only 34.33% on camera-only motion, and GPT-6.1 Sol only 53.33% when neither source moves, so it reports motion in static scenes. Model scale therefore shifts the failing state across models but does not remove it. Training on MoSA does not remove it either, since LoRA and full fine-tuning still fail on Both (52.00% on Molmo2, Table H2), whereas MoSDeR keeps every state at 72.00% or higher on all three backbones (Tab. 1).

Table H4. MoSA test accuracy (%) without MoSA training, by motion state, with MoSDeR Overall for reference. The weakest state of each model is in bold.

ModelNeitherCameraObjectBothOverallMoSDeR
NVILA (8B)1.6732.6763.6785.3345.8376.75
Qwen3 (8B)54.6743.6787.0048.6758.5084.83
Molmo2 (7B)58.6737.6762.0030.0047.0888.08
Qwen3.5-27B50.3334.3394.0079.3364.50n/a
GPT-6.1 Sol53.3390.6777.3392.0078.33n/a
AuthorsOpenReview ↗

Follow-up on our response

Dear Reviewer H7AK,

Thank you again for your review. We understand this is a busy period, and we just wanted to briefly follow up on our response.

Regarding whether the architecture is necessary, we added full fine-tuning (7.76B trainable parameters) and GRPO controls trained on the same data (Fig. 7(a), App. E.1). These improve the single-source states, but on concurrent motion they reach at most 62.00%, compared with 88.67% for MoSDeR, and their errors mostly attribute the motion to the camera alone. We also added a data-scale study (Fig. 7(b)) and results for Qwen3.5-27B, as you suggested (Tab. 1).

We would be grateful to know whether these address your concerns, and we would be happy to clarify anything further.

Best regards,
Authors

Reviewer_U1YdOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether vision-language models can distinguish camera motion from object motion. It proposes MoSDeR, which adds separate branches for the two motion sources and uses them to guide answer generation. The authors also build MoSA from four existing video datasets. Experiments on three models show better motion-source judgments and some gains on broader video question answering.

Critical strengths

The paper addresses a clear problem in video understanding. MoSA provides a useful way to test whether models understand which source causes the observed motion.

They curate a dataset for benchmakring and training through an automatic approach.

MoSDeR outperforms standard fine-tuning on MoSA across three models. The authors also include a baseline with motion-source supervision, which helps check whether the gains come only from extra labels.

Critical weaknesses

The evaluation mainly covers small models. It is unclear whether the same errors remain common in stronger models. Results from larger open models or strong closed models would help establish the scope of the problem. Testing MoSDeR on a stronger open model would also show whether its gains remain.

The dataset covers a limited range of motion. MoSA contains 7,400 target-level samples, but these are not independent videos. The test set comes from 34 acquisition groups. The study also focuses on short clips and rigid objects. It is therefore unclear how well the findings extend to more varied scenes and motion types.

The roles of the two learned branches need stronger evidence. The paper includes ablations and answer-score analyses, but these do not fully show that each branch learns its intended motion source. A test that disables one branch at inference and measures its effect on camera and object judgments separately would provide a clearer explanation of why the method works.

Questions for authors

How do stronger open or closed models perform on MoSA without additional training? Does MoSDeR still improve a stronger open model?

How does performance change with the amount of training data? Does the method work on longer clips or non-rigid objects?

If one motion branch is disabled at inference, does it mainly affect judgments about that source? eg, does disabling the object branch hurt object-motion judgments more than camera-motion judgments?

5 comments
AuthorsOpenReview ↗

Response to Reviewer U1Yd (1/2): route roles and larger open and closed models

Thank you for the positive assessment and the concrete suggestions. Changes are marked in blue in the revised PDF.

Revisions at a glance

PointChangeSectionPage, lines
Q3Route disabling, Fig. 7(c)Sec. 5.2p. 10, L514-519
Q3Exact values, Tab. A16App. E.2pp. 27-28, L1452-1464
Q1Reference rowsTab. 1p. 8, top rows
Q1Failure at larger scaleSec. 3p. 4, L197-199
Q1Failing statesSec. 5.2p. 8, L407-410
Q2Data scale, Fig. 7(b)Sec. 5.2p. 10, L511-513
Q2Exact values, Tab. A17App. E.3p. 28, L1466-1477
ScopeBootstrap intervalsSec. 5.2p. 8, L398-399
ScopeExact values, Tab. A18App. E.4p. 28, L1478-1489
ScopeLonger clips, Tab. A19App. E.5p. 28, L1491-1507
ScopeClip length and rigid targetsConclusionp. 10, L536-538

Q3 Route roles

Following your suggestion, we disable one route of the trained MoSDeR at inference by zeroing its visual injection and decoder branch, without retraining (Molmo2, MoSA test set). Table U1 reports state-wise accuracy, with the change from the full model as a superscript (Fig. 7(c) and App. E.2 of the revised PDF).

Table U1. Disabling one route at inference (Molmo2, MoSA test set). Accuracy (%) by motion state, with the change from the full model as a superscript.

InferenceNeitherCameraObjectBothOverall
Full MoSDeR89.3382.3392.0088.6788.08
Camera off22.33$^{\text{–67.00}}$66.33$^{\textbf{–16.00}}$89.67$^{\text{–2.33}}$50.33$^{\text{–38.33}}$57.17$^{\text{–30.92}}$
Object off65.00$^{\text{–24.33}}$80.33$^{\text{–2.00}}$60.00$^{\textbf{–32.00}}$46.33$^{\text{–42.33}}$62.92$^{\text{–25.17}}$

On the single-source states, disabling the object route lowers Object by 32.00% but Camera by only 2.00%, and disabling the camera route lowers Camera by 16.00% but Object by only 2.33%. On these states, each route therefore mainly carries its own source, in line with Fig. 5(d). Neither and Both require judging both sources, so they drop under either removal, as non-exclusive composition intends. The 67.00% drop on Neither without the camera route shows that this route also supplies the evidence that the camera is static.

Q1 Stronger models

Without MoSA training, Qwen3.5-27B reaches 64.50% Overall and GPT-6.1 Sol 78.33% (Table U2 and Tab. 1). Both exceed the raw 7B and 8B models, yet each still fails on one state, as every raw model does. Qwen3.5-27B reaches only 34.33% on camera-only motion, and GPT-6.1 Sol only 53.33% when neither source moves, so it reports motion in static scenes. The failing state differs across models but persists in both larger models, which points to missing source attribution rather than a single bias that scale removes. Training on MoSA does not remove it either, since fine-tuning still fails on Both (36.33% to 58.33% across backbones, Tab. 1), whereas MoSDeR keeps every state at 72.00% or higher on all three backbones.

Table U2. MoSA test accuracy (%) without MoSA training, by motion state, with MoSDeR Overall for reference. The weakest state of each model is in bold.

ModelNeitherCameraObjectBothOverallMoSDeR
NVILA (8B)1.6732.6763.6785.3345.8376.75
Qwen3 (8B)54.6743.6787.0048.6758.5084.83
Molmo2 (7B)58.6737.6762.0030.0047.0888.08
Qwen3.5-27B50.3334.3394.0079.3364.50n/a
GPT-6.1 Sol53.3390.6777.3392.0078.33n/a
AuthorsOpenReview ↗

Response to Reviewer U1Yd (2/2): data scale, group-bootstrap intervals and longer clips

Q2 Data scale

We train fine-tuning and MoSDeR on Molmo2 with 25%, 50%, 75% and 100% of the state-balanced training set and evaluate on the MoSA test set (Table U3, Fig. 7(b) and App. E.3 of the revised PDF).

Table U3. Overall accuracy (%) vs. training data (Molmo2).

Training data25%50%75%100%
Fine-tuning70.1770.5071.0071.83
MoSDeR71.0880.7585.7588.08
Gap (%)0.9210.2514.7516.25

From 25% to 100% of the data, fine-tuning gains only 1.67%, while MoSDeR gains 17.00% and its lead grows from 0.92% to 16.25%. The source routes thus keep extracting signal from additional labeled data, and the gap widens rather than closes as data grows.

Scope

The 7,400 samples are target-level rather than independent videos (Tab. A5), so our intervals resample acquisition groups (App. A.7). Under this group bootstrap over the 34 test groups (1,000 resamples), the 95% CI of the MoSDeR minus fine-tuning gap excludes zero on all three backbones (Table U4 and App. E.4).

Table U4. MoSDeR minus fine-tuning in Overall accuracy on the MoSA test set, with group-bootstrap 95% CIs.

BackboneGap (%)95% CI (%)
NVILA11.085.58 to 17.00
Qwen311.757.85 to 16.35
Molmo216.2511.80 to 21.81

Clips span 1.0 to 2.5 s because trajectory labels define one motion state per short window. To test longer clips, we resample 122 windows of 5 s from the same AV2 test sequences with the labeling rules of Tab. A2, keeping the 20 input frames, so each window is sampled about half as densely. On Molmo2, MoSDeR reaches 81.15% (99/122) and fine-tuning 66.39% (81/122) (App. E.5). The 14.75% lead matches the 15.00% lead on the standard AV2 windows (Tab. 1), so the lead is preserved on clips twice as long. This agrees with the frame-count sweep in App. D.4, where 12 to 32 frames stay within 3.34% of the selected result. For much longer videos, accuracy depends mainly on how key frames or segments are selected, which is a complementary front end to MoSDeR. Non-rigid and articulated targets are excluded because a single rigid pose cannot define their motion state. OmniVCHall (823 videos, no target boxes) tests transfer to an independent benchmark, with a 7.47% average gain over fine-tuning (Tab. 2).

AuthorsOpenReview ↗

Follow-up on our response

Dear Reviewer U1Yd,

Thank you again for your constructive suggestions. We understand this is a busy period, and we just wanted to briefly follow up on our response.

Following your suggestions, we added the route-disabling analysis (Fig. 7(c)), results for Qwen3.5-27B and GPT-6.1 Sol (Tab. 1), a data-scale study (Fig. 7(b)), group-bootstrap confidence intervals (App. E.4), and an evaluation on longer clips (App. E.5).

We would be grateful to know whether these address your questions, and we would be happy to provide further details.

Best regards,
Authors

Reviewer_PxnFOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies motion-source misattribution in video-language models: although existing models may detect temporal changes, they often fail to determine whether those changes are caused by camera motion, the queried object’s motion, or both. Through target-switching tests, linear probing, hidden-state interventions, and matched-motion experiments, the authors identify unreliable target binding, inconsistent use of source information, and non-additive evidence under concurrent motion. They propose MoSDeR, which combines query-conditioned target binding, separate camera- and object-oriented temporal factors, source-selective visual and decoder adaptation, and non-exclusive composition. The paper also introduces MoSA, a benchmark containing 7,400 target-level samples from four public datasets. Experiments across three VLM backbones show that MoSDeR achieves the best overall accuracy on MoSA and provides some downstream gains on OmniVCHall.

Critical strengths
  1. The problem is valuable and well-defined.
  2. The benchmark well-designed and the results analysis is informative and useful.
  3. The performance gain brought by the training method is consistency across models.
  4. The experiment is comprehensive.
Critical weaknesses
  1. The method part as well as Figure 3 needs more clarification, the current writing is a little bit hard to follow. At least the following parts should be highlighted to help understanding: a) the supervision of each branch b) how to achieve disentanglement.
  2. Based on the current pipeline, what does the model learn? How to evaluate the learned representation for context and object? Is there any solid evidence to showcase that the learn mask can represent the target object?
  3. In the benchmark, since the GT target box is provided, it is hard to claim that this is a "query-conditioned motion reasoning", since MoSA actually evaluate P(y_c, y_c | V, q, Box_GT) not P(y_c, y_c | V, q) .So this description is confusing: "During source-supervised learning, available target boxes instantiate Mq; for general video questions, we predict it from the visual tokens and question stem."
  4. Target/context branches does not mean the real physical camare/motion disentanglement. And the mask-based design is actually an obj/non-obj separation, however, obj itself could contain camera motion semantics. For insrance, target can be actually interpreted as camera and object, context can be interpreted as camera + other objects + background. This design cannot guarantee camera/object disentanglement.
Questions for authors

Please check the weakness.

3 comments
AuthorsOpenReview ↗

Response to Reviewer PxnF (1/2): supervision of each route and how the sources are separated

Thank you for the detailed review. Changes are marked in blue in the revised PDF, including a redrawn Fig. 3 (p. 5) that labels every path with its loss and training stage.

Revisions at a glance

PointChangeSectionPage, lines
W1aLoss and stage of every moduleFig. 3p. 5, L226-229
W1aStage summarySec. 4.1pp. 4-5, L215-233
W1b, W4Source-supervised factorsSec. 4.3p. 6, L299-303
W2Route disabling, Fig. 7(c)Sec. 5.2p. 10, L514-519
W2Exact values, Tab. A16App. E.2pp. 27-28, L1452-1464
W2, W3Mask quality, two settingsSec. 4.2p. 5, L246-258
W3Box-free run, Fig. 7(d)Sec. 5.2p. 10, L520-525
W3Exact values, Tabs. A20-A21App. E.6pp. 28-29, L1509-1542
W3Allocator detailsApp. C.2p. 23, L1232-1241

Supervision of each route (W1a)

Training has a binding step and three training stages (Sec. 4, App. C.3). (1) The target allocator (0.52M parameters on Molmo2), which predicts the soft target mask MqM_q, is trained with balanced BCE on target-box occupancy, never on motion labels or answers, and then frozen. At test time on MoSA, the given box replaces the predicted mask on purpose, so that localization errors do not confound the evaluation of attribution, whereas OmniVCHall uses the predicted mask (see W3). (2) On MoSA, the camera route is trained only with the camera label ycy_c, and the object route only with the object label yoy_o, both derived from trajectories and applied through the decoder answers (Eqs. 5-6). Only the supervised route is active in its step. (3) With both routes frozen, a shared LoRA learns to generate the full answer (Eq. 8). (4) A bounded residual refines the visual route (Eq. 9).

How the sources are separated (W1b, W4)

The mask allocates evidence and does not assign sources. Target regions also carry viewpoint change, and the context can contain other moving objects, as stated in Sec. 4.2. Separation comes from supervision and structure. The camera margin compares answers that differ only in camera state, and each term of the object margin compares answers that differ only in object state, so camera motion alone cannot raise the object margin (Eq. 5). Each route has its own parameters, and composition keeps both routes active, so neither suppresses the other (Eq. 7). The revision therefore calls them source-supervised factors rather than a guaranteed physical disentanglement.

AuthorsOpenReview ↗

Response to Reviewer PxnF (2/2): route-disabling test, mask quality and MoSA without target boxes

What the model learns (W2)

On the 1,200 state-balanced test samples, the camera and object route scores of the four states fall into their expected quadrants (Fig. 5(d)). Replacing target binding by global pooling lowers Overall from 89.17% to 66.50%, mainly on Object (36.67%) and Both (37.00%) (Tab. 3, validation set), and target-switch pairs improve from 12/100 to 52/100 (Tab. A14, validation set). Two new tests on Molmo2 check the routes and the mask directly. Disabling one route at inference, without retraining, mainly removes its own single-source state (Table P1, and Fig. 7(c) and App. E.2 of the revised PDF). Without the camera route, Camera drops by 16.00% and Object by only 2.33%, and without the object route, Object drops by 32.00% and Camera by only 2.00%. Neither and Both require judging both sources and drop under either removal, so both routes contribute together. The predicted MqM_q peaks inside the queried target box in 60.37% of test frames, against 35.42% for the frame center and 8.35% for a uniformly random point, and its IoU doubles that of a fixed center box (Table P1b, Sec. 4.2 and App. E.6).

Table P1. Disabling one route at inference (Molmo2, MoSA test set). Accuracy (%) by motion state, with the change from the full model as a superscript.

InferenceNeitherCameraObjectBothOverall
Full MoSDeR89.3382.3392.0088.6788.08
Camera off22.33$^{\text{–67.00}}$66.33$^{\textbf{–16.00}}$89.67$^{\text{–2.33}}$50.33$^{\text{–38.33}}$57.17$^{\text{–30.92}}$
Object off65.00$^{\text{–24.33}}$80.33$^{\text{–2.00}}$60.00$^{\textbf{–32.00}}$46.33$^{\text{–42.33}}$62.92$^{\text{–25.17}}$

Table P1b. Quality of the predicted target mask (Molmo2, all 1,200 MoSA test samples).

Mask testPredicted MqM_qFrame centerRandom point
Peak or point inside the target box (% of frames)60.3735.428.35
IoU with the target box0.460.22n/a

Target boxes on MoSA (W3)

MoSA evaluates P(yc,yo∣V,q,B)P(y_c, y_o \mid V, q, B) by design. The box only specifies which object is queried and carries no motion, trajectory or answer information, and every method receives the same box track (Tab. A6). This separates attribution from localization. Without boxes, MqM_q is predicted from the visual tokens and question, as on OmniVCHall, where MoSDeR gains 7.47% over fine-tuning on average (Tab. 2). A new run on MoSA replaces the ground-truth box by the mask of one allocator for fine-tuning, 4DP-QA and MoSDeR alike, which directly measures P(yc,yo∣V,q)P(y_c, y_o \mid V, q) (Table P2, Fig. 7(d) and App. E.6). As in the downstream setting, this allocator is fitted only on target boxes of the training set and then frozen, so no test box enters any model.

Table P2. MoSA test accuracy (%) with ground-truth boxes and with the same predicted masks (Molmo2). Mask rows give the change from the box input as a superscript.

MethodInputNeitherCameraObjectBothOverall
Fine-tuningbox87.0075.0073.3352.0071.83
4DP-QAbox92.6765.0099.0078.6783.83
MoSDeRbox89.3382.3392.0088.6788.08
Fine-tuningmask89.00$^{\text{+2.00}}$58.33$^{\text{–16.67}}$65.33$^{\text{–8.00}}$49.33$^{\text{–2.67}}$65.50$^{\text{–6.33}}$
4DP-QAmask94.67$^{\text{+2.00}}$69.33$^{\text{+4.33}}$68.00$^{\text{–31.00}}$60.33$^{\text{–18.33}}$73.08$^{\text{–10.75}}$
MoSDeRmask77.67$^{\text{–11.67}}$82.33$^{\text{+0.00}}$67.33$^{\text{–24.67}}$79.33–9.33^{\text{–9.33}}76.67–11.42^{\text{–11.42}}

With the same predicted masks, MoSDeR remains the most accurate method, ahead of 4DP-QA by 3.58% Overall and 19.00% on Both, and of fine-tuning by 11.17% and 30.00%. Its drop from boxes concentrates on object-only motion (92.00% to 67.33%), which depends on localization. Neither also drops (89.33% to 77.67%), consistent with a misplaced mask passing motion from other regions to the object route when the target is static. Camera-only accuracy does not drop because the camera route reads the soft complement 1−Mq1-M_q, which still covers the scene when the mask is misplaced. Even without boxes, MoSDeR exceeds fine-tuning with ground-truth boxes, by 4.83% Overall (76.67% vs. 71.83%) and 27.33% on Both (79.33% vs. 52.00%). The revision calls MoSA target-specified attribution and separates the two settings in the sentence you quoted.

AuthorsOpenReview ↗

Follow-up on our response

Dear Reviewer PxnF,

Thank you again for your detailed review. We understand this is a busy period, and we just wanted to briefly follow up on our response.

In the revision, we (i) redrew Fig. 3 to label the loss and training stage of every module, (ii) added a route-disabling test indicating that each route mainly carries its own source (Fig. 7(c)), and (iii) added a box-free evaluation on MoSA in which all methods use the same predicted masks (Fig. 7(d), App. E.6).

We would be grateful to know whether these address your concerns, and we would be happy to clarify anything further.

Best regards,
Authors

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response: evidence on architecture, route roles, model scale, data scale and box-free input

We thank all reviewers for their feedback. Reviewers found the problem well motivated and practically important (PxnF, U1Yd, H7AK), the benchmark useful (PxnF, U1Yd), and the gains consistent across three VLM families (all). We uploaded a revised PDF with all changes in blue. The questions shared by several reviewers are answered by the evidence below, collected in Sec. 5.2 (Tab. 1, Fig. 7) and App. E (pp. 27-29).

Evidence for the shared questions

Question (reviewers)Evidence (where)Finding
Architecture (H7AK, U1Yd)Full FT, GRPO (Fig. 7(a))At most 62.00 on Both vs. 88.67
Route roles (PxnF, U1Yd)Disable one route (Fig. 7(c))Own state -32% or -16%, other near -2%
Stronger models (U1Yd, H7AK)Qwen3.5-27B, GPT-6.1 Sol (Tab. 1)One failing state each
Data scale (U1Yd, H7AK)25% to 100% data (Fig. 7(b))FT +1.67%, MoSDeR +17.00%
Robustness (U1Yd)Group bootstrap (App. E.4)All 95% CIs exclude zero
Longer clips (U1Yd)5 s AV2 windows (App. E.5)81.15 vs. 66.39 for FT
No target boxes (PxnF)Same predicted masks (Fig. 7(d))76.67 vs. 73.08 (4DP-QA) and 65.50 (FT)

Text revisions

  • Fig. 3 marks the training stage and loss of every module (PxnF, p. 5).
  • Tab. 1 adds Qwen3.5-27B and GPT-6.1 Sol as references, and Sec. 3 notes that the failure persists in both (U1Yd, H7AK, p. 4, L197-199).
  • Sec. 4.2 separates the target-specified MoSA setting from the box-free setting, and states that the allocator never sees motion labels or answers (PxnF, p. 5, L246-258).
  • Sec. 4.3 explains that source separation comes from single-source margins, and calls the two routes source-supervised factors (PxnF, p. 6, L299-303).
  • Sec. 5.1 and App. C.4 give the fine-tuning configuration (H7AK, p. 7 and p. 24).
  • Sec. 5.2 compares MoSDeR with the reference models of Tab. 1 (U1Yd, H7AK, p. 8, L407-410) and adds Fig. 7 with full fine-tuning and GRPO, data scale, route disabling, and box-free attribution (H7AK, PxnF, U1Yd, pp. 9-10), with exact values in App. E.
  • The conclusion states the short-clip and rigid-target scope (U1Yd, p. 10, L536-538).

Details are in the individual replies.

AuthorsOpenReview ↗

Thank you before the discussion closes

Dear Reviewers,

The discussion period is ending soon, so we wanted to thank you once more for your reviews. Your questions led to several of the new experiments in the revision (Sec. 5.2, Tab. 1, Fig. 7, and App. E), and we think the paper is clearer because of them.

We know this is a busy time. If you have a chance to look at our responses, we would really appreciate knowing whether they address your concerns, and we are happy to answer any remaining questions before the deadline.

Best regards,
Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.