acceptodds
Under review as a conference paper at ICLR 2027

When Left Is Right? Camera-Motion Understanding beyond Screen

Abstract

Can a vision-language model tell left from right? Modern VLMs solve visual puzzles and follow hour-long videos, yet when asked whether the camera moved left or right, they do no better than a coin flip. Adding camera-motion data or geometric representations has been tried, separately, and neither closes the gap. We argue that camera-motion understanding needs two conditions at once: the representation must couple off-screen camera motion with on-screen flow dynamics, and it must be interpreted by the language model rather than trained as a standalone pose or depth estimator. We propose CameraVLM, which pairs a motion-state representation with state-language alignment. Camera tokens capture how the camera moves off screen, flow tokens capture how the subject and background move on screen, and a two-stage training strategy keeps language in the loop so that both are learned together with camera-motion semantics. With only 9B parameters, CameraVLM outperforms all open-source and camera-motion-specific models by a wide margin across three benchmarks and more than 20 baselines, and matches or surpasses GPT-6 Astra as well as human viewers in a 48-participant study.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Spatial Video Language Models, Camera Motion Understanding, Dynamic Spatial Reasoning, Multimodal Large Language Models, Geometric Reasoning, Video Understanding

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_G2EnOpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies camera-motion understanding in vision–language models and proposes CameraVLM, built on Qwen3.5-VL-9B. The method introduces pose-supervised camera tokens and trajectory-supervised flow tokens to jointly represent camera motion and image motion using a shared visual encoder and multimodal backbone. A two-stage training procedure retains the language QA objective throughout, aligning continuous motion states with question semantics. On CameraBench, CineTechBench, and ShotBench, CameraVLM achieves a macro-average accuracy of 82.52%, substantially outperforming its backbone and the open-source models included in the comparison.

Critical strengths
  1. Well-motivated integration of geometric motion states and language interpretation. The paper identifies concrete ambiguities in camera-motion understanding, including camera versus subject motion, pan versus truck, and zoom versus dolly. Its geometric analysis provides a coherent motivation for combining camera states with scene-wide trajectory states. These representations are integrated into a shared multimodal backbone and learned with language supervision throughout training, connecting continuous motion modeling with question answering. The resulting model requires only video and questions at inference.

  2. Strong empirical performance supported by complementary evaluations. CameraVLM substantially outperforms its backbone and the compared open-source baselines across multiple camera-motion benchmarks. The comparison covers general-purpose, geometry-aware, camera-motion-specific, and commercial models, providing broad evidence of the method’s effectiveness. Pose estimation, point tracking, component analyses, qualitative examples, and human evaluation further help assess the learned representations and their practical capabilities. Together, these results provide substantive empirical support for the proposed approach.

Critical weaknesses

Major Weaknesses

  1. The central claim of query-dependent motion understanding lacks direct evaluation. The paper formulates a query as Q = (a, b, F, τ), emphasizing that the same dynamic state can support different descriptions depending on the target, reference entity, reference frame, and temporal interval. Figure 2 and Appendix A further discuss situations involving opposing viewpoints and mirror reflections. However, the reported results are primarily aggregated by camera-motion primitive or category. The paper does not separately evaluate whether the model produces geometrically consistent changes in its answers when these query elements are varied for the same video. Improvements on the existing benchmarks support better camera-motion recognition, but do not independently establish the claimed ability to reinterpret a continuous state under different queries. A targeted paired-query evaluation would strengthen this claim, with separate results for reference changes, temporal-interval changes, and reflection-related cases. If these situations are already covered by the existing benchmarks, their sample counts and corresponding results should be reported explicitly.

  2. The ablations of the two-stage training strategy and component contributions remain confounded. In Table 4, the two-stage variants are trained for 10K iterations per stage, whereas One-stage Training runs only the initial 10K-iteration state-interface stage with the language backbone frozen. Consequently, the difference between 69.0% and 60.8% reflects both the additional training budget and the unfreezing of the language backbone, and cannot be attributed specifically to the ordering of interface adaptation followed by joint training. A budget-matched single-stage baseline that jointly optimizes the model from the beginning would provide a more informative comparison. The authors should also clarify whether “w/o Camera & Flow” removes the supervision, tokens, prediction heads, or the entire corresponding pathways. In addition, a flow-only training variant is missing. The “w/o Flow” variant achieves 72.20% on ShotBench, exceeding the full ablation model’s 70.79%, so the benefit of combining the two state types is not uniform across benchmarks. Although the average results support their complementarity, their individual contributions and the conditions under which they help require clearer characterization.

  3. Direct controlled comparisons with closely related trajectory representations and geometry–language interfaces are missing. Table 1 includes a broad set of baselines, but some closely related approaches cited in Section 2 are not evaluated. TrajTok (Zheng et al., CVPR 2026; arXiv:2602.22779) represents trajectory information as tokens for video understanding, and its TrajVLM training includes CameraBench QA. Geometry-Guided Camera Motion Understanding in VideoLLMs (Feng et al., CVPR Workshops 2026; arXiv:2603.13119) directly investigates injecting geometric camera-motion cues into VideoLLMs and also explores feature distillation. These approaches are relevant to the proposed trajectory representation and geometry-to-language interface, respectively. Their original backbones, training data, and evaluation protocols differ, so their published numbers should not be compared directly with the present results. Where implementations or checkpoints are available, compatible evaluations would help establish the specific advantages of CameraVLM. Alternatively, trajectory-token or structured-motion-prompt controls using the same backbone and training budget would be informative. If such comparisons are infeasible, the authors should explain the concrete constraints and more precisely distinguish the contribution from these already cited approaches.

Minor Weaknesses

  1. Several implementation details needed for reproducibility remain unspecified. Section 3.5 and Appendix B do not explicitly provide the values of λc and λf, the pose coordinate convention and scale normalization, or the sampling proportions of video-only, pose-only, and video+pose inputs. Table 5 also includes Kubric without language annotations and 3DThinker without camera annotations. The authors should explain how the corresponding losses are activated or omitted when supervision is unavailable, and whether the two training stages use the same data-mixture proportions.

  2. The DPT module lacks a citation to its original reference. The flow pathway in Appendix B uses a DPT decoder, but the reference list does not appear to include Ranftl, Bochkovskiy, and Koltun, “Vision Transformers for Dense Prediction,” ICCV 2021. The citation should be added at the first occurrence of this component. Most baselines in the main result tables already have source citations; this comment concerns a specific omission in the implementation description.

  3. Some annotations and experimental settings in the supplementary material require clarification. The asterisk attached to CoWTracker in Table 5 is not explained. Table 7 varies the number of flow tokens, spatial grid, temporal window, and refinement iterations, but does not specify whether these variants are retrained models or inference-time modifications of the same checkpoint. Stating these details would make the interpretation of the sensitivity analysis more precise.

Questions for authors
  1. Query-dependent evaluation: Do the existing evaluation sets contain paired questions that change the target, reference entity, reference frame, or temporal interval for the same video? If so, please report their coverage and performance separately, including reflection-related cases where available. This would clarify how directly the current results support the query-dependent formulation.

  2. Training and component controls: Are results available for a single-stage model jointly trained for the same total 20K iterations as the two-stage ablation model? Please also clarify exactly what is removed in “w/o Camera & Flow” and whether a flow-only training result is available. These controls would help distinguish the effects of training duration, backbone unfreezing, and complementary state supervision.

  3. Comparison with closely related approaches: How does the proposed flow-token and state–language alignment design differ operationally from TrajTok/TrajVLM (Zheng et al., 2026; arXiv:2602.22779) and Geometry-Guided Camera Motion Understanding in VideoLLMs (Feng et al., 2026; arXiv:2603.13119), both cited in Section 2? Are compatible comparisons or matched-backbone controls available? If such comparisons are infeasible, please clarify the implementation, backbone, or evaluation constraints that prevent them.

2 comments

We thank the reviewer for recognizing the importance of jointly modeling camera and scene motion, the integration of geometric states with language understanding, and the strong empirical results. Below, we address each of the reviewer’s concerns in detail.


W1 & Q1: Query-Dependent Motion Reasoning

A: To directly test whether the model dynamically reinterprets identical videos under changing queries, we evaluated 77 same-video paired queries varying reference frames, mirror reflections, and temporal intervals/order. As shown in Table R1, CameraVLM achieves 96.10% overall, directly confirming that our continuous geometric states support query-dependent motion reinterpretation beyond static classification.

Table R1. Query-Dependent Motion Reasoning (Accuracy %)

Query VariationPairsQwen3.5-VLCambrian-PCameraVLM
Reference Frame Shift3046.753.390.0
Mirror Reflection20.00.0100.0
Temporal Interval / Order4586.791.1100.0
Overall7768.874.096.1

W2 & Q2: Two-stage Training and Component Ablations

A: Table R2 adds a budget-matched One-stage Joint baseline and a Flow-only variant under the same 20K training steps. Here, w/o Camera and w/o Flow remove the entire corresponding pathway, including state tokens, prediction heads, and supervision losses.

Staged alignment: At the matched 20K-step budget, Two-stage Full outperforms One-stage Joint by +5.89 pp, confirming that adapting the state interface before joint tuning prevents optimization conflicts between geometric representations and the LLM.

Complementary conditions: Pose models global camera motion and suffices when scenes are dominated by pure camera movement, which is why Pose-only slightly edges out the full model on camera-centric ShotBench without extra local flow noise. In contrast, Flow is required whenever moving objects entangle with camera motion, driving substantial gains on CameraBench and CineTech.

Table R2. Two-stage Training and Component Ablations (Accuracy %)

Training SettingCameraBenchCineTechShotBenchAvg.
Backbone63.654.947.055.2
QA-only67.257.142.155.5
Pose-only64.555.872.264.2
Flow-only65.357.255.459.3
One-stage 10K64.564.053.960.8
One-stage Joint 20K64.166.258.963.1
Two-stage w/o Stage-1 QA54.548.846.649.9
Two-stage Full 20K68.267.970.869.0

W3 & Q3: Comparison with TrajTok/TrajVLM and Geometry-Guided Methods

A: TrajTok has only released its Segmenter and TrajViT-v2 checkpoints without the full TrajVLM checkpoint, and Geometry-Guided has not released its full inference pipeline. We therefore evaluate the released TrajViT-v2 in Table R3, compare on Geometry-Guided's CameraMotionVQA benchmark in Table R4, and run matched-backbone structured and VGGT-guided prompting controls on Qwen3.5-VL in Table R5.

Operational differences: TrajTok compresses video into sparse trajectory tokens for efficient tokenization without modeling camera ego-motion. Geometry-Guided injects external VGGT cues via text prompting or one-way feature distillation without explicit scene-flow tokens. In contrast, CameraVLM jointly learns 3D camera poses and 256-grid full-scene trajectories directly from native VLM features, pooling trajectories into 64 flow tokens that interact with camera and video tokens inside the LLM via two-stage state–language alignment.

Empirical findings: CameraVLM substantially outperforms TrajViT-v2 in Table R3 and all baselines on CameraMotionVQA in Table R4, including the 397B Kling-MLLM. Under the matched Qwen3.5-VL backbone in Table R5, external VGGT-guided prompting brings only modest gains, leaving a large gap to CameraVLM. We will add these comparisons and discussions to the revision.

Table R3. Released TrajViT-v2 Evaluation (Accuracy %)

ModelCameraBenchCineTechShotBenchAvg.
TrajViT-v247.231.031.036.4
CameraVLM88.277.981.582.5

Table R4. CameraMotionVQA from Geometry-Guided (1,228 Questions, Accuracy %)

ModelAccuracy
Qwen3.5-VL-9B27.8
Cambrian-P27.3
TrajViT-v225.6
Kling-MLLM-35B-A3B42.0
Kling-MLLM-27B47.2
Kling-MLLM-397B54.7
CameraVLM59.5

Table R5. Structured and Geometry-Guided Prompting Controls (Accuracy %)

ModelPromptCameraBenchCineTechShotBenchAvg.
Qwen3.5-VLStandard QA66.057.552.658.7
Qwen3.5-VLMotion Instruction Prompt69.258.450.959.5
Qwen3.5-VLVGGT-guided Prompt70.461.362.864.8
CameraVLMStandard QA87.878.881.582.7
CameraVLMMotion Instruction Prompt86.977.980.481.7

W4: Implementation Details

A: We activate or ignore each loss per sample using binary validity masks before averaging over valid batch elements: QA Loss ignores unlabeled text tokens via the cross-entropy ignore index on Kubric, Camera Loss zeros out pose gradients on 3DThinker, and Flow Loss masks out missing or invalid trajectories and frames.

Both stages share the identical data mixture, with input modality sampling ratios following the natural distribution of the combined datasets. We set (λc,λf)=(0.2,0.1)(\lambda_c,\lambda_f)=(0.2,0.1) in Stage 1 and (λc,λf)=(0.4,2.0)(\lambda_c,\lambda_f)=(0.4,2.0) in Stage 2, and represent camera poses in the OpenCV world-to-camera convention anchored at the first frame with translations normalized by mean scene depth. We will include these details in the revision.


W5: DPT Citation

A: Thank you for pointing out the missing citation. We will cite Ranftl, Bochkovskiy, and Koltun, Vision Transformers for Dense Prediction (ICCV 2021), at the first mention of the DPT decoder.


W6: Additional Clarifications

A: The asterisk next to CoWTracker in Table 5 was mistakenly added during formatting. All variants in Table 7 are inference-time modifications using the same trained checkpoint, without retraining. We will remove the asterisk and clarify the experimental settings in the revision.

Reviewer_rh6gOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how VLMs understand movements of the cam and the objects, especially for the cases when the camera motion and object motion produce entangled changes in the video. It adds camera tokens trained to predict camera pose and flow tokens that represent point movement across frames. These are used together with the video and question to generate an answer. After training, these info become all self-generated so added minimal additional overhead. Results show large improvements over the original backbone on CameraBench, CineTechBench, and ShotBench.

Post-rebuttal: 2 -> 3

Critical strengths
  1. The paper targets an important problem. Movement on the screen does not directly tell us how the camera moved, especially when objects also move. Learning camera geometry together with point motion is a reasonable way to address this.
  2. The method shows large improvements across benchmarks, average accuracy increases from 55.16% for the backbone to 82.52%. This is a useful result and suggests that the approach can benefit camera motion understanding beyond one dataset.
Critical weaknesses
  1. The novelty seems limited. Prior work such as Cambrian P already uses camera tokens and joint language and pose training. The tracking branch also follows an existing design. 4DP-QA proposes the concept of True-Motion aiming to disentagle cam and object motion which is relevant to this paper's motivation.
  2. The added flow branch shows some benefit, but the training comparison is not controlled: the one stage model trains for 10K iterations with the LM frozen, while the two stage model gets another 10K iterations with the LM unfrozen. We cannot conclude if the proposed schedule is better than simply training everything jointly for the same amount of time.
  3. The paper's claim about interpreting motion under different references needs more direct evidence. The reported results mainly measure camera motion recognition. It would help to test the same video with questions about different reference objects or time intervals. Also, masking both camera and flow tokens leaves average accuracy unchanged at 82.5%. This does not mean their training is useless, but it makes it unclear whether the model uses these states when answering or mainly benefits from their supervision during training.
Questions for authors
  1. Requested comparison: How does direct joint training perform with the same data and total training budget as the proposed two stage schedule?
  2. In the masking experiment, are the state tokens blocked before they can affect other tokens, or only after their information may already have passed into the shared representation?
  3. The appendix reports pooled human accuracy of 73.16%, below all three benchmark scores in the main table. This is a bit surprising. Please explain how these numbers are obtained.
4 comments

We thank the reviewer for recognizing the importance of disentangling camera and object motion, as well as the substantial improvements across three benchmarks. Below, we address each of the reviewer’s concerns in detail.


W1: Novelty and Distinction from Cambrian-P and 4DP-QA

A: We thank the reviewer for pointing out the valuable 4DP-QA benchmark and dataset work. CameraVLM differs fundamentally from Cambrian-P and 4DP-QA in both motivation and technical formulation, as summarized in Table R0: CameraVLM targets comprehensive cinematic shot understanding involving screen-referenced tracking shots and intrinsic–extrinsic coupling like zoom and dolly, which requires jointly learning the coupled representation of off-screen and on-screen states within the VLM.

Table R0. Comparison of problem space and technical formulation

DimensionCambrian-P4DP-QACameraVLM (Ours)
Primary Goal3D spatial perceptionDataset & SFT for 3D world object motionComprehensive cinematic shot language
Off-screen StateCanonical 3D poseRigid pose under fixed KK — no zoomDedicated Camera Head: 3D pose + dynamic FoV
On-screen StateNoneNo motion head — single-point text coordsDedicated Flow Head: dense scene-wide flow tokens
DisentanglementPose-only alignmentLabel level via fixed reference framesRepresentation level via two-stage alignment

Evidence. Tables R1 and R2 provide direct comparisons with Cambrian-P. Under identical training settings, CameraVLM consistently outperforms Cambrian-P across all camera-motion and general video benchmarks. As the pretrained checkpoint and dataset of 4DP-QA are not yet publicly released, we will thoroughly discuss it in the revision and explore incorporating it in future work.

Table R1. Training and Architecture Comparison (Accuracy %)

ModelCameraBenchCineTechShotBenchAvg.
Cambrian-P64.353.148.155.2
Cambrian-P w/ our training (20K)67.357.868.264.4
CameraVLM (20K)68.267.970.869.0
CameraVLM (Full)88.277.981.582.5

Table R2. Cross-Benchmark Generalization (Accuracy %)

BenchmarkCambrian-PCameraVLM
Camera Motion (3 Benchmarks)55.282.5
CameraMotionVQA27.359.5
Video-MME Short (900 QAs)72.978.7
FAVOR-Bench (1,053 QAs)43.654.4
TempCompass (1,580 QAs)68.274.7

W2 & Q1: Two-stage Training and Component Ablations

A: We added One-stage Joint Training with identical data and total training steps (20K), along with Flow-only and w/o Stage-1 QA ablations, to further validate our training strategy and geometric supervision.

At the matched 20K-step budget, Two-stage Full outperforms One-stage Joint by +5.89 pp, confirming that the gains stem from staged alignment rather than extra training steps. Moreover, Pose-only (+8.68 pp) and Flow-only (+3.83 pp) each improve over QA-only, while coupling both achieves the best performance (68.96%), validating their complementary roles.

Table R3. Two-stage Training and Component Ablations (Accuracy %)

Training SettingCameraBenchCineTechShotBenchAvg.
Backbone63.654.947.055.2
QA-only67.257.142.155.5
Pose-only64.555.872.264.2
Flow-only65.357.255.459.3
One-stage 10K64.564.053.960.8
One-stage Joint 20K64.166.258.963.1
Two-stage w/o Stage-1 QA54.548.846.649.9
Two-stage Full 20K68.267.970.869.0

W3 & Q2: State Utilization and Query-Dependent Motion Reasoning

A: 1. Masking and State Internalization. In Table 6, Camera/Flow tokens and their connectors are removed before LLM prefill. Performance remains high because geometric supervision is internalized into shared representations, consistent with Cambrian-P's findings that explicit spatial states primarily serve as training-time geometric inductive biases.

Since State and Video tokens share visual features, pose/trajectory supervision updates the Vision Encoder and Merger, while Stage-2 joint training further adapts the LLM. Thus, motion information remains accessible through Video tokens even without explicit States. Importantly, removing geometric supervision reduces accuracy from 68.96% to 55.47%.

To validate this, we bidirectionally swapped Vision Encoders between Qwen and CameraVLM, keeping each LLM fixed and removing State tokens at inference. Transferring CameraVLM's visual modules improves Qwen, while the reverse swap degrades CameraVLM. These results support geometric information being internalized across both visual and language representations.

Table R4. Bidirectional Visual-module Swapping (ShotBench, Accuracy %)

LLM BackboneVisual Modules (Encoder + Merger)ShotBench (Full)
Native QwenNative Qwen Enc. + Merger52.6
Native QwenCameraVLM Enc. + Merger58.6
CameraVLM (w/o State)CameraVLM Enc. + Merger81.0
CameraVLM (w/o State)Native Qwen Enc. + Merger74.8

 

2. Role of explicit States on hard samples. On a 200-question hard subset of ShotBench, we applied a stricter intervention than input removal: zeroing State hidden representations and blocking their attention at every LLM layer. Accuracy drops from 67.00% to 62.50% (−4.50 pp).

This suggests that internalized Video tokens suffice for routine samples, while explicit States remain important for resolving motion ambiguity in visually challenging cases.

Table R5. Inference Interventions on ShotBench Hard-200 (Accuracy %)

Inference InterventionShotBench (Hard-200, Accuracy %)
Full Model (Unmasked)67.0
Prefill Input Zeroing (Both States)64.0 (−3.0)
All-layer Attention Blocking (Camera only)64.5 (−2.5)
All-layer Attention Blocking (Flow only)63.0 (−4.0)
All-layer Attention Blocking (Both States)62.5 (−4.5)

 

3. Reference-frame and temporal reinterpretation. To test whether the model consistently updates its motion interpretation when the reference frame or temporal order changes for the same video, we constructed 77 paired queries covering reference-frame shifts (N=30), mirror reflections (N=2), and temporal intervals/order (N=45).

Qwen3.5-VL and pose-only Cambrian-P largely fail under reference-frame shifts and mirror reflections, while CameraVLM maintains high accuracy. This demonstrates its ability to reinterpret motion according to the reference frame and temporal conditions specified by each query.

Table R6. Query-Conditioned Reinterpretation (Accuracy %)

Query-Conditioned ReinterpretationNBackbone (Qwen3.5-VL)Cambrian-PCameraVLM (Ours)
Reference Frame Shift (Ground / Initial Frame)3046.753.390.0
Mirror Reflection Reference20.00.0100.0
Temporal Interval / Order4586.791.1100.0
Overall Weighted Avg.7768.874.096.1

Q3: Clarification of Human Accuracy in Table 8

A: We clarify that 73.16% represents question-level pooled accuracy. Participants were allowed to skip questions when uncertain. Table 1 excludes skipped responses and averages accuracy across three benchmarks, whereas Table 8 includes skipped responses in the total response count and computes pooled accuracy, leading to the reported discrepancy.

We will report both question-level pooled and category-level average accuracies in Table 8 and clarify the evaluation protocols in the revision.

Thank you for the additional experiments. My main concerns have been addressed.

  1. Please clarify the human accuracy calculation. Averaging the displayed category scores gives 71.91%, rather than 73.16%.
  2. Please explain how the paired queries and hard examples were selected, and include the new results in the revision.

Overall, the rebuttal strengthens the paper, and I will raise my score.

Thank you for your follow-up and for reconsidering your score.

Regarding human accuracy, we appreciate the reviewer for pointing this out. Upon rechecking the evaluation statistics, we clarify that 73.16% is the question-level pooled accuracy, while the category-level average is 71.91%. Participants could skip questions when uncertain. Table 1 excludes skipped responses, whereas Table 8 includes them in the total response count, leading to the reported discrepancy. We have updated our previous responses and will report both averages in Table 8 to avoid confusion.

Regarding sample selection, we selected videos using caption keywords and constructed 77 paired queries across three settings: 30 reference-frame queries by changing reference objects or coordinate frames, 2 mirror queries by pairing reflected and real-world motion questions in mirror-containing clips, and 45 temporal queries by changing time intervals or order within the same video. For Hard-200, we selected the 200 ShotBench questions with the smallest log-probability margins between the full model's top two candidate answers.

We will document these selection procedures and include the additional results in the revision.

Reviewer_WHxsOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

CameraVLM augments a Qwen3.5-VL backbone with camera tokens supervised by pose and flow tokens supervised by point trajectories. A two-stage procedure combines geometric objectives with language QA, using joint supervision from motion-specific and general video data. The model reports substantial gains over its backbone across three camera-motion benchmarks and provides state-estimation and human comparisons. The empirical gains are promising, but the causal role of the proposed alignment strategy and the claimed human-level comparison require stronger controls and consistent evaluation statistics.

Critical strengths
  1. The paper targets an important distinction between camera motion and image motion. Pose and point trajectories are complementary signals, and the model integrates both without requiring external pose estimates at inference.

  2. The reported gains over the backbone are large and occur across three datasets. The paper also evaluates the learned pose and tracking states, providing evidence beyond language answers alone, and includes useful qualitative examples of compound motions.

Critical weaknesses
  1. The ablations do not isolate the proposed two-stage state–language alignment. The one-stage model in Table 4 receives only 10K frozen-language-backbone steps, whereas the two-stage model receives 20K steps and an unfrozen second stage. This changes both optimization budget and trainable capacity. The table also lacks a flow-only variant and a direct control for the first-stage QA gradient. Please compare single-stage joint training with staged training under the same data, total steps, and update budget, and clarify the QA-only, pose-only, flow-only, and joint-supervision controls before concluding that the two proposed conditions must hold together.

  2. The evidence does not yet establish query-conditioned use of the explicit states. Masking both state-token groups leaves the average unchanged at 82.5 in Table 6. The paper interprets this as distributed information, but does not specify whether masking occurs before prefill, after states have already mixed with video tokens, or only at the output positions. Moreover, fixed motion-classification benchmarks do not directly test reinterpreting the same state under a changed reference entity, frame, or interval.

Questions for authors
  1. What exactly is masked in Table 6 and at what stage? Can you remove both pathways before any video/state mixing, then compare with state shuffling or temporal perturbation? On a small annotated set of same-video, changed-reference/interval question pairs, does the model change its answers consistently with the specified geometry?

  2. Why is human Overall 73.16 in Table 8 while all three benchmark accuracies in Table 1 exceed 76?

  3. Beyond the training-schedule ablations in Q1, how much of the improvement is attributable to the proposed architecture rather than additional task-specific training? Could you compare CameraVLM with the unmodified backbone fine-tuned on the same QA data and a representative alternative motion/geometry architecture under a matched training-compute budget?

  4. Does the proposed training preserve the backbone's broader capabilities, and do the gains transfer beyond camera-motion classification? The paper mixes general video QA into training to preserve language abilities, but could you provide before/after results on representative general video-understanding and image/language tasks? For finer-grained motion transfer, a benchmark such as FAVOR-Bench[1] is relevant because it includes Camera Motion (CM) alongside other evaluation perspectives. Reporting CM and non-CM results separately, together with general-task results, would help distinguish camera-motion specialization, broader motion-understanding gains, and any capability regression.

[1] FAVOR-Bench: A Comprehensive Benchmark for Fine-Grained Video Motion Understanding

5 comments

We thank the reviewer for recognizing our camera–image motion disentanglement, complementary pose–trajectory design, and strong gains across three benchmarks. Below, we address each of the reviewer’s concerns in detail.


W1: Two-stage Training and Component Ablations

A: We added One-stage Joint Training with identical data and total training steps (20K), alongside Flow-only and w/o Stage-1 QA ablations to isolate the contributions of training strategy, language supervision, and geometric states.

Two-stage alignment: Two-stage outperforms One-stage Joint by 5.89 pp at matched 20K steps, demonstrating the effectiveness of staged State–Language Alignment beyond additional training iterations.

Stage-1 QA supervision: QA Loss maintains compatibility between geometric states and the VLM's language understanding. Geometry-only training in Stage 1 can shift the state interface away from the language model's interpretable feature space, severely degrading subsequent QA and even producing unparseable responses. Removing Stage-1 QA reduces accuracy from 68.96% to 49.93%, below the Backbone.

Camera–Flow complementarity: Pose-only and Flow-only improve over QA-only by 8.68 pp and 3.83 pp, respectively, while joint supervision achieves the highest average accuracy (68.96%), supporting the complementary contributions of Camera Pose and Scene-wide Flow.

Table R1. Two-stage Training and Component Ablations (Accuracy %)

Training SettingCameraBenchCineTechShotBenchAvg.
Backbone (Qwen3.5-VL)63.654.947.055.2
QA-only67.257.142.155.5
Pose-only64.555.872.264.2
Flow-only65.357.255.459.3
One-stage 10K64.564.053.960.8
One-stage Joint 20K64.166.258.963.1
Two-stage w/o Stage-1 QA54.548.846.649.9
Two-stage Full 20K68.267.970.869.0

W2 & Q1: Masking Clarification, State Internalization, and Motion Disambiguation

A: 1. Masking and State Internalization. In Table 6, Camera/Flow tokens and their connectors are removed before LLM prefill. Performance remains high because geometric supervision is internalized into shared representations, consistent with Cambrian-P's findings that explicit spatial states primarily serve as training-time geometric inductive biases.

Since State and Video tokens share visual features, pose/trajectory supervision updates the Vision Encoder and Merger, while Stage-2 joint training further adapts the LLM. Thus, motion information remains accessible through Video tokens even without explicit States. Importantly, removing geometric supervision reduces accuracy from 68.96% to 55.47%.

To validate this, we bidirectionally swapped Vision Encoders between Qwen and CameraVLM, keeping each LLM fixed and removing State tokens at inference. Transferring CameraVLM's visual modules improves Qwen, while the reverse swap degrades CameraVLM. These results support geometric information being internalized across both visual and language representations.

Table R2. Bidirectional Visual-module Swapping (ShotBench, Accuracy %)

LLM BackboneVisual Modules (Encoder + Merger)ShotBench (Full)
Native QwenNative Qwen Enc. + Merger52.6
Native QwenCameraVLM Enc. + Merger58.6
CameraVLM (w/o State)CameraVLM Enc. + Merger81.0
CameraVLM (w/o State)Native Qwen Enc. + Merger74.8

 

2. Role of explicit States on hard samples. On a 200-question hard subset of ShotBench, we applied a stricter intervention than input removal: zeroing State hidden representations and blocking their attention at every LLM layer. Accuracy drops from 67.00% to 62.50%.

This suggests that internalized Video tokens suffice for routine samples, while explicit States remain important for resolving motion ambiguity in visually challenging cases.

Table R3. Inference Interventions on ShotBench Hard-200 (Accuracy %)

Inference InterventionShotBench (Hard-200, Accuracy %)
Full Model (Unmasked)67.0
Prefill Input Zeroing (Both States)64.0 (−3.0)
All-layer Attention Blocking (Camera only)64.5 (−2.5)
All-layer Attention Blocking (Flow only)63.0 (−4.0)
All-layer Attention Blocking (Both States)62.5 (−4.5)

 

3. Reference-frame and temporal reinterpretation. To test whether the model consistently updates its motion interpretation when the reference frame or temporal order changes for the same video, we constructed 77 paired queries covering reference-frame shifts (N=30), mirror reflections (N=2), and temporal intervals/order (N=45).

Qwen3.5-VL and pose-only Cambrian-P largely fail under reference-frame shifts and mirror reflections, while CameraVLM maintains high accuracy. This demonstrates its ability to reinterpret motion according to the reference frame and temporal conditions specified by each query.

Table R4. Query-Conditioned Reinterpretation (Accuracy %)

Query-Conditioned ReinterpretationNBackbone (Qwen3.5-VL)Cambrian-PCameraVLM (Ours)
Reference Frame Shift (Ground / Initial Frame)3046.753.390.0
Mirror Reflection Reference20.00.0100.0
Temporal Interval / Order4586.791.1100.0
Overall Weighted Avg.7768.874.096.1

Q2: Clarification of Human Accuracy in Table 8

A: We clarify that 73.16% represents question-level pooled accuracy. Participants were allowed to skip questions when uncertain. Table 1 excludes skipped responses and averages accuracy across three benchmarks, whereas Table 8 includes skipped responses in the total response count and computes pooled accuracy, leading to the reported discrepancy.

We will report both question-level pooled and category-level average accuracies in Table 8 and clarify the evaluation protocols in the revision.

Q3: Architectural Contributions and Comparisons with Alternative Motion/Geometry Architectures

A: To distinguish architectural gains from task-specific training, we compare the original Qwen3.5-VL backbone, QA-only fine-tuning, Cambrian-P, and CameraVLM, using identical training data and a matched 20K-step budget.

Task-specific fine-tuning vs. architectural contributions. QA-only fine-tuning yields negligible improvement over the original backbone, whereas CameraVLM achieves substantial gains. This demonstrates that task-specific QA training alone cannot explain our improvements, highlighting the contribution of geometric state learning and State–Language Alignment.

Comparison with alternative architectures. Under matched training conditions, CameraVLM outperforms Cambrian-P (68.96% vs. 64.42%), supporting the benefit of jointly modeling Camera Pose and Scene-wide Flow States. We also evaluate TrajViT-v2 and include comparisons with DepthVLM, VLM-3R-7B, and Spatial-MLLM in the main paper.

Together, these results validate the architectural contributions of Camera–Flow joint modeling and two-stage State–Language Alignment, beyond task-specific fine-tuning.

Table R5. Architecture and Training Comparisons (Accuracy %)

SettingModelCameraBenchCineTechShotBenchAvg.
Off-the-shelfQwen3.5-VL63.654.947.055.2
TrajViT-v247.231.031.036.4
Cambrian-P64.353.148.155.2
Matched Fine-tuning 20KFine-tuned Backbone67.257.142.155.5
Cambrian-P67.357.868.264.4
CameraVLM68.267.970.869.0

Q4: General Capability Retention and Motion Transfer

A: We evaluate CameraVLM against Qwen3.5-VL and Cambrian-P on Video-MME, TempCompass, FAVOR-Bench, and CameraMotionVQA to assess general capability retention and transfer beyond camera-motion recognition.

General capability retention (Table R6). CameraVLM maintains comparable overall performance on TempCompass, with improvements on non-camera motion tasks involving Food and Vehicles. On Video-MME, it improves Reasoning and Film & Television, maintains comparable Knowledge performance, and retains strong overall video understanding despite a modest decrease relative to Qwen3.5-VL.

Motion transfer and generalization (Table R7). CameraVLM boosts generic camera-motion understanding on FAVOR-Bench while maintaining strong non-camera motion performance, and substantially outperforms both baselines on the unseen CameraMotionVQA benchmark.

Table R6. General Video Understanding (Accuracy %)

BenchmarkCategoryNQwen3.5-VLCambrian-PCameraVLM
TempCompassOverall1,58074.668.274.7
Food (Non-Camera)15076.768.780.7
Vehicles (Non-Camera)13179.464.983.2
Video-MMEOverall90082.472.978.7
Reasoning16779.670.781.4
Film & Television12082.582.584.2
Knowledge27080.771.580.4

Table R7. Motion Understanding and Cross-Benchmark Generalization (Accuracy %)

BenchmarkCategoryNQwen3.5-VLCambrian-PCameraVLM
FAVOR-BenchGeneric Camera Motion53247.039.352.4
Non-Camera Motion52157.848.056.4
CameraMotionVQAOverall1,22827.827.359.5

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Rebuttal Summary

We thank all three reviewers for their feedback and recognition of our contributions. WHxs, rh6g, and G2En acknowledged the importance of the research problem and the motivation. WHxs and G2En recognized the integration of geometric states with language understanding and the performance gains, while rh6g highlighted the improvements over the original backbone.

We addressed the reviewers' concerns through additional experiments and clarifications:

  1. Two-stage training, component contributions, and query-dependent reasoning (WHxs, rh6g, G2En): We conducted controlled ablations and paired-query evaluations, validating staged alignment, complementary geometric supervision, and query-conditioned motion reasoning.

  2. Role of explicit geometric states (WHxs, rh6g): We conducted state interventions and visual-module swapping experiments, clarifying the contributions of geometric supervision to learned representations and motion reasoning.

  3. Architectural contributions, and related methods (WHxs, rh6g, G2En): We added controlled comparisons with related architectures and geometry-guided methods to distinguish our contributions from existing approaches.

  4. Generalization and capability preservation (WHxs): We evaluated external benchmarks to assess cross-task generalization and clarify capability trade-offs.

  5. Evaluation and reproducibility (WHxs, rh6g, G2En): We clarified human evaluation statistics, sample selection procedures, and implementation details.

We appreciate WHxs and rh6g for acknowledging that their main concerns were addressed and raising their scores, and G2En for recognizing our technical motivation and empirical contributions. We will incorporate the additional experiments, comparisons, and clarifications into the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.