acceptodds
Withdrawn from ICLR 2027

Temporally Grounded Compositional Camera Motion Understanding via Geometric Knowledge Distillation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Understanding camera motion is fundamental to video perception, with applications in spatial intelligence and controllable video generation. Multimodal large language models (MLLMs) provide a natural interface for this task, but existing work typically assigns one or more labels to an entire clip. Such clip-level recognition overlooks two defining properties of real camera motion: it can change within a shot, and multiple movements can occur simultaneously. We therefore formulate camera-motion understanding as temporally grounded, compositional recognition, which requires a model to localize motion-consistent intervals and identify every movement active within each interval. We introduce CamChoreo, a benchmark of real single-shot clips with expert-annotated temporal segments. Its annotations use a compact vocabulary of direction-aware labels, and nearly half of the segments contain compound camera motion, with multiple movement primitives active simultaneously. Recognizing such fine-grained, compositional motion is hard for current MLLMs, whose visual encoders emphasize semantic content rather than the geometric evidence on which camera motion depends. Directly injecting features from a frozen 3D foundation model addresses this gap, but requires running the expensive geometry model on every input; we refer to this baseline as CamInject. We instead propose CamDistill, which distills the same geometric knowledge into lightweight camera tokens during training and removes the 3D model at inference. CamDistill matches the accuracy of direct feature injection without running the 3D teacher at inference. Together, CamChoreo and CamDistill advance camera-motion understanding from clip-level labeling to temporally grounded, compositional recognition.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
MLLM, camera motion undestanding, camera pose estimation, video understanding

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_C6QaOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper formulates camera-motion understanding from clip-level classification to temporally grounded, compositional recognition. The task asks the model to split a video shot into motion-consistent intervals and predict predicts the complete set of direction-aware movements active in each interval.

The paper contributes:

  • CamChoreo. An evaluation benchmark of 4,229 real single-shot YouTube clips with 8,591 expert-annotated segments, using a 20-label taxonomy that includes direction. 57% of clips have multiple segments, and 44.2% of segments have simultaneous movements.

  • An empirical diagnosis. Zero-shot MLLMs and a rule-based baseline on VGGT-Ω poses both perform poorly.

  • CamInject and CamDistill. CamInject, a direct-injection reference baseline, projects camera tokens from a frozen 3D foundation model (VGGT-Ω) into the MLLM before each frame's visual tokens. It works well, but it has to run the 3D model at inference. CamDistill removes this cost. A lightweight Geometry-aware Camera Token Extractor (GCTE) reads intermediate frozen ViT features and predicts one camera token per frame. During training, these tokens are aligned with VGGT-Ω camera tokens by cosine distance, and the 3D model is removed at inference. On Qwen3-VL-4B, both reach 67.5 micro F1. CamDistill adds only about +0.1 s and +1.8 GB at inference, compared with +5.9 s and +4.8 GB for CamInject.

Critical strengths
  • Well-motivated task and benchmark. Existing benchmarks either treat camera movement as one attribute among many or assign a single label set to a whole clip. Neither captures motion that changes within a shot. CamChoreo addresses this gap by combining real web video with camera-specific, direction-aware multi-label annotations over variable-length temporal segments.
  • Useful diagnostic metrics. Separating SegLoc from SegDet shows that the bottleneck is recognizing the full label set, not finding boundaries.
  • A clean, practical method. Using VGGT-Ω as a geometry teacher is a good fit. It supplies exactly the parallax and perspective cues that semantic MLLM encoders lack, and CamInject confirms these features help. Distilling them (CamDistill) only at training time keeps almost all of that gain while removing the 3D model at inference. Model cost, the latency and memory measurements make the efficiency claim concrete.
Critical weaknesses
  1. The task is close to multi-label temporal action localization: the timeline is fully divided into intervals, and each interval gets a set of labels from a closed vocabulary. Here, the 20 camera-motion labels play the role of action classes. Methods for those tasks (e.g. ActionFormer [1], DETR-style [2] temporal detectors) predict boundaries and multi-label classes directly with a lightweight head. CamDistill instead fine-tunes the whole LLM on 43k clips to output this fixed label set in a rigid JSON format, so the LLM essentially acts as a decoder for structured prediction. Hand-written threshold rules on VGGT-Ω poses already reach 63.9 [email protected]. A learned head would likely do much better, and at far lower cost than 10s per clip. This matters for large-scale annotation for video generation, one of the paper's stated uses.

  2. The main comparison is confounded. Table 2 compares zero-shot general-purpose MLLMs and Cam-Motion-7B tuned on CameraBench(with a different taxonomy and output format) against CamDistill/CamInject (which are trained on 43K clips annotated with CamChoreo's own taxonomy and protocol). The micro-F1 margin therefore mainly reflects in-task supervision. Table 3 shows plain SFT (supervised fine-tuning) on the same data already reaches 62.2 (vs 24.2 zero-shot), leaving a 5.3 margin for CamDistill. The claim that "current MLLMs struggle" holds only in the zero-shot setting. Once fine-tuned on the same data, the same MLLM reaches 62.2 micro F1 (Table 3), and CamDistill and CamInject are themselves built on this SFT.

  3. CamChoreo (built from YouTube) is the evaluation-only benchmark and the training data is private (43k Tencent Video clips). Every trained results depends on this private data. However, it is only mentioned after section 5 (Experiments). The paper never explains why it trains on Tencent Video and tests on YouTube.

[1] Zhang, Chen-Lin, Jianxin Wu, and Yin Li. "Actionformer: Localizing moments of actions with transformers." In European conference on computer vision, pp. 492-510. Cham: Springer Nature Switzerland, 2022.

[2] Carion, Nicolas, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. "End-to-end object detection with transformers." In European conference on computer vision, pp. 213-229. Cham: Springer International Publishing, 2020.

Questions for authors
  1. Could a transformer decoder head do the same job as the LLM decoder? For example, a DETR-style set decoder or a per-frame multi-label segmentation head on GCTE tokens, VGGT-Ω tokens, or frozen ViT features. A billion-parameter language decoder appears excessive for 20 outputs, while a lightweight decoder head with a few million parameters can predict in a single forward pass.

2a. The SFT baseline (4B and 8B) should be included in Table 2. The table should be split into zero-shot and supervised rows. Baselines should be fine-tuned on the same Tencent data (e.g. Qwen-VL, Cam-Motion-7B).

2b. How does a MLLM model (8B) with SFT score on CameraBench and CMVQA? Without these, can the transfer in Table 4 be credited to GCTE rather than to fine-tuning on 43K camera-motion clips?

  1. Please explain the reason between the training and testing split on two different platforms. Do authors plan to publish the training data ? If not, how could the main results be reproduced ?
Reviewer_neAyOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates the task of understanding composite camera motions with temporal localization, which requires a model to identify the start and end times of camera motions in a video, as well as the multiple, concurrent motions occurring within each time segment. The authors introduce CamChoreo, a benchmark dataset based on real-world videos, and propose CamDistill. This approach extracts per-frame camera tokens from the intermediate-layer features of a frozen visual encoder, utilizes a 3D foundation model to provide geometric distillation supervision, and subsequently feeds these tokens into a language model to generate predictions. During inference, this method eliminates the need to run the 3D teacher model, thereby enhancing camera motion recognition performance with minimal additional overhead.

Critical strengths
  1. This studied task has practical value.
  2. Factors such as feature layers, module depth, token positions, and teacher selection were examined, which helps in understanding the impact of these various design choices.
Critical weaknesses
  1. Limited methodological novelty. The key components have close precedents: geometry-guided camera-motion understanding [1], learnable per-frame camera tokens with geometric supervision [2], and geometric distillation from 3D foundation models into VLMs [3]. GCTE combines standard attention modules with cosine-based feature distillation. The paper does not sufficiently establish a distinct methodological contribution beyond integrating these existing ideas.
  2. Insufficient attribution of performance gains. CamDistill improves micro F1 over SFT by 5.3 points on the 4B backbone and 3.7 points on the 8B backbone, while adding approximately 110M and 216M trainable parameters, respectively. Although the distillation-weight analysis is useful, comparisons with capacity-matched adapters and alternative supervision targets are needed to isolate the contribution of camera-specific geometric distillation.
  3. Lack of competitive task-specific baselines. Most comparisons involve zero-shot MLLMs rather than alternatives trained on the same data. The pose-only baseline uses hand-designed rules and cannot predict certain categories, while Cam-Motion-7B produces valid outputs for only 19 clips. Comparisons with learned temporal classification and segmentation models would better establish the necessity of the proposed architecture.
  4. Incomplete transfer evaluation and limited reproducibility. The external evaluations omit the SFT baseline trained on the same data, making it unclear whether the improvements arise from geometric distillation or task-specific training. Furthermore, restrictions on the approximately 43,000 Tencent Video training clips limit independent reproduction and verification.
  5. Insufficient validation of the claimed geometric representation. Improvements in recognition scores and a decline in training-set distillation loss do not, by themselves, prove that the student model has truly learned camera geometry. The authors need to verify whether the camera tokens can recover camera poses from unseen videos, or effectively distinguish between camera motion and object motion.
  6. The authors analyze performance degradation on complex samples, yet they fail to adequately demonstrate whether the improvements over SFT are specifically concentrated on these challenging cases. The overall score gains do not clarify whether the method truly addresses the issues highlighted in the paper, namely, the simultaneous occurrence of multiple motions and variations in motion within a single shot.
  7. Heavy reliance on a self-curated training dataset. The method relies on approximately 43,000 specially annotated video clips, but the authors do not clearly commit to releasing the complete training data and annotations. This limits independent reproduction and makes it difficult to assess whether the reported gains hold with publicly available training data.

[1] Geometry-Guided Camera Motion Understanding in VideoLLMs. CVPR 2026 workshop. [2] Cambrian-P: Pose-Grounded Video Understanding. arXiv, 2026. [3] 3D-Aware Vision-Language Models Fine-Tuning with Geometric Distillation. EMNLP 2025 findings.

Questions for authors

Please see the weaknesses.

Reviewer_o4n6OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work has two major contributions: (i) CamChoreo, a dataset of videos with multi-label, segment-level camera motion annotations; (ii) CamDistill, a distillation framework that distills camera features from 3D foundation models instead of keeping 3D foundation models around.

The authors show how their dataset provides richer annotations than past works, and how their distillation approach leads to lower latency and compute with competitive performance.

Critical strengths
  • The proposed dataset does provide temporally rich annotations in the same clip
  • The distillation approach does indeed lead to reduced latency and peak memory, while offering competitive performance compared to SFT and CamInject
Critical weaknesses
  • As per Figure 2 (E), it seems that the majority of the samples in the proposed dataset still have only one sample per video. Even summed up, the number of samples with > 1 label does not exceed 1 label per video. And yet, the model is evaluated on CamChoreo and on other benchmarks, which also have only 1 label per video. Hence, the strength of the contribution of the dataset is a bit diluted.
Questions for authors
  1. Can the authors offer a stratified evaluation of their approach on the videos that have > 1 label per segment? Even if there is not sufficient gain on multi-label segments, it would offer insight into where the current approach stands on multi-label temporal labels.
  2. A follow-up to this question is whether the authors tried a chunk-wise baseline of models trained on single-label datasets that have multiple labels across temporal segments? This would offer insights into how much the geometric features help with temporally varying labels.
  3. Lastly, can the authors offer more details on how they measured peak memory usage? The difference between CamInject and CamDistill is a bit unintuitive, as it is quite low.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.