- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper formulates camera-motion understanding from clip-level classification to temporally grounded, compositional recognition. The task asks the model to split a video shot into motion-consistent intervals and predict predicts the complete set of direction-aware movements active in each interval.
The paper contributes:
-
CamChoreo. An evaluation benchmark of 4,229 real single-shot YouTube clips with 8,591 expert-annotated segments, using a 20-label taxonomy that includes direction. 57% of clips have multiple segments, and 44.2% of segments have simultaneous movements.
-
An empirical diagnosis. Zero-shot MLLMs and a rule-based baseline on VGGT-Ω poses both perform poorly.
-
CamInject and CamDistill. CamInject, a direct-injection reference baseline, projects camera tokens from a frozen 3D foundation model (VGGT-Ω) into the MLLM before each frame's visual tokens. It works well, but it has to run the 3D model at inference. CamDistill removes this cost. A lightweight Geometry-aware Camera Token Extractor (GCTE) reads intermediate frozen ViT features and predicts one camera token per frame. During training, these tokens are aligned with VGGT-Ω camera tokens by cosine distance, and the 3D model is removed at inference. On Qwen3-VL-4B, both reach 67.5 micro F1. CamDistill adds only about +0.1 s and +1.8 GB at inference, compared with +5.9 s and +4.8 GB for CamInject.
Critical strengths
- Well-motivated task and benchmark. Existing benchmarks either treat camera movement as one attribute among many or assign a single label set to a whole clip. Neither captures motion that changes within a shot. CamChoreo addresses this gap by combining real web video with camera-specific, direction-aware multi-label annotations over variable-length temporal segments.
- Useful diagnostic metrics. Separating SegLoc from SegDet shows that the bottleneck is recognizing the full label set, not finding boundaries.
- A clean, practical method. Using VGGT-Ω as a geometry teacher is a good fit. It supplies exactly the parallax and perspective cues that semantic MLLM encoders lack, and CamInject confirms these features help. Distilling them (CamDistill) only at training time keeps almost all of that gain while removing the 3D model at inference. Model cost, the latency and memory measurements make the efficiency claim concrete.
Critical weaknesses
-
The task is close to multi-label temporal action localization: the timeline is fully divided into intervals, and each interval gets a set of labels from a closed vocabulary. Here, the 20 camera-motion labels play the role of action classes. Methods for those tasks (e.g. ActionFormer [1], DETR-style [2] temporal detectors) predict boundaries and multi-label classes directly with a lightweight head. CamDistill instead fine-tunes the whole LLM on 43k clips to output this fixed label set in a rigid JSON format, so the LLM essentially acts as a decoder for structured prediction. Hand-written threshold rules on VGGT-Ω poses already reach 63.9 [email protected]. A learned head would likely do much better, and at far lower cost than 10s per clip. This matters for large-scale annotation for video generation, one of the paper's stated uses.
-
The main comparison is confounded. Table 2 compares zero-shot general-purpose MLLMs and Cam-Motion-7B tuned on CameraBench(with a different taxonomy and output format) against CamDistill/CamInject (which are trained on 43K clips annotated with CamChoreo's own taxonomy and protocol). The micro-F1 margin therefore mainly reflects in-task supervision. Table 3 shows plain SFT (supervised fine-tuning) on the same data already reaches 62.2 (vs 24.2 zero-shot), leaving a 5.3 margin for CamDistill. The claim that "current MLLMs struggle" holds only in the zero-shot setting. Once fine-tuned on the same data, the same MLLM reaches 62.2 micro F1 (Table 3), and CamDistill and CamInject are themselves built on this SFT.
-
CamChoreo (built from YouTube) is the evaluation-only benchmark and the training data is private (43k Tencent Video clips). Every trained results depends on this private data. However, it is only mentioned after section 5 (Experiments). The paper never explains why it trains on Tencent Video and tests on YouTube.
[1] Zhang, Chen-Lin, Jianxin Wu, and Yin Li. "Actionformer: Localizing moments of actions with transformers." In European conference on computer vision, pp. 492-510. Cham: Springer Nature Switzerland, 2022.
[2] Carion, Nicolas, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. "End-to-end object detection with transformers." In European conference on computer vision, pp. 213-229. Cham: Springer International Publishing, 2020.
Questions for authors
- Could a transformer decoder head do the same job as the LLM decoder? For example, a DETR-style set decoder or a per-frame multi-label segmentation head on GCTE tokens, VGGT-Ω tokens, or frozen ViT features. A billion-parameter language decoder appears excessive for 20 outputs, while a lightweight decoder head with a few million parameters can predict in a single forward pass.
2a. The SFT baseline (4B and 8B) should be included in Table 2. The table should be split into zero-shot and supervised rows. Baselines should be fine-tuned on the same Tencent data (e.g. Qwen-VL, Cam-Motion-7B).
2b. How does a MLLM model (8B) with SFT score on CameraBench and CMVQA? Without these, can the transfer in Table 4 be credited to GCTE rather than to fine-tuning on 43K camera-motion clips?
- Please explain the reason between the training and testing split on two different platforms. Do authors plan to publish the training data ? If not, how could the main results be reproduced ?