acceptodds
Desk rejected at ICLR 2027

TCMU: Temporal Camera Motion Understanding

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Camera motion is temporal and compositional: multiple operations can overlap, change speed, and transition within a single shot, whereas shot-level labels leave this structure unspecified. We introduce Temporal Camera Motion Understanding (TCMU), a task that predicts a sequence of temporal segments specifying the types, directions, and speeds of all active camera motions. To support training and evaluation, we construct approximately 292K human-annotated video clips and TCMU-Bench, an independent benchmark of 1,000 shots with 2,590 manually annotated segments. We develop two approaches. TCMU-Geo derives motion timelines from reconstructed geometry by separating depth-independent and depth-dependent components of camera-induced image motion. TCMU-VLM learns to generate these timelines directly from video using the annotated training dataset. On TCMU-Bench, TCMU-Geo achieves 60.7 frame F1, compared with 32.8 for the pose-based SpatialVID baseline. TCMU-VLM achieves 71.5 frame F1, compared with 25.4 for the strongest evaluated zero-shot VLM, as well as 70.0 Event-F1 at temporal IoU 0.5 and 84.9 Boundary-F1 at a 0.5-second tolerance. TCMU-VLM is stronger overall in recognition and temporal localization, while TCMU-Geo performs better on parallax-based motion discrimination and directed-motion recall under dense concurrency. Paired event analysis reveals complementary motion intervals recovered by only one method. As an additional application, geometry-assisted human review of training annotations improves VLM recognition and event localization, although boundary-localization accuracy decreases. We will release TCMU-Bench, the evaluation code, the distilled TCMU-VLM-2B checkpoint, and the complete TCMU-Geo code.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Camera Motion Understanding, Temporal Video Understanding, Vision-Language Models, Geometric Computer Vision, Temporal Localization, Video Benchmark

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_VCT9OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper defines Temporal Camera Motion Understanding, which replaces clip-level labels with temporal segments giving the type, direction, and speed of all concurrent camera motions. It contributes approximately 292K annotated training clips and TCMU-Bench (1,000 shots, 2,590 segments), and develops two approaches, a training-free geometric pipeline (TCMU-Geo) and a fine-tuned Qwen3-VL-8B model distilled to 2B (TCMU-VLM). On TCMU-Bench, TCMU-Geo reaches 60.7 frame F1 against 32.8 for SpatialVID, and TCMU-VLM reaches 71.5 against 25.4 for the strongest zero-shot VLM.

Critical strengths
  • Supervised timeline prediction is accurate and compresses to a small checkpoint. The annotation-trained 8B model reaches 71.5 frame F1 and 70.0 Event-F1 at tIoU 0.5 on TCMU-Bench against 25.4 frame F1 for the strongest zero-shot VLM, and a distilled 2B student reaches 67.0 frame F1. This matters because it establishes that the fine-grained temporal task is learnable with standard supervision and that competitive performance is achievable at 2B scale.

  • The evaluation separates recognition from temporal localization and exposes concrete failure modes. Frame recognition, per-label event localization, and state-boundary localization are scored separately on the same 1,000 shots, with easy, hard, and challenging subsets and condition-level families for parallax discrimination, reconstruction difficulty, and structural complexity. These breakdowns yield concrete diagnoses, such as Geo's 11.5 hold-event recall against the VLM's 80.2, instead of a single aggregate ranking. This matters because the protocol lets TCMU-Bench measure future systems on distinct capabilities rather than one leaderboard score.

Critical weaknesses
  • Task-level novelty undermined by the uncited work CamChoreo. The recent work CamChoreo (arXiv:2608.10932), public since August 11, 2026, formulates nearly the same task, partitioning a shot into motion-consistent intervals and predicting the complete set of concurrent movements in each interval, with an attribute set that includes direction, zero/slow/medium/fast speed tiers, and Static and Unstable states, and a taxonomy that contains all nine TCMU motion types plus Arc, Follow, and Focus Shift. Its benchmark contains 4,229 clips and 8,591 segments, larger than TCMU-Bench, and its evaluation computes micro and macro F1 on a 0.1 s grid with tIoU-based interval metrics, the same protocol family as TCMU's frame, event, and boundary metrics. Both papers use VGGT-Ω to supply geometric evidence and fine-tune Qwen3-VL, with CamChoreo training CamInject and CamDistill on 43,438 clips, while the paper develops TCMU-Geo and TCMU-VLM. The work predates the submission deadline, and the paper neither cites nor discusses it, which undermines the premise that joint interval, concurrency, and speed representation is an open problem. After the overlap is removed, the remaining differences, annotation scale at 292K, a training-free geometric readout, and a geometry-assisted review loop, are incremental. For these reasons, the task-level novelty claim of the paper is not established.

  • Unquantified annotation quality and unsupported speed claims. The training annotations for the approximately 292K clips were quality-controlled through 10% random sampling, and the paper states that this rate describes initial review coverage rather than a measured annotation-accuracy rate. No measured annotation accuracy or inter-annotator agreement is reported for the training data. Reviewing annotations for only about 2.3% of the training videos with human-confirmed corrections raised frame F1 by 4.0, and by 14.5 on the Parallax subset, which indicates material residual label noise. Boundary F1 dropped by 4.0 after the same review even though boundary annotations were unchanged, and the paper offers no explanation. Speed is one of the three core attributes of the task, but the fast tier contains only eight segment-level annotations and 236 label frames at 0.1 s, about 0.1% of the 200,146 label-frame instances, and the highest concurrency level contains only five segments. Because the labels underpin every model result, the data contribution remains unsubstantiated, and a core attribute of the task lacks sample support.

  • The complementarity claim lacks cost and statistical support. The paper presents its geometric and learned routes as complementary and evaluates them as one of its two main contributions, but the evidence required for that comparison is incomplete. No inference cost is reported for either method. TCMU-Geo runs a 1B VGGT-Ω model per shot at 10 fps and 512 input resolution, and TCMU-VLM runs an 8B model with up to 100 input frames, so readers cannot judge whether the training-free route is lighter in practice. The headline comparisons report no confidence intervals or significance tests, the original and reviewed versions trade off across metrics (frame F1 +4.0, boundary F1 -4.0), and the release and distillation use the reviewed version while the main comparisons use the original version, so the version that represents the final method is unclear. The boundary metric uses a 0.5 s tolerance and does not require agreement of adjacent labels, and predictions are endpoint-aligned before scoring, so 84.9 measures timestamp proximity rather than the correctness of the state transitions. Without cost and significance evidence, the complementarity claim cannot be turned into an actionable recommendation.

Questions for authors

What is the minimal essential difference between TCMU and CamChoreo in task definition, attributes, and evaluation protocol, and can the authors provide a direct comparison under the same data or vocabulary? What are the measured annotation accuracy and inter-annotator agreement for the training data, what error rate did the 10% checks uncover, and how many benchmark shots were excluded after tied votes? Why does reviewing annotations for about 2.3% of the training videos lower boundary F1 by 4.0, how do corrected directed-motion labels change the complete-state boundary definition, and which version, original or reviewed, should users adopt? Do the fast-tier and five-segment concurrency results survive when those rare samples are excluded?

Please cite and differentiate CamChoreo, including a direct comparison on task, data, and evaluation. Report latency and memory for both methods and add confidence intervals or significance tests for the headline comparisons. Unify and report both the original and reviewed versions and explain the mechanism behind the boundary drop. Report metrics excluding the 23 generated videos and expand the fast tier and rare concurrency combinations.

Reviewer_fHJ8OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Temporal Camera Motion Understanding (TCMU), a task that predicts a complete temporal partition of a single shot into segments describing the types, directions, and speeds of all active camera motions, including concurrent operations. The authors construct a large human-annotated training set and a separately curated benchmark with multi-reviewer quality control, and develop two approaches: TCMU-Geo, which decodes motion timelines from reconstructed geometry by separating depth-independent and depth-dependent components of camera-induced image motion, and TCMU-VLM, which learns the same structured prediction directly from the annotations. Both approaches outperform their respective baselines, and paired analyses reveal complementary strengths across visual conditions and motion concurrency. However, the methods themselves are relatively standard realizations, and the benchmark scale and evaluation conventions leave some open questions.

Critical strengths

S1. A well-defined new task backed by substantial, high-quality human-annotated resources. The formulation goes beyond clip-level label sets and fixed-grid descriptions by requiring a complete temporal partition with concurrent, typed, directed, and speed-annotated motions. The accompanying training corpus and the independently reviewed benchmark, together with the promised release of the benchmark, evaluation code, a distilled checkpoint, and the Geo code, give the paper lasting value independent of the specific models. This resource contribution is the primary basis for my positive assessment.

S2. The two complementary approaches and their comparative analysis yield genuine insight. Pairing a geometry-driven readout with a learned VLM on a common output space lets the authors show where each representation wins: the geometric method is better under parallax discrimination and dense concurrency, while the VLM is stronger overall and on temporal localization. The subset, concurrency, and paired event-recovery analyses, plus an external check on an existing camera-motion benchmark, make the evaluation more informative than a single leaderboard.

Critical weaknesses

W1. The methods are straightforward realizations relative to the task contribution. TCMU-Geo is a modern assembly of classical parametric motion-field readout on top of a frozen geometry estimator, and TCMU-VLM is standard supervised fine-tuning with a JSON output schema. The paper's advance is the task, data, and benchmark rather than algorithmic novelty, which limits its conceptual contribution and possibly its fit for an audience expecting new learning methods.

W2. Scale and quality caveats temper the strength of the evidence. The benchmark is modest in size, part of the Challenging subset consists of AI-generated videos, quality control on the large training set is based on sampled inspection, and several metric conventions (evaluation after shot-endpoint alignment, a fixed frame grid that can miss short events, boundary matching that ignores label agreement) can flatter or distort the reported scores. These factors make it hard to judge how the results would transfer to noisier, real-world settings.

Questions for authors
  1. Training-set label quality. Quality control on the training corpus uses sampled inspection. What is the estimated label noise rate, and how does it bound the VLM's achievable performance? Did you measure agreement between annotators on the training set itself?
  2. Domain coverage. The training footage is curated film and television content. How do both methods perform on other domains such as egocentric, surveillance, or user-generated video? Is there any cross-domain evaluation?
  3. AI-generated evaluation clips. Part of the Challenging subset is generated by a video model. How were these clips validated as representative and correctly annotated, and do the Challenging-subset conclusions hold without them?
  4. Shot-boundary dependence. The task assumes continuous shots with an external shot detector, and reported numbers are measured after shot-endpoint alignment. How sensitive is the end-to-end pipeline to shot-boundary errors, and can you report raw endpoint prediction accuracy?
  5. Exclusion procedure. Shots with tied reviewer votes on disputed labels are excluded. How many shots were excluded this way, and could this bias the benchmark toward easier cases?
  6. Release scope. You promise the benchmark, evaluation code, the distilled checkpoint, and the Geo code. Will the large human-annotated training set also be released, and under what license?
  7. Speed annotation reliability. Strict-speed matching lowers scores notably for both methods. How reliable are the speed-tier annotations themselves (e.g., inter-annotator agreement on speed), and should speed be evaluated separately from type and direction in the headline metrics?
Reviewer_yMvpOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Temporal Camera Motion Understanding (TCMU), a task that goes beyond shot-level camera motion recognition by requiring temporally localized predictions of motion types, directions, and speeds, including multiple camera operations occurring simultaneously. To support this task, the authors construct a large-scale training dataset with approximately 292K human-annotated video clips and TCMU-Bench, an evaluation benchmark containing 1,000 shots and 2,590 annotated segments.

The paper presents two approaches. TCMU-Geo uses reconstructed camera geometry and motion-field decomposition to derive motion timelines, while TCMU-VLM learns to generate structured predictions directly from videos. Both approaches outperform their respective baselines. The authors also investigate their complementary strengths and use geometric predictions to assist human annotation review.

Critical strengths
  • The paper addresses an interesting and meaningful problem. Existing camera motion recognition benchmarks typically focus on shot-level labels, which cannot fully describe transitions, speed changes, or concurrent motions. The proposed temporal formulation provides a more detailed representation of camera behavior and could be useful for downstream video understanding and generation tasks.

  • The dataset and benchmark are valuable contributions. Constructing temporal annotations for camera motion is challenging, especially when multiple operations overlap. The scale of the training dataset is impressive, and TCMU-Bench provides a reasonably comprehensive evaluation protocol covering recognition, event localization, and boundary prediction. The additional breakdowns by motion type, concurrency, and challenging visual conditions are also useful.

  • The geometric approach is well motivated. Separating depth-independent and depth-dependent image motion provides a reasonable way to distinguish camera rotation, translation, and zoom. I particularly appreciate the controlled comparison in Appendix D.2, where the authors use the same reconstruction backend and motion vocabulary as SpatialVID. This makes the improvement of TCMU-Geo more convincing.

  • The experimental analysis is thorough. The VLM achieves strong recognition and temporal localization results, while the geometry-based approach shows advantages for certain motion types and concurrent movements. Rather than simply claiming that one approach is better, the authors analyze their complementary behavior. The geometry-assisted annotation review is also an interesting application, although its effect on boundary accuracy is mixed.

Critical weaknesses
  • The main evaluation metrics do not fully capture the structured prediction task. TCMU requires each segment to contain motion types, directions, and speeds, but the primary frame-level and event-level metrics do not consider speed correctness. Although strict-speed evaluation is provided in the appendix, the performance drop suggests that speed prediction remains challenging. In addition, Boundary-F1 only evaluates the timing of state changes without requiring the predicted motion labels on either side to be correct. A joint segment-level metric that considers temporal overlap and the complete motion attributes would provide a more comprehensive assessment of the proposed task.

  • The comparison for TCMU-VLM could be strengthened. The improvement over zero-shot VLMs is substantial, but the proposed model is fine-tuned on approximately 292K annotated clips, whereas the main VLM baselines do not receive comparable task-specific supervision. This makes it difficult to determine how much of the gain comes from the large-scale training data versus the structured prediction formulation. A simpler supervised baseline trained on the same annotations, such as frame-level multi-label prediction followed by temporal segmentation, would help clarify the benefit of the proposed approach.

  • The training data and reproducibility could be discussed more carefully. The large-scale human-annotated dataset is central to the paper, but the current description provides limited quantitative evidence of annotation consistency, particularly for motion speed and temporal boundaries, which can be subjective. The authors describe a quality-control process, but reporting annotation agreement or measured error rates would increase confidence in the dataset. Moreover, the release plan includes the benchmark, geometric method, and distilled 2B model, but does not explicitly mention the full training annotations or the main 8B checkpoint. This may make it difficult to reproduce the strongest VLM results.

  • Some aspects of the benchmark coverage and generalization remain limited. Although the benchmark includes diverse motion combinations, certain cases are represented by very few examples. In particular, the fast-speed category and highly concurrent movements have limited support. Consequently, the current results may not fully establish robustness to rare or complex camera motion configurations. A more detailed discussion of these limitations, or additional evaluation on such cases, would strengthen the paper.

Questions for authors
  • Could the authors provide a joint segment-level evaluation that requires agreement on motion type, direction, speed, and temporal boundaries? This would help assess whether the predicted timelines are correct as complete structured outputs.

  • Have the authors considered training a simpler temporal segmentation baseline using the same 292K annotations? Such a comparison would help separate the benefits of large-scale supervision from those of the proposed structured VLM prediction approach.

  • Could the authors provide more information about the annotation quality of the training dataset, particularly agreement on speed categories and temporal boundaries? Are there plans to release the training annotations or the fine-tuned 8B model?

  • The geometry-assisted annotation review improves frame and event F1 but reduces boundary F1, even though the original boundary annotations are unchanged. Could the authors further explain this trade-off and whether the revised labels affect the model's segmentation behavior?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.