acceptodds
Under review as a conference paper at ICLR 2027

Intent-in-the-Loop: Explicit Cinematographic Intent for 3D Dance Camera Synthesis

Abstract

Synthesizing 3D camera trajectories from music and dance requires more than generating plausible camera motion; it requires translating high-level cinematographic intent into precise and dynamically expressive camera movements. Existing methods have made substantial progress by directly mapping music and dance to camera trajectories. However, they leave cinematographic intent implicit and provide no explicit mechanism for verifying whether the generated trajectory reflects that intent. We introduce Intent-in-the-Loop, a closed-loop framework that makes cinematographic intent explicit and verifiable through a plan–generate–align process. First, in plan, we plan shot-level cinematographic intent from music and 3D dance motion. Second, in generate, we generate a camera trajectory in a coarse-to-fine manner conditioned on the planned intent. Third, in align, we recover intent from the output trajectory and align it with its planned intent during training. This loop improves consistency between planned intent and generated trajectories. Extensive experiments on the DCM dataset demonstrate state-of-the-art camera trajectory quality, substantially smoother motion dynamics, and improved GT-intent fidelity, while providing explicit shot-level interpretability. Our source code will be publicly available upon publication.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Camera Trajectory Generation, Virtual Cinematography, Cinematographic Intent, Intent Alignment, Conditional Diffusion Models, Music-and-Dance Conditioning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_juskOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces Intent-in-the-Loop (IITL) for synthesizing 3D camera trajectories from music and dance. A planner predicts shot-level cinematographic intent, then generates a camera trajectory in a coarse-to-fine manner, and a pretrained, frozen aligner provides intent supervision during training. Experiments on DCM evaluate trajectory quality, intent fidelity, and user preferences, with ablations of the alignment losses and model components.

Critical strengths
  • Well-motivated problem formulation and method. Explicitly representing framing scale, visual focus, motion primitives, and motion energy makes camera trajectory generation inspectable, while recovering the attributes from generated trajectories provides a training objective for intent consistency. This formulation addresses a relevant limitation of direct trajectory synthesis and offers a coherent connection between high-level shot planning and camera motion generation.

  • Evidence of practical effectiveness. On DCM, IITL achieves the lowest kinetic and shot-feature FIDs among the compared methods. The experiment results support the method's practical value through improvements in trajectory distributions, perceived quality, and dancer visibility.

Critical weaknesses
  1. The construction of intent labels is insufficiently specified. Section 4.1 and Appendix A.1 describe fixed geometry- and motion-based rules. However, they do not fully specify the rules and thresholds for framing scale, visual focus, and motion primitives, or the computation and normalization of motion energy. A complete specification is central to understanding and reproducing the method.

  2. Limited methodological novelty. Several central mechanisms have close precedents, the contribution is mainly a shot-intent planning and alignment pipeline, with task-specific coarse trajectory construction and residual gating. I believe the novelty does not fit the standards of ICLR.

Questions for authors
  1. Clarification of intent-label construction. Could the authors provide the rules QQ, including the measured quantities, thresholds, normalization, and aggregation within shots for each intent attribute? Please identify which rules are inherited unchanged from the cited methods and which are introduced or modified here.
  2. Clarification of the baseline configurations. How were the Table 2 backbones adapted to the intent tasks? The user study for TemMEGA seems to be missing, making it hard to evaluate proposed method gain over this baseline. The details are needed to attribute the observed differences to the proposed components.
1 comment

Thank you for the constructive feedback.

(Weakness 1) Reproducibility of Intent Labels. For the fixed rules Q, we adopted the numerical settings from the studies cited below without modification. The fixed rules Q use camera discontinuities at shot starts to identify hard cuts, drawing on the distinction between continuous shots and abrupt switches in (Wang et al., 2024b). Scale and focus use camera–dancer geometry, informed by prior work on shot framing and joint visibility (Jiang et al., 2024 and Wang et al., 2024a). Motion primitives and normalized energy are derived from within-shot camera motion, drawing on geometric movement and velocity descriptors (Jiang et al., 2024). To support reproducibility, we will release the implementation of Q and the trajectory refinement code upon publication.

  • (Jiang et al. 2024). Cinematographic Camera Diffusion Model. Computer Graphics Forum.
  • (Wang et al. 2024a). DanceCamera3D: 3D Camera Movement Synthesis with Music and Dance. CVPR.
  • (Wang et al. 2024b). DanceCamAnimator: Keyframe-based Controllable 3D Dance Camera Synthesis. ACM Multimedia.

(Weakness 2) Methodological Contribution. To our knowledge, IITL is the first framework for music and dance driven 3D camera synthesis to use explicit shot-level cinematographic intent both as a generation condition and as a training target for intent recovered from the generated trajectory. This closes the loop by training the generator to realize the same intent it was given. The principle can extend to other conditional generation tasks where the requested attributes can be recovered from the output and used as a training signal. Our quantitative and qualitative evaluations support the effectiveness of this closed-loop formulation in translating explicit cinematographic intent into camera motion.

(Question 1) Intent-Label Construction. Please see weakness 1.

(Question 2) Baseline Configurations and TemMEGA. Table 2 evaluates intent planning and recovery separately from end-to-end camera synthesis. In Table 2(a), we preserve the input modalities listed in the table. We only attach new frame-level intent prediction heads to the output features of each adapted backbone. The heads use the same design across baselines and predict shot boundaries, framing scale, visual focus, camera motion primitives, and camera motion energy. AtomicDance also retains a token output for auxiliary DCM intent-token denoising, while the UniMuMo adaptation reimplements its joint attention for continuous music and dance features. We train each adapted backbone jointly with its new heads on DCM intent labels. In Table 2(b), ASFormer and MS-TCN++ take camera parameters and dancer-relative composition features as input. We replace their action segmentation outputs at each stage with frame-level intent heads, train the backbones and heads jointly on DCM, dataset and evaluate the final-stage predictions. These comparisons assess the adapted backbones on the two intent tasks; they do not isolate the effect of any single IITL component on camera synthesis.

Table 1 reports the published offline TemMEGA scores (Xu et al., 2025, Table 2), obtained using the same DCM dataset and quantitative evaluation protocol. TemMEGA's code was unavailable at the time of our study, so we could not generate trajectories for the user study.

  • (Xu et al. 2025). Smooth and Flexible Camera Movement Synthesis via Temporal Masked Generative Modeling. NeurIPS.
Reviewer_oaGPOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses an important and relatively underexplored problem in 3D dance camera synthesis by explicitly modeling cinematographic intent through a Plan–Generate–Align framework. Experiments on the DCM dataset, including quantitative comparisons, user studies, and ablation analyses, demonstrate improvements in camera generation quality and intent consistency.

Critical strengths

The paper addresses a meaningful limitation of existing dance-camera synthesis methods by explicitly modeling cinematographic intent rather than directly predicting camera trajectories from music and dance motion. The proposed Plan–Generate–Align framework provides a structured and interpretable way to incorporate shot-level decisions into camera generation. This is a valuable contribution to controllable 3D camera synthesis, even though the individual architectural components build on established techniques. The supplementary video also provides convincing qualitative evidence of improved framing and reduced unnatural camera movements.

Critical weaknesses

Explicit cinematographic intent modeling offers some interesting insights, but the main technical components are based on existing methods, and the work provides relatively limited novel insights with broader methodological applicability.

  1. The intent planner shows relatively weak prediction performance in Table 2(a), particularly for shot-boundary detection (F1 = 0.1663), framing scale (F1 = 0.3364), and visual focus (F1 = 0.3821). Moreover, the proposed planner does not consistently outperform existing methods across all intent prediction metrics. Since explicit cinematographic intent planning is a central component of the proposed framework, these results raise concerns about the reliability of the predicted intent. Although the final camera generation results are promising, it remains unclear how much of the improvement comes from accurate intent planning rather than the subsequent diffusion-based refinement and additional trajectory supervision. An oracle-intent ablation would help clarify the contribution of intent prediction to the overall performance.

  2. There is a potential distribution mismatch between the training and subsequent use of the closed-loop Intent Aligner. As described in Section 3.5, the Aligner is pretrained on ground-truth camera trajectories and then frozen. During generator training, however, generated camera trajectories are fed into this frozen Aligner, introducing a distribution shift from real to generated trajectories. If the Aligner produces inaccurate predictions on unrealistic or out-of-distribution trajectories, the Generator may exploit these prediction errors to minimize the intent-alignment loss without genuinely improving cinematographic quality. This raises concerns about the reliability of the proposed closed-loop mechanism and whether the alignment loss effectively enforces the intended cinematographic constraints.

Questions for authors

How well does the proposed method generalize beyond the DCM dataset? Can the proposed cinematographic intent representation transfer to unseen dance styles, multi-character scenes, or other camera trajectory generation tasks?

1 comment

Thank you for the constructive feedback.

(Weakness 1) Methodological Contribution. To our knowledge, IITL is the first framework for music and dance driven 3D camera synthesis to use explicit shot-level cinematographic intent both as a generation condition and as a training target for intent recovered from the generated trajectory. This closes the loop by training the generator to realize the same intent it was given. The principle can extend to other conditional generation tasks where the requested attributes can be recovered from the output and used as a training signal. Our quantitative and qualitative evaluations support the effectiveness of this closed-loop formulation in translating explicit cinematographic intent into camera motion.

(Weakness 2) Planner Accuracy and Oracle Intent. Table 2(a) evaluates shot boundaries against a single reference edit using a strict tolerance of ±3 frames (0.1 s at 30 fps). In the intent recovery setting of Table 2(b), the camera trajectory is available, and hard cuts can produce observable discontinuities. The planner predicts shot boundaries from music and dance alone, and the same inputs can support multiple plausible edits with different cut timings. We therefore interpret boundary F1 relative to DanceMVP, the baseline evaluated under the same protocol. IITL achieves higher boundary F1 under the strict criterion, and its score is largely improved when the tolerance is widened to ±6 frames (0.2 s at 30 fps). Across the four intra-shot metrics reported by all five planners, IITL achieves the best mean rank (1.75). Table R2a shows the per-metric ranks, with 1 indicating the best result.

Table R2a. Intent planner scores and ranks for Table 2(a).

MethodB-F1 ↑Scale F1 ↑Focus F1 ↑Prim. mAP ↑Energy MAE ↓Mean rank ↓
AtomicDance—0.2617 (5)0.2416 (5)0.4509 (5)0.1693 (4)4.75
MotionBERT—0.2886 (4)0.2990 (4)0.4646 (4)0.1708 (5)4.25
DanceMVP0.0478 (2)0.3000 (3)0.4064 (2)0.5800 (1)0.1692 (3)2.25
UniMuMo—0.3213 (2)0.4140 (1)0.5618 (3)0.1681 (2)2.00
IITL0.1663 (1)0.3364 (1)0.3821 (3)0.5702 (2)0.1679 (1)1.75

Direct synthesis takes music and dance as inputs, bypasses explicit intent conditioning, and uses fallback control points. Open-loop uses planned intent and keyframes. Both retain IITL's residual diffusion backbone and keyframe refiner. Both omit the alignment loss and retain the remaining losses, weights, and training settings of IITL. Its improvement in trajectory quality in Table 3 shows the benefit of these planning signals with the diffusion and refinement architecture held fixed.

We evaluated the same trained IITL generator with planned and ground truth intent. In each condition, the supplied intent serves as the evaluation reference (Table R2b).

Table R2b. Intent agreement under planned and oracle conditioning.

Supplied intentB-F1 ↑Scale F1 ↑Focus F1 ↑Prim. mAP ↑Energy MAE ↓
Planned intent0.6680.6870.7290.6540.345
Ground truth intent0.7730.6710.7160.6580.346

(Weakness 3) Distribution Shift in the Intent Aligner. To assess the impact of the potential distribution shift from ground-truth to generated camera trajectories, we evaluate the same Intent aligner on both types of trajectories on DCM dataset. For each trajectory, we compare the aligner’s predictions with intent labels extracted from that same trajectory using the fixed rules Q.

Table R2c. Intent recovery by the frozen Intent Aligner on DCM.

Trajectory given to the AlignerEvaluation referenceScale F1 ↑Focus F1 ↑Prim. mAP ↑Energy MAE ↓
Ground-truth camera trajectoryground truth trajectory0.4920.6640.6310.155
IITL generated trajectorygenerated trajectory0.5180.6520.6380.153

The aligner achieves comparable agreement across all four metrics, suggesting that the potential distribution shift has little effect on measured intent recovery in this evaluation.

(Question 1) Generalization Beyond DCM. We report quantitative results on DCM, the only benchmark available for this study with synchronized music, dance motion, and reference camera trajectories. The supplementary videos show camera trajectories generated by the DCM-trained IITL model for FineDance sequences, including music and dance styles unseen during training. These sequences lack paired reference camera trajectories, so we use them for qualitative evaluation.

The existing dance-camera synthesis methods are all designed for single-dancer scenes, and dedicated methods for multi-character scenes remain limited. We plan to extend IITL to multi-character scenes by incorporating shot-level target selection and group framing into its intent representation.

Reviewer_oUDsOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes an intent-guided framework for generating 3D camera trajectories from music and dance. It plans camera shots, generates and refines trajectories, and uses an intent-recovery network during training to encourage consistency with the plan. Experiments on DCM and a user study show improved framing quality and preference over two existing baselines.

Critical strengths
  • Modeling shot scale, focus, and camera motion makes the generation process easier to understand. The intent-recovery network provides a useful training signal to encourage trajectories to follow the planned camera behavior.
  • The method improves shot FID and is preferred over DanceCamera3D and DanceCamAnimator baselines. The ablations also show that intent alignment improves dancer visibility.
  • The supplementary video contains comparison results with the baselines as well as qualitative results without ground truth trajectories.
Critical weaknesses
  • The experiments measure agreement with the reference camera’s intent, rather than the model’s own plan. Direct plan-adherence results (with & without alignment) are needed to establish whether the proposed loop achieves its intended purpose.
  • Alignment provides only small FID gains, without reported run-to-run variability, and the direct-synthesis ablation is insufficiently described. - The smoothness evaluation also excludes only reference cuts, so generated cuts at other times may be counted as within-shot jerk.
  • Some important implementation details remain unclear, including the intent-extraction thresholds and constraints on interpolation weights and residual gates. These omissions affect reproducibility and make the claimed behavior of the refinement stage difficult to assess.
  • The smoothness evaluation excludes only reference shot boundaries. Generated cuts at other times may therefore be counted as within-shot jerk, mixing differences in shot timing with actual motion instability.
Questions for authors
  • Can the authors report agreement between the planned attributes and attributes extracted from generated trajectories using the fixed rules for both full IITL and the model without alignment? An evaluation using ground-truth intent as the supplied plan would also help distinguish planner errors from execution failures.
  • What exactly is removed or replaced in the direct-synthesis variant in Table 3? Does it retain the same residual-diffusion backbone, keyframe refinement, training losses, and comparable capacity? Can the authors provide repeated-run variability for the full and open-loop models?
  • Do the dynamics improvements persist when generated hard cuts are excluded as well, while cut accuracy is evaluated separately? This would help determine whether Table 5 primarily reflects smoother continuous motion or better agreement with reference cut locations.
  • Please specify the intent-extraction rules and thresholds. Also, are the delta and gamma parameters constrained after the linear projection in Eq. 32?
1 comment

Thank you for the constructive feedback.

(Weakness 1) Planned Intent Agreement. Using DCM dataset, we compare intent from the generated trajectories with generator’s input intent plan, rather than ground truth intent. IITL outperforms the open-loop variant on all five metrics (Table R1a). These results indicate that the closed-loop generator produces camera trajectories that more faithfully reflect the planned cinematographic intent.

Table R1a. Planned intent agreement on DCM.

MethodB-F1 ↑Scale F1 ↑Focus F1 ↑Prim. mAP ↑Energy MAE ↓
Explicit intent conditioning (open-loop)0.5960.5860.6220.5820.347
IITL (closed-loop)0.6680.6870.7290.6540.345

(Weakness 2) FID Variability, Direct Synthesis, and Within-Shot Dynamics. We did independent runs across 30 different random seeds, IITL achieved FIDₖ = 3.181 ± 0.042 (mean ± SD), and open-loop achieved 3.204 ± 0.048.

Direct synthesis takes music and dance alone and uses fallback control points. Open-loop uses explicit intent conditioning and planned keyframes. Both share IITL’s residual diffusion backbone, keyframe refiner, and training settings. Both omit the intent alignment loss and retain all other loss terms and weights. With the backbone and refiner held fixed, intent conditioning and planned keyframes improve trajectory quality in Table 3. Generated hard cuts outside the original ground-truth exclusion regions can contribute to jerk. 0We recomputed Table 5 with a shared mask that also excludes neighborhoods around detected generated hard cuts, retaining the ground-truth distance scale for NTJ. Among synthesized methods, IITL still has the lowest NTJ and AJ, and DanceCamera3D has the lowest ZoomJ and OscRMS (Table R1c). Table 4 separately reports overall shot-boundary F1 with a ±3-frame tolerance.

Table R1c. Within-shot jerk and oscillation on DCM with generated hard cuts also excluded.

MethodNTJ ↓AJ ↓ZoomJ ↓OscRMS ↓
Ground truth77.01821.8191.2170.352
DanceCamera3D1903.3641147.510281.8350.308
DanceCamAnimator8107.8964407.1511629.9460.494
IITL (Ours)485.341282.852757.9190.331

(Weakness 3) Fixed rules Q and Eq. (32). As described in Appendix A.1, we use a threshold of 0.5 to decode the intent planner’s sigmoid outputs. For the fixed rules Q, we adopted the numerical settings from the studies cited below without modification. The fixed rules Q use camera discontinuities at shot starts to identify hard cuts, drawing on the distinction between continuous shots and abrupt switches in (Wang et al., 2024b). Scale and focus use camera–dancer geometry, informed by prior work on shot framing and joint visibility (Jiang et al., 2024 and Wang et al., 2024a). Motion primitives and normalized energy are derived from within-shot camera motion, drawing on geometric movement and velocity descriptors (Jiang et al., 2024).

After linear projection, we apply softplus to obtain positive framewise increments and normalize their cumulative sum within each keyframe interval. This gives a nondecreasing interpolation weight in [0, 1] that is shared across all camera parameter channels. We use a scaled sigmoid to constrain each component of the residual gate γⱼ to (0, 0.35). We will revise Eqs. (14) and (32) to reflect the shared scalar weight and the post-projection constraints. To support reproducibility, we will release the implementation of Q and the trajectory refinement code upon publication.

  • (Jiang et al. 2024). Cinematographic Camera Diffusion Model. Computer Graphics Forum.
  • (Wang et al. 2024a). DanceCamera3D: 3D Camera Movement Synthesis with Music and Dance. CVPR.
  • (Wang et al. 2024b). DanceCamAnimator: Keyframe-based Controllable 3D Dance Camera Synthesis. ACM Multimedia.

(Weakness 4). Within-Shot Dynamics. Please see Weakness 2.

(Question 1). Please see Weakness 1.

We also supplied ground truth intent to the same trained IITL generator at inference (Table R1b). B-F1 is 0.773 under ground truth intent conditioning and 0.668 under planned intent conditioning. The four intra-shot metrics differ by at most 0.016.

Table R1b. Intent agreement with conditioned intent.

Conditioned intentB-F1 ↑Scale F1 ↑Focus F1 ↑Prim. mAP ↑Energy MAE ↓
Planned intent0.6680.6870.7290.6540.345
Ground truth-derived intent0.7730.6710.7160.6580.346

(Question 2). Please see Weakness 2.

(Question 3). Please see Weakness 2 for the within-shot dynamics results.

(Question 4). Please see Weakness 3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.