- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper introduces Cutalio, a framework for music-driven video editing that predicts clip durations and cut positions given a fixed sequence of video clips and a music track. The method formulates editing as constrained boundary-trajectory prediction and combines multimodal masked prediction with differentiable dynamic programming to enforce global constraints and provide planning feedback. It further employs progressive reveal, multi-trajectory sampling, and model-based ranking for test-time scaling. The authors construct a dataset of reference and augmented editing trajectories and introduce relational metrics to evaluate duration allocation and acoustic correspondence. Experiments investigate modality contributions, data augmentation, inference scaling, and conditional completion with known cut positions. The results demonstrate the feasibility of the framework, although its overall performance remains below that of the fine-tuned Qwen3-Omni baseline.
Critical strengths
1. Meaningful task formulation. The paper addresses a practical yet relatively underexplored aspect of music-driven video editing: jointly determining clip durations and cut positions under global constraints. Formulating this problem as boundary-trajectory prediction is well motivated and provides a useful perspective on structured temporal editing.
2. Unified framework for constrained prediction. The integration of multimodal masked prediction with differentiable dynamic programming enables joint boundary prediction and conditional completion around known cuts. The paper also provides systematic analyses of modality contributions, data augmentation, and inference-time scaling, offering useful insights into the proposed framework.
Critical weaknesses
1. Limited methodological novelty and insufficient evidence of architectural advantages. The proposed framework primarily combines established techniques, including masked prediction, differentiable dynamic programming, and progressive reveal. Although their integration is reasonable, the paper does not sufficiently establish the necessity or distinct advantages of the proposed architecture over simpler alternatives. More importantly, Cutalio underperforms the fine-tuned Qwen3-Omni baseline on the overall JointH4 metric, even with substantially increased inference-time computation. This weakens the empirical justification for the proposed method.
2. Insufficient validation of actual editing quality. The proposed evaluation metrics primarily measure correspondence with reference editing trajectories rather than perceptual editing quality. Given that the paper explicitly motivates the task by the existence of multiple valid editing solutions, similarity to a single reference may not reliably reflect rhythmic alignment, temporal coherence, or overall editing preference. The absence of human evaluation or other independent quality assessments makes it difficult to determine whether the proposed method produces genuinely better edits.
Questions for authors
Q1. Architectural necessity. Can the authors provide a more direct comparison between the proposed differentiable planning feedback and simpler alternatives, such as independent boundary prediction followed by constrained DP decoding? How much improvement can be specifically attributed to feedback through differentiable DP?
Q2. Comparison with foundation models. The fine-tuned Qwen3-Omni baseline achieves higher JointH4 than all reported Cutalio configurations. What are the main advantages of the proposed specialized architecture beyond geometric validity? Could the authors provide additional evidence supporting its effectiveness, particularly under more comparable computational budgets?
Q3. Perceptual editing quality. Since multiple valid editing trajectories may exist, how well do the proposed reference-based metrics correlate with human judgments of editing quality? Can the authors provide a user study comparing rhythmic alignment, temporal coherence, and overall preference across the generated edits?
Q4. Evaluation metrics. How do the authors ensure that RelOrder and RelRatio do not unfairly penalize alternative editing trajectories that differ from the reference but remain perceptually valid? Have the proposed metrics been validated against human preferences or established editing-quality criteria?
Q5. Test-time scaling. The improvements from increasing inference-time computation are relatively limited and non-monotonic. For example, D8/K16 achieves nearly the same JointH4 as D4/K4 despite requiring substantially more network predictions. What explains this behavior? Can the authors disentangle the contributions of progressive reveal, multi-trajectory sampling, and model-based ranking?
Q6. Generalization and practical constraints. The experiments use a uniform five-second virtual source-clip duration budget rather than actual source-specific durations. How well does the framework generalize to realistic editing scenarios with heterogeneous clip lengths and more complex constraints?
6 comments
Thank you for highlighting a central research question for TTS: how far additional inference computation can improve quality, and what limits those gains. To elaborate on Q5, we discuss the relationship between training supervision and inference scaling using the depth comparisons and augmentation ablation in Tables 1 and 3 and Appendix D.1 of the original paper.
Additional response to Q5: training augmentation and depth saturation
Holding the sample count K fixed, the original main table gives the following common-valid JointH4 scores (n=282), with P2 at both reveal depths:
| K | D4/P2 | D8/P2 |
|---|---|---|
| 4 | 0.6081 | 0.6026 |
| 16 | 0.6051 | 0.6080 |
The effect of additional reveal depth is small and depends on the sampling budget. The augmentation results help us interpret this saturation.
Independent-start augmentation retains the original reference and adds alternative timing trajectories for the same ordered clips and music, each with its own S0. Evaluation uses the original references. Training matches original-example updates, and checkpoints are selected by baseline validation before TTS. The following results use fixed-support JointH4 on 302 reference-eligible combos:
| Training data | P2 baseline | D4/P2/K4 | D8/P2/K16 |
|---|---|---|---|
| GT only | 0.5896 | 0.5984 | 0.5844 |
| GT + independent start | 0.5809 | 0.6054 | 0.6075 |
Independent-start augmentation gives a slightly lower P2 baseline but higher scores at both TTS budgets. Comparing each model's gain over its own P2 baseline, the augmented model gains 0.0318 more than GT-only at D8/P2/K16, with a paired source-clustered 95% CI of [0.0057, 0.0592] (302 combos, 197 sources; Appendix D.1). GT-only also improves at D4/P2/K4; augmentation strengthens the benefit at the higher tested budget.
Existing augmentation has already improved TTS performance. Our view is that raising the TTS ceiling further requires higher-quality, more diverse training trajectories. In future work, we will focus on providing such supervision for the same footage combination. We will also clarify the connection between augmentation and TTS in the revised paper.
I thank the authors for responding to my questions. However, I still think the presentation of this paper to be unclear for human understanding, and I will maintain my score at this stage.
Thank you for your careful review and recognition of the task and framework.
Q1. Architecture and feedback
DP marginals feed back to joint cut prediction. We trained λ_fb=0 from scratch, matching predictor, DP decoding, data, seed, optimizer, 20 epochs and inference budgets. Both models were valid on 343/343.
Fixed-support JointH4: 302 combos/197 sources; Δ=feedback−no feedback; paired source-clustered 95% bootstrap CIs.
At matched epochs, feedback improves P2 significantly; D4/K4 is not significant. With the best P2-validation no-feedback checkpoint (e17), both CIs include zero.
Training-only duration priors + DP, with/without beat/downbeat rewards, give fixed-support JointH4 0.3675/0.4511, both 343/343 valid. Rules were frozen before evaluation input export, with no evaluation tuning. Against the stronger rule, P2/D4 gain 0.1297/0.1542, 95% CIs [0.1040, 0.1562]/[0.1306, 0.1795], under the same DP.
Q2. Qwen and architectural advantages
Qwen SFT generates timestamps autoregressively without hard path constraints (21/343 invalid). Cutalio predicts boundaries jointly; DP enforces order, anchors and durations (343/343 valid).
Supports: 282/302/322/304. Fixed-support JointH4 scores invalid outputs zero (20 Qwen cases). Qwen leads on common-valid JointH4; Cutalio's acoustic means are higher.
The demo linked in the original paper illustrates our motivation: once footage, narrative order and music are chosen, users or agents need coordinated cut timing and repeated revision. Completion lets them move/add anchors, retain approved cuts and jointly regenerate the rest. Each call uses the current footage, music and anchors, without carrying the editing history.
Interactive demo — anchor editing and cut completion:
cutalio-demo.pages.dev
The 32.09M-parameter planner averages 0.20 s (P2) and 1.61 s (D4/K4) on A100 80GB with cached features, excluding feature extraction.
Q3. Perceptual quality
We randomly sampled 50 combos/50 sources. Three authors voted with model identities/scores hidden and A/B order randomized; one aggregate vote per pair/dimension. Four Qwen outputs were invalid. D4/K4 wins/ties/losses:
Mean per-combo metrics for the rated pairs:
Qwen leads on metrics and overall votes in this subset. Against P2, D4/K4 gets more duration/transition votes; music favors P2. Renders keep main-table cuts, slowing insufficient footage with music at original speed.
Q4. Metric validity
RelOrder compares beat-duration ordering; RelRatio measures adjacent beat-duration ratio error. Cuts preserving these relations can score equally well; perceptually valid edits with different relations can be penalized.
Across 96 pairs, metric differences correlate with votes: RelRatio–duration/transitions Spearman ρ=0.549 [0.396, 0.678]; SeqTrend–music ρ=0.389 [0.223, 0.542] (95% source-clustered CIs; ties included; RelRatio oriented toward lower error). Higher JointH4 matches decisive overall votes in 30/38 Qwen pairs and 32/39 P2 pairs.
Q5. TTS saturation and components
On the common-valid set, P2→D4/K4 raises JointH4 from 0.5829 to 0.6081 (+0.0252). D8/K16 reaches 0.6080 with 205.22 versus 30.61 network predictions (6.7×): gains saturate. RelRatio improves (0.8091→0.7533), while RelOrder falls (0.6961→0.6858).
Progressive reveal conditions later predictions on selected cuts; sampling explores feasible alternatives; self-ranking chooses the final path. Model probability may rank paths differently from reference metrics, potentially causing non-monotonic scores. The D/K sweep measures their combined effect.
Q6. Actual footage constraints
The 5 s virtual budget hides explicit reference-duration cues. DP accepts actual per-shot lengths with minimum rate 1.0×. With unchanged checkpoints/features/sampling, P2 and D4/K4 each rendered 45/50 existing evaluation combos at original speed without footage overrun. Three failures had shots<0.2 s; two had no feasible grid path.
Thank you again for identifying these issues, especially in our presentation. We will clarify the motivation, explain metrics before formulas, and simplify the technical exposition.