acceptodds
Under review as a conference paper at ICLR 2027

LET RHYTHM TELL YOUR STORY

Abstract

In audio-driven video editing, jointly determining ordered video clips' relative durations and positions on a given music timeline is a cross-modal structured prediction problem with multiple valid solutions and global constraints. We formulate this as constrained boundary-trajectory prediction, jointly specifying relative clip durations and timeline positions. Our primary evaluation uses a given start time. We introduce Cutalio, a unified framework for trajectory data construction, modeling, inference, and relational evaluation. Using real edits and independent-start augmentations, we train a multimodal masked model that fuses video, music, and rhythmic evidence. The model jointly predicts unknown boundaries; differentiable dynamic programming computes distributions over feasible paths under configurable outer constraints, including per-clip available source durations, an optional target edit duration, and known cuts, and feeds these distributions back into subsequent predictions. At inference, the framework can generate a full timeline from the given start or complete it around fixed cuts. Progressive reveal, multi-trajectory sampling, and model-based ranking provide test-time scaling. Because absolute cut error against a single reference cannot adequately assess multiple valid arrangements, we introduce pairwise relational metrics for duration ordering, duration ratios, local acoustic changes, and sequence-level acoustic trends. Experiments show that music content and rhythmic cues are complementary, known anchors aid completion, and independent-start augmentation helps primarily under multi-trajectory inference.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Music-Guided Video Editing, Multimodal Temporal Modeling, Masked Prediction, Differentiable Planning, Test-Time Scaling

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_3cnfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces Cutalio, a framework for music-driven video editing that predicts clip durations and cut positions given a fixed sequence of video clips and a music track. The method formulates editing as constrained boundary-trajectory prediction and combines multimodal masked prediction with differentiable dynamic programming to enforce global constraints and provide planning feedback. It further employs progressive reveal, multi-trajectory sampling, and model-based ranking for test-time scaling. The authors construct a dataset of reference and augmented editing trajectories and introduce relational metrics to evaluate duration allocation and acoustic correspondence. Experiments investigate modality contributions, data augmentation, inference scaling, and conditional completion with known cut positions. The results demonstrate the feasibility of the framework, although its overall performance remains below that of the fine-tuned Qwen3-Omni baseline.

Critical strengths

1. Meaningful task formulation. The paper addresses a practical yet relatively underexplored aspect of music-driven video editing: jointly determining clip durations and cut positions under global constraints. Formulating this problem as boundary-trajectory prediction is well motivated and provides a useful perspective on structured temporal editing.

2. Unified framework for constrained prediction. The integration of multimodal masked prediction with differentiable dynamic programming enables joint boundary prediction and conditional completion around known cuts. The paper also provides systematic analyses of modality contributions, data augmentation, and inference-time scaling, offering useful insights into the proposed framework.

Critical weaknesses

1. Limited methodological novelty and insufficient evidence of architectural advantages. The proposed framework primarily combines established techniques, including masked prediction, differentiable dynamic programming, and progressive reveal. Although their integration is reasonable, the paper does not sufficiently establish the necessity or distinct advantages of the proposed architecture over simpler alternatives. More importantly, Cutalio underperforms the fine-tuned Qwen3-Omni baseline on the overall JointH4 metric, even with substantially increased inference-time computation. This weakens the empirical justification for the proposed method.

2. Insufficient validation of actual editing quality. The proposed evaluation metrics primarily measure correspondence with reference editing trajectories rather than perceptual editing quality. Given that the paper explicitly motivates the task by the existence of multiple valid editing solutions, similarity to a single reference may not reliably reflect rhythmic alignment, temporal coherence, or overall editing preference. The absence of human evaluation or other independent quality assessments makes it difficult to determine whether the proposed method produces genuinely better edits.

Questions for authors

Q1. Architectural necessity. Can the authors provide a more direct comparison between the proposed differentiable planning feedback and simpler alternatives, such as independent boundary prediction followed by constrained DP decoding? How much improvement can be specifically attributed to feedback through differentiable DP?

Q2. Comparison with foundation models. The fine-tuned Qwen3-Omni baseline achieves higher JointH4 than all reported Cutalio configurations. What are the main advantages of the proposed specialized architecture beyond geometric validity? Could the authors provide additional evidence supporting its effectiveness, particularly under more comparable computational budgets?

Q3. Perceptual editing quality. Since multiple valid editing trajectories may exist, how well do the proposed reference-based metrics correlate with human judgments of editing quality? Can the authors provide a user study comparing rhythmic alignment, temporal coherence, and overall preference across the generated edits?

Q4. Evaluation metrics. How do the authors ensure that RelOrder and RelRatio do not unfairly penalize alternative editing trajectories that differ from the reference but remain perceptually valid? Have the proposed metrics been validated against human preferences or established editing-quality criteria?

Q5. Test-time scaling. The improvements from increasing inference-time computation are relatively limited and non-monotonic. For example, D8/K16 achieves nearly the same JointH4 as D4/K4 despite requiring substantially more network predictions. What explains this behavior? Can the authors disentangle the contributions of progressive reveal, multi-trajectory sampling, and model-based ranking?

Q6. Generalization and practical constraints. The experiments use a uniform five-second virtual source-clip duration budget rather than actual source-specific durations. How well does the framework generalize to realistic editing scenarios with heterogeneous clip lengths and more complex constraints?

6 comments

Thank you for your careful review and recognition of the task and framework.

Q1. Architecture and feedback

DP marginals feed back to joint cut prediction. We trained λ_fb=0 from scratch, matching predictor, DP decoding, data, seed, optimizer, 20 epochs and inference budgets. Both models were valid on 343/343.

Fixed-support JointH4: 302 combos/197 sources; Δ=feedback−no feedback; paired source-clustered 95% bootstrap CIs.

BudgetFeedback e20No feedback e20Δ [95% CI]
P2/K10.58090.5651+0.0158 [0.0032, 0.0283]
D4/P2/K40.60540.6033+0.0021 [−0.0159, 0.0194]

At matched epochs, feedback improves P2 significantly; D4/K4 is not significant. With the best P2-validation no-feedback checkpoint (e17), both CIs include zero.

Training-only duration priors + DP, with/without beat/downbeat rewards, give fixed-support JointH4 0.3675/0.4511, both 343/343 valid. Rules were frozen before evaluation input export, with no evaluation tuning. Against the stronger rule, P2/D4 gain 0.1297/0.1542, 95% CIs [0.1040, 0.1562]/[0.1306, 0.1795], under the same DP.

Q2. Qwen and architectural advantages

Qwen SFT generates timestamps autoregressively without hard path constraints (21/343 invalid). Cutalio predicts boundaries jointly; DP enforces order, anchors and durations (343/343 valid).

MethodJointH4 common-validJointH4 fixedLocalTrendSeqTrend
Qwen SFT0.62920.58760.25260.3572
Cutalio P20.58290.58090.28150.4299
Cutalio D4/K40.60810.60540.26570.4058

Supports: 282/302/322/304. Fixed-support JointH4 scores invalid outputs zero (20 Qwen cases). Qwen leads on common-valid JointH4; Cutalio's acoustic means are higher.

The demo linked in the original paper illustrates our motivation: once footage, narrative order and music are chosen, users or agents need coordinated cut timing and repeated revision. Completion lets them move/add anchors, retain approved cuts and jointly regenerate the rest. Each call uses the current footage, music and anchors, without carrying the editing history.

Interactive demo — anchor editing and cut completion:

cutalio-demo.pages.dev

The 32.09M-parameter planner averages 0.20 s (P2) and 1.61 s (D4/K4) on A100 80GB with cached features, excluding feature extraction.

Q3. Perceptual quality

We randomly sampled 50 combos/50 sources. Three authors voted with model identities/scores hidden and A/B order randomized; one aggregate vote per pair/dimension. Four Qwen outputs were invalid. D4/K4 wins/ties/losses:

ComparatorPairsMusic fitDuration/transitionsOverall
Qwen4616/6/2412/9/2515/8/23
P25017/12/2125/9/1622/11/17

Mean per-combo metrics for the rated pairs:

nModelRelOrder ↑RelRatio ↓LocalTrend ↑SeqTrend ↑JointH4 ↑
46D4/K40.70510.94620.21680.33610.5995
46Qwen0.76830.62720.28320.41810.6653
50D4/K40.68800.94050.22100.34570.5974
50P20.57001.06030.24150.37940.5245

Qwen leads on metrics and overall votes in this subset. Against P2, D4/K4 gets more duration/transition votes; music favors P2. Renders keep main-table cuts, slowing insufficient footage with music at original speed.

Q4. Metric validity

RelOrder compares beat-duration ordering; RelRatio measures adjacent beat-duration ratio error. Cuts preserving these relations can score equally well; perceptually valid edits with different relations can be penalized.

Across 96 pairs, metric differences correlate with votes: RelRatio–duration/transitions Spearman ρ=0.549 [0.396, 0.678]; SeqTrend–music ρ=0.389 [0.223, 0.542] (95% source-clustered CIs; ties included; RelRatio oriented toward lower error). Higher JointH4 matches decisive overall votes in 30/38 Qwen pairs and 32/39 P2 pairs.

Q5. TTS saturation and components

On the common-valid set, P2→D4/K4 raises JointH4 from 0.5829 to 0.6081 (+0.0252). D8/K16 reaches 0.6080 with 205.22 versus 30.61 network predictions (6.7×): gains saturate. RelRatio improves (0.8091→0.7533), while RelOrder falls (0.6961→0.6858).

Progressive reveal conditions later predictions on selected cuts; sampling explores feasible alternatives; self-ranking chooses the final path. Model probability may rank paths differently from reference metrics, potentially causing non-monotonic scores. The D/K sweep measures their combined effect.

Q6. Actual footage constraints

The 5 s virtual budget hides explicit reference-duration cues. DP accepts actual per-shot lengths with minimum rate 1.0×. With unchanged checkpoints/features/sampling, P2 and D4/K4 each rendered 45/50 existing evaluation combos at original speed without footage overrun. Three failures had shots<0.2 s; two had no feasible grid path.

Thank you again for identifying these issues, especially in our presentation. We will clarify the motivation, explain metrics before formulas, and simplify the technical exposition.

Thank you for highlighting a central research question for TTS: how far additional inference computation can improve quality, and what limits those gains. To elaborate on Q5, we discuss the relationship between training supervision and inference scaling using the depth comparisons and augmentation ablation in Tables 1 and 3 and Appendix D.1 of the original paper.

Additional response to Q5: training augmentation and depth saturation

Holding the sample count K fixed, the original main table gives the following common-valid JointH4 scores (n=282), with P2 at both reveal depths:

KD4/P2D8/P2
40.60810.6026
160.60510.6080

The effect of additional reveal depth is small and depends on the sampling budget. The augmentation results help us interpret this saturation.

Independent-start augmentation retains the original reference and adds alternative timing trajectories for the same ordered clips and music, each with its own S0. Evaluation uses the original references. Training matches original-example updates, and checkpoints are selected by baseline validation before TTS. The following results use fixed-support JointH4 on 302 reference-eligible combos:

Training dataP2 baselineD4/P2/K4D8/P2/K16
GT only0.58960.59840.5844
GT + independent start0.58090.60540.6075

Independent-start augmentation gives a slightly lower P2 baseline but higher scores at both TTS budgets. Comparing each model's gain over its own P2 baseline, the augmented model gains 0.0318 more than GT-only at D8/P2/K16, with a paired source-clustered 95% CI of [0.0057, 0.0592] (302 combos, 197 sources; Appendix D.1). GT-only also improves at D4/P2/K4; augmentation strengthens the benefit at the higher tested budget.

Existing augmentation has already improved TTS performance. Our view is that raising the TTS ceiling further requires higher-quality, more diverse training trajectories. In future work, we will focus on providing such supervision for the same footage combination. We will also clarify the connection between augmentation and TTS in the revised paper.

I thank the authors for responding to my questions. However, I still think the presentation of this paper to be unclear for human understanding, and I will maintain my score at this stage.

Reviewer_DTbbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies music-driven cut planning. The model receives an ordered list of video clips, a full music track, and a start time S0, and predicts the internal cut points and the END time on the audio-token grid. Cutalio is a 32.09M-parameter masked Transformer. It fuses frozen Qwen3-Omni video and audio features with madmom rhythm descriptors and scores (boundary, candidate position) pairs. A soft, differentiable dynamic program computes forward–backward marginals over feasible boundary paths, and these marginals are fed back to a second prediction pass (P2). Test-time scaling combines progressive reveal, K sampled trajectories, and model self-ranking. Training data are 16,341 combos mined automatically from 5,827 public fan-edit sources, augmented with independent start points. Evaluation uses 343 validation combos from 215 held-out sources and four relational metrics (RelOrder, RelRatio, LocalTrend, SeqTrend) with their harmonic mean JointH4. The paper also reports modality, augmentation, and oracle-anchor experiments.

Critical strengths

S1. Structured feasibility is a real advantage over free-form timestamp generation. The DP enforces order, anchor, and duration constraints, giving 100% geometric validity. Fine-tuned Qwen3-Omni reaches 93.9%: 21 of 343 combos fail, and 26 further gaps are shorter than 0.2 s (Table 1 note, App. D.3). Given anchors, completion improves markedly. MAE drops from 1.83 s to 0.69 s and END error from 3.54 s to 1.06 s, better than DP-only (1.42 s) (Table 4). However, this experiment uses n = 32 and an older model.

S2. The evaluation is unusually transparent. The paper gives worked metric examples (App. C), paired source-clustered CIs, separate common-valid and fixed-support accounting, a full sweep (Table 8), and failure cases (Fig. 4B), and it states openly that there is no untouched test set. The relational metrics are more interpretable than absolute cut-time error. Cutalio also has better acoustic correspondence than Qwen3-Omni SFT: LocalTrend 0.2815 vs. 0.2526 and SeqTrend 0.4299 vs. 0.3572 (Table 1).

Critical weaknesses

W1. The headline mechanism, differentiable DP feedback, is never isolated. The final model has no ablation that removes the feedback at matched compute: no λ_fb = 0 variant, no single pass, no stop-gradient through the DP, and no DP used only at decoding. This is so even though p.2 (l.83) lists feedback among the aspects examined. The diagnostic ablations in Table 6 use an older "historical" model, and P2 and TTS initialise the feedback differently (App. B). Meanwhile, on the common-valid set, no Cutalio budget exceeds the fine-tuned Qwen3-Omni baseline on JointH4. Qwen3-Omni SFT scores 0.6292, the best Cutalio budget (D4/P2/K4) 0.6081, and the CI of the paired difference is [−0.0010, 0.0434]. There is also no rule-based floor, such as a beat/downbeat-snapped duration prior decoded with the same DP. Readers therefore cannot tell how much of the 0.58–0.61 JointH4 range comes from learning rather than from the DP and rhythm features. This is "not shown" rather than "shown false", but it is the main contribution of the paper.

W2. The evidence base and metric validity are limited.

  • All results come from the same 343-combo set that is used for checkpoint selection and for choosing inference budgets, with a single training seed. The authors themselves state that there is no untouched test (p.8 l.419). Source-clustered CIs do not remove selection bias from repeated development on this set.
  • JointH4 is author-designed. It compares predictions against a single reference edit, although several good edits can differ, and it has not been validated against human judgement. Its composition drives the conclusions. TTS improves Order/Ratio in 10/10 settings but lowers LocalTrend and SeqTrend in 9/10 (Table 1), which the text describes only as "fluctuate".
  • Whether Cutalio beats Qwen depends on the accounting. The paper does not report Qwen under its fixed-support convention (invalid outputs scored 0). My rough estimate is ≈0.59, which would place Qwen below the best TTS settings (0.6054 / 0.6075 in Table 8). A blind human comparison is needed to decide which accounting reflects editing quality.

Secondary concerns (less decisive than W1–W2):

W3. The experimental setting departs from the motivating problem. The input clips are shots cut from the finished edit, and per-source duration limits are replaced by a uniform 5 s virtual budget (p.7). The "available footage length" constraint emphasised in the motivation is therefore not tested. Appendix B also acknowledges that clip content carries duration cues. Without a video-free or shuffled-video control, it is unclear how much the model uses visual content rather than duration leakage and rhythm.

Questions for authors
  1. Ground-truth definition and leakage. Are combo clips always contiguous segments of the original edit, with the original timing? Appendix A says "not a contiguity restriction". In that case, what is the ground truth? Ground-truth gaps can reach the 10 s / 16-beat cap (App. D.3), so why does the 5 s budget not make the ground truth infeasible? Is the train/validation split deduplicated by song, not only by source? Synthetic ground truth or song overlap would lower my score.
  2. Feedback ablation. Please train a no-feedback control (λ_fb = 0 or stop-gradient) with the same recipe and compute, and report paired CIs at P2 and D4/K4. A significantly positive difference would move me towards acceptance.
  3. Rule-based floor. Please add a rule-based baseline (beat/downbeat-snapped duration prior + the same DP) and report Qwen3-Omni under fixed-support accounting as well.
  4. Human validation. A small blind pairwise study (≈50 combos) would show whether ≈0.02 JointH4 differences, and the "better ratios but worse acoustic correspondence" trade-off of TTS, match real preferences. A second training seed for Table 3 would also help.
  5. Independent evaluation. Can you evaluate a frozen configuration on sources never used during development?

I would raise my score to 3 if Q1 is answered cleanly and Q2/Q4 come out positive. I would lower it if Q1 reveals synthetic ground truth or song-level leakage.

Minor issues

  • p.3 (l.120–121) says cost is reported in forward calls and runtime, but no runtime is reported (Table 1 note: "not latency").
  • p.9 says augmentation helps in "all ten" TTS settings, but GT-only is reported for only two budgets (Tables 3/7).
  • The independent start-point view exists for only ~30% of samples (4,965 / 16,340, App. D.3), and its effect on Cutalio should be discussed.
  • Tables 4 and 6 use the historical model rather than the final one.
  • Errors from shot detection and automatic ground-truth parsing are not audited.
  • Ethics/data note (not a flag): The dataset is mined from public fan edits containing film, anime, and game footage with commercial music, and possibly identifiable dancers. The paper does not redistribute media and includes an ethics statement (p.10). The authors should still state the source platform(s) and the terms-of-service / copyright basis for collecting and using these data.

Overall. The formulation and the transparent evaluation are valuable, but the central methodological contribution is not isolated. All evidence comes from a development set with a single seed and an unvalidated composite metric, and the method does not exceed the single external baseline on the main metric.

3 comments

Thank you for your careful review and suggestions, especially checking Qwen's fixed-support score: we obtain 0.5876. Replies follow your question order.

Q1. GT and leakage

Combos are contiguous original-edit shots, preserving order/timing; labels map boundaries to the candidate grid. Independent-start augmentation is training-only; evaluation uses original references. We will correct "not a contiguity restriction." We split by source video; songs do not repeat across sources, with no train/validation song overlap.

Main-table limits are virtual 5 s, rate≥0.5× (5/0.5=10 s), and ≤16 beats, hiding explicit reference-duration cues. With actual lengths and rate≥1.0× on 50 existing combos, P2 and D4/K4 each rendered 45/50 without footage overrun; three failures had shots<0.2 s and two had no feasible grid path.

Q2. Feedback ablation

We trained λ_fb=0 from scratch with DP decoding, matching data, seed, optimizer, 20 epochs and inference budgets. Both models were valid on 343/343. Fixed-support JointH4 uses 302 combos/197 sources; Δ=feedback−no feedback, with paired source-clustered 95% CIs (2,000 resamples).

BudgetFeedback e20No feedback e20Δ [95% CI]
P2/K10.58090.5651+0.0158 [0.0032, 0.0283]
D4/P2/K40.60540.6033+0.0021 [−0.0159, 0.0194]

Feedback significantly improves P2 at matched epochs; D4/K4 is not significant. Selecting the no-feedback checkpoint by its best P2 validation score (e17) gives CIs including zero at both budgets.

Q3. Rule baselines and fixed-support Qwen

Two rules use training-only median/MAD beat-duration priors + DP, with/without beat/downbeat rewards (weights 1 and 2, width 0.25 beats). Rules/code were frozen before evaluation input export, without evaluation-based tuning. Both use the main-table grid, S0 and DP, yielding 343/343 valid outputs.

MethodJointH4 common-valid (n=282)Fixed-support (n=302)
Duration prior + DP0.44680.4511
Prior + beat/downbeat + DP0.36190.3675
Cutalio P2/K10.58290.5809
Cutalio D4/P2/K40.60810.6054
Qwen SFT0.62920.5876

Against the stronger rule, P2 gains 0.1297 [0.1040, 0.1562] and D4/K4 0.1542 [0.1306, 0.1795] in fixed-support JointH4 (paired source-clustered 95% CIs; 20,000 resamples). Cutalio also beats both rules on all four components.

Of Qwen's 21 invalid outputs, 20 lie in fixed support and score zero; Cutalio is valid throughout. Qwen leads on common-valid JointH4; D4/K4 leads under fixed-support accounting. We will report both and validity rates.

Q4. Human validation

We randomly sampled 50 combos/50 sources. Three authors voted with model identities/scores hidden and randomized A/B order; one aggregate outcome per pair/dimension. Four Qwen outputs were invalid. D4/K4 wins/ties/losses:

ComparatorPairsMusic fitDuration/transitionsOverall
Qwen4616/6/2412/9/2515/8/23
P25017/12/2125/9/1622/11/17

Mean original scores on the same human-evaluation pairs (per-combo scores averaged):

nModelRelOrder ↑RelRatio ↓LocalTrend ↑SeqTrend ↑JointH4 ↑
46D4/K40.70510.94620.21680.33610.5995
46Qwen0.76830.62720.28320.41810.6653
50D4/K40.68800.94050.22100.34570.5974
50P20.57001.06030.24150.37940.5245

Qwen scores higher on this subset and receives more preference votes. Against P2, D4/K4 gets more duration/transition votes; music favors P2. This trade-off will replace "fluctuate."

Across 96 pairs, metric-difference/preference Spearman correlations are RelRatio–duration/transitions ρ=0.549 [0.396, 0.678] and SeqTrend–music ρ=0.389 [0.223, 0.542] (95% source-clustered CIs; ties included; RelRatio oriented toward lower error). Higher JointH4 agrees with decisive overall votes in 30/38 Qwen pairs and 32/39 P2 pairs. For 0.01≤|ΔJointH4|≤0.03, four pairs favor the higher score, seven tie, and one favors the lower score.

Renders retain main-table cuts, slowing insufficient footage while keeping music at original speed.

Q5. Independent evaluation

We will include an independent evaluation of a frozen configuration on sources not used during development in the final version.

Other comments

We will report cached-feature planning time on A100 80GB: P2 0.20 s, D4/K4 1.61 s, excluding feature extraction; limit augmentation claims to the two reported budgets; discuss the ~30% independent-start coverage; and label Tables 4/6 as historical-model results. We will specify source platforms and collection/use terms and copyright basis, without redistributing media.

Thank you for the additional experiments. Q1 is answered clearly (contiguous original shots, source-level split, no song overlap), and the rule-based floor, the fixed-support Qwen score (0.5876) and the actual footage-length test are useful additions.

My conditions for raising were positive results on Q2 and Q4, and these are mixed. Feedback helps at P2 (+0.016) but not at the main D4/P2/K4 budget (+0.002, CI includes zero), and neither difference holds when the no-feedback model uses its best checkpoint. The human study supports the relational metrics, but the raters are the authors, Qwen SFT is preferred over D4/K4 overall (15/8/23), and pairs that differ by 0.01–0.03 JointH4 are mostly ties. I am therefore keeping my rating at 2. Ratings from non-author listeners and an evaluation on untouched sources would help most in a future version.

Thank you for your careful follow-up and for acknowledging the clarifications and additional experiments. We appreciate your clear assessment of the evidence and the remaining concerns.

We are now collecting new sources for an independent evaluation and will freeze the model checkpoints and inference settings before testing. We also plan to obtain A/B ratings from non-author participants. Your comments have helped us prioritize these next steps and clarify how we should present the feedback gains across inference budgets and checkpoint choices. Thank you again for your thoughtful engagement with our work.

Reviewer_oiD2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The authors propose Cutalio, a multimodal modeling framework capable of predicting alignment points between sets of video clips and a musical track. The model uses a differentiable DP mechanism to reason over the joint distribution of possible alignments. They train both small models from scratch and a fine-tune large models, finding their approach to be effective in modeling the task.

Critical strengths
  • Proposed Cutalio framework is interesting and may apply to tasks beyond the video-to-music editing and alignment task proposed here.
  • Experiments show that even very small models can produce effective results under this framework
Critical weaknesses

There are a few notable weaknesses in the work:

Lack of agentic baseline. It feels almost cliche to say, but these days, frontier agents are capable of many tasks, including these types of multimodal editing tasks. It's essential to compare to these baselines, if for no other reason than to calibrate expectations on quantitative performance for a task-specific model. I generally expect small task-specific models to perform worse, but even in that case, an evaluation of inference costs between agents and task-specific approaches can also provide justification for the latter.

Artificially-constrained editing task. The proposed task here feels artificially narrow. Real-world video-music synchronization has far more flexibility: changing clip ordering, time stretching, adding additional scenes (e.g. text chyrons), edits within a scene, jumps and cutpoints for music not just video, etc. The proposed task representation of start time / end times for ordered clips is quite inflexible in a relative sense.

No human evaluation. A human comparison of editing (or even LLM-as-a-judge as a crude backup) is essential for measuring the qualitative performance of the proposed method / baselines.

Minor concerns:

  • Writing is a bit dense and overly technical in some places. Formalisms are appreciated, but also need to convey intuition. E.g., in section 5, give a high-level explanation of each evaluation metric before launching into the technical details.
  • Missing citations to other differentiable algorithms like CTC which optimize over alignments
Questions for authors

N/A (please focus on responding to critical weaknesses)

3 comments

Thank you for your careful review and for recognizing the framework's potential and small-model results. We respond in order.

1. Codex baseline and cost

We added zero-shot gpt-6-astra (medium) in Codex on all 343 evaluation combos. It receives fixed S0, four frames per shot, audio-analysis tools (beat/downbeat, onset, energy, spectrum), and a constraint checker: 300 s and at most three checks per case. It uses the same 5 s virtual footage budget; actual clip durations, FPS, original timestamps and GT are hidden because footage lengths from finished edits can reveal reference timing.

MethodRelOrder ↑RelRatio ↓LocalTrend ↑SeqTrend ↑JointH4 fixed ↑
Codex0.63520.84910.13990.22030.5585
Cutalio P2/K10.65880.97150.28150.42990.5809
Cutalio D4/P2/K40.69610.80910.26570.40580.6054
Qwen SFT0.74050.68130.25260.35720.5876

Components use main-table common-valid sets: Order n=282, Ratio/Seq n=304, Local n=322. Following another reviewer's suggestion, fixed-support JointH4 uses 302 reference-eligible combos and scores invalid outputs zero. Common-valid JointH4 (n=282): 0.5541/0.5829/0.6081/0.6292 in row order. D4/K4 exceeds Codex by 0.0469, paired source-clustered 95% CI [0.0229, 0.0713]. Both Cutalio settings also beat Codex on LocalTrend/SeqTrend, with 95% CIs excluding zero (paired supports: 343/324).

Codex yields 343/343 valid outputs, averaging 78.14 s/case including tool calls. Cutalio's 32.09M-parameter planner averages 0.20 s (P2) and 1.61 s (D4/K4) on A100 80GB with cached features; these times exclude frozen feature extraction.

2. Task scope and architecture

The demo linked in the original paper illustrates our motivation: users or agents choose footage, order and music; Cutalio times shots to music with optional anchors. Reordering, retiming, captions, within-shot edits and music splicing occur upstream; Cutalio receives the updated inputs.

Interactive demo — anchor editing and cut completion:

cutalio-demo.pages.dev

Completion supports repeated editing: users move/add anchors, retain confirmed boundaries, and regenerate the remaining cuts. The evaluated Qwen baseline generates timestamps as text without hard constraints on the complete path. Cutalio predicts boundaries jointly and applies DP constraints. Qwen has 21/343 invalid outputs; 22 further geometrically valid outputs contain intervals below 0.2 s. Cutalio is valid on all 343 cases.

3. Human evaluation

We randomly sampled 50 combos from 50 sources. Three authors voted with model identities/scores hidden and A/B order randomized; one aggregate outcome was saved per pair and dimension. Four Qwen outputs were invalid, leaving 46 D4/K4–Qwen pairs and 50 D4/K4–P2 pairs. D4/K4 wins/ties/losses:

ComparatorMusic fitDuration/transitionsOverall
Qwen16/6/2412/9/2515/8/23
P217/12/2125/9/1622/11/17

Same-pair means of per-combo scores:

SubsetModelOrder ↑Ratio ↓Local ↑Seq ↑JointH4 ↑
n=46D4/K40.70510.94620.21680.33610.5995
n=46Qwen0.76830.62720.28320.41810.6653
n=50D4/K40.68800.94050.22100.34570.5974
n=50P20.57001.06030.24150.37940.5245

Across 96 pairs, RelRatio correlates with duration/transition preference at Spearman ρ=0.549, 95% CI [0.396, 0.678]; SeqTrend correlates with music preference at ρ=0.389, CI [0.223, 0.542]. These use metric differences and aggregate votes, include ties, orient RelRatio toward lower error, and cluster CIs by source. Within the 46 Qwen pairs, LocalTrend/SeqTrend correlations with music preference are 0.493/0.557 (both Holm-adjusted p<0.001). Qwen receives more preference votes on this subset, consistent with its better metric values. Among decisive overall votes, higher JointH4 agrees in 30/38 (78.9%) Qwen pairs and 32/39 (82.1%) P2 pairs.

Renders retain main-table cuts, slowing footage when insufficient while keeping music at original speed. Slow motion can affect preference. Original-speed deployment instead supplies actual footage lengths as DP duration constraints.

Other comments

All metrics compare with the reference: RelOrder measures longer/shorter duration ordering in beats; RelRatio measures adjacent beat-duration ratio error; LocalTrend compares the direction and magnitude of musical changes around corresponding cuts; SeqTrend compares acoustic variation across the cut sequence. JointH4 takes a per-combo harmonic mean after mapping components to higher-is-better scales. We will explain these intuitions before formulas and clarify the workflow. We will cite CTC (Graves et al., 2006), which sums over alignments using forward–backward computation; our DP infers constrained boundary-path marginals and feeds them back to the predictor.

Thank you for the response and for the additional experiments. I appreciate the addition of the agentic baseline. However, the response does not really address my concern about the task scope. Moreover, the added human study is conducted by authors so it's intrinsically biased, and it also indicates that baselines may be preferred over the proposed approach. Accordingly, I will maintain the overall rating from my initial review.

Thank you for your reply. Please allow us to restate and elaborate on our motivation. Once the footage, its order, and the music have been selected, usually by a user or an agent, the editor still needs to decide how much of the overall timeline each shot should occupy, where transitions should fall, and when the edit should end. The selected music may last several minutes, while the finished video uses only part of it. Starting from a given position in the music, Cutalio uses visual content, music, and surrounding shots to allocate shot durations and predict transitions and the endpoint. Editors can also specify key transitions and have Cutalio plan the timing around them, as illustrated by the complete workflow in our demo. This completion mechanism also supports subsequent revisions: editors can move or add anchors, retain confirmed cuts, and regenerate the remaining boundaries.

Joint prediction and DP allow Cutalio to coordinate cuts while respecting anchors and configured duration limits. Cutalio produced valid timing plans for all 343 evaluation cases. The evaluated Qwen SFT baseline generates timestamps as text without enforcing these constraints across the complete sequence and produced 21 invalid plans. Maintaining valid timing during generation and revision is a practical advantage of our architecture. The trajectory formulation, data pipeline, and relational metrics also provide a common framework for studying this task and comparing different approaches.

For human validation, we used randomized A/B comparisons with model identities and scores hidden. Three authors voted on each pair; two had not participated in dataset construction. Four invalid Qwen outputs among the 50 sampled cases left 46 pairs for comparison. On these same pairs, all four component metrics favored Qwen: its mean JointH4 was 0.6653 versus 0.5995 for Cutalio D4/K4. Overall votes likewise favored Qwen in 23 pairs and Cutalio in 15, with 8 ties. Across the 38 decisive comparisons, the higher-JointH4 edit was preferred in 30. The human preferences therefore agree with the metric results on this subset, providing evidence for the metrics’ relevance to perceived editing quality.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Revised Manuscript and Summary of Changes

Thank you to all three reviewers for their careful reading and constructive suggestions. We have revised the manuscript accordingly. The main changes are summarized below.

  1. Improved readability. Following oiD2 and 3cnf's suggestions, we reorganized the motivation, data construction, and method presentation; clarified the purpose of each module and the notation; and added intuitive explanations of the evaluation metrics. We also clarified how users or agents work with Cutalio to create and repeatedly revise edits.

  2. Clarified the data and evaluation setup. In response to DTbb, we clarified clip contiguity, original reference timing, data splitting, and the virtual duration budget, and expanded the explanation of common-valid and fixed-support reporting.

  3. Added baseline comparisons. Following oiD2's suggestion, we added a Codex baseline and inference-time measurements. Following DTbb's suggestion, we added rule-based baselines and reported Qwen's fixed-support results with invalid outputs assigned zero (Table 1).

  4. Added feedback and TTS analyses. To address DTbb and 3cnf's questions about the architecture, we added a no-feedback control trained with the same recipe and reported paired confidence intervals (Table 2). In response to 3cnf's Q5, we compared candidate-selection strategies at different candidate counts to examine the gains and saturation of TTS (Table 3 and Appendix E.9).

  5. Added human validation. Following suggestions from all three reviewers, we reported a blind pairwise preference evaluation of rendered edits, the original metric scores on the same samples, and correlations between metrics and human preferences. We also described the slow-motion rendering settings (Table 5).

  6. Evaluated actual footage-length constraints. In response to DTbb and 3cnf's concerns about practical editing conditions, we added rendering results using actual footage lengths as constraints at original playback speed (Section 6.6).

We also added relevant references, including CTC, and provided full experimental settings and statistical results in the appendix to facilitate verification.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.