acceptodds
Under review as a conference paper at ICLR 2027

Towards Flexible and Controllable Music Generation for Universal Video

Abstract

Video-to-Music (V2M) generation task creates music that matches given visual content to improve audience experience. However, current methods have limitations in editing flexibility, conditional controllability, and generalizability. To meet varied user needs for general video use cases, we first construct Universe-MV, a music-video dataset for general scenarios, using large-scale web videos. We then train CueMuse, a general-purpose V2M framework supporting multi-condition control and editing, on Universe-MV. The framework includes V-Beater, a visual rhythm extraction module, and a conditional music generation backbone. Our model detects visual rhythm frames at 12 fps and can achieve precise alignment between video and music. At the same time, CueMuse can take text and editable video rhythm timestamps as inputs to satisfy additional professional requirements. Both subjective and objective experiments suggest that CueMuse achieves competitive performance and generally outperforms prior models in music-video alignment, music generation quality, and musicality. See our demo at https://anonymous.4open.science/r/demo-CueMuse.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video-to-music generation, Multimodal generation, Music generation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_BqYGOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents CueMuse, a flexible and controllable video-to-music generation framework designed for diverse video scenarios. The authors construct Universe-MV, a multi-category video-music dataset, and introduce V-Beater, a visual rhythm detection module that identifies shot transitions and salient motion events at 12 FPS. Building upon a modified MusicGen backbone, CueMuse incorporates video semantics, motion features, textual instructions, and editable beat timestamps to support multi-conditional music generation. The framework is trained through a two-stage strategy to improve video-music alignment and rhythm controllability. Extensive quantitative comparisons, ablation studies, and subjective evaluations demonstrate competitive performance in music quality, cross-modal alignment, and temporal synchronization.

Critical strengths

1. Practical and flexible video-to-music generation framework. The paper addresses an important practical problem by supporting diverse video scenarios and multiple optional control conditions. In particular, combining automatic visual rhythm detection with user-editable beat timestamps provides useful flexibility for different creative requirements.

2. Comprehensive implementation and competitive experimental performance. The authors construct a multi-category video-music dataset and evaluate CueMuse against several representative baselines using objective metrics and human evaluations. The results demonstrate competitive music quality and video-music alignment, supporting the practical effectiveness of the proposed framework.

Critical weaknesses

1. Limited methodological novelty. The proposed framework largely combines established components, including pretrained visual encoders, optical-flow-based rhythm extraction, a modified MusicGen backbone, and feature-level conditioning. While the resulting system is practically useful, the methodological advances beyond existing conditional music generation approaches are relatively incremental.

2. Insufficient isolation and validation of the core contributions. The strong performance may partly result from the extensively pretrained MusicGen-LLaMA backbone rather than the proposed video conditioning and rhythm control mechanisms. Moreover, although the paper evaluates rhythm editing stability, the current experiments do not sufficiently establish the advantages of V-Beater over simpler rhythm extraction alternatives or quantify the trade-off between controllability and music quality. Stronger controlled comparisons would make the central contributions more convincing.

Questions for authors

Q1. Contribution of the music generation backbone. How much of the performance improvement can be attributed to the stronger MusicGen-LLaMA backbone versus the proposed video conditioning framework? A comparison using the same backbone with simpler video conditioning mechanisms would help isolate the methodological contribution.

Q2. Effectiveness of V-Beater. How does V-Beater compare with simpler visual rhythm extraction approaches, such as optical-flow acceleration peak detection or shot-transition-based methods, in terms of detection accuracy and downstream music synchronization?

Q3. Rhythm controllability and music quality. Table 3 shows that enabling rhythm control degrades several music quality and alignment metrics. Could the authors provide a more detailed analysis of the trade-off between rhythm adherence and music quality, particularly under different control densities or strengths?

Q4. Generalization across video categories. Given the emphasis on universal video scenarios, how consistently does CueMuse perform across different video categories, particularly those with weak or ambiguous visual rhythms? Category-wise results would help substantiate the generalization claim.

Q5. Evaluation of fine-grained temporal synchronization. The paper emphasizes fine-grained visual rhythm detection and precise beat alignment. Could the authors clarify how the claimed temporal precision translates into actual music-beat synchronization accuracy, and whether the improvement remains consistent across different types of motion events?

Reviewer_QEGvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes CueMuse, a flexible and controllable video-to-music generation framework. It uses V-Beater to extract visual rhythms and supports video, text, and editable rhythm timestamps as generation conditions. The authors also construct Universe-MV dataset and Universe-Bench benchmark. Experiments show that CueMuse achieves competitive performance in music-video alignment, music generation quality, and musicality.

Critical strengths

1.Flexible and practical controllability. CueMuse supports video, text prompts, automatically detected visual rhythms, and user-edited rhythm timestamps. This makes the framework more practical and flexible than some prior methods using only video conditions. 2.Strong experimental performance. Comprehensive subjective and objective experiments show competitive performance in music-video alignment, music generation quality, and musicality.

Critical weaknesses

1.CueMuse accepts different control signals, but the experiments do not cover cases where they disagree. For example, a text may request a mood different from that suggested by the video, or edited timestamps may disagree with V-Beater. The paper therefore does not establish whether user input can override the video conditions. 2.The paper mainly evaluates CueMuse on the self-constructed Universe-Bench. Although a customized benchmark can provide complementary evidence, the paper should also report results on the public test sets, such as VidMuse’s V2M-Bench. Otherwise, the reported results are difficult to compare directly with prior work and may be affected by benchmark-selection bias.

Questions for authors
  1. How does CueMuse behave when text prompts conflict with the semantic content or emotion of the input video? When user-specified rhythm timestamps conflict with the automatically detected visual rhythms, which condition does the model prioritize?
  2. Why are the main results reported primarily on the author-constructed Universe-Bench rather than also on the public benchmarks, such as V2M-Bench?
  3. How much of the performance improvement comes from the proposed multi-condition framework, as opposed to the MusicGen-LLaMA backbone and its large-scale internal training data?
  4. How sensitive is CueMuse to key hyperparameters, such as the rhythm-window width, V-Beater frame rate, minimum inter-peak distance, and rhythm-related loss weights?
  5. Several closely related controllable video-to-music methods, including VMB, Spatial-Temporal Decomposition and Alignment, MTCV2M, and CPVC, are neither discussed nor compared. Given that these works have already explored multi-attribute, temporal, and fine-grained control, what is the key methodological novelty of CueMuse, and how does it differ from them in terms of controllable attributes, temporal granularity, user editability, and condition-injection mechanisms?
Reviewer_R7rNOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies general-purpose video-to-music generation and introduces CueMuse, which supports conditioning on video, text descriptions, and editable beat timestamps. The authors construct the Universe-MV video–music dataset, train an approximately 1.6B-parameter MusicGen-LLaMA backbone, and propose V-Beater to predict rhythmic events and shot transitions from visual features at 12 fps. Semantic, motion, and beat features are aligned with the music-token timeline, and staged fine-tuning supports automatic scoring and rhythm editing. Evaluation on the 396-sample Universe-Bench covers music quality, semantic relevance, and synchronization, supplemented by human evaluation, training-data ablations, OES-Com evaluation, and an editing-stability experiment on 50 samples. The results support the system's competitiveness across several video-scoring settings. They also show that enforcing rhythmic constraints can reduce quality.

Critical strengths
  1. Optional control conditions address practical, potentially conflicting creative requirements. The system lets users specify music descriptions and rhythmic conditions separately. Table 3 shows that adding rhythm control changes FAD from 2.2010 to 2.3166, and the authors explicitly acknowledge this cost. This provides empirical support for making such control optional and is useful for the design of video-scoring tools (pp. 6–9, Table 3).

  2. The evaluation covers music quality, video–music relationships, and subjective preference, with strong overall results. In the video-only setting, CueMuse achieves an FAD of 2.2010, compared with 2.5096 for VidMuse. With text and video, its FAD is 1.8376. Table 9 reports the highest mean scores across all four human-evaluation dimensions, and the appendices include OES-Com results and training-data-scale ablations. These experiments provide quantitative evidence for end-to-end generation performance (pp. 9, 22–24, Tables 2, 9, 10, and 12).

Critical weaknesses
  1. The benefit of V-Beater as a central technical contribution has not been sufficiently isolated. Removing the motion branch and retraining increases FAD from 2.2010 to 2.3737 and lowers some alignment scores in Table 3. This ablation within the same system supports the usefulness of motion features. However, it does not independently evaluate V-Beater's detection of rhythmic events or compare learned detection against conditions based only on shot transitions or simple optical-flow envelopes. V-Beater's detection accuracy remains unreported. The detector's incremental value for fine-grained control therefore remains unclear beyond the contribution of the motion-feature branch (pp. 4–7, 9, 16, Tables 3–5).

  2. The main evidence for fine-grained rhythm control does not yet directly establish the visual alignment objective emphasized by the paper. The Sync Score in Appendix H measures agreement between the beats of generated music and reference music. Alignment with visual events is assessed only indirectly. Table 11 tests responses to specified timestamps on 50 samples with global temporal-shift perturbations. Original-condition recall is 0.767 within a tolerance of ±80 ms. Reasonable additional beats may lower precision. Nevertheless, the current results do not disentangle the effects of difficult samples, beat density, and control strength. The human MOS results support overall preference. Perceptual success at the specified visual timing points after editing requires separate evaluation. This limits the evidence for fine-grained, editable visual rhythm control as a core contribution (pp. 22–25, Tables 9 and 11, Appendices H and J).

Questions for authors
  1. Could the authors report V-Beater's detection performance on independently annotated videos? Please compare shot-transition-only and simple optical-flow rhythm conditions using the same CueMuse backbone and training configuration. This comparison would clarify the learned detector's independent value.

  2. How were the specified timestamps in Table 11 obtained, and how were the 50 samples selected? Do they cover different beat densities and irregular rhythms? Could the authors report the relationship between per-sample control success and musical naturalness? Please identify the results that directly validate alignment with visual events.

  3. What counting units underlie the 10K entries in Table 1 and the 102,776 entries in Table 6? Were Universe-MV and Universe-Bench separated at the levels of original videos, musical tracks, and near-duplicate clips? Please also reconcile the backbone configurations described in Appendix C and Table 4.

  4. How many participants contributed to Table 9, how many valid ratings were collected per video, and what do the reported ± terms represent? Could the authors provide confidence intervals for the main MOS differences using videos as the statistical unit, so that the stability of the overall preference result can be assessed?

1 comment

I appreciate the author's detailed response. Having carefully reviewed the clarifications, which resolve most of my concerns, I would like to increase my initial rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.