- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents CueMuse, a flexible and controllable video-to-music generation framework designed for diverse video scenarios. The authors construct Universe-MV, a multi-category video-music dataset, and introduce V-Beater, a visual rhythm detection module that identifies shot transitions and salient motion events at 12 FPS. Building upon a modified MusicGen backbone, CueMuse incorporates video semantics, motion features, textual instructions, and editable beat timestamps to support multi-conditional music generation. The framework is trained through a two-stage strategy to improve video-music alignment and rhythm controllability. Extensive quantitative comparisons, ablation studies, and subjective evaluations demonstrate competitive performance in music quality, cross-modal alignment, and temporal synchronization.
Critical strengths
1. Practical and flexible video-to-music generation framework. The paper addresses an important practical problem by supporting diverse video scenarios and multiple optional control conditions. In particular, combining automatic visual rhythm detection with user-editable beat timestamps provides useful flexibility for different creative requirements.
2. Comprehensive implementation and competitive experimental performance. The authors construct a multi-category video-music dataset and evaluate CueMuse against several representative baselines using objective metrics and human evaluations. The results demonstrate competitive music quality and video-music alignment, supporting the practical effectiveness of the proposed framework.
Critical weaknesses
1. Limited methodological novelty. The proposed framework largely combines established components, including pretrained visual encoders, optical-flow-based rhythm extraction, a modified MusicGen backbone, and feature-level conditioning. While the resulting system is practically useful, the methodological advances beyond existing conditional music generation approaches are relatively incremental.
2. Insufficient isolation and validation of the core contributions. The strong performance may partly result from the extensively pretrained MusicGen-LLaMA backbone rather than the proposed video conditioning and rhythm control mechanisms. Moreover, although the paper evaluates rhythm editing stability, the current experiments do not sufficiently establish the advantages of V-Beater over simpler rhythm extraction alternatives or quantify the trade-off between controllability and music quality. Stronger controlled comparisons would make the central contributions more convincing.
Questions for authors
Q1. Contribution of the music generation backbone. How much of the performance improvement can be attributed to the stronger MusicGen-LLaMA backbone versus the proposed video conditioning framework? A comparison using the same backbone with simpler video conditioning mechanisms would help isolate the methodological contribution.
Q2. Effectiveness of V-Beater. How does V-Beater compare with simpler visual rhythm extraction approaches, such as optical-flow acceleration peak detection or shot-transition-based methods, in terms of detection accuracy and downstream music synchronization?
Q3. Rhythm controllability and music quality. Table 3 shows that enabling rhythm control degrades several music quality and alignment metrics. Could the authors provide a more detailed analysis of the trade-off between rhythm adherence and music quality, particularly under different control densities or strengths?
Q4. Generalization across video categories. Given the emphasis on universal video scenarios, how consistently does CueMuse perform across different video categories, particularly those with weak or ambiguous visual rhythms? Category-wise results would help substantiate the generalization claim.
Q5. Evaluation of fine-grained temporal synchronization. The paper emphasizes fine-grained visual rhythm detection and precise beat alignment. Could the authors clarify how the claimed temporal precision translates into actual music-beat synchronization accuracy, and whether the improvement remains consistent across different types of motion events?
I appreciate the author's detailed response. Having carefully reviewed the clarifications, which resolve most of my concerns, I would like to increase my initial rating.