acceptodds
Under review as a conference paper at ICLR 2027

VMEval: A Multi-Dimensional and Interpretable Framework for Video-Music Harmony Evaluation

Abstract

Recent advances in audio-visual learning have primarily focused on cross-modal generation tasks such as video-to-music generation. However, the field still lacks a systematic and objective framework for evaluating video-music harmony, particularly one that simultaneously supports quantitative scoring and qualitative interpretation. Existing approaches are limited to either global compatibility estimation for cross-modal retrieval or descriptive analysis with multimodal large language models (MLLMs) without targeted supervision for harmony evaluation. In this work, we propose VMEval, a multi-dimensional and interpretable framework for video-music harmony evaluation. We decompose video-music harmony into six structured features including rhythm, emotion, dynamics, structure, atmosphere, and content, which provide complementary views of cross-modal consistency and collectively cover the major perceptual aspects. Based on these features, VMEval learns fine-grained cross-modal consistency and jointly produces quantitative dimension-specific harmony scores and interpretable natural-language explanations. VMEval can be applied to downstream tasks including cross-modal retrieval and the evaluation of video-to-music generation models. Extensive experiments show that VMEval achieves competitive performance on cross-modal retrieval and demonstrates strong agreement with human judgments in evaluating video-music harmony.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
multimodal large language models, video-music harmony

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_gnWBOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This work develops VMEval, a multi-dimensional framework for evaluating harmony between videos and music. The system disentangles harmony into six perceptual dimensions. By incorporating pseudo-supervised feature alignment, adaptive weighting strategies and global style modeling, it can output quantitative metrics as well as natural-language explanations. Experimental results reveal better cross-modal retrieval performance and higher consistency with human subjective evaluations.

Critical strengths

This paper targets a meaningful, practically useful task: evaluating video-music harmony, which carries importance for music recommendation and video-to-music generation. The research motivation is solid, and the problem merits further investigation. Moreover, the proposed framework delivers both quantitative scores and interpretable explanations, bringing a valuable viewpoint for fine-grained video-music harmony assessment.

Critical weaknesses

1. Ambiguous technical contributions and design rationale. The introduced framework assembles multiple established techniques: feature decomposition, cross-attention, adaptive weighting, and global style modeling. Still, the manuscript lacks adequate justification for this specific component combination and fails to explicitly verify the unique gains brought by such integration, leaving its primary technical innovations underspecified.

2. Inadequate validation for annotations and interpretability claims. The method depends substantially on pseudo-labels produced via Gemini-2.5-Pro, but the work does not thoroughly examine the trustworthiness and quality of these generated annotations. In addition, the experimental evaluation focuses largely on cross-modal retrieval and aggregate harmony metrics. It lacks direct validation to confirm whether the predicted per-dimension scores and corresponding importance weights match human perceptual perceptions, which undermines empirical support for its fine-grained interpretability claim.

Questions for authors

Q1. What justifications support the six harmony dimensions, and how is their semantic independence validated? These six dimensions are manually predefined. Several categories including emotion, atmosphere and dynamics may share considerable semantic overlap. What evidence backs this decomposition strategy, and how robust is the framework when adopting alternative dimension taxonomies?

Q2. How trustworthy are the pseudo-supervised feature scores and importance weights generated by Gemini-2.5-Pro? While the authors present a high approval rate for annotations, this metric cannot directly prove the correctness of separate scores or weight values. Could the authors supply human assessments at the dimension level, and analyze how noise and biases within pseudo-labels impact final predictions?

Q3. Does this video-music harmony evaluator generalize well to unseen datasets and evaluation scenarios? The evaluation mainly builds on cross-modal retrieval and a limited pool of human subjective ratings. Still, strong retrieval results do not equate to good perceptual harmony: multiple music pieces can fit a single video equally well. How do the authors verify that VMEval learns broadly applicable perceptual compatibility, instead of overfitting to dataset-specific matching patterns?

Reviewer_sXWiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes VMEval, a framework for evaluating video-music harmony. VMEval decomposes harmony into six fine-grained features, and predicts a consistency score for each feature with an importance weight for each video-music pair. VMEval also adds a global style term to produce an overall matching score of the video-music pair. A Qwen-2.5-Omni decoder is trained to generate a natural-language explanation from these intermediate predictions. These intermediate fine-grained supervision comes from using Gemini-2.5-Pro to create pseudo-annotations on HarmonySet.

Experiments include video-to-audio and audio-to-video retrieval on HarmonySet and EmoMV, using both automatic and human evaluations.

Critical strengths
  • The paper is easy to follow, and the motivation is well presented.

  • It is a good idea to have fine-grained evaluation of video-music harmony, which has increasing importance for audio-visual generation applications. Weighting the aspects separately for each example is intuitive and sound.

Critical weaknesses
  • My biggest concern is on generalization. Specifically,

    • VMEval is a model-based evaluator based on Qwen-2.5-Omni. It is unclear whether the results hold beyond this.

    • The pseudo-supervision comes from Gemini-2.5-Pro and it is unknown if VMEval still works with other annoators.

    • VMEval involves training several modules on the HarmonySet, so its results may not generalizing beyond it. Authors also provide results on EmoMV, however, EmoMV also contains contents from Youtube, making it similar to HarmonySet, the claimed generalization is not convincing to me.

  • Authors claimed VMEval can be used for evaluting generation models (line27), but the experiments only contain retrieval tasks.

  • In general, VMEval is not showing strong emperical results. It falls behind the baselines on R@1 on V2A retrieval on Harmonyset and ties on EmoMV. And on A2V retrieval, it falls behind baselines on MedR and AvgR.

  • Though the idea of feature decomposition is interesting, but the ablation shows that it is actually not effective (MedR is 15 in V2A for both w/ and w/o FD in Table3; A2V results show similar patterns). So the main performance contributor is still the coarse style alignment, rather than the fine-grained features VMEval proposes.

  • Some technical details are missing. E.g., there is no information about how the baselines are trained/tuned (author only describe as "re-trained on the same HarmonySet") -- Gemini-2.5-pro is also listed as baseline, and it is not an open-weight model, which is confusing.

Questions for authors
  • have you tried with other models to create the pseudo-annotations?
  • what about VMEval's performance on datasets not derived from Youtube?
  • how did you train (or maybe prompt?) the baselines? Could you explain in detail?
Reviewer_tbxPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents VMEval, a framework for quantitative and qualitative video–music harmony evaluation. It combines six feature-specific consistency scores with adaptive weights and a global style compatibility term to produce an overall score. A LoRA-fine-tuned Qwen decoder generates textual explanations from the intermediate predictions and audiovisual representations. Experiments report improved retrieval performance and lower error against human ratings than the evaluated baselines.

Critical strengths

The framework provides both quantitative scores and qualitative explanations of video–music harmony. Its feature-specific scores and pair-dependent weights offer a useful way to inspect different aspects of compatibility.

Critical weaknesses
  1. The retrieval results provide limited evidence of reliable harmony evaluation. With 40 candidates, random R@1 is 2.5%, compared with VMEval’s 5.5% on HarmonySet and 4.1% on EmoMV. Moreover, retrieving the original soundtrack does not necessarily establish that the model can reliably judge perceptual compatibility.

  2. Section 3.2 describes scores and weights for six dimensions, whereas Appendix A.2 specifies only four. The mapping between these supervision labels and the six predicted dimensions is not explained.

  3. Although Equation 11 defines the final score, the training objective and score calibration remain unclear. Please specify the losses used to supervise the feature scores, weights, and global style module, and explain how predictions are mapped to the valid scoring range.

  4. Ablations removing these intermediate predictions and interventions modifying their values would help assess whether the explanations faithfully reflect the scoring process.

Questions for authors
  1. How are the four-dimensional pseudo-labels mapped to the six predicted dimensions?

  2. How is score comparability across video–music pairs validated when different pairs use different feature weights?

  3. What is the complete training objective for quantitative scoring, and how is the output calibrated to the valid scoring range?

  4. Do the explanations change consistently when the predicted feature scores or weights are modified?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.