acceptodds
Withdrawn from ICLR 2027

AUDITING MUSIC EVALUATORS ACROSS ROLES: FROM CONTROL VALIDITY TO TEACHING UTILITY

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Music evaluators assess attribute control, predict listener preferences, and provide training supervision. Evidence for one use does not by itself justify another. We develop a framework that connects evaluator selection and improvement through role-specific comparisons of control validity, direct preference quality, and teaching utility. A log-loss counterexample shows that better direct preference predictions need not yield a more beneficial one-step update to a constrained student, even with common examples and soft targets. Control evaluations show how label priors and text truncation affect measured agreement, and use waveform replacement to test whether judgments depend on the audio. For musicality preferences, complementary assessment scores improve direct prediction. Separate, fully fold-isolated teacher–student comparisons identify modest reductions in student log loss relative to matched human-only training; some gains persist after temperature calibration of both supervised and human-only students. On common human targets, comparisons among supervised students inform teacher selection, while comparisons with matched human-only training test whether supervision helps. Human-feedback adaptation lowers the log loss of MuQ-based preference scorers on held-out content relative to continued training on the original labels and temperature calibration, closing the evaluation–improvement loop for direct preference quality. The framework grounds evaluator selection and improvement in evidence matched to the intended role and, for supervision, to the student and learning procedure.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Music Evaluation, Reward Model Evaluation, Human Preference Learning, Teacher–Student Learning, Pseudo-Labeling, Role-Specific Evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_bWyzOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper audits music evaluators in three roles: checking control, predicting human preferences, and supervising another model. It combines shortcut tests, preference-score fusion and teacher–student experiments, then improves preference prediction using human feedback.

Critical strengths

I like the practical audit: label-prior baselines, waveform replacement, and token-truncation checks expose ways an apparently good evaluator can miss the actual task.

The experimental controls are unusually thorough: fold isolation, matched human-only training and symmetric calibration make the modest supervision gains more credible. The authors also report failed tests candidly.

Critical weaknesses

The teaching story remains fairly narrow. Gains are below 1% in development NLL, use one MuQ student architecture, and remain inconclusive on the official benchmark. The pre-specified teacher-ranking reversal is also unresolved.

The stronger feedback results are exploratory, following inspection and repeated annotation of the same pool. The listener holdout helps, but involves just six people; both feedback-versus-frozen comparisons miss the adjusted significance threshold and significance testing may miss adjustment.

Questions for authors

How sensitive are the teacher rankings to the training schedule selected using External 4D? Could that choice materially affect the utility comparison?

Do the small supervision gains remain convincing when uncertainty includes retraining, rather than conditioning on the fitted ensembles?

Reviewer_4pevOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether music evaluators are reliable for three different uses: checking attribute control, predicting human preferences, and providing useful supervision to a student preference model. It combines label-prior, text-truncation, and donor-audio checks with score fusion, matched teacher–student experiments, and human-feedback adaptation. A theoretical counterexample illustrates why better direct preference prediction need not lead to better student learning. Empirically, combining evaluator scores improves direct prediction, while supervision produces modest student NLL gains and leaves the primary teacher-ranking reversal unresolved. Human-feedback adaptation provides clearer gains in held-out preference prediction, although its subsequent teaching benefit is not tested.

Critical strengths
  1. The paper provides thorough analyses of a timely problem in music generation. Label-prior, text-truncation, and donor-audio checks expose concrete pitfalls that raw agreement can hide, making the control audit useful for practitioners.
  2. The matched human-only controls, fold isolation, and calibration comparisons strengthen the preference-learning analysis. Human-feedback adaptation also improves held-out preference prediction beyond scalar calibration.
Critical weaknesses
  1. Limited novelty and actionable takeaway. The distinction between teacher prediction quality and downstream usefulness is already motivated by distillation and reward-model evaluation, which the paper acknowledges. The constrained one-step counterexample clarifies this distinction, but offers limited guidance for selecting a teacher without training each student. What generalizable insight does the music study add?
  2. Narrow and modest evidence for teaching utility. The main study fixes one MuQ student architecture with frozen encoder features and a particular hard pseudo-label/weighting procedure; scaling across student capacity and human/pseudo-label budgets is not established. Table 3 shows only 0.62–0.97% relative NLL gains with little accuracy change, official-test utility is inconclusive, and the primary External-4D/AudioBox ranking reversal is not established. NLL is appropriate for preference probabilities, and accuracy/Brier results are also provided, but these do not establish improved generated-music quality. The stronger feedback-adaptation results assess direct prediction, leaving its downstream teaching benefit open.
Questions for authors
  1. Clarification: beyond the acknowledged teacher–student/reward-model literature, what concrete teacher-selection decision can the framework support without a full student-training comparison?
  2. Evidence request: do the teacher ordering and gains over ZERO persist with different student capacities or human/pseudo-label budgets? This would help distinguish a broader finding from the current learning procedure.
  3. Clarification: given the inconclusive primary reversal and official-test utility results, what is the strongest supported downstream claim? Is there evidence that the adapted evaluator improves subsequent student learning or human-rated generated music?
Reviewer_TJnJOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

I am unable to summarize the contributions of the paper because, despite my best attempts, i largely do not understand it. (see below)

Critical strengths

The authors work on the important and timely topic of automatic music generation. They experiment with state-of-the art machine listening tools such as CLAP, MERT, MuQ-MuLan, and TuneJury. They evaluate their approach on multiple benchmarks: CMI-RewardBench, Music Arena, SongEval, SongBench. They have collected aesthetic judgments of generated music from 12 paid contributors with 1–10 years of musical experience.

Critical weaknesses

(1) The main weakness of the paper is clarity. The language is often vague and occasionally abstruse. Key elements are missing from the abstract and the introduction. What is the problem? What makes the problem interesting? What makes the problem difficult? What is your solution? These questions are not clearly answered in the introduction. Rather, the authors state the following question: "What evidence justifies using an evaluator in each of these roles?" and conclude the introduction with "Human feedback improves held-out preference prediction beyond these controls, closing the loop for a specified task, population, and scoring policy." In my opinion there is a blatant mismatch between question and answer.

Terms like "condition judgments", "common human targets", "label priors", "matched training", "source condition", "deterministic donor" "waveform dependence", "separately fitted positive temperature", are presented without definition. Terms like "usefulness", "utility", "validity", "quality", "advantage" seem to be used nearly interchangeably.

The authors write as if the reader was already familiar with the current literature, not just of machine learning for audio as a whole, but of particular software tools and datasets.

A flowchart explaining the protocol would have greatly improved the clarity of the proposed method.

The rest of the paper is poorly organized, with numerous repetitions of almost exactly the same claim. Appendices are long and too wordy, with a lack of focus. More than once, i have felt like the paper might have been completed in a rush, with little attention to clarity.

(2) The second weakness of the paper is scope. The authors do not present an algorithm for representation learning. From what i understand, they use a well-established statistical model (Bradley-Terry) on a collection of already-in-place technological tools. Moreover, it is not clear to what extent the proposed algorithm is specific to music listening or whether it could also apply, with suitable modifications, to other kinds of aesthetic judgments. Section 2 briefly alludes to music source separation but the paper is not written in a way that can potentially reach a wide audience at ICLR, beyond the specific use case of music generation.

Questions for authors

(1) If i wanted to replicate your findings, what is the minimum number of already-in-place AI models i would have to install and pipe together (in addition to human labor)? It seems to me that this number is large, which puts the robustness and generalizability of the presented technique at risk.

(2) Can you walk me through Section 4.2 once again? I must confess that i do not understand these paragraphs at all, despite my best efforts:

To test gains beyond scalar confidence adjustment, we calibrate each student and ZERO with a separately fitted positive temperature, preserving threshold decisions. In each fold, temperatures fitted to inner-ensemble validation predictions transfer to the corresponding outer ensembles. [...] Some gains thus persist under symmetric calibration, conditional on fitted ensembles and transferred temperatures. Separate evaluation uses the original frozen 25-member student ensembles, distinct from these refits, on three-listener judgments unused in teacher or student fitting.

Reviewer_1DnFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper audits music evaluators across three roles: assessing whether audio satisfies a control description, predicting human preferences, and providing useful supervision for a student preference model. It combines control-validity diagnostics, multidimensional score fusion, teacher–student comparisons, and adaptation using additional human feedback. A theoretical counterexample shows that better direct predictive quality need not imply better teaching utility. Empirically, score fusion improves direct preference prediction, some teachers produce small student NLL improvements under a fold-isolated protocol, and human-feedback adaptation improves prediction on held-out content. The prespecified primary test of a reversal between direct evaluator ranking and teaching ranking remains unresolved. Here, teaching utility concerns student preference prediction, rather than improvements to a music generator.

Critical strengths
  1. An interesting and consequential distinction between evaluating well and teaching well. Direct agreement with human preferences and usefulness as a training signal are different objectives, and the paper is right to discourage substituting one for the other. The theoretical counterexample makes the absence of a universal implication explicit, while the audit framework gives this distinction a concrete form in music evaluation. This is the strongest conceptual contribution.
  2. Careful controls and unusually explicit reporting of limitations. The fold-isolated student comparisons, matched human-only ZERO baseline, symmetric temperature calibration, and same-target comparisons address important sources of misleading conclusions. The paper also documents the earlier leakage issue and its corrected experiment, and acknowledges that the primary ranking-reversal test is not confirmed. These are substantive strengths of the empirical methodology, even though I find the resulting contribution insufficient for acceptance.
Critical weaknesses
  1. The central distinction is compelling, but the empirical study provides limited actionable knowledge about teaching utility. In the same-target primary comparison, External 4D has a clear direct NLL advantage over AudioBox, but the student difference is only +0.000144, with an interval of [−0.001440, 0.001662]. The reannotations do not resolve the teaching ordering. There are informative secondary comparisons, including 10D versus 22D, but they do not establish a broadly reliable teacher-selection rule or identify when direct predictive quality becomes a misleading selection criterion. An inconclusive primary test is not itself a reason to reject an audit paper. My concern is that, after accounting for this uncertainty, the demonstrated new knowledge is largely a caution against assuming transfer across roles, with limited evidence about how practitioners should make a better choice.
  2. The positive results do not sufficiently establish the practical value of separating the roles. Under the fully fold-isolated protocol, the student NLL improvements over ZERO are approximately 0.62%–0.97%, with little change in accuracy; after symmetric calibration, only AudioBox and 10D survive the reported Holm correction.
Questions for authors
  1. Given that the prespecified primary reversal remains unresolved, what is the strongest supported empirical conclusion about when direct evaluator quality is insufficient for teacher selection?
  2. Do the existing held-out results support a reproducible benefit from using the audit to choose or adapt a teacher, relative to choosing by direct NLL? If no such decision-level validation is available, how should the demonstrated student gains be interpreted in practical terms, and which claims are limited to direct preference prediction?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.