- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper audits music evaluators in three roles: checking control, predicting human preferences, and supervising another model. It combines shortcut tests, preference-score fusion and teacher–student experiments, then improves preference prediction using human feedback.
Critical strengths
I like the practical audit: label-prior baselines, waveform replacement, and token-truncation checks expose ways an apparently good evaluator can miss the actual task.
The experimental controls are unusually thorough: fold isolation, matched human-only training and symmetric calibration make the modest supervision gains more credible. The authors also report failed tests candidly.
Critical weaknesses
The teaching story remains fairly narrow. Gains are below 1% in development NLL, use one MuQ student architecture, and remain inconclusive on the official benchmark. The pre-specified teacher-ranking reversal is also unresolved.
The stronger feedback results are exploratory, following inspection and repeated annotation of the same pool. The listener holdout helps, but involves just six people; both feedback-versus-frozen comparisons miss the adjusted significance threshold and significance testing may miss adjustment.
Questions for authors
How sensitive are the teacher rankings to the training schedule selected using External 4D? Could that choice materially affect the utility comparison?
Do the small supervision gains remain convincing when uncertainty includes retraining, rather than conditioning on the fitted ensembles?