- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper examines the extent to which text descriptions and tags agree in music datasets. They propose a method to sanity check those tags by attempting to predict them from associated text descriptions. The findings suggest
Critical strengths
None
Critical weaknesses
Poor writing. The biggest issue with this paper is that the writing is entirely incoherent. It is almost impossible to read. Even as early as the third sentence of the abstract:
"Yet a scorer can remain above label prevalence after complete label sets are reassigned among records. FullSongEval audits a candidate rule that accepts text–label pairings when a paired fold interval for a scorer’s average-precision margin over prevalence has a positive lower endpoint. We test original-pair acceptance and scorer sensitivity to an injected label-linked cue."
Too many undefined terms, impossible to follow what the authors are trying to say. The writing clarity problems persist throughout the paper. I am left with very little understanding of the work after reading.
Title. The title "FullSongEval" seems largely irrelevant to the proposed work. Why does predicting labels from text have anything to do with "full songs"?
Little impact or novelty. It seems like the main point of the paper is to propose a method for flagging noisy music tags based on associated text captions. However, no evidence is presented to suggest that this improves the performance of any downstream AI systems (e.g., classifiers, generative models) trained on the denoised data. Moreover, there are no new methods presented here. Accordingly, it's not clear what the contribution is.
Questions for authors
N/A
The primary text–label studies use MusicCaps and MTG-Jamendo. The paper also reports scoped follow-ups on 1,300 MARD album reviews, 546 Song Describer captions tied to MTG recordings, and 8,000 FMA catalog records. These checks use different protocols and are not independent replications of the primary study, so the results support a conclusion about the tested prevalence rule and controls, not a general acceptance rule for music datasets.
For the four fixed complete-label-set reassignments, prevalence remains above zero while the tie-aware random-label reference has intervals crossing zero. In a separate conditional check with 200 independent per-label allocations, prevalence accepts 124 MusicCaps draws and 200 MTG draws; the tie-aware comparison accepts 0 and 3. Other results depend on the control: within-author caption rotation crosses zero even against prevalence, MARD's cue response is too small to establish scorer sensitivity, and Song Describer reuses MTG recordings. The FMA check is a separate catalog, with exact-metadata overlap sensitivity reported in the appendix.
One downstream check trains audio-feature genre classifiers on original or reassigned MTG tags and tests them against independently collected consensus genres for the same recordings. Across 106 supported blocks, mean AP is 0.7927 for original tags and 0.4365 and 0.4304 for the reassigned tags. This is evidence about genre prediction from released 30-second features; it does not measure full-song quality, retrieval, or generative-system performance. The current manuscript needs clearer terminology and story organization; no broader utility test is reported.