acceptodds
Under review as a conference paper at ICLR 2027

FullSongEval: Auditing Text–Label Associations Before Music Evaluation

Abstract

Public music datasets pair text with labels such as genre. Before using these labels in evaluation, researchers may ask whether the text predicts them. Yet a scorer can remain above label prevalence after complete label sets are reassigned among records. FullSongEval audits a candidate rule that accepts text–label pairings when a paired fold interval for a scorer’s average-precision margin over prevalence has a positive lower endpoint. We test original-pair acceptance and scorer sensitivity to an injected label-linked cue. We then refit the scorer after two fixed label-set reassignments per dataset and ask whether the rule rejects them. On MusicCaps and MTG-Jamendo, the scorer detects the cue, but the rule accepts both reassignments. On the same MTG tracks, audio-feature genre classifiers trained with original tags predict independently collected consensus genres more accurately than classifiers trained with reassigned tags: mean average precision is 0.7927 versus 0.4365 and 0.4304. A diagnostic finite-sample random-label reference accounts for tied scores. In a separate 8,000-record Free Music Archive check fixed before source inspection, this reference separates original pairings from two reassignment controls, while the prevalence rule accepts all three. Exceeding prevalence is insufficient to approve these text–label pairings for later music evaluation.

Primary area:
datasets and benchmarks
Keywords:
music metadata, text–label associations, dataset validation, average precision, label reassignment, negative controls, music information retrieval

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_xWhpOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper examines the extent to which text descriptions and tags agree in music datasets. They propose a method to sanity check those tags by attempting to predict them from associated text descriptions. The findings suggest

Critical strengths

None

Critical weaknesses

Poor writing. The biggest issue with this paper is that the writing is entirely incoherent. It is almost impossible to read. Even as early as the third sentence of the abstract:

"Yet a scorer can remain above label prevalence after complete label sets are reassigned among records. FullSongEval audits a candidate rule that accepts text–label pairings when a paired fold interval for a scorer’s average-precision margin over prevalence has a positive lower endpoint. We test original-pair acceptance and scorer sensitivity to an injected label-linked cue."

Too many undefined terms, impossible to follow what the authors are trying to say. The writing clarity problems persist throughout the paper. I am left with very little understanding of the work after reading.

Title. The title "FullSongEval" seems largely irrelevant to the proposed work. Why does predicting labels from text have anything to do with "full songs"?

Little impact or novelty. It seems like the main point of the paper is to propose a method for flagging noisy music tags based on associated text captions. However, no evidence is presented to suggest that this improves the performance of any downstream AI systems (e.g., classifiers, generative models) trained on the denoised data. Moreover, there are no new methods presented here. Accordingly, it's not clear what the contribution is.

Questions for authors

N/A

1 comment

The primary text–label studies use MusicCaps and MTG-Jamendo. The paper also reports scoped follow-ups on 1,300 MARD album reviews, 546 Song Describer captions tied to MTG recordings, and 8,000 FMA catalog records. These checks use different protocols and are not independent replications of the primary study, so the results support a conclusion about the tested prevalence rule and controls, not a general acceptance rule for music datasets.

For the four fixed complete-label-set reassignments, prevalence remains above zero while the tie-aware random-label reference has intervals crossing zero. In a separate conditional check with 200 independent per-label allocations, prevalence accepts 124 MusicCaps draws and 200 MTG draws; the tie-aware comparison accepts 0 and 3. Other results depend on the control: within-author caption rotation crosses zero even against prevalence, MARD's cue response is too small to establish scorer sensitivity, and Song Describer reuses MTG recordings. The FMA check is a separate catalog, with exact-metadata overlap sensitivity reported in the appendix.

One downstream check trains audio-feature genre classifiers on original or reassigned MTG tags and tests them against independently collected consensus genres for the same recordings. Across 106 supported blocks, mean AP is 0.7927 for original tags and 0.4365 and 0.4304 for the reassigned tags. This is evidence about genre prediction from released 30-second features; it does not measure full-song quality, retrieval, or generative-system performance. The current manuscript needs clearer terminology and story organization; no broader utility test is reported.

Reviewer_rTxJOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates a very specific problem: evaluating the quality of data vs. labels in music datasets where text describes each song, and the labels are song genres. More accurately, it asks these text–label associations are meaningful. The paper, which is mostly a statistical report, shows that a straight forward attempt to evaluate the dataset quality fails when tested on random data-label assignments, if considering simply the quality of a detector trained on the data. Instead, it proposes a tie-aware random-label reference approach that seems to perform better in this aspect. While the paper identifies a legitimate problem, it is written in a manner that is next to impossible to follow, performs little analysis on a couple of datasets and deduces a larger conclusions that is not sufficiently backed up, and provides an overall minimal contribution to a niche problem.

Critical strengths

The paper makes the important distinctions between a classifier outperforming a trivial baseline and that text–label associations are meaningful. They also suggest an interesting and useful experiment of testing the set with reassigned labels. The proposed tie-aware is also might be interesting, although it is difficult to assess.

Critical weaknesses

First and foremost, the writing is difficult to follow. Many terms are used without being explained, sentences leave significant logical gaps between them, and no motivation is presented for the specific choices or the overall problem. The paper required several passes to parse.

Maybe because of the way it is presented and maybe not, another major weakness seems to be the limited novelty and significance of the contribution. The central result is highlights that a straight forward examination, using prevalence-based AUPRC, is insufficient, and that a tie-aware random-label reference yields better results for two cases. This is a useful observation, but it is not developed enough. Alternative approaches in the literature are not discussed, generalizable insights are not drawn, and two cases are far too narrow to be signifiant.

As mentioned earlier, another weakness of the paper is its evaluation. Evaluation would be stronger if more cases could be demonstrated. Perhaps some augmentations over the existing experiments and another set of synthetic experiments would boost the claim for robustness of the method. In addition, other alternatives should also be demonstrated, taken from literature of similar tasks in other domains perhaps. Other mechanisms in the experiment could also be investigated, such as the scoring models, dataset characteristics, or reassignment mechanisms.

Questions for authors

For a future version, I would recommend for the authors to significantly improve the story telling aspect of the exposition, clearly motivating the problem, justifying the choice of for the solution, and demonstrating the effectiveness of the result on downstream tasks. This additionally means developing the initial interesting idea further, perhaps until it can be applied to a downstream task, such as model training, retrieval, etc.

1 comment

The tested object is a rule for deciding whether text–label pairs are credible before later music evaluation. On MusicCaps and MTG-Jamendo, the rule accepts the original pairs and all four fixed reassignment controls. This is the specific failure established by the primary audit.

The paper does not evaluate a full-song music system or compare with existing music-evaluation methods. The later tie-aware random-label comparison separates the fixed reassignment controls in these datasets, but it was developed after the primary audit and is presented as a diagnostic. It is not a generally validated replacement rule. A clearer distinction between this association audit and music-system evaluation remains needed in the manuscript.

Reviewer_zWtUOpenReview ↗
Rating
1
Confidence
1
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper propose FullSongEval, which is a new approach for audio evaluation. The proposed method is used for MusigCaps and MTG-Jamendo.

Critical strengths

None

Critical weaknesses
  • I am not super familiar with music evaluation. However, the approach is quite naive and there must exist similar approach to evaluate song dataset.
  • The paper is not well written and difficult to follow.
Questions for authors

To my knowledge, there exist large number of music evaluation approaches. What is the key advantages over existing methods? Also, the comparison to other approaches are necessary.

1 comment

The paper tests a candidate rule for accepting text–label pairs before later music evaluation. It asks whether the rule accepts original pairs, responds to added label-linked cues, and rejects label-reassignment controls. On MusicCaps and MTG-Jamendo, it accepts the original pairs and all four fixed reassignment controls; that is the failure demonstrated by this test.

The paper does not introduce or benchmark a full-song evaluation system, so its results do not establish superiority over existing music-evaluation methods. A later tie-aware random-label comparison separates the fixed reassignment controls in these datasets, but it was developed after the primary audit and is presented as a diagnostic, not a generally validated replacement rule. The manuscript needs clearer positioning relative to the closest existing checks and a clearer distinction between this audit and music-system evaluation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.