- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper proposes an Audio Language Model (ALM) to evaluate text generated music clips based on multidimensional natural language rubric prompts. 2 generator models MusicGen-small and ACE-step v1 and three evaluators CLAP, SongEval and Audiobox are used in the comparative evaluation experiments. Its empirical results show improvement on the music quality evaluation scores against many text to music generators , especially for multi-aspect of prompts over many single metric evaluators. It claim that the method can be useful in post-training of music generation. But ALM rubric tends to be complementary and is weaker than task specific measurement and still cannot be used as independent stand alone evaluation method for music generation.
Critical strengths
The paper shows empirically rubric ALM can be used in assisting the evaluation of text-to-music generation as a complementary method to task-specific evaluation method. It shed some light in the multi-dimensional evaluation of alignment and quality of music generation in terms of text-prompts.
Critical weaknesses
The result in Figure 1, the relative score changes using ALM rubric rater, although tend to be positive, the improvement almost less than 2% which is not very significant. In Figure 2, the paper compares the relative score change by DPO against DiffusionNFT under post training, DPO shows not significant improvement while DiffusionNFT shows variant (positive negative) change. It implies it is still not very stable to claim the improvement using the rubric metric for post-training of music generation.
Whether the multi-dimension metrics are independent or dependent can be an issue to clarify that why joint prompting and single dimensional prompting perform differently. But the paper did not address the issue. For example, in the paper it stated: “Music Flamingo collapses its dimension scores under joint prompting but improves all three external evaluators under dimension-wise prompting”. This might imply that the Rubric prompts might not be independent.
The experimental results still consist many factors (optimizer, rater, prompting strategy) that can harm the justification of the claim as to how the ALM rubric method can be used properly in improving the evaluation score against different setup of evaluations. The users might have to searching for different optimizers, raters and prompting strategies in order to find better results.
Minor concerns:
“For the general text-to-music experiments, we use a seven-dimensional rubric covering prompt alignment (w = 0.25), style and instrumentation (0.15), melody and motif (0.10), rhythm and groove (0.10), structure and ending (0.15), production quality (0.15), and overall musicality (0.10);” Again here the weight for each dimension is subjectively specified in ad hoc way and without justification. Besides “style and instrument” are two dimensions, while “overall musicality” can be related and dependent to the rest of music dimension, why it is only weighted as 0.10?
Questions for authors
“Each dimension specifies the quality to be assessed, a list of audible evidence the rater should listen for (for example, pulse clarity and timing stability for rhythm), five score anchors at 0, 2, 5, 8, and 10, and a weight wk.” I wonder how the five score anchors are chosen. Why not just linear “0 2 4 6 8 10”?
4 comments
Response to Reviewer 8QxN (2/3): stability and rubric dimensions
Weakness 1
In Figure 2, the paper compares the relative score change by DPO against DiffusionNFT under post training, DPO shows not significant improvement while DiffusionNFT shows variant (positive negative) change. It implies it is still not very stable to claim the improvement using the rubric metric for post-training of music generation.
We thank the reviewer for pointing this out: the results in Figure 2 do look unstable, with changes of both signs, and the paper did not explain them. In short, the apparent instability is not randomness: with a reliable rater, DiffusionNFT improves the evaluators consistently; the negative results come from unreliable raters, which can be identified before training; and the DPO results reflect a training-precision limitation rather than the rubric reward. In more detail, the sign is set by the rater rather than by run-to-run instability. No Qwen3-Omni run lowers any evaluator at the end of training, under either prompting strategy, and repeating the Qwen3-Omni joint run twice more with the same configuration again raises CLAP and Audiobox in all three runs (Audiobox significantly in each). The negative cells come from two identifiable rater failures: Qwen2-Audio's reward is anti-correlated with CLAP over training, and Music Flamingo under joint prompting returns seven identical scores on 99.6% of clips (Weakness 2); both can be detected before training (Weakness 3). The ACE-Step DPO runs show no significant change for a technical reason. We trained them in bf16 to keep the training time and memory of the 3.5B-parameter model manageable, and at that precision many of their preference margins round to exactly zero, so the loss does not move. With fp32 and a larger learning rate the same objective learns, but its gains remain smaller than those of DiffusionNFT (our response to Reviewer eJAx). We will add this explanation to §5.
Weakness 2
Whether the multi-dimension metrics are independent or dependent can be an issue to clarify that why joint prompting and single dimensional prompting perform differently. But the paper did not address the issue. For example, in the paper it stated: "Music Flamingo collapses its dimension scores under joint prompting but improves all three external evaluators under dimension-wise prompting". This might imply that the Rubric prompts might not be independent.
This is a good point, and we agree that the paper should have analyzed whether the rubric dimensions are independent, since this bears on the difference between joint and dimension-wise prompting. In short, the dimensions are correlated, as musical qualities naturally are, but not redundant, and the difference between joint and dimension-wise prompting comes from one rater's failure under joint prompting, not from this correlation. For the Qwen raters, scores still vary across dimensions within a clip: seven identical scores occur on 0.9–3% of clips with the evaluation prompt and on about 20% with the training prompt. The difference for Music Flamingo is a rater failure: given all dimensions in one request, it returns seven identical scores on 99.6% of clips, whereas asked one dimension at a time it does so on 8%. Dimension-wise prompting therefore recovers the per-dimension signal for this rater. Correlated dimensions are not redundant, however: with the same rater, replacing the rubric by a single overall-quality question lowers SongEval and Audiobox significantly, whereas the multi-dimensional rubric improves all three evaluators (our response to Reviewer eJAx, Question 5). We will add this analysis to the paper.
Response to Reviewer 8QxN (3/3): practical guidance and rubric weights
Weakness 3
The experimental results still consist many factors (optimizer, rater, prompting strategy) that can harm the justification of the claim as to how the ALM rubric method can be used properly in improving the evaluation score against different setup of evaluations. The users might have to searching for different optimizers, raters and prompting strategies in order to find better results.
We agree that the number of factors makes it harder to see how the method should be used, and that the paper should give clearer guidance. Our recommendation for each factor is as follows.
Optimizer. We recommend DiffusionNFT. Because offline DPO gives weak gains, we moved to online optimization. Flow-GRPO was the first online method we tried, on the tempo task, and its results were not good enough, which is why we switched to DiffusionNFT. For this response we also ran Flow-GRPO on instrument and key, and again on tempo, with the same rewards, prompts and training budget as DiffusionNFT. It comes close on tempo but barely moves instrument and key accuracy.
Table 1. ACE-Step v1, attribute tasks with objective rewards (instrument tagger, key detector, beat tracker), same reward, prompts and training budget for both optimizers: change in attribute accuracy relative to the untrained model at the end of training. Bold: the better optimizer.
| Task | DiffusionNFT | Flow-GRPO |
|---|---|---|
| Instrument accuracy | +0.124 | +0.005 |
| Key accuracy | +0.069 | +0.002 |
| Tempo accuracy | +0.232 | +0.176 |
Rater. Results are bounded by the rater, and a reliable rater can be checked before training. As discussed under Weakness 1, the sign of the changes under DiffusionNFT is set by the rater: Qwen3-Omni improves all three evaluators under both prompting strategies, while Qwen2-Audio, whose prompt-alignment score is unrelated to CLAP on untrained-model clips (within-prompt correlation -0.02, vs 0.23 for Qwen3-Omni), loses CLAP. Such checks on untrained-model clips take minutes and need no training, and a better rater can be swapped in without changing the method.
Prompting. The rubric is flexible by design. Users choose the rubric that matches their intention. For raters that collapse their scores under joint prompting, dimension-wise prompting is the safe default (Weakness 2).
Minor concern
"For the general text-to-music experiments, we use a seven-dimensional rubric covering prompt alignment (w = 0.25), style and instrumentation (0.15), melody and motif (0.10), rhythm and groove (0.10), structure and ending (0.15), production quality (0.15), and overall musicality (0.10);" Again here the weight for each dimension is subjectively specified in ad hoc way and without justification. Besides "style and instrument" are two dimensions, while "overall musicality" can be related and dependent to the rest of music dimension, why it is only weighted as 0.10?
We thank the reviewer for raising this; we tested the weights directly by retraining. The weights set the balance between prompt adherence and musical quality, but the gains do not hinge on them: retraining DiffusionNFT with uniform weights also improves the evaluators and lowers none, whereas rewarding prompt alignment alone, or replacing the rubric by a single overall-quality question, lowers SongEval and Audiobox (our response to Reviewer eJAx, Question 5). "Style and instrumentation" is a single dimension that assesses genre and instrumentation together. Because "overall musicality" overlaps with the other dimensions, raising its weight would mostly re-weight qualities that are already covered. We will clarify both in the main text.
We hope this addresses the reviewer's concerns, and would be grateful if the reviewer would consider revisiting the score.
Three-seed results: the rubric gains are stable across seeds
As the discussion closes, we would like to share a result that directly addresses the reviewer's concern about stability (Figure 2, Weakness 1). We trained the main DiffusionNFT comparison with three independent seeds (details in our reply to Reviewer eJAx; revised Tables 1 and 9).
The gains are stable. With a reliable rater, the rubric reward improves CLAP, SongEval and Audiobox in each of the three seeds, each with an interval excluding zero (mean CLAP +0.031 [+0.003, +0.060], Audiobox +0.434 [+0.208, +0.660]). No other reward we re-ran does so: SongEval-only and Audiobox-only each lower at least one other evaluator in every seed. The positive and negative changes in Figure 2 therefore come from the rater, which our pre-training checks identify on untrained-model clips, not from instability of the method.
Why the multi-dimensional rubric matters. The same ALM asked one holistic question gives seed-dependent outcomes (one seed improves all three evaluators, the other two lower SongEval and Audiobox; this updates our earlier single-run statement in parts 2/3 and 3/3), whereas the multi-dimensional rubric improves them consistently. The correlated dimensions therefore still add value: they make the reward reliable.
Revised PDF. The revision is uploaded, with all changes from the original submission shown in blue.
We would be grateful if the reviewer would consider these results in their final assessment.
Response to Reviewer 8QxN (1/3): score anchors and size of the gains
We are grateful to the reviewer for a detailed review. We appreciate their recognition of the paper's clarity and of its contribution to the multi-dimensional evaluation of music generation. We also value the reviewer's questions, particularly around the stability of the results and the design of the rubric. Here, we respond to each point that was raised.
Question 1
We thank the reviewer for this question. We chose five levels (absent, weak, adequate, strong, exceptional) at integer positions, with "adequate" at the midpoint. The anchors are descriptions at some integer values, but the rater may return any score in [0,10], including decimals. A linear 0/2/4/6/8/10 scale would need six descriptions, and five evenly spaced ones would sit at 0/2.5/5/7.5/10.
The placement has a small effect. To check, we asked Qwen3-Omni to score the same 512 clips three times, each time with the same five descriptions placed differently: at 0/2/5/8/10 (as in the paper), evenly at 0/2.5/5/7.5/10, and with only the two ends labeled (0 and 10). The ranking of the clips stays very similar across the three versions (Spearman correlation 0.84–0.87). For reference, scoring the same clips twice with the paper's placement gives identical scores on the same GPU, and a Spearman correlation of 0.92 when only the GPU type changes. The placement therefore changes the ranking only slightly more than switching hardware does, so its effect on the reward is small.
Weakness 1
We agree that the gains from offline DPO in Figure 1 are small. A replication with three training seeds and matched preference pairs confirms that offline DPO changes the evaluators only slightly for every preference source, so it cannot reliably separate reward sources (our response to Reviewer eJAx, Question 1). The absolute gains are small because offline DPO learns only from a fixed set of preference pairs sampled once from the untrained model, and this is the reason the paper proposes online optimization with DiffusionNFT, which scores fresh samples at every update. With DiffusionNFT on ACE-Step, the Qwen3-Omni rubric improves CLAP by 8.4% (p=1×10^-6), SongEval by 5.4% and Audiobox by 5.4% (both p<10^-30, paired t-test) relative to the untrained model, even though none of the three evaluators is part of the reward. Gains of this size are still meaningful: current text-to-music models already produce high-quality music, and raising established evaluators without lowering any of them is hard, as the single-metric rewards show. We will emphasize this in the revision.