acceptodds
Under review as a conference paper at ICLR 2027

Rubric-Based Optimization for Text-to-Music Generation

Abstract

Post-training text-to-music generation requires reward signals that capture multiple aspects of musical quality beyond what any single automatic metric can measure. We study structured, rubric-based rewards from pretrained audio-language models (ALMs) as training signals for both autoregressive and diffusion-based music generators. An ALM scores each generated clip against the rubric; we rank candidates generated for the same text prompt by their scores and convert these rankings into preference pairs for DPO on both MusicGen-small and ACE-Step v1, and additionally use the rubric scores directly as scalar rewards for DiffusionNFT on ACE-Step v1. On MusicCaps, rubric-based optimization improves CLAP, SongEval, and Audiobox-Aesthetics simultaneously, with the strongest gains obtained by DiffusionNFT on ACE-Step. By contrast, on MusicGen-small, building preferences from any one of these automatic evaluators produces clear cross-metric trade-offs: the targeted evaluator improves while other independent evaluators deteriorate. We further study tempo, key, and instrumentation, where precise objective rewards are available. Directly optimizing these specialized rewards reliably improves the target attributes, whereas ALM rubrics provide only partial transfer for tempo and instrumentation and no measurable improvement for key. Together, these results suggest a practical division of labor: ALM rubrics are effective for broad perceptual qualities that are difficult to formalize, while specialized objective rewards remain preferable when reliable measurements are available.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
text-to-music generation, audio-language models, reward optimization, preference optimization, direct preference optimization, diffusion models, music generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_8QxNOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes an Audio Language Model (ALM) to evaluate text generated music clips based on multidimensional natural language rubric prompts. 2 generator models MusicGen-small and ACE-step v1 and three evaluators CLAP, SongEval and Audiobox are used in the comparative evaluation experiments. Its empirical results show improvement on the music quality evaluation scores against many text to music generators , especially for multi-aspect of prompts over many single metric evaluators. It claim that the method can be useful in post-training of music generation. But ALM rubric tends to be complementary and is weaker than task specific measurement and still cannot be used as independent stand alone evaluation method for music generation.

Critical strengths

The paper shows empirically rubric ALM can be used in assisting the evaluation of text-to-music generation as a complementary method to task-specific evaluation method. It shed some light in the multi-dimensional evaluation of alignment and quality of music generation in terms of text-prompts.

Critical weaknesses

The result in Figure 1, the relative score changes using ALM rubric rater, although tend to be positive, the improvement almost less than 2% which is not very significant. In Figure 2, the paper compares the relative score change by DPO against DiffusionNFT under post training, DPO shows not significant improvement while DiffusionNFT shows variant (positive negative) change. It implies it is still not very stable to claim the improvement using the rubric metric for post-training of music generation.

Whether the multi-dimension metrics are independent or dependent can be an issue to clarify that why joint prompting and single dimensional prompting perform differently. But the paper did not address the issue. For example, in the paper it stated: “Music Flamingo collapses its dimension scores under joint prompting but improves all three external evaluators under dimension-wise prompting”. This might imply that the Rubric prompts might not be independent.

The experimental results still consist many factors (optimizer, rater, prompting strategy) that can harm the justification of the claim as to how the ALM rubric method can be used properly in improving the evaluation score against different setup of evaluations. The users might have to searching for different optimizers, raters and prompting strategies in order to find better results.

Minor concerns:

“For the general text-to-music experiments, we use a seven-dimensional rubric covering prompt alignment (w = 0.25), style and instrumentation (0.15), melody and motif (0.10), rhythm and groove (0.10), structure and ending (0.15), production quality (0.15), and overall musicality (0.10);” Again here the weight for each dimension is subjectively specified in ad hoc way and without justification. Besides “style and instrument” are two dimensions, while “overall musicality” can be related and dependent to the rest of music dimension, why it is only weighted as 0.10?

Questions for authors

“Each dimension specifies the quality to be assessed, a list of audible evidence the rater should listen for (for example, pulse clarity and timing stability for rhythm), five score anchors at 0, 2, 5, 8, and 10, and a weight wk.” I wonder how the five score anchors are chosen. Why not just linear “0 2 4 6 8 10”?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 8QxN (1/3): score anchors and size of the gains

We are grateful to the reviewer for a detailed review. We appreciate their recognition of the paper's clarity and of its contribution to the multi-dimensional evaluation of music generation. We also value the reviewer's questions, particularly around the stability of the results and the design of the rubric. Here, we respond to each point that was raised.

Question 1

"Each dimension specifies the quality to be assessed, a list of audible evidence the rater should listen for (for example, pulse clarity and timing stability for rhythm), five score anchors at 0, 2, 5, 8, and 10, and a weight wk." I wonder how the five score anchors are chosen. Why not just linear "0 2 4 6 8 10"?

We thank the reviewer for this question. We chose five levels (absent, weak, adequate, strong, exceptional) at integer positions, with "adequate" at the midpoint. The anchors are descriptions at some integer values, but the rater may return any score in [0,10], including decimals. A linear 0/2/4/6/8/10 scale would need six descriptions, and five evenly spaced ones would sit at 0/2.5/5/7.5/10.

The placement has a small effect. To check, we asked Qwen3-Omni to score the same 512 clips three times, each time with the same five descriptions placed differently: at 0/2/5/8/10 (as in the paper), evenly at 0/2.5/5/7.5/10, and with only the two ends labeled (0 and 10). The ranking of the clips stays very similar across the three versions (Spearman correlation 0.84–0.87). For reference, scoring the same clips twice with the paper's placement gives identical scores on the same GPU, and a Spearman correlation of 0.92 when only the GPU type changes. The placement therefore changes the ranking only slightly more than switching hardware does, so its effect on the reward is small.

Weakness 1

The result in Figure 1, the relative score changes using ALM rubric rater, although tend to be positive, the improvement almost less than 2% which is not very significant.

We agree that the gains from offline DPO in Figure 1 are small. A replication with three training seeds and matched preference pairs confirms that offline DPO changes the evaluators only slightly for every preference source, so it cannot reliably separate reward sources (our response to Reviewer eJAx, Question 1). The absolute gains are small because offline DPO learns only from a fixed set of preference pairs sampled once from the untrained model, and this is the reason the paper proposes online optimization with DiffusionNFT, which scores fresh samples at every update. With DiffusionNFT on ACE-Step, the Qwen3-Omni rubric improves CLAP by 8.4% (p=1×10^-6), SongEval by 5.4% and Audiobox by 5.4% (both p<10^-30, paired t-test) relative to the untrained model, even though none of the three evaluators is part of the reward. Gains of this size are still meaningful: current text-to-music models already produce high-quality music, and raising established evaluators without lowering any of them is hard, as the single-metric rewards show. We will emphasize this in the revision.

AuthorsOpenReview ↗

Response to Reviewer 8QxN (2/3): stability and rubric dimensions

Weakness 1

In Figure 2, the paper compares the relative score change by DPO against DiffusionNFT under post training, DPO shows not significant improvement while DiffusionNFT shows variant (positive negative) change. It implies it is still not very stable to claim the improvement using the rubric metric for post-training of music generation.

We thank the reviewer for pointing this out: the results in Figure 2 do look unstable, with changes of both signs, and the paper did not explain them. In short, the apparent instability is not randomness: with a reliable rater, DiffusionNFT improves the evaluators consistently; the negative results come from unreliable raters, which can be identified before training; and the DPO results reflect a training-precision limitation rather than the rubric reward. In more detail, the sign is set by the rater rather than by run-to-run instability. No Qwen3-Omni run lowers any evaluator at the end of training, under either prompting strategy, and repeating the Qwen3-Omni joint run twice more with the same configuration again raises CLAP and Audiobox in all three runs (Audiobox significantly in each). The negative cells come from two identifiable rater failures: Qwen2-Audio's reward is anti-correlated with CLAP over training, and Music Flamingo under joint prompting returns seven identical scores on 99.6% of clips (Weakness 2); both can be detected before training (Weakness 3). The ACE-Step DPO runs show no significant change for a technical reason. We trained them in bf16 to keep the training time and memory of the 3.5B-parameter model manageable, and at that precision many of their preference margins round to exactly zero, so the loss does not move. With fp32 and a larger learning rate the same objective learns, but its gains remain smaller than those of DiffusionNFT (our response to Reviewer eJAx). We will add this explanation to §5.

Weakness 2

Whether the multi-dimension metrics are independent or dependent can be an issue to clarify that why joint prompting and single dimensional prompting perform differently. But the paper did not address the issue. For example, in the paper it stated: "Music Flamingo collapses its dimension scores under joint prompting but improves all three external evaluators under dimension-wise prompting". This might imply that the Rubric prompts might not be independent.

This is a good point, and we agree that the paper should have analyzed whether the rubric dimensions are independent, since this bears on the difference between joint and dimension-wise prompting. In short, the dimensions are correlated, as musical qualities naturally are, but not redundant, and the difference between joint and dimension-wise prompting comes from one rater's failure under joint prompting, not from this correlation. For the Qwen raters, scores still vary across dimensions within a clip: seven identical scores occur on 0.9–3% of clips with the evaluation prompt and on about 20% with the training prompt. The difference for Music Flamingo is a rater failure: given all dimensions in one request, it returns seven identical scores on 99.6% of clips, whereas asked one dimension at a time it does so on 8%. Dimension-wise prompting therefore recovers the per-dimension signal for this rater. Correlated dimensions are not redundant, however: with the same rater, replacing the rubric by a single overall-quality question lowers SongEval and Audiobox significantly, whereas the multi-dimensional rubric improves all three evaluators (our response to Reviewer eJAx, Question 5). We will add this analysis to the paper.

AuthorsOpenReview ↗

Response to Reviewer 8QxN (3/3): practical guidance and rubric weights

Weakness 3

The experimental results still consist many factors (optimizer, rater, prompting strategy) that can harm the justification of the claim as to how the ALM rubric method can be used properly in improving the evaluation score against different setup of evaluations. The users might have to searching for different optimizers, raters and prompting strategies in order to find better results.

We agree that the number of factors makes it harder to see how the method should be used, and that the paper should give clearer guidance. Our recommendation for each factor is as follows.

Optimizer. We recommend DiffusionNFT. Because offline DPO gives weak gains, we moved to online optimization. Flow-GRPO was the first online method we tried, on the tempo task, and its results were not good enough, which is why we switched to DiffusionNFT. For this response we also ran Flow-GRPO on instrument and key, and again on tempo, with the same rewards, prompts and training budget as DiffusionNFT. It comes close on tempo but barely moves instrument and key accuracy.

Table 1. ACE-Step v1, attribute tasks with objective rewards (instrument tagger, key detector, beat tracker), same reward, prompts and training budget for both optimizers: change in attribute accuracy relative to the untrained model at the end of training. Bold: the better optimizer.

TaskDiffusionNFTFlow-GRPO
Instrument accuracy+0.124+0.005
Key accuracy+0.069+0.002
Tempo accuracy+0.232+0.176

Rater. Results are bounded by the rater, and a reliable rater can be checked before training. As discussed under Weakness 1, the sign of the changes under DiffusionNFT is set by the rater: Qwen3-Omni improves all three evaluators under both prompting strategies, while Qwen2-Audio, whose prompt-alignment score is unrelated to CLAP on untrained-model clips (within-prompt correlation -0.02, vs 0.23 for Qwen3-Omni), loses CLAP. Such checks on untrained-model clips take minutes and need no training, and a better rater can be swapped in without changing the method.

Prompting. The rubric is flexible by design. Users choose the rubric that matches their intention. For raters that collapse their scores under joint prompting, dimension-wise prompting is the safe default (Weakness 2).

Minor concern

"For the general text-to-music experiments, we use a seven-dimensional rubric covering prompt alignment (w = 0.25), style and instrumentation (0.15), melody and motif (0.10), rhythm and groove (0.10), structure and ending (0.15), production quality (0.15), and overall musicality (0.10);" Again here the weight for each dimension is subjectively specified in ad hoc way and without justification. Besides "style and instrument" are two dimensions, while "overall musicality" can be related and dependent to the rest of music dimension, why it is only weighted as 0.10?

We thank the reviewer for raising this; we tested the weights directly by retraining. The weights set the balance between prompt adherence and musical quality, but the gains do not hinge on them: retraining DiffusionNFT with uniform weights also improves the evaluators and lowers none, whereas rewarding prompt alignment alone, or replacing the rubric by a single overall-quality question, lowers SongEval and Audiobox (our response to Reviewer eJAx, Question 5). "Style and instrumentation" is a single dimension that assesses genre and instrumentation together. Because "overall musicality" overlaps with the other dimensions, raising its weight would mostly re-weight qualities that are already covered. We will clarify both in the main text.

We hope this addresses the reviewer's concerns, and would be grateful if the reviewer would consider revisiting the score.

AuthorsOpenReview ↗

Three-seed results: the rubric gains are stable across seeds

As the discussion closes, we would like to share a result that directly addresses the reviewer's concern about stability (Figure 2, Weakness 1). We trained the main DiffusionNFT comparison with three independent seeds (details in our reply to Reviewer eJAx; revised Tables 1 and 9).

The gains are stable. With a reliable rater, the rubric reward improves CLAP, SongEval and Audiobox in each of the three seeds, each with an interval excluding zero (mean CLAP +0.031 [+0.003, +0.060], Audiobox +0.434 [+0.208, +0.660]). No other reward we re-ran does so: SongEval-only and Audiobox-only each lower at least one other evaluator in every seed. The positive and negative changes in Figure 2 therefore come from the rater, which our pre-training checks identify on untrained-model clips, not from instability of the method.

Why the multi-dimensional rubric matters. The same ALM asked one holistic question gives seed-dependent outcomes (one seed improves all three evaluators, the other two lower SongEval and Audiobox; this updates our earlier single-run statement in parts 2/3 and 3/3), whereas the multi-dimensional rubric improves them consistently. The correlated dimensions therefore still add value: they make the reward reliable.

Revised PDF. The revision is uploaded, with all changes from the original submission shown in blue.

We would be grateful if the reviewer would consider these results in their final assessment.

Reviewer_eJAxOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies off-the-shelf audio-language models (ALMs) as reward signals for post-training text-to-music generators. The ALM is prompted with a hand-written rubric (natural-language definitions, listening cues, and score anchors for seven dimensions such as prompt alignment, rhythm, and production quality), returns a 0–10 score per dimension, and the weighted mean of these scores serves as the reward. The reward is used in two ways: to rank candidates into preference pairs for DPO on MusicGen-small and ACE-Step v1, and as a scalar reward for online DiffusionNFT on ACE-Step v1. Three raters (Music Flamingo, Qwen2-Audio, Qwen3-Omni) and two prompting strategies (all dimensions in one request vs. one request per dimension) are compared on MusicCaps, with CLAP, SongEval, and Audiobox-Aesthetics as held-out evaluators. The main findings are that rubric-based DPO on MusicGen-small improves all three evaluators simultaneously, whereas DPO on any single evaluator improves it at the expense of the others; that DiffusionNFT yields much larger changes than DPO on ACE-Step, with outcomes strongly dependent on the rater and prompting strategy; and that for tempo, key, and instrumentation, optimizing a dedicated audio analyzer directly is far more effective than the rubric reward. A small listening test prefers the best DiffusionNFT model over the untrained model in 69% of judgments.

Critical strengths
  1. Clear and well-organized presentation. The paper is easy to follow: it states two explicit research questions up front (whether rubric rewards can improve several aspects of generation at once, and whether a rubric is still the right reward when an attribute can be measured directly) and structures the experiments around them. Each experiment opens with its setup and the question it answers, and the figures and captions are informative, with full numerical results provided in the appendix. The authors are also careful to qualify their claims, e.g., noting that the DPO and DiffusionNFT comparison involves separately tuned configurations and does not isolate which difference is responsible.
  2. Candid reporting of failure modes. Rather than presenting only successful configurations, the paper documents where rubric rewards fail and why: Music Flamingo returning the same score for all seven dimensions on more than 98% of clips under joint prompting while its own ratings indicate improvement; Qwen2-Audio raising SongEval and Audiobox-Aesthetics at the cost of caption adherence; the rater's own reward continuing to rise while external evaluators decline; and no measurable transfer for key. These observations are valuable to the community as concrete cautions for anyone using audio-language models as reward signals.
Critical weaknesses
  1. The ACE-Step DPO baseline fails a basic sanity check, which undermines the optimizer comparison in §5.2. On MusicGen, each single-metric DPO run improves its target metric, as expected. On ACE-Step, the CLAP- and SongEval-ranked runs decrease their own target metric (Table 1), and some rubric runs retain very few preference pairs after margin filtering (Appendix A.10). This suggests the single-timestep flow-matching DPO surrogate barely optimizes anything. As a result, (i) the claim that DiffusionNFT makes "far better use of the rubric" than DPO (§5.2) may reflect a weak baseline rather than an advantage of online optimization, and (ii) the ACE-Step row of Figure 1 says little about rubric vs. single-metric preferences, so §5.1 effectively rests on MusicGen alone. A stronger offline baseline (e.g., a properly tuned Diffusion-DPO with multiple timesteps) or an online alternative such as Flow-GRPO would make this comparison meaningful.
  2. Under the optimizer that does work, the rubric reward is unreliable, leaving the central claim inconclusive. DiffusionNFT itself appears to optimize effectively: with the objective reward it doubles the tempo score and triples key accuracy within a few dozen updates (§5.3). With the rubric reward, however, it substantially decreases CLAP in three of the six configurations (Music Flamingo with joint prompting and both Qwen2-Audio runs; for Qwen2-Audio, "roughly a quarter to a third of the untrained model's score," §5.2), and only three of six improve all three external evaluators. This pattern suggests the optimizer faithfully maximizes whatever the rater rewards, including generic polish over caption adherence (Qwen2-Audio) or a single collapsed score (Music Flamingo, joint). It directly contradicts the Discussion's claim (line 450) that resistance to narrow optimization is "a central advantage of rubric rewards." Moreover, there is no a priori way to identify a good rater/prompting configuration without consulting the very external evaluators the rubric is meant to replace, and no single-metric DiffusionNFT baseline is provided to show that the rubric does better than a single metric under online optimization. Taken together with Weakness 1, the only setting that shows the claimed balanced gains is MusicGen with DPO, where the improvements are small (Table 2) and reported from a single run without confidence intervals.
  3. Positioning relative to closely related work could be strengthened. I may be missing distinctions the authors have in mind, but several closely related works do not seem to be discussed, and engaging with them would help clarify the paper's contribution. Resonate (https://arxiv.org/abs/2603.11661) uses an off-the-shelf large audio-language model as the reward for online RL (Flow-GRPO) on a flow-matching text-to-audio model and also reports that online RL outperforms offline DPO, which seems close to the setup and optimizer finding of §5.2. Rubric-prompted judges as rewards have also been explored for LLM post-training (Rubrics as Rewards, https://arxiv.org/abs/2507.17746) and text-to-image generation (RubricRL, https://arxiv.org/abs/2511.20651, which likewise scores each criterion independently, similar to dimension-wise prompting). The ACE-Step DPO surrogate (Appendix A.4) appears closely related to Diffusion-DPO (https://arxiv.org/abs/2311.12908), and earlier audio preference-optimization work with automatic rewards (e.g., Tango 2, https://arxiv.org/abs/2404.09956; TangoFlux, https://arxiv.org/abs/2412.21037) might also be relevant. A short discussion of how the present work differs, for instance the focus on music, the use of untrained judges, and the analysis of rubric vs. objective rewards, would make the novelty clearer to readers.
Questions for authors

(i) Questions

  1. In §5.1, "the same top-2×bottom-2 pairing" seems to refer to the same pairing rule applied to the same six candidates, while the actual preference pairs differ across preference sources, since each source ranks the candidates differently. Is this correct? If so, please clarify the wording. Please also report the margin m, learning rate, and resulting number of preference pairs for the single-metric baselines (these are given only for the rubric runs in Appendix A.10), and explain how the margin was set given that CLAP, SongEval, and Audiobox-Aesthetics are on very different scales from the 0–10 rubric score.
  2. Related to Weakness 1: the ACE-Step run trained on CLAP-ranked preferences ends up with a lower CLAP score than the untrained model (Figure 1, bottom row, "CLAP score" panel; Table 1). Is this within evaluation noise, given that the untrained model's own scores vary across evaluation runs (Table 4)? Was the flow-matching DPO surrogate verified to improve its own training metric in any setting?
  3. Related to Weakness 2: please show the rater's own rubric reward over training for the six MusicCaps DiffusionNFT runs. This would show whether the reward rose in all six runs, including those where CLAP dropped, i.e., whether the optimizer succeeded and the reward was exploited.
  4. Can you add single-metric DiffusionNFT baselines (CLAP, SongEval, Audiobox-Aesthetics) on ACE-Step, so that the rubric vs. single-metric comparison of §5.1 is also made under the online optimizer, where effects are large?
  5. The rubric weights are set by hand "to emphasize adherence to the prompt while keeping every quality in play" (§4.1), and no justification is given for the weights of the attribute rubrics (Appendix A.2). How sensitive are the results to these choices? For example, how do uniform weights, or the prompt-alignment dimension alone, compare? This would also clarify whether the gains come from the multi-dimensional structure of the rubric or mainly from the prompt-alignment dimension.

(ii) Suggestions

  1. Add a short paragraph to the method section explaining what rubrics are and how they are used to prompt the ALM. This would make the paper more self-contained at the cost of little added space (simplifies things for people unfamiliar with this concept).
7 comments
AuthorsOpenReview ↗

Response to Reviewer eJAx (1/5): MusicGen DPO pairing and replication

We thank the reviewer for an exceptionally thorough and constructive review. We appreciate their recognition of the paper's clarity and of our candid reporting of failure modes. We also value the reviewer's suggestions for improvement, particularly around stronger baselines and weight ablations. Here, we respond to each point that was raised.

Question 1 and Weakness 2

Question 1: In §5.1, "the same top-2×bottom-2 pairing" seems to refer to the same pairing rule applied to the same six candidates, while the actual preference pairs differ across preference sources, since each source ranks the candidates differently. Is this correct? If so, please clarify the wording. Please also report the margin m, learning rate, and resulting number of preference pairs for the single-metric baselines [...], and explain how the margin was set given that CLAP, SongEval, and Audiobox-Aesthetics are on very different scales from the 0–10 rubric score.

Weakness 2: Taken together with Weakness 1, the only setting that shows the claimed balanced gains is MusicGen with DPO, where the improvements are small (Table 2) and reported from a single run without confidence intervals.

Pairing rule. We thank the reviewer for the careful reading; it is indeed the same rule applied to the same candidates. Six candidates per prompt are sampled once; each source ranks them, and each of its two highest-ranked candidates is paired with each of its two lowest-ranked (exact ties are dropped). The candidates and the rule are shared, but the pairs differ between sources. We will reword §5.1 accordingly.

Margins and pair counts. We agree that one margin cannot be applied across these scales. The margin discards pairs whose two scores are closer than m, and we applied it only to the rubric sources, whose scores share the rater's 0–10 scale: m = 2.0–2.5 (650–3,132 pairs), with β 0.5, a learning rate of 1e-7 to 3e-7 and 2 epochs. CLAP, SongEval and Audiobox have no common scale on which one threshold means the same thing, so the single-metric sources used no margin and kept every top-2×bottom-2 pair (17,664 pairs), with β 0.75, a learning rate of 5e-7 and 1 epoch. The two groups therefore also differ in pair count, and the paper's MusicGen rows come from one run each: with caption-paired 95% bootstrap intervals, every single-metric source trades off significantly, but only two of the nine rubric changes are significant.

Matched replication. To compare the sources fairly, we re-ran every source with the same number of pairs, 3,000, taken as that source's widest-margin pairs with the margin measured relative to the spread of its own scores, with identical training settings, three training seeds and settings fixed before training.

Table 1. MusicGen-small DPO replication: three training seeds per source, 3,000 pairs for every source, settings fixed before training. Mean change over the three seeds, with a caption-paired 95% bootstrap interval pooled over seeds. Bold: interval excludes zero. †Part of that source's reward.

Preference sourceΔCLAPΔSongEvalΔAudiobox
Rubric, Music Flamingo-0.0006 [-0.0036, +0.0024]-0.003 [-0.016, +0.010]-0.025 [-0.051, +0.001]
Rubric, Qwen2-Audio-0.0012 [-0.0042, +0.0016]+0.000 [-0.013, +0.013]-0.026 [-0.053, +0.001]
Rubric, Qwen3-Omni+0.0016 [-0.0017, +0.0049]+0.001 [-0.013, +0.016]-0.017 [-0.046, +0.011]
Random pairs-0.0017 [-0.0045, +0.0012]-0.002 [-0.014, +0.011]-0.037 [-0.063, -0.012]
CLAP only+0.0061† [+0.0025, +0.0097]+0.004 [-0.012, +0.019]-0.068 [-0.102, -0.036]
SongEval only+0.0035 [+0.0000, +0.0068]+0.037† [+0.022, +0.052]+0.000 [-0.031, +0.030]
Audiobox only+0.0005 [-0.0029, +0.0041]+0.002 [-0.013, +0.017]+0.075† [+0.043, +0.106]

With matched pairs, offline DPO changes MusicGen-small only slightly for every source: no change exceeds 3% of the untrained model's score. Each single-metric source mainly raises the metric it is trained on, and no rubric source changes any evaluator significantly. Offline DPO on MusicGen therefore cannot reliably separate reward sources, and we will revise §5.1 to say so. The comparison the paper rests on is online DiffusionNFT, which scores fresh samples at every update and where the effects are much larger: the rubric improves all three evaluators while every single-metric reward lowers at least one (Question 4), and rewarding a single criterion instead of the full rubric lowers them (Question 5).

AuthorsOpenReview ↗

Response to Reviewer eJAx (2/5): ACE-Step DPO and Flow-GRPO

Question 2 and Weakness 1

Question 2: Related to Weakness 1: the ACE-Step run trained on CLAP-ranked preferences ends up with a lower CLAP score than the untrained model (Figure 1, bottom row, "CLAP score" panel; Table 1). Is this within evaluation noise, given that the untrained model's own scores vary across evaluation runs (Table 4)? Was the flow-matching DPO surrogate verified to improve its own training metric in any setting?

Weakness 1: On ACE-Step, the CLAP- and SongEval-ranked runs decrease their own target metric (Table 1), and some rubric runs retain very few preference pairs after margin filtering (Appendix A.10). This suggests the single-timestep flow-matching DPO surrogate barely optimizes anything. A stronger offline baseline (e.g., a properly tuned Diffusion-DPO with multiple timesteps) or an online alternative such as Flow-GRPO would make this comparison meaningful.

CLAP drop. Thank you for checking this; both drops are within evaluation noise. CLAP varies even for a fixed model, because it is computed on a random 10 s crop of each clip (per-clip standard deviation 0.032 from the crop alone). The CLAP-ranked run changes CLAP by -0.0043 and the SongEval-ranked run changes SongEval by -0.0019; neither change is significant. We will add 95% intervals to Figure 1 and Table 1 so that changes within noise are visible.

Flow-matching DPO. We share this concern: the paper's ACE-Step DPO runs barely optimized. They used bf16 weights and losses to save training time and memory, and at that precision 44–75% of the policy-minus-reference margins are exactly zero, so the loss stays at ln 2. In a diagnostic on a small preference set, switching to fp32 alone is not enough; fp32 together with a 10× larger learning rate makes the same objective fit its preferences (accuracy 0.94 on training pairs, 0.88 on held-out pairs) and raises Audiobox by 0.12 (95% interval 0.05–0.21). Drawing four timesteps per pair, as the reviewer suggests, with a larger β, fits further (1.00 / 0.88; Audiobox +0.16). These gains are still smaller than those of online DiffusionNFT, which is why the paper relies on DiffusionNFT.

Relation to MR-FlowDPO and Diffusion-DPO. We agree that the surrogate is closely related to Diffusion-DPO; relating it to these models is natural, and we built on them. Our DPO objective is the flow-matching DPO used for music by MR-FlowDPO, which the paper cites. MR-FlowDPO adapts Diffusion-DPO to flow-matching text-to-music generation: a logistic loss on how much more the policy than a frozen reference reduces the flow-matching error of the preferred sample relative to the rejected one, at one shared timestep per pair. It differs only in implementation choices: we measure the error in x_0 space (the velocity error times σ^2) and sample timesteps from ACE-Step's logit-normal density. We will describe and cite it as such.

Flow-GRPO. Flow-GRPO was the first online optimizer we tried, on the tempo task, and its results were not good enough, which is why we switched to DiffusionNFT. For this response we also ran it on instrument and key, and again on tempo, with the same rewards, prompts and training budget as the paper's DiffusionNFT runs. It comes close on tempo but barely moves instrument and key accuracy.

Table 2. ACE-Step v1, attribute tasks with objective rewards (instrument tagger, key detector, beat tracker), same reward, prompts and training budget for both optimizers: change in attribute accuracy relative to the untrained model at the end of training. Bold: the better optimizer.

TaskDiffusionNFTFlow-GRPO
Instrument accuracy+0.124+0.005
Key accuracy+0.069+0.002
Tempo accuracy+0.232+0.176

We will add this comparison to §5.2.

AuthorsOpenReview ↗

Response to Reviewer eJAx (3/5): rater reward and single-metric baselines

Question 3

Related to Weakness 2: please show the rater's own rubric reward over training for the six MusicCaps DiffusionNFT runs. This would show whether the reward rose in all six runs, including those where CLAP dropped, i.e., whether the optimizer succeeded and the reward was exploited.

Thank you for this suggestion; we will add this table to the paper.

Table 3. Change relative to the untrained model at the end of training. Rater reward: the rater's weighted rubric score (0–1 scale) on the validation prompts; CLAP, SongEval and Audiobox: on the test prompts, as in Question 4. Bold: the rater that passes the pre-training checks.

Run (DiffusionNFT, rubric reward)ΔRater rewardΔCLAPΔSongEvalΔAudiobox
Music Flamingo, joint+0.055-0.0393-0.225-0.921
Music Flamingo, dimension-wise+0.011+0.0127+0.053+0.192
Qwen2-Audio, joint+0.077-0.0581+1.198+0.885
Qwen2-Audio, dimension-wise+0.105-0.0758+0.606+0.514
Qwen3-Omni, joint+0.076+0.0080+0.020+0.418
Qwen3-Omni, dimension-wise+0.086+0.0180+0.106+0.338

The rater's own reward rises in all six runs, so the optimizer succeeds every time. Where CLAP drops, the reward was exploited, as the reviewer suspected: with Qwen2-Audio, and with Music Flamingo under joint prompting, the reward rises while CLAP falls. With Qwen3-Omni the reward rises together with all three evaluators. These failures can be detected before training, from clips of the untrained model only: Music Flamingo under joint prompting gives seven identical scores on 99.6% of clips, and Qwen2-Audio's prompt-alignment score is unrelated to CLAP (within-prompt correlation -0.02, versus 0.23 for Qwen3-Omni). We will revise the claim at line 450 accordingly: with a reliable rater, the rubric reward avoids the trade-offs of single-metric rewards (Question 4).

Question 4 and Weakness 2

Question 4: Can you add single-metric DiffusionNFT baselines (CLAP, SongEval, Audiobox-Aesthetics) on ACE-Step, so that the rubric vs. single-metric comparison of §5.1 is also made under the online optimizer, where effects are large?

Weakness 2: [...] and no single-metric DiffusionNFT baseline is provided to show that the rubric does better than a single metric under online optimization.

We agree that this comparison matters, and we ran it. We trained DiffusionNFT on ACE-Step with CLAP, SongEval or Audiobox alone as the reward, under the paper's training budget, and evaluated every run at the end of training, as for the rubric.

Table 4. ACE-Step v1 post-trained with DiffusionNFT: change relative to the untrained model at the end of training.

RewardΔCLAPΔSongEvalΔAudiobox
Qwen3-Omni rubric, dimension-wise+0.0180+0.106+0.338
CLAP only+0.0345-0.007-0.123
SongEval only-0.1264-0.443-1.466
Audiobox only-0.0216-0.158+0.700

Training on one evaluator does not improve the music as judged by the others: each single-metric reward lowers at least one other evaluator, and SongEval-only lowers all three. The Qwen3-Omni rubric, which contains none of the three evaluators, improves all three.

Reviewer_hpbdOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies rubric-based rewards from off-the-shelf audio-language models (ALMs) as post-training signals for text-to-music generation. An ALM scores each generated clip on a seven-dimension rubric; the scores become preference pairs for DPO on MusicGen-small and ACE-Step v1, and scalar rewards for DiffusionNFT on ACE-Step. The main findings are: (1) on MusicGen-small, DPO preferences built from any single automatic metric improve that metric but degrade at least one other, while rubric-based preferences improve CLAP, SongEval, and Audiobox-Aesthetics simultaneously; (2) on ACE-Step, DiffusionNFT extracts far more from the same rubric than the DPO surrogate, but the outcome depends heavily on the rater and the prompting strategy, and a rising rater score does not guarantee rising external metrics; (3) for tempo, key, and instrumentation, directly optimizing a dedicated measurement clearly outperforms rubric rewards. The authors conclude with a division of labor: rubrics for perceptual qualities, objective rewards wherever a reliable measurement exists.

Critical strengths
  1. This paper is very thorough and careful. The grid of two generators, two optimizers, three raters, and two prompting strategies under one shared protocol is more complete than most post-training papers in this area, and this completeness is exactly what makes the conclusions credible rather than anecdotal. The engineering rigor is also rare: rater screening before training (score collapse, attribute separation, counterfactual swaps), strict output parsing, full rubrics and rating prompts in the appendix, and complete hyperparameter reporting.

  2. Several findings are genuinely impressive. The clearest is that rubric preferences improve CLAP, SongEval, and Audiobox-Aesthetics simultaneously on MusicGen-small while every single-metric run trades off; anyone who has tried multi-objective preference optimization knows this is not easy to get. The negative results are equally valuable: Music Flamingo collapsing all seven dimension scores into one value under joint prompting while its own reward keeps rising, and the longer run where external metrics fall as the rater's score keeps increasing.

Critical weaknesses

The work is essentially an empirical study, and the methodological contribution is thin. Both main ingredients are off the shelf: the rubric-based ALM reward follows the line of PAM, AudioJudge, and AnyAudio-Judge, and the two alignment methods (DPO and DiffusionNFT) are existing optimizers applied as-is. The findings point to useful directions, but the paper stops at documenting them.

To be clear, the empirical study itself is highly detailed and carefully done. But the contribution ends up being a thorough map of the design space rather than a new method or a new understanding of why things go wrong. For this reason, I lean toward a weak rejection, though I wouldn't mind if they were accepted.

Questions for authors

Why was there no comparison against a weighted combination of automatic metrics as the preference source, either the three evaluators themselves (CLAP, SongEval, AudioBox) with proper normalization, or a disjoint set of off-the-shelf rewards such as the MR-FlowDPO combination? Given that multi-metric scalarization is known to be difficult in DPO, this baseline seems necessary to separate "broad rewards transfer" from "ALM rubrics specifically transfer".

5 comments
AuthorsOpenReview ↗

Response to Reviewer hpbd (1/2): combined-metric baseline

We thank the reviewer for a careful and generous review. We appreciate their recognition of the thoroughness and rigor of the study and the value of its negative results. We also value the reviewer's suggestions, particularly around the combined-metric baseline and the positioning of our contribution. Here, we respond to each point that was raised.

Question 1

Why was there no comparison against a weighted combination of automatic metrics as the preference source, either the three evaluators themselves (CLAP, SongEval, AudioBox) with proper normalization, [...]? Given that multi-metric scalarization is known to be difficult in DPO, this baseline seems necessary to separate "broad rewards transfer" from "ALM rubrics specifically transfer".

This is a good question, and we ran the requested baseline (Table 1). First, a clarification of the role of the three evaluators. Our central claim is not that rubric rewards raise CLAP, SongEval and Audiobox. Post-training music generation needs a reward that captures aspects of musical quality that no single automatic metric measures, and we propose structured ALM rubrics as that reward. The three metrics are independent evaluators, not the objective: in the rubric experiments the reward is only the ALM's weighted dimension scores, so CLAP, SongEval and Audiobox are never part of it, and they test whether optimizing the rubric brings improvements that an outside measure confirms. Once a metric is part of the reward, a higher score on it shows that the optimizer fit the reward, not that the music improved.

For the baseline, we trained DiffusionNFT on ACE-Step with the equal-weight sum of the three z-normalized evaluators as the reward and, as further baselines, with each evaluator alone. All runs use the paper's training budget and are evaluated at the end of training, as for the rubric, with no checkpoint selected on the evaluators.

Table 1. ACE-Step v1 post-trained with DiffusionNFT: change relative to the untrained model at the end of training. The multi-metric row is the mean of three runs.

RewardΔCLAPΔSongEvalΔAudiobox
Qwen3-Omni rubric, dimension-wise+0.0180+0.106+0.338
CLAP only+0.0345-0.007-0.123
SongEval only-0.1264-0.443-1.466
Audiobox only-0.0216-0.158+0.700
Multi-metric (z-sum of CLAP, SongEval, Audiobox)-0.049-0.253-0.426

Each single-metric reward lowers at least one of the other evaluators, and SongEval-only lowers all three, including SongEval itself. The combined reward does not avoid this: by the end of training it is below the untrained model on all three. The Qwen3-Omni rubric, which contains none of the three evaluators, improves all three. Metric rewards, single or combined, end up fitted to the metrics they contain, while a reliable ALM rubric improves evaluators it was never trained on. The rubric is also not a re-weighting of the metrics: within a prompt, the rubric and the combined reward rank the same candidates very differently (Kendall τ≤0.19).

Rubric versus the ALM alone. The cleanest test of "ALM rubrics specifically transfer" keeps the ALM and removes the rubric. With the same rater (Qwen3-Omni), budget and seed, we replaced the rubric by a single overall-quality question. The optimizer raises the rater's own score (+0.112 on a 0–1 scale), yet SongEval (-0.170) and Audiobox (-0.587) fall significantly and CLAP does not improve; rewarding only the rubric's prompt-alignment dimension gives the same picture (-0.112 and -0.569). The same ALM improves the independent evaluators only when its judgment is structured by the multi-dimensional rubric.

AuthorsOpenReview ↗

Response to Reviewer hpbd (2/2): MR-FlowDPO and contribution

Question 1

[...] or a disjoint set of off-the-shelf rewards such as the MR-FlowDPO combination?

We thank the reviewer for bringing up MR-FlowDPO, which we built on: our ACE-Step DPO objective follows its flow-matching formulation. Its reward combination, however, is not disjoint from our evaluators: two of its three rewards are a music CLAP text-alignment score and Audiobox-Aesthetics production quality. Training on it would put CLAP and Audiobox into the reward, so it could not test transfer. More generally, any combination of automatic metrics, disjoint or not, works against our central claim: such metrics capture only part of musical quality, and optimizing them rewards the measured proxies rather than the qualities a rubric describes. Our experiments ask whether a reward improves quality as judged by evaluators it was not trained on. Putting the evaluators into the reward would make the evaluation in-domain: their scores would then show how well the optimizer fit the reward, not whether the reward generalizes. We will discuss MR-FlowDPO in the revision.

Weakness 1

The work is essentially an empirical study, and the methodological contribution is thin. Both main ingredients are off the shelf: the rubric-based ALM reward follows the line of PAM, AudioJudge, and AnyAudio-Judge, and the two alignment methods (DPO and DiffusionNFT) are existing optimizers applied as-is. The findings point to useful directions, but the paper stops at documenting them.

To be clear, the empirical study itself is highly detailed and carefully done. But the contribution ends up being a thorough map of the design space rather than a new method or a new understanding of why things go wrong.

We are glad the reviewer found the study careful, and we agree that we do not propose a new optimization algorithm. DPO and DiffusionNFT are used as published, and we keep them fixed so that differences between runs come from the reward. Our contribution is on the reward side: how to build a reward for post-training music generation from structured rubrics scored by an off-the-shelf ALM. PAM, AudioJudge and AnyAudio-Judge show that ALMs can be good evaluators of audio; we use them as training rewards for text-to-music generation. To our knowledge, this is the first work to use ALM rubric rewards for reinforcement learning of music generation models; related work with ALM rewards targets general audio (e.g., Resonate for text-to-audio). With DiffusionNFT, such a reward improves all three independent evaluators, which no metric reward we tested does within the same training budget.

The study also explains what goes wrong with the rewards in common use. The single automatic metrics that most music-generation work relies on capture only part of musical quality: optimizing any one of them lowers at least one of the others, so a gain on one metric need not mean better music. Combining them does not solve this (Question 1). The study also shows why a structured rubric is needed: when the same ALM is asked one holistic question, or only about prompt alignment, the optimizer raises the rater's score while the independent evaluators fall (Question 1). Unstructured or single-criterion judgments are exploited by the optimizer; a reliable rater's multi-dimensional rubric is not, within the same budget.

We hope these results and clarifications address the reviewer's concerns about the baseline and the contribution.

AuthorsOpenReview ↗

Three-seed results for Question 1: the rubric transfers, metric rewards do not

As the discussion closes, we would like to share a result that directly strengthens our answer to the reviewer's Question 1. We trained the DiffusionNFT comparison with three independent seeds (details in our reply to Reviewer eJAx; revised Tables 1 and 9).

The rubric transfers; metric rewards do not, in every seed. The Qwen3-Omni rubric reward, which contains none of the evaluators, improves CLAP, SongEval and Audiobox in each of the three seeds (mean CLAP +0.031 [+0.003, +0.060], Audiobox +0.434 [+0.208, +0.660]). SongEval-only and Audiobox-only each lower at least one other evaluator in every seed, and SongEval-only lowers all three. This separates "ALM rubrics specifically transfer" from "broad rewards transfer" with independent seeds.

Why the rubric's structure matters. The same ALM asked one holistic question gives seed-dependent outcomes (one seed improves all three evaluators, the other two lower SongEval and Audiobox; this updates our earlier single-run result in "Rubric versus the ALM alone"), whereas the structured rubric improves them consistently. This bears on the reviewer's point about a new understanding of why things go wrong: metric rewards are exploited (SongEval-only's own reward peaks and then collapses in every seed), and an unstructured ALM judgment is unreliable, while a multi-dimensional rubric scored by a reliable rater is neither.

Revised PDF. The revision is uploaded, with all changes from the original submission shown in blue.

We would be grateful if the reviewer would consider these results in their final assessment.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of new results and planned revisions

We thank all three reviewers for their careful and constructive reviews. In response, we ran the requested experiments and summarize the main new results here; details are in our response to each reviewer.

1. Metric-reward baselines under DiffusionNFT (hpbd Q1; eJAx Q4). We trained DiffusionNFT on ACE-Step with CLAP, SongEval or Audiobox alone, and with their z-normalized sum, under the paper's training budget. Each single-metric reward lowers at least one of the other evaluators, and the combined reward ends below the untrained model on all three. The Qwen3-Omni rubric, which contains none of these evaluators, improves all three (CLAP +8.4%, SongEval +5.4%, Audiobox +5.4%).

2. The rubric structure, not the ALM alone, drives the gains (hpbd Q1; eJAx Q5; 8QxN). With the same rater, budget and seed, replacing the rubric by a single overall-quality question, or rewarding prompt alignment alone, raises the rater's own score but lowers SongEval and Audiobox. With uniform weights, the rubric still improves the evaluators and lowers none.

3. Rater reliability (eJAx Q3; 8QxN). In all six rubric runs the optimizer raises the rater's reward. Where CLAP falls, an unreliable rater's reward was exploited, and these raters can be identified before training from clips of the untrained model.

4. Offline DPO (eJAx Q1, Q2; 8QxN). A matched replication with three seeds per source shows that offline DPO on MusicGen cannot reliably separate reward sources, so we revise the MusicGen claim in §5.1 and rest the main claim on online DiffusionNFT. Under the same budget, DiffusionNFT also clearly outperforms Flow-GRPO.

We will incorporate these results into the revised paper, together with the requested clarifications (pairing rule, margins, score anchors, a description of the rubric) and the related work raised by the reviewers (Resonate, Rubrics as Rewards, RubricRL, Diffusion-DPO, MR-FlowDPO, Tango 2, TangoFlux). We are happy to answer any further questions during the discussion.

AuthorsOpenReview ↗

Revised PDF uploaded

We have uploaded a revised PDF with changes shown in blue. The revision adds the analyses the reviewers requested: metric-reward baselines and same-ALM ablations under DiffusionNFT (§5.2, Table 1), three independent seeds for the main comparison (Tables 1 and 9), confidence intervals for all results (Tables 2–4), the matched DPO replication and precision diagnosis (Tables 7–8), and the requested related work and clarifications (§2, §4). Note that the single-metric DPO table our responses called "Table 1" is now Table 2.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.