acceptodds
Under review as a conference paper at ICLR 2027

Reinforcement Learning for Music Generation with Pairwise Reward Modeling

Abstract

Improving text-to-music models through reinforcement learning (RL) requires a reward model that can clearly quantify the quality of a text-music pair. In this domain, this usually includes () how high-quality the outputs sound, and () how well they align with the provided text. We introduce *DuetJudge*, an on-policy pairwise reward model that learns to directly compare two instrumental pieces conditioned on the same caption, without assigning independent scalar scores to each. *DuetJudge* is functionally a ternary classifier that outputs either one piece wins or they are a tie. Particularly, *DuetJudge* prioritizes text-music alignment, and then considers perceptual quality of the music. *DuetJudge* achieves 71% agreement with human preferences on our internal test set, outperforming all existing score-based baselines. To use *DuetJudge* as a reward model in text-to-music RL, we () design a reward score based on aggregated pairwise preference probabilities, and () introduce classifier-free guidance (CFG)-aware policy optimization for text-to-music models trained with CFG. We test *DuetJudge* using multiple in-distrubtion and out-of-distribution text-to-music models, and find systematic improvements in both automatic metrics and human ratings. Ablation studies further favor direct pairwise rewards over a Bradley-Terry scalar baseline under the same training settings. These results confirm the effectiveness of our proposed *DuetJudge*, our data curation and augmentation methods, our reward model training recipes. and our text-to-music RL training recipes. An anonymous demo is available at https://anonymous.4open.science/w/duetjudge-demo-8E08/.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
text-to-music, music generation, reinforcement learning, reward modeling

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_WRbDOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes a method to evaluate music alignment as well as its music quality in text-to-music generation problem by using a comparative DuetJudge approach. The comparative preference scores of two audio clips are expressed in terms of probability scores that are used as reward for training RL to optimize the policy. The experimental data and procedures are clearly described and the results are slightly better in comparison to other baselines. They provide a feasible method that deal with the difficulty of the quantize the music quality and alignment of text-to-music generation problem.

Critical strengths

The paper proposes a comparative duetjudge concept as well as a RL framework into training of the evaluation model on music generation in terms of text-alignment and music quality. The experimental data and procedures are carefully and well presented and the results are slightly better than previous baselines. The method provide an alternative music evaluation method to deal with otherwise difficult to assess the specification alignment of prompt text and generated music quality.

Critical weaknesses

The correctness of RL policy learning relies on accurate reward models. converting comparative probability to a scalar reward score tends to be weak in theoretical justification. Is the converting mapping linear or non-linear (I suspect) is not clear. Intuitively, high probability indicates the high reward (or preference), low probability represents low reward. But the corresponding function is not clear by converting probability into actual reward for RL optimization. It can be sensitive if the mapping functions differ. How many comparisons are enough to collect the sufficient data for constructing the accurate reward score for all correct judgement of music quality is not addressed. The DuetJudge reward score concept somewhat borrows from Bradely-Terry model , it also inherits the weakness of the model such as the cyclic preference comparison (transitivity might not hold) problem, context sensitivity problem. In the sense, the novelty of the paper adopting Duet-Judge approach is not much difference than BT model. But the paper adopts the probability reward score and then refine with RL to obtain slightly better performance against DuetJudge-BTRM as shown in the ablation study results in Table 4 of the paper.

The comparative preference scoring of audio clips is based on 10 seconds which is a local score. How well can local preference rewards on text alignment or music quality can be accumulated into overall evaluation on a global preference concept on the generated music is still a concern. Can RL optimization model really learn the global concept quality by accumulating the local preference reward is not clearly justified. Because RL is based on dynamic programming principle, where global optimality score relies on accumulation of local optimality score while global music properties or quality might not follow. When the local scores are assessed according to the local preference with respect to certain global metric, it may not reflect accurately that cause bias. Therefore the accumulate local scores cannot reflect correct global metric score on the whole music sequence evaluation. For example if supposed a global uniformity is to be evaluated, assuming clip uniformity x == y > z, if one sequence consists of alternating clips x, y, x, y, x, y while another sequence consists of z, z, z, z, z, z, then apparently each local uniformity comparison clip wise no matter x or y is greater than z, however the global uniformity score the latter sequence is greater than former one.

To deal with Tie, there is extension called Bradley-Terry- Davidson model that assign probability score to TIE situations. The paper did not discuss with this extension.

Questions for authors

Table 2 The tie tends to be higher than other base-line models, does that imply the proposed model tend to avoid ambiguities so that the performance tend to be better (like recall and precision tradeoff)? Also as the paper writes: “We predict Tie if |P(A)−P(B)|<0.05 or P(Tie) >0.25; otherwise, we choose A or B according to which has the higher probability.”, the threshold parameter of TIE can affect the performance evaluation of the model. It is not justified how the threshold is decided (optimally?)

Reviewer_cPdnOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DuetJudge, a pairwise reward model for text-to-music generation. It compares two music clips and predicts a preference or tie. The authors construct a large-scale preference dataset and use DuetJudge to provide rewards for RL training. Experiments on several music generation models show improvements in both automatic metrics and human evaluations.

Critical strengths

The paper provides reasonably comprehensive experiments across different model sizes and architectures. The improvements in human evaluations suggest that the proposed reward model is practically useful for music generation.

Critical weaknesses
  1. Limited novelty. DuetJudge essentially follows a standard pairwise LLM-as-a-Judge formulation adapted to audio. The alignment-first criterion is mainly implemented through data construction, while the reward aggregation and RL optimization largely reuse existing techniques. I find the methodological contribution rather limited.

  2. Insufficient justification of the main contribution. The experiments do not convincingly demonstrate that joint pairwise judgment itself is responsible for the improvements, rather than the carefully constructed training data and preference supervision. The additional computational cost of pairwise comparisons is also not sufficiently justified.

Questions for authors
  1. What is the key methodological novelty of DuetJudge beyond adapting existing pairwise LLM-as-a-Judge approaches to audio?

  2. Can the authors better isolate the benefit of joint pairwise inference from the effects of training data and supervision, for example through a more strictly controlled pointwise baseline?

  3. How much additional computational cost does pairwise reward modeling introduce during RL training, and is the improvement worth this cost?

Reviewer_EW3bOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper presents a new reward model, DuetJudge, which predicts human preference labels for pairs of audio, prioritizing alignment. It provides evidence that DuetJudge both better aligns with human preferences and also improves music generation to a greater degree than past reward models via RLHF.

Critical strengths
  1. Reasonable and effective new strategies for scaling up pairwise data collection for this task, using a mixture of human and synthetic labeling.
  2. Interesting results suggesting that RLHF benefits from rewards based on pairwise judgments (Eq. 2).
Critical weaknesses

1. Lack of effective ablation between the three main contributions: new data recipes, pairwise reward modeling, and pairwise RLHF. Right now, there is no way of knowing which is contributing the most. Experiments that would help:

  1. Train both a standard Bradley–Terry (BT) reward model and a pairwise model using the same base model and the same data.
  2. Disentangle data recipes from pairwise reward modeling: run the evaluation suite (Table 2) on the standard BT model and compare it to the pairwise model. Which is better?
  3. Disentangle pairwise reward modeling from pairwise RLHF: run RLHF with
    • (1) standard BT scores as the reward, from the BT model;
    • (2) Eq. 2 rewards from the BT model;
    • (3) Eq. 2 rewards from the pairwise model.
    • Configurations (1) and (2) can also be compared with existing BT models, e.g., TuneJury.

2. No details on the human annotation process. How were 908 hours of labels collected for the "Holistic Preference" set? Were annotators paid? What was their level of musical expertise? There is documentation for the human evaluation via Prolific, but not for the core data annotation.

3. Not stated whether the models or data will be released upon publication. This raises concerns about reproducibility (there is no reproducibility statement) and about usefulness to the community without these resources.

Questions for authors

Questions

  1. Is DuetJudge-BTRM (Table 4) a standard BT reward model trained on the same data and base model as DuetJudge? If so, how does it perform on the Table 2 evaluation suite?
  2. Which of the three contributions (new data recipes, pairwise reward modeling, pairwise RLHF) do you believe contributes most to the gains, and what evidence supports this?
  3. Who annotated the Holistic Preference set? How many annotators were there, how were they recruited, were they paid (and at what rate), and what qualifies them as "music experts"?
  4. Will the DuetJudge weights, the judgment data, and the training code be released upon publication? If only some, which ones? Can you add a reproducibility statement?
  5. Is Eq. 2 novel? If prior work has used similar group-averaged pairwise rewards, how does Eq. 2 differ?

Suggestions (minor concerns)

  • Missing citations: FAD (Kilgour et al., Interspeech 2019) and MAD (Huang et al., ISMIR 2025) for using degradations to provide ground-truth ranking labels, and a citation for the claim in Figure 1 that pointwise evaluators are "prone to reward hacking as an RL reward."
  • No related work section; in particular, Equation 2 needs related work.
  • No runtime cost considerations in the evaluation (e.g., DuetJudge appears to require two forward passes per pair, due to the definition of pijp_{ij}).
  • Could explore twin networks to remove the asymmetry of the pairwise model.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.