acceptodds
Under review as a conference paper at ICLR 2027

Creating Music as a Go Player: a Process Reward Model of Music Generation

Abstract

Evaluating and aligning music generation systems remains challenged by the gap between static, outcome-level labels and the intrinsically dynamic, process-dependent nature of musical quality. Existing post-training methods rely primarily on coarse global rewards, restricting fine-grained policy optimization and leaving models incapable of localized error correction. Inspired by value function temporal difference (TD) learning in computer Go (e.g., AlphaGo)—where dense intermediate values are derived from sparse outcomes—we introduce MusicGo, a process reward model that assigns dense temporal rewards to partial musical sequences. By framing process reward estimation as predicting the likelihood that a prefix leads to a high-quality completion within a TD framework, MusicGo converts sparse track-level preferences into fine-grained process supervision with minimal human process annotations. Correspondingly, we release MusPR, the first dataset for training and validating process reward models in music generation. Furthermore, we evaluate MusicGo in offline and online reinforcement learning settings and present PassGRPO (Passage-level Group Relative Policy Optimization), an enhanced GRPO variant that executes policy updates at passage granularity to produce structurally coherent generations. By establishing frame-level reward diagnostics and passage-wise gradient updating, our framework lays a precedent for targeted music revision, localized inpainting, and fine-grained interactive editing. We make our audio samples available here.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Process Reward, Music Reward Model, Music Evaluation, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_J8EgOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a framework for autoregressive music generation. The contributions are three folds: 1. MusicGo, a process reward model inspired by value function learning in Alpha Go. 2. a new dataset with annotated coarse quality and fine-grained highlight and degradation. 3. a passage-level variant of GRPO that interleaves music generation, reward evaluation, and policy updates, enabling localized optimization before a complete track is generated.

Critical strengths
  1. The paper addresses the challenge of learning fine-grained process rewards for music generation, where quality is subjective, temporally dependent, and difficult to evaluate using only outcome-level supervision. The proposed MusPR dataset combines large-scale coarse quality labels with localized human annotations of musical highlights and degradations. This provides a potentially valuable resource for studying process reward modeling in music generation, a relatively underexplored area compared with language-based reasoning tasks.
  2. The paper demonstrates the usefulness of temporally resolved rewards across several settings, including inference-time sampling, reinforcement learning, and localized rewriting. In particular, the human preference results suggest that passage-level reward guidance can outperform whole-track reward-based selection, while PassGRPO shows encouraging improvements over Process GRPO in direct listening comparisons. These findings support the practical value of fine-grained reward modeling, even though the individual contributions of TD regularization and local supervision remain insufficiently established.
Critical weaknesses
  1. A central claim of the paper is that MusicGo learns fine-grained process rewards through AlphaGo-inspired TD learning. However, the training objective combines global quality supervision, synthetic degradation signals, explicit human-annotated local rewards, and TD regularization. Without appropriate ablation studies, it remains unclear whether the improvements are attributable to TD learning or primarily to the local supervision. Moreover, the proposed TD loss appears to enforce consistency between adjacent value predictions rather than directly learning expected future returns through Bellman updates. This raises concerns about the paper's central methodological contribution. Comparisons against global-outcome-only value learning and conventional temporal smoothing would be particularly important.

  2. PassGRPO introduces several modifications simultaneously, including passage-level advantages, immediate policy updates, reward-guided continuation selection, and a final song-level optimization step. However, the experiments do not adequately disentangle these components. In particular, it is unclear whether the improvements over Process GRPO arise from passage-level policy optimization or simply from reward-guided selection among multiple candidate continuations. Furthermore, the performance gains are not consistent across passage lengths and reward model configurations. These limitations make it difficult to assess the necessity and general effectiveness of the proposed optimization algorithm, which constitutes another major contribution of the paper.

Questions for authors

Q1. What is the specific contribution of TD learning to MusicGo?

Q2. Which components of PassGRPO account for its performance improvements?

Q3. How do the authors explain the inconsistent improvements across reward model stages and passage lengths?

In Table 2, Stage-II MusicGo does not consistently outperform Stage-I MusicGo, and PassGRPO does not uniformly improve over Process GRPO across different passage lengths. Could the authors clarify the reasons for these variations and provide statistical uncertainty estimates for the main human preference comparisons? This would help assess the robustness and generality of the reported improvements.

3 comments

We sincerely thank the reviewer for the positive assessment and constructive suggestions. We conducted two additional controlled evaluations to further investigate the contributions of TD regularization and immediate passage-level optimization.

1. Contribution of TD Regularization

We conducted a controlled TD vs. no-TD ablation, removing only the TD term while keeping all other training objectives, data, model architecture, and downstream generation settings unchanged.

Across 192 judgments, the TD variant achieves 93 wins (48.44%), 58 losses (30.21%), and 41 ties (21.35%). Excluding ties, it achieves a 61.59% preference rate (95% listener-clustered bootstrap 95% CI: [52.87%, 69.33%]; track-clustered CI: [54.93%, 67.74%]). These results provide perceptual evidence for the contribution of TD regularization beyond the remaining objectives.

2. Contribution of Immediate Passage-level Updates

To isolate the contribution of immediate passage-level updates, we conducted a controlled Full PassGRPO vs. Delayed-update comparison.

Delayed-update postpones each passage-level policy update until the entire track has been generated, so subsequent passages are sampled without incorporating gradients from earlier passages.

Both variants share the same frozen Stage-II MusicGo, 1,195 training prompts, sampling configuration (G=4G=4, L=400L=400, temperature =0.9=0.9, top-k=50k=50), continuation-selection parameters (ε=0.15\varepsilon=0.15, τ=0.4\tau=0.4), learning rate (5×10−75\times10^{-7}), and one training epoch. Their KL coefficients are also identical: βin=0.08\beta_{\mathrm{in}}=0.08 for passage-level updates and β=0.02\beta=0.02 for the final song-level update.

The only procedural difference is when passage-level updates are applied: Full PassGRPO updates immediately after each passage, whereas Delayed-update postpones these updates until the complete track has been sampled. The resulting optimizer-step counts are 22,791 and 22,386, respectively, a difference of approximately 1.8%.

Across 24 tracks, 8 listeners, and 192 judgments, Full PassGRPO achieves 93 wins (48.44%), 56 losses (29.17%), and 43 ties (22.40%). Excluding ties, it achieves a 62.42% preference rate (95% listener-clustered bootstrap 95% CI: [56.08%, 68.63%]; track-clustered CI: [54.48%, 70.00%]). These results support the contribution of immediate passage-level optimization while controlling for continuation selection and KL settings.

3. Experimental Scope

Our existing experiments report objective evaluation results across different passage lengths and reward model stages. The additional listening tests provide complementary perceptual evidence for the principal design choices.

We acknowledge that further component-wise ablations would provide a more comprehensive understanding of individual design choices, and will endeavor to extend these analyses in the revised work. We sincerely appreciate the reviewer's continued engagement and hope these additional controlled results address the concerns.

Reviewer_J8EgOpenReview ↗

The rebuttal has partially addressed my concerns. I remain

I appreciate the author's feedback. The rebuttal partially addressed my concern about the effectiveness of TD and ablation study. Overall, this paper contains interesting idea and formulation for music generation. Including the experimental analysis of TD in the final version can strengthen the paper. Also, it is better to add more ablation studies to showcase the effectiveness of each component. I therefore maintain my positive rating.

We sincerely thank the reviewer for their positive assessment, constructive feedback, and continued support. We will incorporate the additional TD analysis into the final version and further strengthen the component-wise ablation studies where feasible.

Reviewer_qhZwOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a method for training a reward function, and a corresponding RL technique based on GRPO, aimed at improving the quality of music generated by a pretrained audio LLM (i.e., an autoregressive audio-codec token prediction model). The authors propose modeling the evolving quality throughout the musical sequence, using global quality labels, synthetic degradations, and local human annotations of highlights and errors to shape the reward function. They use their reward model for inference-time guidance and RL post-training, and compare a variety of alignment techniques, including methods derived from DPO and GRPO. The authors report improved human preference in some settings, although automatic evaluation results are mixed. However, the comparisons do not isolate the effects of the proposed changes. It is therefore difficult to assess the reliability of the improvements or attribute them to the proposed method.

Critical strengths

The authors identify a natural problem in reward modeling for generative music models. Terminal-reward models may not capture local degradations, and therefore RL phases may fail to address these local degradations. The proposed approach of using a PRM, trained using a mixture of global preference labels as well as local degradation identifications, is natural. Moreover, the authors release the human labels, which could be a useful resource for future work. The authors make a reasonable attempt to compare their reward model with different RL algorithms, including time-intensive human listening tests, which is appreciated.

Critical weaknesses

There are several areas that this paper could use serious improvement.

Contributions:

There are some questions about what should be taken away from this paper. The issues identified by the authors are well studied and relevant in many different areas in which RL techniques are applied. Reward shaping, temporal difference learning, value and advantage estimation, and PRMs have significant precedent. The authors mainly build on existing approaches. Reward shaping in particular is well studied [1]. Although this paper is framed as a music application paper, I feel that the authors need to establish more clearly what the music-specific contributions are. The dataset and the construction of temporal supervision are useful contributions. However, as it stands currently, I feel that the main takeaway is that shaping rewards using track-level labels and local degradation annotations can help improve generations after RL. I judge these findings as slightly below the bar for ICLR. The experiments do not yet make clear which of the proposed changes are responsible for these improvements.

[1] Ng, A. Y., Harada, D., and Russell, S. (1999). Policy invariance under reward transformations: Theory and application to reward shaping. ICML.

Evaluations:

The authors propose a reward function which specifies per-token rewards. This is compatible with standard PPO-derived policy gradient optimization techniques. Moreover, PPO typically uses GAE as well as value estimation, so PPO using the same MusicGo rewards would be an important comparison to include in the ablation experiments. From my perspective, this would help establish the benefits of PassGRPO over a standard approach to temporal credit assignment.

The experimental results are also somewhat mixed, making it hard to draw concrete conclusions about the proposed changes. The results of table 3 suggest a more modest benefit than is claimed l415-417. Furthermore, when listening to the supplement, from my perspective it is quite hard to hear massive differences between the different techniques, however this is of course subjective.

Presentation:

The introduction and abstract could benefit from some presentation improvements. The paper focuses on a procedure for developing a reward model for an autoregressive audio-codec LLM, and a downstream RL technique. The introduction should explain more concretely which aspects of musical quality the techniques aim to improve, how the track-level labels and local human annotations inform the reward model, and briefly summarize the main results. It could also better situate these contributions within existing approaches to reward shaping, TD learning, and RL for music generation.

There are also some minor presentation concerns, which I'll list here:

l25-26: 'enhanced GRPO variant'. What is enhanced here?

l69-70: 'offering a paradigm for interactive human-AI co-creation'. Be more concrete.

l141: 'Suno (Muse; Jiang et al., 2026)'. Citation formatting unclear.

l173-175: Awkwardly integrated Llama and LeVo citations.

Questions for authors

Did you compare PassGRPO to PPO? What were the results?

1 comment

We sincerely thank the reviewer for the thoughtful feedback. We clarify our music-specific contributions and provide additional experimental evidence.

Q1. Music-specific Contributions and Novelty

Our contribution is not a new general-purpose RL algorithm, but a process-reward framework tailored to subjective, long-form music generation, where intermediate quality is difficult to evaluate and local improvements do not necessarily guarantee global musical coherence.

Our contributions are threefold:

  • MusPR: A dataset containing 36,098 tracks, with 1,281 unique tracks and 1,274 effective fine-grained annotations, providing timestamped human feedback on musical highlights and degradations.
  • MusicGo: A causal prefix-level reward model integrating global quality supervision, synthetic degradations, temporal regularization, and human process feedback.
  • PassGRPO: A passage-level optimization framework that interleaves generation, reward evaluation, and policy updates.

Our contribution lies in adapting and integrating established mechanisms to address the temporal and perceptual challenges of music generation, rather than claiming entirely new reward-shaping or RL principles.

Q2. TD Ablation and MusicGo Evaluation

Our TD objective encourages discounted temporal consistency across adjacent segment boundaries, while Stage II incorporates additional human-annotated temporal supervision.

To isolate the contribution of TD regularization, we conducted a controlled TD vs. no-TD ablation, removing only the TD term while keeping all other training and generation settings unchanged. Across 192 judgments, the TD variant receives 93 wins, 58 losses, and 41 ties. Excluding ties, it achieves a 61.59% preference rate (95% listener-clustered bootstrap 95% CI: [52.87%, 69.33%]), providing perceptual evidence for the contribution of TD regularization.

Additionally, on MusPR-S2-Eval (n=60n=60), the pooled Stage-II MusicGo used for RL assigns average scores of 0.655 (95% CI: [0.624, 0.686]) to highlights and 0.396 (95% CI: [0.375, 0.413]) to degradations. This demonstrates clear region-level score discrimination, although precise event localization remains challenging.

Q3. PPO+GAE and Additional PassGRPO Evaluation

We have not conducted a separate PPO+GAE comparison. Our primary focus is GRPO-based optimization, which estimates advantages through relative rewards among sampled candidates without requiring an additional trained value critic. This avoids the extra memory, computation, and optimization complexity of critic training, particularly relevant to long-form autoregressive music generation.

Process GRPO is therefore a more directly relevant baseline for examining passage-level optimization, as it shares the group-relative framework and frozen MusicGo reward model with PassGRPO.

To further investigate immediate passage-level updates, we conducted a Full PassGRPO vs. Delayed-update comparison. Delayed-update postpones passage-level policy updates until the complete track has been generated, so subsequent passages cannot benefit from earlier updates during generation. Both variants retain identical sampling, continuation selection, KL coefficients, and other training hyperparameters.

Across 24 tracks, 8 listeners, and 192 judgments, Full PassGRPO achieves 93 wins, 56 losses, and 43 ties, yielding a 62.42% non-tie preference rate (95% listener-clustered bootstrap 95% CI: [56.08%, 68.63%]). The two variants differ by approximately 1.8% in optimizer steps.

We sincerely appreciate the reviewer's feedback and respectfully invite reconsideration of the overall assessment in light of these clarifications and additional results.

Reviewer_o6UgOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to provide feedback on music quality during generation, rather than assigning a single score to a finished track. It introduces MusicGo, a causal reward model that scores audio prefixes, and MusPR, a dataset combining track-level quality labels with timestamped human annotations of highlights and degradations. MusicGo is trained first on coarse labels and synthetic degradations, then refined using human feedback. The paper also proposes PassGRPO, which alternates between generating candidate passages, scoring them, and updating the policy before continuing the track. Experiments cover reward-guided sampling, policy optimization, personalization, and local rewriting. Some listening-test results are encouraging, but the evaluation does not yet establish how well the reward model identifies local quality changes or which parts of PassGRPO account for its gains.

Critical strengths
  1. A track-level score gives little indication of where a generated song goes wrong. Collecting timestamped highlights and degradations is a sensible way to study this problem. The annotation protocol includes reaction-time calibration and event clustering, and it leaves unmarked regions unannotated rather than treating them as negative examples. This dataset could be a useful contribution if its composition and splits are clarified.
  2. In Table 3, Stage-II PassGRPO with L = 400 wins against the corresponding Process GRPO model in 66.48% of judgments and loses in 26.70%. This direct comparison is a positive result. It gives a reason to take the approach seriously, even though the missing sample counts and experimental controls limit what can be concluded from it.
Critical weaknesses
  1. Section 6.3 mainly evaluates the terminal score wT against a track-level binary label. That does not show whether intermediate scores, or their changes, correctly identify local highlights, degradations, and recovery. I could not find a quantitative evaluation against held-out temporal annotations. The rewriting experiments have a related limitation. MusicGo identifies low-scoring regions, selects candidate edits, and then supplies the main metrics used to report improvement. Higher Join WR or Last WR therefore shows that the edits receive better scores from the selection model; it does not independently establish that listeners prefer the edits or find the transitions more natural. Table 5 does not report a listening test for this claim. The training description also leaves an important gap. Section 3 constructs dense targets ending at VT, but Section 4 does not clearly explain how those targets enter the optimization objective. The stated TD loss encourages discounted consistency between neighboring values. Its contribution, beyond the local annotation losses and other regularizers, remains unclear without an ablation.
  2. Listening tests are the primary evaluation, but the paper does not adequately report the number of tracks and listeners in each comparison, repeated judgments per track, aggregation procedure, or uncertainty. PassGRPO also changes several things at once: update frequency, continuation selection, advantage construction, and KL settings. Without matched budgets and ablations, the improvement cannot be attributed specifically to updating the policy passage by passage. Some claims are stronger than the reported results. Section 6.3 says personalized steering consistently outperforms both B0 and the pooled model. In Table 4, however, annotator A's model wins against B0 in 33.33% of judgments and loses in 41.67%; against the pooled model, its wins and losses are equal. Annotator B also has equal win and loss rates against the pooled model. Similarly, Table 3 reports equal win and loss rates for Stage-II Process GRPO against Sample GRPO, which does not support a general claim that process supervision systematically outperforms outcome supervision. The dataset counts also need reconciliation. The main text reports 1,274 tracks with fine-grained annotations, while Appendix C.2.3 reports 1,270 complete annotations and 1,281 unique tracks. The relationship between the pooled evaluation size of n = 64 and the per-annotator holdout of n = 60 is not explained.
Questions for authors
  1. How do the dense targets and terminal target VT constructed in Section 3 enter the losses in Section 4? How are continuous human ratings and the Stage-I pairwise loss handled during Stage II? What quantity is wt intended to estimate?
  2. Do you have held-out results measuring agreement between local score changes and human temporal annotations? Are there ablations removing the TD term or the local human annotations, and independent listening-test results for the rewriting experiments?
  3. Could you provide the raw win/loss/tie counts, track counts, listener counts, and confidence intervals for the main A/B comparisons? Are PassGRPO and Process GRPO matched for sampling, reward evaluation, and optimization budgets? Does the advantage remain when continuation selection and KL settings are controlled?
1 comment

We sincerely thank the reviewer for the constructive feedback and for recognizing the value of MusPR and PassGRPO. We provide the following clarifications and additional results.

1. Clarification of MusicGo Training Objectives

Stage I combines binary quality supervision, synthetic degradation constraints, and temporal regularization. Stage II retains the complete Stage-I objective and adds human highlight supervision. The continuous work-quality ratings are normalized to construct the terminal reference VTV_T and the corresponding temporal trajectory. Its polarity provides binary supervision for the inherited alignment and classification objectives. Stage II retains TD regularization and the Stage-I pairwise objective, while degradation and highlight annotations provide additional local supervision through Lsoft\mathcal{L}_{\mathrm{soft}} and LH\mathcal{L}_H, respectively.

Importantly, wt=σ(ut)w_t=\sigma(u_t) is a bounded causal prefix-level quality estimate, not a Bradley–Terry preference probability. We will clarify these supervision pathways in the revision.

2. Temporal Evaluation, TD Ablation, and Rewriting

On MusPR-S2-Eval (n=60n=60), the pooled Stage-II MusicGo checkpoint used for RL assigns average scores of 0.655 (95% CI: [0.624, 0.686]) to highlights and 0.396 (95% CI: [0.375, 0.413]) to degradations. This clear score separation demonstrates its ability to distinguish human-annotated high-quality and degraded regions. Nevertheless, precise event localization remains challenging.

We conducted a controlled TD vs. no-TD ablation, removing only the TD term while keeping all other training and generation settings unchanged. Across 192 judgments, TD achieves 93 wins, 58 losses, and 41 ties. Its non-tie preference rate is 61.59% (95% listener-clustered bootstrap CI: [52.87%, 69.33%]; track-clustered CI: [54.93%, 67.74%]).

We also conducted an independent rewriting A/B test on 24 tracks with 6 listeners (144 judgments per comparison):

  • Continue-SFT vs. Original: 84/45/15 wins/losses/ties (58.33%/31.25%/10.42%).
  • Cloze-SFT vs. Original: 52/49/43 (36.11%/34.03%/29.86%).
  • Continue-SFT vs. Cloze-SFT: 79/28/37 (54.86%/19.44%/25.69%).

These results support Continue-SFT over the original, while Cloze-SFT shows less consistent benefits.

3. PassGRPO Evaluation and Experimental Controls

Our main A/B evaluation covers 24 tracks, 16 listeners, and 352 judgments. Stage-II PassGRPO (L=400L=400) achieves 234 wins (66.48%), 94 losses (26.70%), and 24 ties (6.82%) against Process GRPO. The all-judgment win rate has a 95% listener-clustered bootstrap CI of [61.93%, 71.02%], consistent with the track-clustered CI of [61.65%, 71.02%].

Both methods share the SFT initialization, frozen Stage-II MusicGo, sampling configuration (G=4G=4, temperature =0.9=0.9, top-k=50k=50), learning rate (5×10−75\times10^{-7}).

To more directly examine immediate passage-level updates, we conducted an additional Full PassGRPO vs. Delayed-update comparison. Delayed-update postpones each passage-level policy update until the entire track has been generated, so subsequent passages are sampled without incorporating gradients from earlier passages. Both variants retain identical sampling, continuation-selection parameters, KL coefficients, and other training hyperparameters. The only procedural difference is whether passage-level policy updates are applied immediately or postponed until the complete track has been sampled.

Across 24 tracks, 8 listeners, and 192 judgments, Full PassGRPO achieves 93 wins, 56 losses, and 43 ties, corresponding to a 62.42% non-tie preference rate (95% listener-clustered bootstrap CI: [56.08%, 68.63%]; track-clustered CI: [54.48%, 70.00%]).

4. Dataset Statistics and Clarifications

MusPR contains 1,281 unique tracks, of which 1,274 have effective fine-grained annotations. The previously reported 1,270 annotations and pooled evaluation size of n=64n=64 were typographical errors; the correct evaluation size is n=60n=60. We apologize and will correct these inconsistencies.

We also acknowledge that personalized steering and process supervision do not uniformly outperform their respective baselines across all configurations. We will moderate these claims accordingly. We sincerely appreciate the reviewer's detailed feedback and hope these clarifications and additional results address the main concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.