- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a framework for autoregressive music generation. The contributions are three folds: 1. MusicGo, a process reward model inspired by value function learning in Alpha Go. 2. a new dataset with annotated coarse quality and fine-grained highlight and degradation. 3. a passage-level variant of GRPO that interleaves music generation, reward evaluation, and policy updates, enabling localized optimization before a complete track is generated.
Critical strengths
- The paper addresses the challenge of learning fine-grained process rewards for music generation, where quality is subjective, temporally dependent, and difficult to evaluate using only outcome-level supervision. The proposed MusPR dataset combines large-scale coarse quality labels with localized human annotations of musical highlights and degradations. This provides a potentially valuable resource for studying process reward modeling in music generation, a relatively underexplored area compared with language-based reasoning tasks.
- The paper demonstrates the usefulness of temporally resolved rewards across several settings, including inference-time sampling, reinforcement learning, and localized rewriting. In particular, the human preference results suggest that passage-level reward guidance can outperform whole-track reward-based selection, while PassGRPO shows encouraging improvements over Process GRPO in direct listening comparisons. These findings support the practical value of fine-grained reward modeling, even though the individual contributions of TD regularization and local supervision remain insufficiently established.
Critical weaknesses
-
A central claim of the paper is that MusicGo learns fine-grained process rewards through AlphaGo-inspired TD learning. However, the training objective combines global quality supervision, synthetic degradation signals, explicit human-annotated local rewards, and TD regularization. Without appropriate ablation studies, it remains unclear whether the improvements are attributable to TD learning or primarily to the local supervision. Moreover, the proposed TD loss appears to enforce consistency between adjacent value predictions rather than directly learning expected future returns through Bellman updates. This raises concerns about the paper's central methodological contribution. Comparisons against global-outcome-only value learning and conventional temporal smoothing would be particularly important.
-
PassGRPO introduces several modifications simultaneously, including passage-level advantages, immediate policy updates, reward-guided continuation selection, and a final song-level optimization step. However, the experiments do not adequately disentangle these components. In particular, it is unclear whether the improvements over Process GRPO arise from passage-level policy optimization or simply from reward-guided selection among multiple candidate continuations. Furthermore, the performance gains are not consistent across passage lengths and reward model configurations. These limitations make it difficult to assess the necessity and general effectiveness of the proposed optimization algorithm, which constitutes another major contribution of the paper.
Questions for authors
Q1. What is the specific contribution of TD learning to MusicGo?
Q2. Which components of PassGRPO account for its performance improvements?
Q3. How do the authors explain the inconsistent improvements across reward model stages and passage lengths?
In Table 2, Stage-II MusicGo does not consistently outperform Stage-I MusicGo, and PassGRPO does not uniformly improve over Process GRPO across different passage lengths. Could the authors clarify the reasons for these variations and provide statistical uncertainty estimates for the main human preference comparisons? This would help assess the robustness and generality of the reported improvements.
3 comments
The rebuttal has partially addressed my concerns. I remain
I appreciate the author's feedback. The rebuttal partially addressed my concern about the effectiveness of TD and ablation study. Overall, this paper contains interesting idea and formulation for music generation. Including the experimental analysis of TD in the final version can strengthen the paper. Also, it is better to add more ablation studies to showcase the effectiveness of each component. I therefore maintain my positive rating.
We sincerely thank the reviewer for their positive assessment, constructive feedback, and continued support. We will incorporate the additional TD analysis into the final version and further strengthen the component-wise ablation studies where feasible.
We sincerely thank the reviewer for the positive assessment and constructive suggestions. We conducted two additional controlled evaluations to further investigate the contributions of TD regularization and immediate passage-level optimization.
1. Contribution of TD Regularization
We conducted a controlled TD vs. no-TD ablation, removing only the TD term while keeping all other training objectives, data, model architecture, and downstream generation settings unchanged.
Across 192 judgments, the TD variant achieves 93 wins (48.44%), 58 losses (30.21%), and 41 ties (21.35%). Excluding ties, it achieves a 61.59% preference rate (95% listener-clustered bootstrap 95% CI: [52.87%, 69.33%]; track-clustered CI: [54.93%, 67.74%]). These results provide perceptual evidence for the contribution of TD regularization beyond the remaining objectives.
2. Contribution of Immediate Passage-level Updates
To isolate the contribution of immediate passage-level updates, we conducted a controlled Full PassGRPO vs. Delayed-update comparison.
Delayed-update postpones each passage-level policy update until the entire track has been generated, so subsequent passages are sampled without incorporating gradients from earlier passages.
Both variants share the same frozen Stage-II MusicGo, 1,195 training prompts, sampling configuration (G=4, L=400, temperature =0.9, top-k=50), continuation-selection parameters (ε=0.15, τ=0.4), learning rate (5×10−7), and one training epoch. Their KL coefficients are also identical: βin=0.08 for passage-level updates and β=0.02 for the final song-level update.
The only procedural difference is when passage-level updates are applied: Full PassGRPO updates immediately after each passage, whereas Delayed-update postpones these updates until the complete track has been sampled. The resulting optimizer-step counts are 22,791 and 22,386, respectively, a difference of approximately 1.8%.
Across 24 tracks, 8 listeners, and 192 judgments, Full PassGRPO achieves 93 wins (48.44%), 56 losses (29.17%), and 43 ties (22.40%). Excluding ties, it achieves a 62.42% preference rate (95% listener-clustered bootstrap 95% CI: [56.08%, 68.63%]; track-clustered CI: [54.48%, 70.00%]). These results support the contribution of immediate passage-level optimization while controlling for continuation selection and KL settings.
3. Experimental Scope
Our existing experiments report objective evaluation results across different passage lengths and reward model stages. The additional listening tests provide complementary perceptual evidence for the principal design choices.
We acknowledge that further component-wise ablations would provide a more comprehensive understanding of individual design choices, and will endeavor to extend these analyses in the revised work. We sincerely appreciate the reviewer's continued engagement and hope these additional controlled results address the concerns.