- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates why reinforcement learning is more effective than SFT at improving compositional generalization in large language models, namely, combining existing atomic skills to solve more complex problems. The authors decompose training mechanisms along two axes, data source and loss function, to separately examine the effects of on-policy sampling, negative gradients, the group-mean baseline, and standard-deviation normalization. Experiments on string-function composition and competition math assess whether models can generalize from shallower function compositions or easier math problems to more difficult tasks. The results show that, under the paper’s experimental settings, combining on-policy sampling with negative gradients supports sustained and stable generalization gains. The group-mean baseline contributes substantially on the string-function task, while standard-deviation normalization provides smaller additional gains. Through gradient analysis, the authors explain how contrasting successful and failed trajectories can highlight critical reasoning steps that determine success or failure, while on-policy data directs updates toward errors the model would actually make. Analysis of training dynamics also shows that on-policy SFT may lead to entropy collapse and shorter responses, whereas applying negative gradients to off-policy data may cause sharp increases in gradient norms and degenerate outputs.
Critical strengths
Clear analytical framework and informative component decomposition. The paper organizes post-training methods along two axes, data source and loss function, and progressively compares the effects of negative gradients, the group-mean baseline, and standard-deviation normalization. It also identifies task-dependent contributions of these components, helping clarify the performance differences between SFT and GRPO.
Experiments cover both controlled and realistic tasks, with findings extending beyond GRPO. String-composition tasks allow explicit control over composition depth, while competition math evaluates generalization from easier to harder problems. A simple REINFORCE objective with positive and negative rewards also achieves sustained gains, providing broader empirical support for the effectiveness of combining on-policy sampling with negative gradients.
Theoretical interpretation and training-dynamics analysis complement each other. The authors use gradient analysis to explain the role of group centering while examining response length, entropy, and gradient norms to analyze generalization performance and training instability across methods. These analyses provide concrete insights for understanding and diagnosing post-training behavior.
Critical weaknesses
Insufficient empirical evidence for the core mechanism. The explanation that gradients concentrate on critical reasoning steps relies on specific assumptions, while response length, entropy, and overall gradient norms provide only indirect evidence and cannot directly confirm this mechanism. The appendix also notes that, under strictly on-policy sampling, several appropriately corrected estimators have the same expected gradient. Therefore, performance differences may also arise from finite-sample effects and optimization dynamics.
Limited coverage of mathematical evaluation. The paper primarily evaluates on difficulty-based splits within the DeepScaleR dataset, without validation on independent standard mathematical benchmarks. It therefore remains unclear whether its conclusions generalize to other mathematical tasks.
Questions for authors
Can you provide experimental evidence that gradient updates indeed concentrate on critical reasoning steps? Given that some corrected estimators have the same expected gradient under strictly on-policy sampling, how can the proposed mechanism be distinguished from finite-sample effects and optimization dynamics in explaining the performance differences?
Are there results on independent standard mathematical benchmarks beyond DeepScaleR, such as MATH500, AIME24 and AIME25, that support the advantages of combining on-policy sampling with negative gradients? Do the paper’s main conclusions and the relative performance of different methods generalize to these benchmarks?
Does the collapse of on-policy SFT depend on the specific training settings? Does it persist when the learning rate or training duration is changed, or entropy regularization is introduced? Under these settings, do methods using negative gradients retain their advantage?