acceptodds
Withdrawn from ICLR 2027

Dissecting Reinforcement Learning: Mechanisms Behind Compositional Reasoning in LLMs

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Reinforcement learning with verifiable rewards (RLVR) has become the standard way to train large reasoning models, yet it is unclear which of its components account for its advantage over supervised fine-tuning (SFT). We study this question through the lens of compositional generalization: how well the model learns to compose multiple atomic skills to solve harder tasks. Our main contribution involves organizing post-training methods along two axes. The data axis classifies methods by the source of training data (teacher, frozen checkpoint, or on-policy models). The loss axis, in the binary reward setting, considers how different loss functions place weight on correct and incorrect samples during gradient updates. Through this unifying perspective, we are able to isolate the effect of on-policy data, negative gradients, the group-mean baseline, and standard-deviation normalization individually. On a string-function task and on competition math, we find that neither on-policy data nor negative gradients suffices alone. However, when combined, they improve average accuracy and allow the model to generalize beyond the compositional depths or difficulty of math on which it was trained. We provide a theoretical intuition for why this is the case: methods that give weight to negative examples allow the gradient to concentrate at critical points which distinguish successes from failures, while on-policy data ensures that these updates concentrate on wrong decisions the model would actually make. Consistent with this account, we find that training dynamics and results correspond to several predicted failure cases. Positive-only training with off-policy data such as teacher SFT is stable but lacks generalization, while on-policy positive-only training can lead to entropy collapse and shortened responses which eventually degrade performance. Lastly, applying negative rewards to off-policy data destabilizes training with large gradient norms and produces degenerate outputs. Combining on-policy sampling with negative gradients avoids these observed failure modes and supports sustained generalization.

Primary area:
reinforcement learning
Keywords:
reinforcement learning with verifiable rewards, compositional generalization, GRPO, negative gradients, on-policy training

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_GMuiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates why reinforcement learning is more effective than SFT at improving compositional generalization in large language models, namely, combining existing atomic skills to solve more complex problems. The authors decompose training mechanisms along two axes, data source and loss function, to separately examine the effects of on-policy sampling, negative gradients, the group-mean baseline, and standard-deviation normalization. Experiments on string-function composition and competition math assess whether models can generalize from shallower function compositions or easier math problems to more difficult tasks. The results show that, under the paper’s experimental settings, combining on-policy sampling with negative gradients supports sustained and stable generalization gains. The group-mean baseline contributes substantially on the string-function task, while standard-deviation normalization provides smaller additional gains. Through gradient analysis, the authors explain how contrasting successful and failed trajectories can highlight critical reasoning steps that determine success or failure, while on-policy data directs updates toward errors the model would actually make. Analysis of training dynamics also shows that on-policy SFT may lead to entropy collapse and shorter responses, whereas applying negative gradients to off-policy data may cause sharp increases in gradient norms and degenerate outputs.

Critical strengths

Clear analytical framework and informative component decomposition. The paper organizes post-training methods along two axes, data source and loss function, and progressively compares the effects of negative gradients, the group-mean baseline, and standard-deviation normalization. It also identifies task-dependent contributions of these components, helping clarify the performance differences between SFT and GRPO.

Experiments cover both controlled and realistic tasks, with findings extending beyond GRPO. String-composition tasks allow explicit control over composition depth, while competition math evaluates generalization from easier to harder problems. A simple REINFORCE objective with positive and negative rewards also achieves sustained gains, providing broader empirical support for the effectiveness of combining on-policy sampling with negative gradients.

Theoretical interpretation and training-dynamics analysis complement each other. The authors use gradient analysis to explain the role of group centering while examining response length, entropy, and gradient norms to analyze generalization performance and training instability across methods. These analyses provide concrete insights for understanding and diagnosing post-training behavior.

Critical weaknesses

Insufficient empirical evidence for the core mechanism. The explanation that gradients concentrate on critical reasoning steps relies on specific assumptions, while response length, entropy, and overall gradient norms provide only indirect evidence and cannot directly confirm this mechanism. The appendix also notes that, under strictly on-policy sampling, several appropriately corrected estimators have the same expected gradient. Therefore, performance differences may also arise from finite-sample effects and optimization dynamics.

Limited coverage of mathematical evaluation. The paper primarily evaluates on difficulty-based splits within the DeepScaleR dataset, without validation on independent standard mathematical benchmarks. It therefore remains unclear whether its conclusions generalize to other mathematical tasks.

Questions for authors

Can you provide experimental evidence that gradient updates indeed concentrate on critical reasoning steps? Given that some corrected estimators have the same expected gradient under strictly on-policy sampling, how can the proposed mechanism be distinguished from finite-sample effects and optimization dynamics in explaining the performance differences?

Are there results on independent standard mathematical benchmarks beyond DeepScaleR, such as MATH500, AIME24 and AIME25, that support the advantages of combining on-policy sampling with negative gradients? Do the paper’s main conclusions and the relative performance of different methods generalize to these benchmarks?

Does the collapse of on-policy SFT depend on the specific training settings? Does it persist when the learning rate or training duration is changed, or entropy regularization is introduced? Under these settings, do methods using negative gradients retain their advantage?

Reviewer_ETmVOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates why RLVR improves compositional generalization over SFT by decomposing post-training into rollout sources and loss functions. The authors argue that combining on-policy sampling with negative gradients is essential for generalization, while group-mean baselines further improve credit assignment.

Critical strengths
  • The two-axis framework provides a clear overview of post-training methods and where this paper sits
  • The controlled synthetic task and training-dynamics analyses are useful for studying RLVR
  • This paper is overall well-presented with clear figures
Critical weaknesses
  • The central mechanistic claim is not adequately supported. Appendix B.4 shows that positive-only and negative-gradient estimators have equivalent expected on-policy gradient directions up to scaling under the stated assumptions. This doesn't fully explain the empirical conclusions.
  • Eq. (1) includes sequence-length normalization, whereas Eqs. (2)–(4) do not explicitly include it. The loss variants also differ in gradient magnitude. Without consistent normalization and effective update-scale controls, the improvements in Table 6 cannot be cleanly attributed to individual components.
  • The negative-sample objective on fixed off-policy data is unbounded below, making training collapse unsurprising. The absence of stabilized off-policy baselines, such as DPO or appropriately corrected policy-gradient methods, weakens the claim that on-policy sampling is necessary.
  • Table 6 averages accuracy over L1–L5, although L1 is used for initialization and L2 for training. Recomputing the contributions on unseen depths L3–L5 changes the relative contributions of negative gradients and group centering. This substantially changes the paper's central component-level interpretation.
Questions for authors

See weaknesses.

Reviewer_LmHHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies which components of RL training help LLMs solve harder reasoning tasks. It compares different data sources and loss functions. Experiments on string-function composition and math show that on-policy training with negative updates gives the most sustained gains.

Critical strengths
  1. The paper offers an interesting view of SFT and RL by studying data sources and loss functions within a unified framework.
  2. The paper explores the effects of different training components and examines how they relate to accuracy and training stability.
Critical weaknesses

The main weakness is the gap between the experimental comparisons and the explanations drawn from them. The paper shows which training settings work better, but does not clearly establish how the components produce these gains. Several conclusions rely on plausible explanations with limited direct evidence:

  1. Table 6/7 adds components in one order and assigns the gain to them. These percentages depend on the tested sequence and do not account for interactions between components.
  2. Section 4 gives a possible explanation based on the gradient updates, suggesting that learning focuses on critical reasoning steps. The experiments measure accuracy and training behavior without checking whether this happens. They therefore do not establish that the proposed mechanism explains the gains. The claim in Section 5.3 that these results “empirically validate” the theoretical interpretation is too strong.

Overall, i think the paper provides a useful framework and explores the effects of several training components, however, my main concern is that its conclusions go beyond the evidence. These gaps limit what the paper tells us about why RL improves compositional generalization.

Questions for authors
  1. How do interactions between components affect the percentage contributions reported in Table 6 and 7?

  2. Could the author provide more direct evidence supports the proposed explanation in Section 4?

  3. Eq (1) includes a length-normalization factor (1/T), while Eq(2)–(4) do not. Is the same length normalization used across methods in the implementation? If not, how does this difference affect the comparisons?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.