- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies the learning dynamics of reinforcement learning for LLM reasoning. The authors observe that RL reduces sampled reasoning-route diversity and can either increase or decrease coverage. They further introduce a two-stage theoretical model that separates strategy selection from strategy execution. Within this framework, they analyze how shifts in strategy preferences can produce transfer or forgetting and broaden or narrow finite-budget coverage.
Critical strengths
The paper addresses an important question: how RL changes the reasoning capabilities of LLMs beyond improvements in average accuracy. Its theoretical framework offers a plausible account of why RL can decrease Pass@K on some tasks while increasing it on others. Theoretical modeling provides a valuable perspective on these phenomena.
Critical weaknesses
- The connection between the empirical route analysis and the two-stage theoretical model remains insufficiently established. In particular:
- The empirical analysis identifies and counts solution routes separately for each question, whereas the theoretical model assumes a shared strategy library across training and evaluation prompts.
- The model represents strategy selection and execution using separately parameterized linear-softmax policies over frozen features.
- The empirical measurements do not determine the separate contributions of strategy selection and execution to the observed performance changes.
Questions for authors
- How do the question-specific routes identified in the empirical analysis correspond to the shared strategy library assumed by the theory?
- Could the authors further analyze both route-selection frequencies and correctness conditional on route before and after RL?
2 comments
Part 2/2
W3: The empirical measurements do not determine the separate contributions of strategy selection and execution to the observed performance changes. Could the authors further analyze both route-selection frequencies and correctness conditional on route before and after RL?
A3: Thanks for raising this point. We agree that the original empirical measurements do not determine the separate contributions of strategy and execution. We now address this in two ways.
Separate contributions (General Response GR1). We cut each response right after the sentence where it commits to its strategy and let either model complete it, so the strategy and the executing model can be swapped independently. On 92 math questions, with the executor held fixed, starting from the RL model's strategy raises accuracy by 6.9% [+4.2%, +9.7%]. With the strategy held fixed, the RL executor raises it by 9.2% [+6.6%, +12.0%]. Both strategy selection and execution contribute to the gain.
Route frequencies and conditional correctness. We also computed the quantities the reviewer asked for, using 16 scored responses per model and route labels assigned without seeing correctness. On the same 92 math questions, of the +14.4% accuracy gain, +10.9% [+7.0%, +14.5%] comes from higher accuracy within a route and +3.4% from shifting between routes.
We are not arguing that strategy selection is the only change under RL, or that execution stays fixed, and we do not use these experiments to prove the theory. They motivate the two-stage framework, which is a simplification of the empirical setting; we will say so in the limitations. In the revision we will report route frequencies and conditional accuracies with their counts for all 240 questions. Conditional accuracy on a route one model never used is left undefined rather than set to zero.
References
[1] Chen, Fan, et al. “The Coverage Principle: How Pre-Training Enables Post-Training.” International Conference on Learning Representations, 2026.
[2] Huang, Yu, et al. “On the Emergence of Implicit Curriculum in RLVR Learning Dynamics.” International Conference on Machine Learning, 2026.
[3] Chen, Xingwu, et al. “Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection.” International Conference on Learning Representations, 2026.
[4] Razin, Noam, et al. “What Makes a Reward Model a Good Teacher? An Optimization Perspective.” Advances in Neural Information Processing Systems, 2025.
Part 1/2
W1: The empirical analysis identifies and counts solution routes separately for each question, whereas the theoretical model assumes a shared strategy library across training and evaluation prompts. How do the question-specific routes identified in the empirical analysis correspond to the shared strategy library assumed by the theory?
A1: We agree that question-specific routes alone do not establish the shared library assumed by the theory. To address this limitation, we now build this correspondence explicitly (General Response GR2, Step 1). We grouped the 1,020 routes of all 240 questions into a shared strategy set of 801 families, which serve as our empirical counterpart of the strategy library [M]. Routes of different questions join a family when they use the same decisive method; each family has a written operational criterion that the LLM annotator applies, and a second judge audits the grouping. 31% of routes (314 of 1,020) share a family with a route from another question, and 51 families span two or more benchmarks. For example, exhaustive finite candidate evaluation ("enumerates every object in an explicit finite candidate space and directly tests or scores each complete candidate") appears on 30 questions across five benchmarks, in both math and code. Since these families are built from evaluation questions, we assume, as in the theory, that training and evaluation prompts share the same library, so the strategies RL reweights on training prompts are drawn from these families.
This shared library does not change the per-question results in Figure 4, since they are computed within each question and the grouping only merges routes from different questions. We also used this strategy set to label 64 responses per model on the 92 math questions (General Response GR2, Step 2): the gap in distinct routes is −0.51 [−0.83, −0.20], which strengthens our finding that RL uses fewer routes per question.
W2: The model represents strategy selection and execution using separately parameterized linear-softmax policies over frozen features.
A2: We thank the reviewer for raising this point. Linear-softmax policies over frozen features are widely adopted in theoretical analyses of autoregressive models and RL [1, 2], and our setup is strictly more general and relies on weaker assumptions than prior works [3, 4]. Specifically, our formulation generalizes the tabular softmax models in [3, 4]: one-hot features recover tabular logits, while shared features allow us to analyze how updates on training prompts transfer to unseen prompts. Moreover, our transfer results hold under correlated source features, requiring only linear independence rather than orthogonality under the same answer realizability condition.