acceptodds
Under review as a conference paper at ICLR 2027

How RL Reshapes LLM Reasoning: Transferability, Coverage, and Scaling Laws

Abstract

Recent studies on reinforcement learning (RL) report seemingly conflicting evidence about large language model (LLM) reasoning. Training on mathematics can improve performance in other domains, yet gains in Pass@1 can coincide with lower Pass@ than the base model. This raises a fundamental question: does RL expand an LLM's reasoning boundary, or merely reweight its existing reasoning space? We revisit these phenomena across Qwen and Gemma model families, revealing that cross-domain transfer and forgetting both exist, while high-budget coverage increases on some tasks and degrades on others. Detailed analysis of solution traces before and after RL indicates a shift in the reasoning strategies the model employs, motivating a two-stage autoregressive policy model that separates strategy selection from problem-specific execution. Within this framework, we prove how RL's implicit bias reshapes strategy preferences, allowing gains on some tasks while suppressing strategies required by others. This mechanism can also broaden or narrow finite-budget coverage even without expanding strategy support. We further provide theoretical justifications for log-sigmoid and log-linear scaling laws in RL compute, and evaluate their predictive power. Together, these results connect changes in strategy selection to cross-domain transfer, reasoning coverage, and compute scaling.

Primary area:
reinforcement learning
Keywords:
reasoning transferability, training dynamics

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_XAXVOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the learning dynamics of reinforcement learning for LLM reasoning. The authors observe that RL reduces sampled reasoning-route diversity and can either increase or decrease coverage. They further introduce a two-stage theoretical model that separates strategy selection from strategy execution. Within this framework, they analyze how shifts in strategy preferences can produce transfer or forgetting and broaden or narrow finite-budget coverage.

Critical strengths

The paper addresses an important question: how RL changes the reasoning capabilities of LLMs beyond improvements in average accuracy. Its theoretical framework offers a plausible account of why RL can decrease Pass@K on some tasks while increasing it on others. Theoretical modeling provides a valuable perspective on these phenomena.

Critical weaknesses
  • The connection between the empirical route analysis and the two-stage theoretical model remains insufficiently established. In particular:
    • The empirical analysis identifies and counts solution routes separately for each question, whereas the theoretical model assumes a shared strategy library across training and evaluation prompts.
    • The model represents strategy selection and execution using separately parameterized linear-softmax policies over frozen features.
  • The empirical measurements do not determine the separate contributions of strategy selection and execution to the observed performance changes.
Questions for authors
  • How do the question-specific routes identified in the empirical analysis correspond to the shared strategy library assumed by the theory?
  • Could the authors further analyze both route-selection frequencies and correctness conditional on route before and after RL?
2 comments

W1: The empirical analysis identifies and counts solution routes separately for each question, whereas the theoretical model assumes a shared strategy library across training and evaluation prompts. How do the question-specific routes identified in the empirical analysis correspond to the shared strategy library assumed by the theory?

A1: We agree that question-specific routes alone do not establish the shared library assumed by the theory. To address this limitation, we now build this correspondence explicitly (General Response GR2, Step 1). We grouped the 1,020 routes of all 240 questions into a shared strategy set of 801 families, which serve as our empirical counterpart of the strategy library [M][M]. Routes of different questions join a family when they use the same decisive method; each family has a written operational criterion that the LLM annotator applies, and a second judge audits the grouping. 31% of routes (314 of 1,020) share a family with a route from another question, and 51 families span two or more benchmarks. For example, exhaustive finite candidate evaluation ("enumerates every object in an explicit finite candidate space and directly tests or scores each complete candidate") appears on 30 questions across five benchmarks, in both math and code. Since these families are built from evaluation questions, we assume, as in the theory, that training and evaluation prompts share the same library, so the strategies RL reweights on training prompts are drawn from these families.

This shared library does not change the per-question results in Figure 4, since they are computed within each question and the grouping only merges routes from different questions. We also used this strategy set to label 64 responses per model on the 92 math questions (General Response GR2, Step 2): the gap in distinct routes is −0.51 [−0.83, −0.20], which strengthens our finding that RL uses fewer routes per question.

W2: The model represents strategy selection and execution using separately parameterized linear-softmax policies over frozen features.

A2: We thank the reviewer for raising this point. Linear-softmax policies over frozen features are widely adopted in theoretical analyses of autoregressive models and RL [1, 2], and our setup is strictly more general and relies on weaker assumptions than prior works [3, 4]. Specifically, our formulation generalizes the tabular softmax models in [3, 4]: one-hot features recover tabular logits, while shared features allow us to analyze how updates on training prompts transfer to unseen prompts. Moreover, our transfer results hold under correlated source features, requiring only linear independence rather than orthogonality under the same answer realizability condition.

W3: The empirical measurements do not determine the separate contributions of strategy selection and execution to the observed performance changes. Could the authors further analyze both route-selection frequencies and correctness conditional on route before and after RL?

A3: Thanks for raising this point. We agree that the original empirical measurements do not determine the separate contributions of strategy and execution. We now address this in two ways.

Separate contributions (General Response GR1). We cut each response right after the sentence where it commits to its strategy and let either model complete it, so the strategy and the executing model can be swapped independently. On 92 math questions, with the executor held fixed, starting from the RL model's strategy raises accuracy by 6.9% [+4.2%, +9.7%]. With the strategy held fixed, the RL executor raises it by 9.2% [+6.6%, +12.0%]. Both strategy selection and execution contribute to the gain.

Route frequencies and conditional correctness. We also computed the quantities the reviewer asked for, using 16 scored responses per model and route labels assigned without seeing correctness. On the same 92 math questions, of the +14.4% accuracy gain, +10.9% [+7.0%, +14.5%] comes from higher accuracy within a route and +3.4% from shifting between routes.

We are not arguing that strategy selection is the only change under RL, or that execution stays fixed, and we do not use these experiments to prove the theory. They motivate the two-stage framework, which is a simplification of the empirical setting; we will say so in the limitations. In the revision we will report route frequencies and conditional accuracies with their counts for all 240 questions. Conditional accuracy on a route one model never used is left undefined rather than set to zero.

References

[1] Chen, Fan, et al. “The Coverage Principle: How Pre-Training Enables Post-Training.” International Conference on Learning Representations, 2026.

[2] Huang, Yu, et al. “On the Emergence of Implicit Curriculum in RLVR Learning Dynamics.” International Conference on Machine Learning, 2026.

[3] Chen, Xingwu, et al. “Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection.” International Conference on Learning Representations, 2026.

[4] Razin, Noam, et al. “What Makes a Reward Model a Good Teacher? An Optimization Perspective.” Advances in Neural Information Processing Systems, 2025.

Reviewer_fmczOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates whether RL expands LLM reasoning capabilities or primarily reweights existing reasoning strategies. The authors examine cross-domain transfer, forgetting, Pass@N coverage, and RL compute scaling across different models and benchmarks. They propose a two-stage theoretical framework separating strategy selection from answer generation to provide a unified explanation of these phenomena.

Critical strengths

The attempt to connect transferability, reasoning coverage, and scaling laws within a unified theoretical framework is potentially interesting.

Critical weaknesses
  1. The central idea that RL reshapes existing reasoning patterns has already been studied in Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection. Similarly, Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? and ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models have investigated the relationship between RL and reasoning coverage. Cross-domain transfer has also been examined in Does Math Reasoning Improve General LLM Capabilities? Understanding Transferability of LLM Reasoning. The paper combines these established findings without clearly demonstrating what fundamentally new insight its framework provides.
  2. The proposed strategy-selection mechanism is not convincingly validated. Changes in Pass@N or observed reasoning routes cannot distinguish strategy reweighting from changes in execution reliability or other model behaviors. Strategies absent from 16 base-model samples are not necessarily new. Appendix F also acknowledges that the observations cannot uniquely identify strategy selection. Thus, the theory offers a plausible interpretation rather than a convincing explanation of the underlying mechanism.
  3. The theoretical analysis relies on restrictive assumptions, including fixed strategy sets, frozen features, and orthogonality conditions. Its key theoretical quantities are not measured in actual LLMs, and the resulting predictions are not directly validated. Similarly, the scaling-law analysis mainly establishes local approximations of existing functional forms, without demonstrating a distinctive mechanism behind empirical scaling behavior
Questions for authors
  1. What is the fundamental novelty of the proposed framework compared with Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection? What new insights or predictions does this framework provide?

  2. How can the authors establish that strategy reweighting is the primary mechanism behind the observed changes, rather than improvements or degradation in strategy execution? In particular, how can 16 base-model samples reliably characterize the model's existing strategy space?

  3. Can the theoretical quantities, such as feature overlap and valid strategy sets, be measured in actual LLMs to predict transfer and forgetting? Without such validation, how does the theory move beyond a qualitative explanation of existing observations?

  4. What is the additional explanatory value of the scaling-law theory, given that both functional forms have been studied previously and the current guarantees are restricted to local approximation regimes?

3 comments

W1: The central idea that RL reshapes existing reasoning patterns has already been studied in Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection. Similarly, Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? and ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models have investigated the relationship between RL and reasoning coverage. Cross-domain transfer has also been examined in Does Math Reasoning Improve General LLM Capabilities? Understanding Transferability of LLM Reasoning. The paper combines these established findings without clearly demonstrating what fundamentally new insight its framework provides.

A1: We thank the reviewer for pointing out the connection to previous works. Our major contribution is to develop a theoretical framework to understand the popular observations in RL, and this theory provides new insights:

  • For transferability, although [1] also considers reasoning pattern selection, the theoretical insights are fundamentally different. That analysis shows how RL sharpens toward a single pattern; we show that RL can preserve multiple supporting strategies in exactly their initial preference. This offers a mechanism for the transfer and retention observed in prior work [2]: useful strategies remain available even if the base model rarely selects them. The key reason for such difference is that we study the vanishing KL limit, whereas [1] sets KL to zero in its training dynamics. Technically, our analysis also handles a much more challenging setting: we characterize how training a linear softmax policy on multiple prompts affects OOD prompts, whereas their analysis uses a tabular policy for a single prompt and has no generalization. Moreover, in our theory, the strategy policy πθ\pi_\theta and execution policy πw\pi_w are trained jointly, whereas [1] trains only πθ\pi_\theta
  • The Pass@k findings in [3] and [4] motivate our analysis. Our contribution is to explain both outcomes from first principles through RL's implicit bias. In our model, RL can suppress strategies that solve some prompts while amplifying rare valid strategies on others. This can shrink or expand coverage at a finite sampling budget even when the strategy library is unchanged.

We will make these insights and distinctions more explicit in the revision.

W2: The proposed strategy-selection mechanism is not convincingly validated. Changes in Pass@N or observed reasoning routes cannot distinguish strategy reweighting from changes in execution reliability or other model behaviors. Strategies absent from 16 base-model samples are not necessarily new. Appendix F also acknowledges that the observations cannot uniquely identify strategy selection. Thus, the theory offers a plausible interpretation rather than a convincing explanation of the underlying mechanism.

A2: We thank the reviewer for raising this distinction. Our goal is to provide a theoretical framework for understanding transfer, forgetting, and changes in Pass@N. We show that RL's implicit bias can produce these outcomes through strategy reweighting under our model's assumptions. This establishes one sufficient mechanism. We agree that Pass@N and route counts alone cannot separate strategy reweighting from execution, so we ran two additional analyses.

(1) More traces. We are not able to analyze every prompt with 1,024 responses due to limited resources, but we raised the budget from 16 to 64 responses per model (as in [1]) on 92 curated math questions (General Response GR2). With 64 base samples, the share of RL responses on routes unobserved in base falls from 14.9% to 4.4% [2.8%, 6.2%] (AIME24 24.6% → 6.0%, AIME25 22.2% → 7.1%). RL places about 96% of its responses on routes that base model also uses. Since the base model produces most of these routes with more draws, we will remove the Figure 4(b) experiment in the revision and replace it with the strategy-selection experiments in GR1.

(2) Strategy selection vs. execution. A route-fixed intervention (General Response GR1) shows that both stages matter. With the executor held fixed, starting from the RL model's strategy raises accuracy by 6.9% [+4.2%, +9.7%]. With the strategy held fixed, the RL executor raises it by 9.2% [+6.6%, +12.0%] (92 math questions). We are not arguing that strategy selection is the only change under RL, or that execution stays fixed, and we do not use these experiments to prove the theory. They motivate the two-stage framework, which separates strategy selection from execution and is then used to prove our results on transfer, forgetting and changes in Pass@N. The framework is a simplification of the empirical setting, and we will say so in the limitations.

W3: The theoretical analysis relies on restrictive assumptions, including fixed strategy sets, frozen features, and orthogonality conditions. Its key theoretical quantities are not measured in actual LLMs, and the resulting predictions are not directly validated. Similarly, the scaling-law analysis mainly establishes local approximations of existing functional forms, without demonstrating a distinctive mechanism behind empirical scaling behavior

A3: We thank the reviewer for raising both points.

  • Our assumptions in theoretical analysis generalize prior work rather than introducing new restrictions. First, while our strategy set is fixed, it can be completely arbitrary as in [1]. And unlike [1], which holds execution accuracy fixed on a single training prompt, our framework jointly learns both strategy selection and execution across multiple prompts. Second, the frozen features are standard in theoretical analyses [1, 5, 6], and we replace the simplified tabular parameterization of [1, 6] with an autoregressive linear-softmax policy, which is a much weaker and more challenging setting. Finally, as remarked in the paper, our analysis only requires source features to be linearly independent; orthogonality is adopted purely for ease of exposition (and encompasses the tabular setting in [1, 6] as a special case).

  • We agree that both functional forms are established. Our theoretical contribution is to explain how they arise from policy-gradient dynamics. In the strategy selection formulation, we derive how the probability of choosing a valid strategy evolves during training and how that change carries over to target prompts. The resulting local error bounds explain why both laws can fit the same early-stage data. Moreover, we provide new emprical observations of scaling laws on extrapolation regime (Fig. 5). As we stated in the paper, the log-sigmoid and log-linear fits are often nearly indistinguishable within the fitting regime yet diverge substantially when extrapolating, with prediction reliability depending heavily on the task domain (e.g., holding much more reliably on in-domain math tasks). Previous works such as [7,8] are restricted to held-out data drawn from the training distribution and didn't do a symstematic comparison on log-sigmoid and log-linear law.

Q4: What is the additional explanatory value of the scaling-law theory, given that both functional forms have been studied previously and the current guarantees are restricted to local approximation regimes?

A4: While both functional forms have been studied heuristically, our analysis provides both theoretical and empirical explanatory value:

  • Our theoretical contribution explains how both functional forms emerge from policy-gradient dynamics and why they are locally indistinguishable. Rather than postulating these curves phenomenologically, we derive how the probability of choosing a valid strategy evolves during training and how that change carries over to target prompts in the strategy selection regime. Crucially, the resulting local error bounds are a feature rather than a limitation: they mathematically explain why two distinct functional forms (log-sigmoid and log-linear) can fit the same early-stage training data nearly identically.
  • We provide new empirical observations on scaling laws in the extrapolation regime. As shown in the paper (Fig. 5), while log-sigmoid and log-linear fits are often nearly indistinguishable within the fitting regime, they lead to markedly different predictive behavior when extrapolating. Furthermore, we show that extrapolation reliability largely depends on the task domain—specifically, extrapolation is significantly more reliable on in-domain math tasks. This goes beyond prior works such as [7, 8], which are restricted to held-out data drawn from the training distribution and lack a systematic comparison between log-sigmoid and log-linear laws.

References

[1] Chen, Xingwu, et al. “Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection.” International Conference on Learning Representations, 2026.

[2] Huan, Maggie Ziyu, et al. “Does Math Reasoning Improve General LLM Capabilities? Understanding Transferability of LLM Reasoning.” International Conference on Machine Learning, 2026.

[3] Yue, Yang, et al. “Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?” Advances in Neural Information Processing Systems, 2025.

[4] Liu, Mingjie, et al. “ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models.” Advances in Neural Information Processing Systems, 2025.

[5] Chen, Fan, et al. “The Coverage Principle: How Pre-Training Enables Post-Training.” International Conference on Learning Representations, 2026.

[6] Razin, Noam, et al. “What Makes a Reward Model a Good Teacher? An Optimization Perspective.” Advances in Neural Information Processing Systems, 2025.

[7] Khatri, Devvrit, et al. “The Art of Scaling Reinforcement Learning Compute for LLMs.” International Conference on Learning Representations, 2026.

[8] Tan, Zelin, et al. “Scaling Behaviors of LLM Reinforcement Learning Post-Training: An Empirical Study in Mathematical Reasoning.” Annual Meeting of the Association for Computational Linguistics, 2026.

Reviewer_wguKOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether reinforcement learning (RL) truly expands the reasoning boundaries of large language models (LLMs) or merely reweights an existing reasoning space. The authors find that cross-domain transfer and forgetting occur simultaneously, with coverage exhibiting both gains and losses. They propose a two-stage policy model—decoupling policy selection from execution, to provide a theoretical explanation of how RL reshapes policy preferences, and they further discuss the computational scaling laws of RL.

Critical strengths
  1. The paper systematically reveals the coexistence of in-domain gains, cross-domain transfer and forgetting, and divergent Pass@N coverage changes across Qwen and Gemma models and multiple benchmarks.

  2. Based on solution-trajectory analysis, the paper proposes a two-stage autoregressive model separating strategy selection from execution, providing a unified theoretical framework for explaining RL’s implicit preference reshaping, transfer, and forgetting.

  3. The paper provides theoretical justifications for log-sigmoid and log-linear scaling laws in RL compute and evaluates their predictive power on real training extrapolation, strengthening the link between empirical laws and mechanistic explanations.

P.S. I don’t fully understan the theoretical analysis, so I’m not in a position to properly evaluate that part of the content.

Critical weaknesses
  1. Scaling-law extrapolation is less stable on coding and planning tasks, some fitted parameters are weakly identified, and the paper only establishes local-window approximations, so it does not yet support global or cross-model-scale RL scaling predictions.

  2. Strategy identification mainly relies on LLM-extracted route classifications, and some coverage analyses lack retained full RL traces, so sample size and annotation reliability limit causal conclusions about changes in strategy selection.

Questions for authors
  1. How is the finite strategy library [M] defined in the context of real-world LLMs, and can the "reasoning paths" extracted from an LLM be reliably mapped to the strategy variables z?

  2. For 12.7% of RL responses, the route is absent from the 16 base samples, but could the base model produce these routes under a larger sampling budget, and is defining “absent route” with only 16 samples too strong?

  3. AIME has only 30 questions and BoxNet only 100; is the statistical power sufficient for detecting changes in Pass@N coverage, and should confidence intervals, effect sizes, and multiple-comparison corrections be reported?

3 comments

Thanks for your careful review and constructive feedback! We address your concerns as follows.

W1: Scaling-law extrapolation is less stable on coding and planning tasks; the paper only establishes local-window approximations

A1: We agree that the theoretical guarantees are local and do not certify extrapolation beyond the approximation window. Their purpose is to derive explicit approximation bounds from specified strategy-selection dynamics. The later-checkpoint evaluations address a separate empirical forecasting question. We will clarify this distinction and add chronological fitting-window and forecast-horizon sensitivity using the existing checkpoint scores, with a last-observation baseline and benchmark-specific errors. Coding and planning failures will remain visible. These checks require no additional training and will be described as retrospective robustness analyses rather than new independent tests.

W2: Sample size and annotation reliability limit causal conclusions about changes in strategy selection.

A2: We thank the reviewer for raising both points, and we tested each concern directly.

Sample size. We raised the budget from 16 to 64 responses per model on 92 curated math questions and labeled the new responses with a shared strategy set (General Response GR2). The narrowing in Figure 4(a) becomes clearer, not weaker. RL minus base distinct routes per question goes from −0.26 [−0.56, +0.03] at 16 samples to −0.51 [−0.83, −0.20] at 64, and is negative on each benchmark (MATH500 −0.34, AIME24 −0.50, AIME25 −0.77).

Annotation reliability. An independent annotator, Claude Opus 5.5, re-labeled 64 responses drawn across all six benchmarks (32 base, 32 RL). It saw each question's route list but not the original labels, the model or correctness. It agreed with the pipeline on 60/64 responses (94%), and on 44/48 for questions with two or more routes to choose from.

Q1: How is the finite strategy library M defined in the context of real-world LLMs, and can the "reasoning paths" extracted from an LLM be reliably mapped to the strategy variables z?

A3: We decompose a reasoning path into a strategy and its execution, mirroring the two stages of our model: the strategy policy πθ\pi_\theta selects z∈[M]z \in [M], and the execution policy πw\pi_w carries it out. [M][M] is a modeling assumption: a finite set of reusable strategies at a chosen level of abstraction, such as "binary search" or "set up and solve a system of equations". A similar pattern abstraction appears in [1].

How responses are mapped to strategies. We identify a response's strategy by its representation and decisive solution step. Responses to the same question with the same strategy form a route, and the steps that carry out the strategy constitute its execution. To map strategies across questions, we group routes from different questions that use the same decisive method into a shared strategy set of 801 families, our empirical counterpart of [M][M] (General Response GR2). 31% of routes share a family with a route from another question. Reliability: labeling is blind to the model and to correctness, every grouping is audited by a second judge, and an independent annotator from a different model family agrees with 60 of 64 route labels (see A2). The route-fixed intervention in GR1 uses this decomposition directly: it fixes the strategy prefix and swaps the model that executes it.

Q2: For 12.7% of RL responses, the route is absent from the 16 base samples, but could the base model produce these routes under a larger sampling budget, and is defining “absent route” with only 16 samples too strong?

A4: We agree that 16 samples is a weak basis for "absent". We drew 48 more responses per model for every math question with route labels (92 questions, 64 responses per model) and labeled them with the shared strategy set, opening new entries where none matched (General Response GR2).

RL responses whose route is absent from the base responsesMATH500 (39)AIME24 (28)AIME25 (25)All 92
16 samples per model3.4%24.6%22.2%14.9% [10.4, 20.0]
64 samples per model1.4%6.0%7.1%4.4% [2.8, 6.2]

This drop rests on base responses that were assigned to routes previously seen only in RL. Most routes that looked absent at 16 samples are therefore rare base routes, not new ones. We will therefore remove the Figure 4(b) experiment in the revision and replace it with the strategy-selection experiments in GR1.

Q3: AIME has only 30 questions and BoxNet only 100; Is the statistical power sufficient for detecting changes in Pass@N coverage?

A5: AIME and BoxNet are established benchmarks for Pass@N evaluation. Prior studies report coverage on AIME [2, 3, 4] and BoxNet [3, 4], including a 100-question BoxNet evaluation in [3].

References

[1] Chen, Xingwu, et al. “Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection.” International Conference on Learning Representations, 2026.

[2] Yue, Yang, et al. “Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?” Advances in Neural Information Processing Systems, vol. 38, 2025.

[3] Liu, Mingjie, et al. “ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models.” Advances in Neural Information Processing Systems, vol. 38, 2025.

[4] Wu, Fang, et al. “The Invisible Leash: Why RLVR May or May Not Escape Its Origin.” arXiv, version 5, 2026, arXiv:2507.14843.

Thanks for your experiments and your effort! The authors have addressed most of my concerns, so I have decided to maintain my positive rating.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response (Part 1/2)

General response GR1: both strategy selection and execution matter

GR1. RL changes both strategy selection and execution: a route-fixed intervention. Reviewers fmcz (W2) and XAXV (W3) asked whether RL's gains come from strategy selection or from execution. To separate the two, we conduct an intervention experiment similar to that of Chen et al. [1], holding the strategy fixed and swapping the model that executes it.

Setup. We use 92 curated math questions (MATH500, AIME24, AIME25) and all 16 sampled responses per question from the base model and from the math-RL model, 2,798 responses with an identified route. Using the route annotator (which has no access to correctness), we truncate each response right after the sentence where it commits to a strategy (at a median of 21% into the response). Both models then complete every strategy prefix 4 times with the original decoding settings and 16k-token budget, and we grade the completed response. This gives a 2×2 design per question: strategy chosen by {base, RL} × executed by {base, RL}. The strategy-selection effect is the change in accuracy from swapping which model chose the strategy, with the executor fixed. The execution effect is the change from swapping the executor, with the strategy fixed. Each is averaged over both levels of the other factor, so the two add up to the total change.

Strategy chosen by → executed bybase → basebase → RLRL → baseRL → RLStrategy-selection effectExecution effect
Accuracy (%)38.248.045.754.3+6.9 [+4.2, +9.7]+9.2 [+6.6, +12.0]

Effects are reported in percentage points with 95% paired bootstrap intervals over questions. When each model finishes its own prefixes, it closely matches its original accuracy (base: 38.2% vs. 40.0%; RL: 54.3% vs. 54.3%), confirming that truncating and continuing the responses does not distort the comparison.

The results indicate that both stages matter. With the executor held fixed, starting from the RL model's strategy instead of the base model's raises accuracy by 6.9%, even though the rest of the solution is written by the same model. With the strategy held fixed, the RL executor raises accuracy by 9.2%. Neither stage alone accounts for the RL model's gain. This is aligned with our two-stage framework, in which RL jointly updates a strategy policy πθ\pi_\theta and an execution policy πw\pi_w.

By comparison, [1] did a similar experiment with a different protocol and found stable per-pattern accuracy. They keep a base response up to its final-answer sentence and let the RL model write only the answer, so the base model has already done almost all of the execution and per-pattern accuracy is stable almost by construction. By truncating right where the model commits to a strategy, our experiment leaves the actual execution open, allowing us to cleanly measure the effect of both stages.

AuthorsOpenReview ↗

General Response (Part 2/2)

General response GR2: shared family library and 64 samples

GR2. Two steps added to the submission: a shared strategy set and 64 samples per model. Reviewers wguK (W2, Q2) and fmcz (W2) asked whether 16 samples per model are enough to call a route "absent" from the base model. Reviewers XAXV (W1) and wguK (Q1) asked how question-specific routes relate to the shared strategy library assumed by the theory. In response to these concerns, we conducted two additional experiments:

  1. We built a shared strategy set by grouping routes across questions.
  2. We extended the analysis to 64 responses per model on the math benchmarks, and labeled the new responses with this shared strategy set.

Step 1: A shared strategy set across questions. In the original submission, the 32 responses (16 base, 16 RL) for each question were grouped into routes: two responses share a route when they use the same representation and the same decisive solution step. Because routes were defined separately for each question, routes from different questions were unrelated. Here, we group all 1,020 routes from the 240 questions into a shared strategy set of 801 families, placing routes from different questions into the same family when they rely on the same decisive method. For example, simultaneous equation solving ("represents mutually dependent unknowns with simultaneous algebraic equations and solves or reduces the resulting system") groups routes from 8 questions across MATH500, AIME24, AIME25, and CodeContests. We find that 31% of the routes (314 of 1,020) belong to a family shared across multiple questions.

Step 2: 64 samples per model, labeled with the shared strategy set. We drew 48 more responses per model for each of the 92 math questions with route labels in MATH500, AIME24 and AIME25, 8,832 new responses in total. We then used the shared strategy set from Step 1 to label them and recomputed the numbers in Figure 4.

Figure 4 with 64 samples (92 math questions).

(a) RL − base, 16 samples(a) RL − base, 64 samples(b) RL absent from base, 16 samples(b) RL absent from base, 64 samples
MATH500−0.16−0.343.4%1.4%
AIME24−0.23−0.5024.6%6.0%
AIME25−0.45−0.7722.2%7.1%
All−0.26 [−0.56, +0.03]−0.51 [−0.83, −0.20]14.9% [10.4, 20.0]4.4% [2.8, 6.2]

Panel (a) shows the difference in distinct routes per question (base 3.05 →\rightarrow RL 2.79 at 16 samples; 3.96 →\rightarrow 3.45 at 64 samples), comparing equal numbers of identifiable responses from each model

  • (a) RL's narrowing grows with the budget. At 64 samples, our original message that "RL uses fewer routes" holds and is even strengthened: the difference is significant overall and negative on every benchmark.
  • (b) Most "absent" routes appear with more base samples. The share of RL responses on routes absent from base falls from 14.9% at 16 samples to 4.4% at 64. This indicates that most of the previously absent routes were simply not covered by the first 16 base samples. We will remove the Figure 4(b) experiment in the revision and replace it with the strategy-selection experiments in General Response GR1.

Due to the time limit, these results only cover the math benchmarks. For the revision, we will also sample 64 responses per model on GPQA and the two coding benchmarks and include them in the paper.

References

[1] Chen, Xingwu, et al. “Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection.” International Conference on Learning Representations, 2026.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.