acceptodds
Under review as a conference paper at ICLR 2027

Linear Dynamics in the RLVR Training of Large Language Models

Abstract

Reinforcement learning with verifiable rewards (RLVR) has driven significant performance gains in reasoning-oriented large language models (LLMs), yet its internal training dynamics remain largely a black box. In this work, we perform a comprehensive trajectory-level analysis of RLVR and uncover a striking regularity: across various model families, RL algorithms, and training configurations, RLVR consistently enters a robust linear regime, where both parameter weights and output log-probabilities, measured rigorously via teacher-forced evaluation, evolve in a highly linear manner (). Through controlled experiments and theoretical analysis, we demonstrate that this linearity is not a coincidence, but stems from the high-variance, noisy nature of RLVR training signals, which act as a low-pass filter to concentrate optimization along a stable, low-dimensional drift. Moreover, we show that this linear structure is not merely descriptive but powerfully predictive and actionable. Specifically, weight-space extrapolation matches the performance of standard RL optimization while achieving a 6.1× training speedup through periodic re-grounding. Meanwhile, output-space extrapolation serves as a lightweight intervention that effectively bypasses late-stage model collapse, consistently outperforming standard RL across mathematical and coding benchmarks, with an average performance improvement of 4.2%. Our code is available at https://anonymous.4open.science/r/RLVR-Linearity-ICLR-31CD.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Reinforcement Learning with Verifiable Rewards, Large Language Models, Training Dynamics

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_eexAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary
  1. This paper finds that, across diverse RLVR training settings, model parameters and token log-probabilities on fixed text evolve approximately linearly with training steps, and suggests that high-variance reward signals may contribute to a stable update direction.
  2. Building on this observation, the authors use weight- and output-space extrapolation, along with periodic re-grounding that alternates genuine RL updates with weight extrapolation, to reduce training steps and improve reasoning performance.
Critical strengths
  1. The authors systematically examine trajectory linearity across different model scales, architectures, RL algorithms, and training hyperparameters, providing broad empirical support for its robustness and consistency across configurations.
  2. The authors analyze the linearity phenomenon at multiple levels, including model parameters, outputs, and intermediate activations, and use controlled experiments to examine potential contributing factors, providing valuable insights into RLVR training dynamics.
Critical weaknesses
  1. Figure 2 presents extensive experimental evidence supporting the paper’s central claim, but key details of these experiments are insufficiently described in both the main text and the appendices.
  2. Table 2 provides limited evidence for consistent gains across benchmarks. The improvements are most pronounced on AIME24.
Questions for authors

Q1. Figure 2 provides central empirical evidence for the paper’s claimed linearity, but the training-step intervals used to compute R² for each experimental setting are not clearly specified. Could the authors clarify how these fitting windows were selected and whether the reported high degree of linearity remains robust to changes in the fitting interval?

Q2. Table 2 shows limited performance gains on LiveCodeBench and MATH500, while the improvements on AIME25 are also modest—for example, the proposed method’s accuracy increases from 29.79% at 200 actual RL steps to 31.20% at 1200 steps. The gains appear more pronounced on AIME24. Could the authors explain this difference across benchmarks, particularly given that the re-grounding schedules were selected using AIME24?

Q3. How sensitive are the results in Table 2 to the learning rate? Do the gains from periodic re-grounding persist across different learning rates, and does the method still outperform standard RL when the learning rate is independently tuned for each method?

Reviewer_8ryJOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the training dynamics of large language models under reinforcement learning with verifiable rewards (RLVR). The authors identify an approximately linear regime in both model parameter trajectories and output log-probabilities, reporting high linearity across different model families, RL algorithms, and training configurations. To explain this phenomenon, they hypothesize that noisy, high-variance RLVR training signals suppress unstable update directions and produce a consistent long-term optimization drift, supported by controlled noise-injection experiments. Building on these observations, the paper explores linear extrapolation in both weight and output spaces to predict future model states from earlier checkpoints. It further proposes periodic re-grounding, which alternates between standard RL updates and weight extrapolation to reduce training computation. Experiments on mathematical reasoning and coding benchmarks report up to a 6.1× reduction in gradient-based training steps and an average relative performance improvement of 4.2% through output-space extrapolation.

Critical strengths

Clear presentation and well-organized research narrative. The paper is clearly written and logically structured, following a coherent progression from identifying an empirical phenomenon, to investigating its underlying mechanisms, and finally leveraging the findings for practical applications. This organization makes the research easy to follow and effectively connects the empirical observations, explanatory analysis, and proposed applications.

Critical weaknesses

1. Limited novelty beyond prior work. Previous studies, particularly Cai et al. (ICLR 2026), have already identified linear dynamics in RLVR parameter updates and exploited these properties for training acceleration through checkpoint extrapolation. The main additional contributions of this paper appear to be the analysis of token log-probability linearity and an explanation for the observed dynamics. However, token-level linearity may largely be a consequence of linear parameter evolution rather than a fundamentally new phenomenon. The paper does not sufficiently distinguish its contributions from prior work or provide direct comparisons with existing extrapolation methods such as AlphaRL.

2. Insufficient theoretical support for the proposed explanation. The paper attributes RLVR's linear dynamics to noisy, high-variance training signals that suppress unstable update directions. While this hypothesis is interesting, the supporting evidence is primarily empirical, particularly through noise-injection experiments. The paper does not establish a rigorous theoretical explanation or demonstrate that this mechanism is the fundamental cause of the observed linearity. Consequently, its claimed mechanistic contribution remains insufficiently substantiated.

Questions for authors

1. Novelty. What are the key contributions of this paper beyond prior work, particularly Cai et al. (ICLR 2026), which has already investigated linear RL dynamics and checkpoint extrapolation?

2. Extrapolation methods. Beyond periodic re-grounding, how does the proposed weight extrapolation method fundamentally differ from existing approaches such as AlphaRL, and why are direct experimental comparisons missing?

3. Combining extrapolation strategies. Can weight-space and output-space extrapolation be combined to further improve performance or training efficiency?

Reviewer_wrF6OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates linearity in reinforcement learning with verifiable rewards (RLVR). Through extensive controlled experiments, the authors provide empirical evidence of linearity in both output and parameter spaces. They further explore output-space and weight-space extrapolation for RL acceleration and intervention, evaluating these approaches on AIME and LCB.

Critical strengths
  1. The empirical analysis in Section 3 is thorough, covering multiple model families, RL algorithms, learning rates, and optimizers.
Critical weaknesses
  1. The paper’s findings are primarily empirical, with limited theoretical grounding to explain the observed phenomena.

  2. The novelty of the findings and the proposed acceleration method appears limited. On Predictability of Reinforcement Learning Dynamics for Large Language Models (ICLR 2026) already studies rank-1 linear dynamics in RL-induced parameter updates and exploits their predictability to accelerate training. The authors should clarify the additional insights and methodological contributions of their work beyond this prior study.

  3. The evaluation is limited to mathematics and LCB. Extending it to a broader range of tasks and domains would help establish the generality of the findings.

Questions for authors
  1. Regarding the experiments on optimizer choice, have you considered Muon? AdamW and SGD appear to produce updates dominated by a small number of singular directions, whereas Muon may yield updates with a more uniform singular-value spectrum. Could the reported linearity weaken or disappear when using Muon for RLVR? Including this comparison would help clarify whether the observed behavior depends on the optimizer.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.