- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
- This paper finds that, across diverse RLVR training settings, model parameters and token log-probabilities on fixed text evolve approximately linearly with training steps, and suggests that high-variance reward signals may contribute to a stable update direction.
- Building on this observation, the authors use weight- and output-space extrapolation, along with periodic re-grounding that alternates genuine RL updates with weight extrapolation, to reduce training steps and improve reasoning performance.
Critical strengths
- The authors systematically examine trajectory linearity across different model scales, architectures, RL algorithms, and training hyperparameters, providing broad empirical support for its robustness and consistency across configurations.
- The authors analyze the linearity phenomenon at multiple levels, including model parameters, outputs, and intermediate activations, and use controlled experiments to examine potential contributing factors, providing valuable insights into RLVR training dynamics.
Critical weaknesses
- Figure 2 presents extensive experimental evidence supporting the paper’s central claim, but key details of these experiments are insufficiently described in both the main text and the appendices.
- Table 2 provides limited evidence for consistent gains across benchmarks. The improvements are most pronounced on AIME24.
Questions for authors
Q1. Figure 2 provides central empirical evidence for the paper’s claimed linearity, but the training-step intervals used to compute R² for each experimental setting are not clearly specified. Could the authors clarify how these fitting windows were selected and whether the reported high degree of linearity remains robust to changes in the fitting interval?
Q2. Table 2 shows limited performance gains on LiveCodeBench and MATH500, while the improvements on AIME25 are also modest—for example, the proposed method’s accuracy increases from 29.79% at 200 actual RL steps to 31.20% at 1200 steps. The gains appear more pronounced on AIME24. Could the authors explain this difference across benchmarks, particularly given that the re-grounding schedules were selected using AIME24?
Q3. How sensitive are the results in Table 2 to the learning rate? Do the gains from periodic re-grounding persist across different learning rates, and does the method still outperform standard RL when the learning rate is independently tuned for each method?