acceptodds
Under review as a conference paper at ICLR 2027

Beyond Reward Sparsity: How Sample Difficulty Shapes Reasoning in RLVR

Abstract

Reinforcement learning with verifiable rewards (RLVR) has become a powerful approach for improving the reasoning capabilities of large language models, yet how training sample difficulty shapes the learning process remains poorly understood. Existing analyses often view difficulty through the lens of reward sparsity, but reward availability alone does not explain why some samples improve reasoning while others lead to unstable updates. In this work, we present a learning-dynamics analysis of sample difficulty in RLVR. We distinguish static sample difficulty, measured by base-policy solvability, from dynamic learning-signal availability under the evolving policy, and derive a per-update decomposition of RLVR optimization into three components: the training signal, its transfer through shared parameters, and the resulting prediction change. Through controlled experiments on MATH, we show that different difficulty regimes induce distinct learning behaviors: frontier samples provide stable improvements, while hard samples can produce harmful updates despite retaining non-trivial signals. Using sparse autoencoders and feature interventions, we further reveal that difficulty-dependent training reshapes internal feature dynamics, with samples of similar difficulty producing heterogeneous feature changes depending on their realized rollout groups. These results provide a mechanistic understanding of how sample difficulty governs RLVR learning dynamics beyond reward sparsity.

Primary area:
interpretability and explainable AI
Keywords:
Reinforcement learning, Language model

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_4QPQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Studying the impact of problem difficulty on RL problems, the paper group problems with a base model's empirical success probability, and controlled training runs compare easy, frontier, and hard subsets. The main idea is that frontier problems improve performance, whereas training only on very hard problems can degrade easier-problem performance even when within-group advantage signals become more frequent.

Critical strengths
  1. The controlled difficulty-group experiment separates the availability of a nonzero learning signal from whether that signal transfers beneficially. The observation that hard examples can provide optimization signal while harming broader performance is useful.
  2. The signal–transfer–readout decomposition provides a coherent vocabulary for discussing why reward sparsity alone is not an adequate explanation of RLVR learning dynamics.
Critical weaknesses
  1. The empirical scope is too narrow to support the paper’s broad mechanistic claims: the central results rely on one model family and one mathematical dataset, with effects of only a few percentage points and limited statistical analysis. It is unclear whether the phenomenon generalizes beyond this particular RLVR regime.
  2. The mechanistic evidence is incomplete and largely post hoc. The one-step decomposition is a standard first-order expansion; SAE features are selected and semantically interpreted after training; the reported feature effects are very small; and a headline claim about predicting post-update feature changes is not backed by the corresponding quantitative results in the main paper. As written, the analysis redescribes the observed behavior more than it explains it.
Questions for authors
  1. How does this work differ from prior studies of sample difficulty in RLVR, such as DAPO's dynamic sampling (Yu et al., 2025), online difficulty filtering (Bae et al., 2025), and learnability-based curricula (Foster & Foerster, 2025)? These already select intermediate-difficulty prompts or discard zero-variance groups. Beyond confirming that frontier samples are useful, what does the signal × transfer × readout decomposition predict that these methods do not, and can you show a case where it changes which samples should be used?
  2. Does the hard-example degradation replicate across substantially different model families, datasets, and RL algorithms when difficulty labels and sample counts are fixed prospectively?
  3. Were the SAE features and their semantic labels selected before examining the downstream effects? If not, what held-out procedure demonstrates that the feature-level interpretation is not post hoc?
  4. Does the same picture hold for other RLVR objectives (Dr. GRPO, RLOO, PPO with a critic)?
9 comments
AuthorsOpenReview ↗

Our Response (Part I/V)

We thank the reviewer so much for the insightful comments and appreciation of our work. We address your concerns and questions in the following point by point. We also make the corresponding changes in the revision following your suggestions.


Response to Weaknesses Part

  1. Comment: "The empirical scope is ... regime."

    Response: We thank the reviewer for this important concern. In the revision, we extended the experiments along each axis the reviewer raises: model scale and family, training data, benchmarks, and RLVR regime. All settings use the same difficulty-based comparison protocol, with difficulty recalibrated for each base model and each subset compared with a natural control matched in the number of distinct questions and prompt presentations. The results are summarized in Section 3.2 (Figure 2) and Appendix D.2 (Table 5), with the setup in Appendix D.1 and full results in Appendix D.2 (Tables 6–9). All entries below are accuracy differences in percentage points relative to the control.

    (1) Model scale and family. We trained Qwen2.5-Math-7B and DeepSeek-Math-7B-Instruct, a different model family, on MATH with GRPO (Appendix D.2; Table 6).

    ModelComparisonProbeMATH500AIME24AIME25AMCMinerva MathOlympiadBench
    Qwen2.5-Math-1.5B (original)Frontier − control+1.63+2.80+0.31+1.05+2.64+0.37+2.98
    Hard − control−1.83−2.00−0.10−0.21+0.11−2.94−1.34
    Qwen2.5-Math-7BFrontier − control+0.27+1.60+1.25+1.56+0.45−0.73+1.34
    Hard − control−0.47−0.40−1.66−0.31−0.52−5.52−1.34
    DeepSeek-Math-7B-InstructFrontier − control+2.41+1.60+1.35+0.42+0.42+1.84+0.74
    Hard − control−0.98−0.60−0.52−0.83−1.46+0.73−0.45

    (2) Training data. We trained the same three models on DAPO-Math, a second mathematical dataset whose integer answers allow reliable rule-based verification (Appendix D.2; Table 7).

    ModelComparisonProbeMATH500AIME24AIME25AMCMinerva MathOlympiadBench
    Qwen2.5-Math-1.5BFrontier − control+3.45+3.60+2.09+0.84+2.93+1.83−0.30
    Hard − control−1.44−0.60−1.77−0.31−0.53−0.37−1.39
    Qwen2.5-Math-7BFrontier − control+0.84+2.20+0.52+0.21+1.28+1.84+0.61
    Hard − control−2.19−0.60−2.61−1.87−0.80−4.05−0.60
    DeepSeek-Math-7B-InstructFrontier − control+0.33+0.80+0.21+0.21+0.79+1.10+1.48
    Hard − control−1.12−1.00−0.32−0.21−1.06−2.57−1.04

    (3) Beyond mathematical RLVR. We trained Qwen2.5-3B-Instruct on Sokoban, a multi-turn agentic planning task, and evaluated it on a held-out probe, the Sokoban test set, and FrozenLake, which is never used for training (Appendix D.2; Table 9).

    ComparisonEasyF-easyF-hardHardIntract.ProbeSokoban testFrozenLake
    Frontier − control+14.73+41.86+27.46+18.84+3.05+18.88+13.86+9.57
    Hard − control−3.10−10.85−0.71+0.10−0.87−3.25−5.47−1.76

    On Sokoban, hard-only training leaves its own stratum essentially unchanged but lowers the easier strata, whereas frontier-only training improves every stratum and transfers to FrozenLake. The degradation on easier strata is consistent with the MATH pattern in Figure 1(d).

    Effect sizes and consistency. On the held-out probe, frontier-only training is above its control and hard-only training is below it in all seven settings above, and each pattern holds in 43 of 45 evaluations (Appendix D.2; Tables 5–7 and 9). The four exceptions across the two regimes are all below one point. These counts describe consistency across the evaluated configurations; we do not treat the configurations or their shared benchmarks as independent statistical replications. The effects are also not uniformly small: hard-only training lowers Minerva Math for Qwen2.5-Math-7B by 5.52 and 4.05 points after MATH and DAPO-Math training, and on Sokoban it lowers frontier-easy success by 10.85 points. All differences are measured against natural controls matched in distinct questions and prompt presentations, providing controlled comparisons of difficulty-conditioned training under the evaluated settings.

AuthorsOpenReview ↗

Our Response (Part II/V)

Weaknesses Part (continuing)


  1. Comment: "The mechanistic evidence ... explains it."

    Response: We address these concerns as follows.

    1. Scope of the decomposition. The first-order expansion itself is standard. Our contribution is its application to GRPO to separate the realized group-relative signal, its transfer through shared parameters, and the observation-specific readout (Section 3.3; Proposition 1, Eq. 9). This provides testable predictions about individual updates. The cumulative relation retains per-step residuals and does not establish an accurate reconstruction of the full training trajectory (Appendix B.1, Eq. 43; Appendix E.1, Eq. 47).

    2. Feature selection and interpretation. We acknowledge that feature discovery is exploratory. The selected feature panel is retained for later diagnostics, and the one-step calibration uses fresh rollouts with sampling seeds disjoint from screening (Section 5.2; Appendix D.6). Appendix D.7 details the selection chronology and the separate freezing of context labels before held-out label testing. We do not claim that the intervention feature set was selected before all downstream accuracy outcomes were observed. The revised descriptions are positional or formatting associations, not established reasoning functions (Appendix D.7; Table 16).

    3. Small feature effects. Section 4.1 reports mean activation, firing rate, and differences from matched controls in the frozen shared SAE coordinates (Eqs. 11–13; Figure 4). Appendix D.8 quantifies the modest endpoint effects (Table 17). We also test behavioral consequences through joint training-time interventions on four selected feature directions, removed during evaluation (Section 5.3; Table 1; Appendix D.3). Steering changes the benchmark-average difference from −1.08 to +0.52 percentage points for hard-only training and from +1.69 to −0.15 for frontier-only training. These results support behavioral relevance of the selected directions, without establishing unique feature mediation (Appendix D.8).

    4. Quantitative prediction results. Across the 48 one-step trials, the prediction along the actual AdamW displacement achieves MAE 3.84×10−43.84\times10^{-4}, Pearson r=0.99396r=0.99396, Spearman ρ=0.98892\rho=0.98892, and 97.73% sign agreement (43/44 informative trials). The prediction and measurement are defined in Section 5.2, Eq. 19, and the full statistics are reported in Appendix D.6, Table 13. Error and correlation statistics include all 48 trials. The MAE is approximately 32% of the mean absolute observed feature change, so high correlation and sign agreement coexist with non-negligible magnitude error (Appendix D.8).

AuthorsOpenReview ↗

Our Response (Part III/V)

Response to Questions


  1. Comment: "How does this ... be used?"

    Response: We thank the reviewer for this question. We agree that preferring intermediate-difficulty prompts is established, and we do not claim it as our contribution. These approaches select prompts using success-based statistics of the current policy. For binary rewards, DAPO's dynamic sampling retains mixed-reward groups, online difficulty filtering prioritizes an intermediate success band, and learnability-based scores such as p(1−p)p(1-p) favor intermediate success probabilities.

    Our distinction is between fixed reference difficulty and evolving signal availability (Section 3.1, Eqs. 5–6), followed by a local decomposition into signal, transfer, and readout (Section 3.3; Proposition 1, Eq. 9). Success-based selection affects the availability of group-relative signals, but a scalar success rate does not determine the full realized signal Gτ\mathcal{G}_\tau, the transfer Kτ\mathcal{K}_\tau, or the readout Tτ\mathcal{T}_\tau. We provide three complementary demonstrations of information beyond success-based selection.

    1. Similar signal magnitude does not imply the same effect. In Appendix C.1, the frontier-hard and easy source groups have similar logit-space signal norms (0.00901 vs. 0.00914; Table 2) but opposite signed effective transfer onto the same frontier-hard response (κ=−2.12\kappa=-2.12 vs. +1.78+1.78; Figure 6(b), with the definition in Eq. 45). Their local predicted effects therefore have opposite signs. This shows that signal magnitude alone does not determine transfer; it does not imply that these groups have identical success rates or that every success-based criterion would rank them identically. This comparison concerns local predictions on fixed responses, with finite-step calibration restricted to the hard and frontier-hard sources (Appendices C.1 and C.3; Figure 6).

    2. Signal availability does not guarantee useful training. During hard-only training, the learning-signal availability plotted in Figure 1(b) rises from 0.255 to 0.479, while hard-only training still reduces accuracy on easier held-out strata (Section 3.1; Figure 1(d)). Similarly, the score p(1−p)p(1-p) assigns the same value to current success probabilities p=0.2p=0.2 and p=0.8p=0.8; this symmetry alone gives no information about transfer to a fixed observation. Reference-difficulty labels, however, are defined by the base policy and remain fixed as current success changes.

    Across all ten settings, hard-only training is below its matched control on the held-out probe (Section 3.2, Figure 2; Appendix D.2, Table 5). The deficit persists under Dr. GRPO and under PPO with a learned critic, for which all-failure groups need not have zero advantages (Appendices D.1–D.2; Table 8). Training-time steering further provides intervention-based evidence that the selected feature directions affect downstream performance, without establishing that they uniquely mediate the hard-only deficit (Section 5.3; Table 1; Appendices D.3 and D.8). These findings motivate assessing training utility beyond signal availability; they do not establish that every hard prompt is harmful or that a particular online filter should reject every such prompt.

    3. Within reference-difficulty bands, update directions are heterogeneous. Among the frontier prompts analyzed in Section 4.2, the sign of the mean sample–feature influence (Eq. 14) varies within each stratum: 57.0% negative and 43.0% positive for frontier-easy, and 45.3% negative, 51.6% positive, and 3.1% zero for frontier-hard. Meanwhile, 93.4% and 68.8% of the respective rollout groups have mixed rewards (Figure 5). Thus, a shared difficulty label or the presence of reward variation does not determine whether an update strengthens or suppresses a fixed feature. The local analysis distinguishes these directions conditional on the realized rollout group. The 48 fresh-rollout trials also show that accurate prediction along a realized update does not yet establish a consistent screening advantage over matched sources (Section 5.2; Appendix D.6, Tables 13–15).

Reviewer_breeOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper discusses the question on "How does sample difficulty shape the learning dynamics of RLVR", particularly in the aspect of non-uniform group score in GRPO. 1. The authors investigated on different difficulty levels, pointing out that non-uniform group rewards provide a learning signal but do not necessarily lead to useful updates. 2. They provide a theory on how training signal transfer 3. They induct an experiment using SAE, results shows that feature dynamics differ across difficulty levels, even within same group, the same features could either be strengthened or weakened

Critical strengths
  1. Controlled experiments show that frontier-only improves more on acc, while hard tasks can have negative impact.
  2. The author provides informative results on both theoretical and experimental aspects
Critical weaknesses
  1. In sec 5.2, the authors do not causally connect SAE feature changes to accuracy gains or losses, which does not fulfill the work.
  2. “ReasonScore,” the SAE setup, and the 48 one-step results explaination are not good enough and are unconvincing.
Questions for authors
  1. What is the definition of "ReasonScore" in figure 3? What is the relation with features?
  2. Do interventions on identified features change the model’s accuracy? I may like to change my score if the authors got insightful results 3.What are the 48 one-step trials’ prediction errors and sign agreement, and do screened sources outperform controls?
5 comments
AuthorsOpenReview ↗

Our Response (Part I/III)

We thank the reviewer for the valuable comments. Below we address your concerns and questions point by point.


Response to Weaknesses Part

  1. Comment: "In sec 5.2, ... the work."

    Response: We thank the reviewer for this comment. We added Section 5.3 and Table 1 (p. 10), which intervene on the identified SAE feature directions during RLVR training and evaluate downstream accuracy with the intervention removed.

    Specifically, we pre-apply the feature shifts associated with each training regime, aiming to reduce their incorporation into the model weights. For hard-only training, steering reduces the probe deficit from −1.83 to −0.39 percentage points and improves the benchmark mean from −1.08 to +0.52. For frontier-only training, steering removes the original gains: the probe difference drops from +1.63 to −0.26, and the benchmark mean from +1.69 to −0.15. Accuracy improves across all seven evaluations for hard-only training and decreases across all seven for frontier-only training.

    These interventions provide evidence linking the selected feature directions during training to subsequent accuracy gains and losses.

  2. Comment: " 'ReasonScore' ... are unconvincing."

    Response: We thank the reviewer for highlighting these issues. We have revised the presentation in three respects:

    • ReasonScore. Our previous ReasonScore results were computed using the metric proposed by Galichin et al. (2026) and have been moved to the appendix. The main text now reports mean feature activation, firing rate, and changes relative to matched controls (Section 4.1, Eqs. 11–13), which directly characterize feature evolution.
    • SAE setup. We clarify that the shared SAE is trained on base-model and intermediate-checkpoint activations. Its dictionary and normalization remain frozen, with identical evaluation sequences and token positions across checkpoints.
    • 48 one-step trials. Section 5.2 clarifies that these comprise 12 questions × four fresh rollout groups, including matched controls and sampling seeds disjoint from screening. Each trial resets the model and AdamW state. Eq. 19 compares predicted and observed feature changes along the actual optimizer displacement, making explicit that the experiment tests local feature-change prediction.
AuthorsOpenReview ↗

Our Response (Part II/III)

Response to Questions Part

  1. Comment: "What is the ... features?"

    Response: ReasonScore in the original Figure 3 follows Galichin et al. (2026) and measures each SAE feature’s association with reasoning-related vocabulary contexts:

    ReasonScorei=μi+∑jμj+Hiα−μi−∑jμj−\mathrm{ReasonScore}_i = \frac{\mu_i^{+}}{\sum_j\mu_j^{+}}H_i^\alpha - \frac{\mu_i^{-}}{\sum_j\mu_j^{-}}

    where μi+\mu_i^{+} and μi−\mu_i^{-} are feature ii’s mean activations within reasoning-vocabulary context windows and outside them, respectively. HiH_i is the normalized entropy of its activation distribution across reasoning words, and α\alpha controls the penalty for concentrating on only a few words.

    Thus, ReasonScore assigns a scalar score to each SAE feature: higher values indicate stronger relative association with these contexts. It does not directly measure absolute feature strength or establish a causal contribution to reasoning accuracy.

    In the revision, we have moved the ReasonScore results to the appendix and replaced them in the main text with mean feature activation, firing rate, and changes relative to matched controls (Section 4.1, Eqs. 11–13), making the measured feature changes more explicit.

  2. Comment: "Do interventions on ... outperform controls?"

    Response: We thank the reviewer for this important question. Intervening on the identified features during RLVR training changes held-out accuracy, and the direction of the change follows the direction of the intervention. We added this experiment in the new Section 5.3 (Table 2).

    Section 5.1 perturbs feature directions only in the frozen base model, so it measures local changes in response likelihood rather than accuracy. To test accuracy, we intervene throughout training instead. Following preventive steering[1], we pre-apply the feature shifts that a training regime induces, so that the optimizer does not need to encode them in the weights. For the set S\mathcal{S} of the eight layer-13 features in Figure 3, we extend Eq. 16 to

    hθ,λℓ(o)=hθℓ(o)+s∑f∈Sλfdf\mathbf{h}^{\ell}_{\theta,\lambda}(o) = \mathbf{h}^{\ell}_{\theta}(o) + s \sum_{f \in \mathcal{S}} \lambda_f \mathbf{d}_f

    at every token position during RLVR training, with λf>0\lambda_f > 0 for features the regime strengthens and λf<0\lambda_f < 0 for those it weakens. Hard-only training weakens four reasoning-related features (e.g., conclusion connectives and proof verification) and strengthens four surface-level features (e.g., copula filler and LaTeX subscripts), and frontier-only training shifts them in the opposite direction. The intervention is removed at evaluation, so accuracy reflects only what the weights have learned.

    Accuracy differences in points relative to the size-matched natural control, for Qwen2.5-Math-1.5B trained with GRPO on MATH:

    TrainingProbeMATH500AIME24AIME25AMCMinerva MathOlympiadBenchBench. avg.
    Hard-only−1.83−2.00−0.10−0.21+0.11−2.94−1.34−1.08
    Hard-only + steering−0.39−0.20+1.25+0.41+1.73−0.37+0.29+0.52
    Frontier-only+1.63+2.80+0.31+1.05+2.64+0.37+2.98+1.69
    Frontier-only + steering−0.260.00−1.88+0.52+1.05−0.73+0.15−0.15

    The effect on accuracy is direction-specific. Steering raises hard-only accuracy on all seven evaluations, by 0.62 to 2.57 points, and lowers frontier-only accuracy on all seven, by 0.53 to 2.83 points. The two regimes use the same features with opposite signs and produce opposite accuracy changes, which a generic perturbation of the hidden states would not explain.

    Steering removes the difficulty-specific effect but keeps the benefit of RLVR training. Both steered models end up close to the natural control, within 0.4 points on the probe and 0.6 points on the benchmark average. They still improve as much as training on natural data does, so steering does not simply prevent learning. What disappears is the part of the effect that depends on difficulty, in both directions. This suggests that the difficulty-dependent generalization changes are carried largely by these feature shifts.

    The results suggest a mechanistic reading of why hard samples hurt. The frontier-only gain appears to require the weights to strengthen the reasoning-related features: when the hook supplies this strengthening during training, the weights no longer acquire it, and the gain vanishes once the hook is removed. Conversely, the hard-only deficit arises when the weights learn to suppress these features and amplify surface-level ones; when the hook absorbs this pressure, the deficit largely disappears. Hard samples thus appear to harm generalization not by lacking signal, but by pushing the weights to rewrite reasoning-related representations that easier problems also rely on.

AuthorsOpenReview ↗

Our Response (Part III/III)

Questions Part (continuing)


  1. Comment: "What are ... outperform controls?"

    Response: Across the 48 one-step trials, the prediction along the actual AdamW displacement achieves an MAE of 3.8×10−43.8\times10^{-4} in feature-activation units, with Pearson r=0.994r=0.994 and Spearman ρ=0.989\rho=0.989. Sign agreement is 97.7% (43/44 informative trials); error and correlation statistics include all 48 trials.

    However, screened sources do not consistently outperform controls on the targeted feature change. For example, one positive-screened source produces a mean change of +8.25×10−4+8.25\times10^{-4}, compared with +1.09×10−3+1.09\times10^{-3} for its random-matched control. Another positive-screened source yields a negative mean change of −6.70×10−4-6.70\times10^{-4} under fresh rollouts.

    These results support local prediction conditional on the realized update.

References

[1] Chen R, Arditi A, Sleight H, et al. Persona vectors: Monitoring and controlling character traits in language models.

Reviewer_AN8aOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates the learning dynamics of RLVR from the perspective of sample difficulty. Unlike previous methods that use static difficulty (e.g., success rate), this paper considers dynamic learning-signal availability. Specifically, this paper conducts controlled math reasoning experiments, where it derives a per-step decomposition for the GRPO update. This allows a finer investigation of a non-zero reward of a sample and its effects on the model. It also uses frozen sparse autoencoders (SAEs) to show that different difficulty regimes drive feature changes and model outputs. Overall, this paper provides an interesting perspective for RLVR.

Critical strengths
  1. The design of the decomposition of GRPO per‑step influence (into signal, transfer, and readout components) is clean and reveals why some hard samples are harmful (via the transfer kernel).

  2. The SAE-based analysis also helps to measure feature shifts w.r.t. sample difficulty, providing a feasible way beyond success rates to develop new learning methods.

Critical weaknesses
  1. The decomposition of the influence in GRPO is based on the per-step first-order Taylor expansion, which makes it hard to describe the evolution of the model during training.

  2. Only one model is investigated in this paper (Qwen2.5‑Math‑1.5B), it is unclear whether the findings in this paper apply to other models across different scales, architectures, GRPO variants, and domains.

  3. The link between feature shifts and performance degradation remains incompletely validated. Sec. 4.1 reports that hard‑sample training tends to weaken existing reasoning‑related features. Sec. 5.1 investigates how perturbing these feature directions alters model predictions. However, these experiments are performed on the frozen pretrained model, not during RLVR training.

Questions for authors
  1. All your experiments are conducted with Qwen2.5‑Math‑1.5B. Have you considered more models across different domains to verify your claims? Otherwise, your discovered patterns might only hold for this particular model/data.

  2. Could you provide training-time steering results to test whether feature shifts are indeed responsible for the generalization drop?

7 comments
AuthorsOpenReview ↗

Our Response (Part I/IV)

We thank the reviewer so much for the valuable comments and appreciation of our work and efforts. Below we address your concerns and questions point by point.


Response to Weaknesses Part

  1. Comment: "The decomposition ... training."

Response: We thank the reviewer for this insightful comment. We agree that a first-order expansion at a single checkpoint alone cannot characterize the model's evolution throughout training. In the revised manuscript, we clarify how per-step influence relates to cumulative changes and provide additional checkpoint-based measurements of training signals, internal features, and local predictive accuracy.

  • From local influence to cumulative changes. Under the local update map of Proposition 1, summing per-step changes along the realized training trajectory gives
\log\boldsymbol{\pi}_{\theta^T}(o) -\log\boldsymbol{\pi}_{\theta^0}(o) &= -\sum_{\tau=0}^{T-1}\eta_\tau \mathcal{T}_\tau(o) \mathcal{K}_\tau(o,\mathcal{U}_{x_\tau}^{\tau}) \mathcal{G}_\tau(x_\tau) \\\\ &\quad +\sum_{\tau=0}^{T-1}\boldsymbol{r}_\tau. \end{aligned}$$ where, the signal, transfer, and readout components are evaluated at each current model state, allowing their contributions to change direction and reinforce or cancel over training. Importantly, this is a cumulative decomposition with explicit Taylor remainders, not a claim that the entire trajectory can be predicted from an initial first-order expansion (Appendix B.1, Eq. 43; Appendix E.1, Eq. 47). For hard-threshold SAE readouts, the residuals also account for gate switching. - **Evolution of training signals and internal features.** Appendix E.2 (pp. 32–33) and Fig. 9 (p. 34) track 66,816 rollout groups across three training arms and three seeds. The mixed-group fraction decreases from 51.0% to 29.0% in easy-only training, remains high in frontier-only training (86.8% to 77.6%), and increases from 35.8% to 46.9% in hard-only training. On 64 fixed prompts per arm, the logit-space signal magnitude also exhibits distinct trends: it decreases by 73.0% and 20.3% in easy-only and frontier-only training, respectively, but increases by 3.19× in hard-only training. Furthermore, Appendix E.3 (p. 33) and Fig. 10 (p. 35) track 11 fixed SAE features across three checkpoints; 16 of 33 feature–arm sequences reverse their activation-change direction between consecutive intervals, demonstrating non-monotonic feature evolution. - **State-dependent local responses and predictive accuracy.** Appendix E.7 (Fig. 17, p. 44) shows that the conditional mean effective transfer to a fixed easy probe in frontier-only training changes from +5.93 to −1.46 and −3.68 across three checkpoints, despite similar signal magnitudes. We further validate local predictions at each checkpoint using controlled SGD updates with two step sizes (Appendix E.8, pp. 43–44; Fig. 18, p. 45). First-order predictions achieve 99.42%–99.43% sign agreement for output readouts and 98.43%–99.16% for SAE readouts, among effects above the $10^{-7}$ resolution threshold. Magnitude prediction is less uniform (pooled SAE correlations: 0.663 and 0.840), with gate crossings accounting for most absolute feature error. Overall, these additions clarify that our decomposition is a local analytical tool whose components evolve with the model state. Its cumulative interpretation retains explicit residuals, while direct checkpoint measurements demonstrate non-monotonic training dynamics and validate local directional predictions at multiple stages. We have clarified this scope in Section 3.3 and Appendix E.
AuthorsOpenReview ↗

Our Response (Part II/IV)

Weaknesses Part (continuing)


  1. Comment: "Only one ... and domains."

    Response: We thank the reviewer for this important point and agree that a single model is insufficient to support general claims. In the revision, we added experiments that vary each component the reviewer mentions: model scale, architecture, GRPO variants and other RL objectives, and the task domain. The results are summarized in the new Section 3.2 (Figure 2) and Appendix D.2 (Table 5), with the setup in Appendix D.1 and full results in Appendix D.2 (Tables 6–9).

  • Model scale and architecture. We trained Qwen2.5-Math-7B (larger scale) and DeepSeek-Math-7B-Instruct (a different model family with its own pretraining corpus and architecture) on MATH with GRPO.
ModelComparisonProbeMATH500AIME24AIME25AMCMinerva MathOlympiadBench
Qwen2.5-Math-1.5B (original)Frontier − control+1.63+2.80+0.31+1.05+2.64+0.37+2.98
Hard − control−1.83−2.00−0.10−0.21+0.11−2.94−1.34
Qwen2.5-Math-7BFrontier − control+0.27+1.60+1.25+1.56+0.45−0.73+1.34
Hard − control−0.47−0.40−1.66−0.31−0.52−5.52−1.34
DeepSeek-Math-7B-InstructFrontier − control+2.41+1.60+1.35+0.42+0.42+1.84+0.74
Hard − control−0.98−0.60−0.52−0.83−1.46+0.73−0.45

The asymmetry holds for both new models: frontier-only training improves on its control, and hard-only training falls below it on the held-out probe. For Qwen2.5-Math-7B, the probe-level effects are smaller, but hard-only training lowers all seven evaluations.

  • GRPO variants and other RL objectives. We trained Qwen2.5-Math-1.5B on MATH with Dr. GRPO, RLOO, and PPO with a learned critic, reusing the original difficulty labels since the base model is unchanged.
ObjectiveComparisonProbeMATH500AIME24AIME25AMCMinerva MathOlympiadBench
Dr. GRPOFrontier − control+1.43+1.80+1.98+0.83+1.43+1.47+0.74
Hard − control−2.02−1.20−1.25−0.73−1.84−1.10−2.55
RLOOFrontier − control+0.98+0.60+1.56+2.29+2.48+3.31+3.57
Hard − control−0.32−0.20−1.44−0.42−1.16−0.73−0.15
PPOFrontier − control+2.09+1.20+1.66+1.57+0.30+3.31+1.48
Hard − control−0.39−1.80−2.40−0.62−2.94−1.48−1.19

The asymmetry holds on every evaluation under all three objectives. Notably, the hard-only deficit persists under Dr. GRPO, which removes the standard-deviation normalization that upweights hard prompts with rare successes, and under PPO, whose learned critic gives non-zero advantages even when all rollouts fail. The harm of hard samples is therefore neither an artifact of GRPO's reweighting nor merely a consequence of reward sparsity.

  • Domain. We trained Qwen2.5-3B-Instruct with GRPO on Sokoban, a multi-turn agentic planning task, and evaluated it on each stratum of a held-out probe, the Sokoban test set, and FrozenLake, which is never used for training.
ComparisonEasyF-easyF-hardHardIntract.ProbeSokoban testFrozenLake
Frontier − control+14.73+41.86+27.46+18.84+3.05+18.88+13.86+9.57
Hard − control−3.10−10.85−0.71+0.10−0.87−3.25−5.47−1.76

The asymmetry is much stronger here. As on MATH (Figure 1(d)), hard-only training leaves its own stratum essentially unchanged but lowers the easier strata and both external tasks, whereas frontier-only training improves every stratum, including the hard one, and transfers to FrozenLake.

Additional experiment: an extra training dataset. We also trained all three math models on DAPO-Math with GRPO.

ModelComparisonProbeMATH500AIME24AIME25AMCMinerva MathOlympiadBench
Qwen2.5-Math-1.5BFrontier − control+3.45+3.60+2.09+0.84+2.93+1.83−0.30
Hard − control−1.44−0.60−1.77−0.31−0.53−0.37−1.39
Qwen2.5-Math-7BFrontier − control+0.84+2.20+0.52+0.21+1.28+1.84+0.61
Hard − control−2.19−0.60−2.61−1.87−0.80−4.05−0.60
DeepSeek-Math-7B-InstructFrontier − control+0.33+0.80+0.21+0.21+0.79+1.10+1.48
Hard − control−1.12−1.00−0.32−0.21−1.06−2.57−1.04

Hard-only training is below its control on all 21 evaluations. Since DAPO-Math answers are integers that rule-based verification checks reliably, the deficit is unlikely to stem mainly from verifier errors.

AuthorsOpenReview ↗

Our Response (Part III/IV)

Weaknesses Part (continuing)


  1. Comment: "The link ... RLVR training."

    Response: We thank the reviewer for this insightful comment. We agree that Section 5.1 perturbs feature directions only in the frozen base model and therefore cannot establish that the feature shifts in Section 4.1 contribute to the generalization changes. To test this link directly, we added a training-time intervention in the new Section 5.3 (Table 1), with the intervention configuration detailed in Appendix D.3.

    • Method. Following preventive steering[1], we pre-apply shifts in the directions associated with a training regime, aiming to reduce the need for the optimizer to encode those shifts in the weights. Let S={1614,603,2235,1019}\mathcal{S}=\{1614,603,2235,1019\} denote the four selected features shown in Figure 4. Extending Eq. 15 to all features in S\mathcal{S}, as in Eq. 20, we apply

      hθ,λℓ(o)=hθℓ(o)+s∑f∈Sλfdf\mathbf{h}^{\ell}_{\theta,\boldsymbol{\lambda}}(o) = \mathbf{h}^{\ell}_{\theta}(o) + s \sum_{f \in \mathcal{S}} \lambda_f \mathbf{d}_f

      at every token position oo throughout RLVR training, where df\mathbf{d}_f is the SAE decoder direction of feature ff, ss is the SAE scale factor, and the sign of λf\lambda_f follows the direction of the shift: λf>0\lambda_f > 0 for features the regime strengthens and λf<0\lambda_f < 0 for those it weakens. We jointly intervene on these four layer-13 features using the frozen shared SAE decoder directions. For each regime, each coefficient's sign follows that feature's shift relative to the corresponding natural control (Appendix D.3; Table 10). The intervention is removed at evaluation, so the reported accuracy measures the resulting trained model without a test-time activation offset. This tests whether pre-applying the selected regime-associated feature shifts during training changes downstream generalization.

    • Results. Accuracy differences in points for Qwen2.5-Math-1.5B trained with GRPO on MATH:

ComparisonProbeMATH500AIME24AIME25AMCMinerva MathOlympiadBench
Hard-only − control−1.83−2.00−0.10−0.21+0.11−2.94−1.34
Hard-only w/ steering − control−0.39−0.20+1.25+0.41+1.73−0.37+0.29
Steered − unsteered+1.44+1.80+1.35+0.62+1.62+2.57+1.63
Frontier-only − control+1.63+2.80+0.31+1.05+2.64+0.37+2.98
Frontier-only w/ steering − control−0.260.00−1.88+0.52+1.05−0.73+0.15
Steered − unsteered−1.89−2.80−2.19−0.53−1.59−1.10−2.83

Steering largely cancels the effect of each regime. For hard-only training, the deficit on the held-out probe shrinks from −1.83 to −0.39 points, the benchmark average changes from −1.08 to +0.52, and accuracy improves on all seven evaluations. Conversely, pre-applying the frontier-induced shifts removes the frontier-only gain, from +1.63 to −0.26 on the probe, and lowers accuracy on all seven evaluations.

These results provide intervention-based evidence that jointly steering the selected feature directions during training affects downstream performance. As clarified in Appendices D.3 and D.8, this experiment uses a single training seed and does not include random-direction controls or separate-feature interventions; it therefore does not establish that these directions uniquely mediate the generalization changes.

[1] Chen R, Arditi A, Sleight H, et al. Persona vectors: Monitoring and controlling character traits in language models.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers and Summary of Changes of the Revision

We sincerely thank all reviewers for their valuable comments, thoughtful questions, and constructive suggestions. In response, we have substantially revised the paper to strengthen the empirical evidence, clarify the scope of our theoretical analysis, and provide more details about our feature measurements and intervention procedures. For ease of reference, all changes in the revised manuscript are highlighted in blue.

The central message of our work is that sample difficulty affects RLVR in more ways than simply determining whether a useful reward signal is available. It also influences the direction of parameter updates, how these updates transfer across samples, how internal representations change, and ultimately how well the model generalizes. To better understand these effects, we make three main contributions:

  • A framework for understanding learning dynamics. We distinguish between the fixed difficulty of a reference sample and the changing availability of learning signals during training. Based on this distinction, we analyze GRPO updates from three perspectives: signal availability, update transfer, and their effects on model representations.
  • Controlled comparisons across training regimes. We show that training exclusively on hard samples can hurt generalization, even when learning signals remain available. In contrast, frontier-only training leads to more consistent improvements across the settings we evaluate.
  • Feature-level analysis and interventions. We examine how different features change in response to actual rollout groups and use controlled interventions to test whether selected feature directions have a causal effect on model behavior.

The main changes in the revised manuscript are summarized below:

  1. Expanded robustness experiments (Section 3.2 and Appendix D). We extend our experiments to cover different model sizes and families, MATH and DAPO-Math, four training algorithms (GRPO, PPO, RLOO, and Dr. GRPO), and agent tasks. Across ten experimental configurations, frontier-only and hard-only training consistently show opposite trends relative to their corresponding natural-data controls on held-out probes.
  2. Clarified the relationship between local influence and training dynamics (Section 3.3 and Appendices B.1 and E.1). We now make the connection between per-step changes and cumulative changes more explicit by expressing the latter as a trajectory-dependent sum that retains approximation residuals. We also clarify that the first-order expansion is intended as a local analytical tool, rather than an accurate reconstruction of the entire training trajectory.
  3. Added direct measurements and multi-checkpoint calibration (Appendix E). We now track both the availability and magnitude of training signals, measure actual feature activations at checkpoints 0, 58, and 116, and independently evaluate local predictions at each checkpoint. We distinguish between feature changes observed during actual training and those produced by controlled diagnostic updates. We also report both the agreement between predictions and measurements and the limitations caused by numerical precision and SAE gate switching.
  4. Added training-time feature interventions and their effects on accuracy (Section 5.3 and Table 1). We intervene on selected difficulty-associated SAE feature directions during training and remove these interventions before evaluation. For hard-only training, the benchmark-average performance difference relative to the control changes from −1.08 to +0.52 percentage points. For frontier-only training, it changes from +1.69 to −0.15 percentage points. These results provide intervention-based evidence that the selected feature directions during training are behaviorally relevant to downstream performance.
  5. Revised feature measurements and clarified experimental procedures (Sections 4 and 5). In the main text, we replace the previous ReasonScore results with more directly interpretable measurements: mean activation, firing rate, and feature changes relative to the control. The ReasonScore results are retained in the appendix. We also clarify the shared, frozen SAE measurement protocol and provide a more detailed description of the 48 one-step trials, including source selection, control conditions, fresh rollouts, optimizer resets, and predictions based on the actual parameter displacement. Finally, we more carefully distinguish between descriptive feature associations, the predictive accuracy of local analyses, and evidence linking feature changes to downstream behavior.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.