- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Studying the impact of problem difficulty on RL problems, the paper group problems with a base model's empirical success probability, and controlled training runs compare easy, frontier, and hard subsets. The main idea is that frontier problems improve performance, whereas training only on very hard problems can degrade easier-problem performance even when within-group advantage signals become more frequent.
Critical strengths
- The controlled difficulty-group experiment separates the availability of a nonzero learning signal from whether that signal transfers beneficially. The observation that hard examples can provide optimization signal while harming broader performance is useful.
- The signal–transfer–readout decomposition provides a coherent vocabulary for discussing why reward sparsity alone is not an adequate explanation of RLVR learning dynamics.
Critical weaknesses
- The empirical scope is too narrow to support the paper’s broad mechanistic claims: the central results rely on one model family and one mathematical dataset, with effects of only a few percentage points and limited statistical analysis. It is unclear whether the phenomenon generalizes beyond this particular RLVR regime.
- The mechanistic evidence is incomplete and largely post hoc. The one-step decomposition is a standard first-order expansion; SAE features are selected and semantically interpreted after training; the reported feature effects are very small; and a headline claim about predicting post-update feature changes is not backed by the corresponding quantitative results in the main paper. As written, the analysis redescribes the observed behavior more than it explains it.
Questions for authors
- How does this work differ from prior studies of sample difficulty in RLVR, such as DAPO's dynamic sampling (Yu et al., 2025), online difficulty filtering (Bae et al., 2025), and learnability-based curricula (Foster & Foerster, 2025)? These already select intermediate-difficulty prompts or discard zero-variance groups. Beyond confirming that frontier samples are useful, what does the signal × transfer × readout decomposition predict that these methods do not, and can you show a case where it changes which samples should be used?
- Does the hard-example degradation replicate across substantially different model families, datasets, and RL algorithms when difficulty labels and sample counts are fixed prospectively?
- Were the SAE features and their semantic labels selected before examining the downstream effects? If not, what held-out procedure demonstrates that the feature-level interpretation is not post hoc?
- Does the same picture hold for other RLVR objectives (Dr. GRPO, RLOO, PPO with a critic)?
9 comments
Our Response (Part II/V)
Weaknesses Part (continuing)
-
Comment: "The mechanistic evidence ... explains it."
Response: We address these concerns as follows.
-
Scope of the decomposition. The first-order expansion itself is standard. Our contribution is its application to GRPO to separate the realized group-relative signal, its transfer through shared parameters, and the observation-specific readout (Section 3.3; Proposition 1, Eq. 9). This provides testable predictions about individual updates. The cumulative relation retains per-step residuals and does not establish an accurate reconstruction of the full training trajectory (Appendix B.1, Eq. 43; Appendix E.1, Eq. 47).
-
Feature selection and interpretation. We acknowledge that feature discovery is exploratory. The selected feature panel is retained for later diagnostics, and the one-step calibration uses fresh rollouts with sampling seeds disjoint from screening (Section 5.2; Appendix D.6). Appendix D.7 details the selection chronology and the separate freezing of context labels before held-out label testing. We do not claim that the intervention feature set was selected before all downstream accuracy outcomes were observed. The revised descriptions are positional or formatting associations, not established reasoning functions (Appendix D.7; Table 16).
-
Small feature effects. Section 4.1 reports mean activation, firing rate, and differences from matched controls in the frozen shared SAE coordinates (Eqs. 11–13; Figure 4). Appendix D.8 quantifies the modest endpoint effects (Table 17). We also test behavioral consequences through joint training-time interventions on four selected feature directions, removed during evaluation (Section 5.3; Table 1; Appendix D.3). Steering changes the benchmark-average difference from −1.08 to +0.52 percentage points for hard-only training and from +1.69 to −0.15 for frontier-only training. These results support behavioral relevance of the selected directions, without establishing unique feature mediation (Appendix D.8).
-
Quantitative prediction results. Across the 48 one-step trials, the prediction along the actual AdamW displacement achieves MAE , Pearson , Spearman , and 97.73% sign agreement (43/44 informative trials). The prediction and measurement are defined in Section 5.2, Eq. 19, and the full statistics are reported in Appendix D.6, Table 13. Error and correlation statistics include all 48 trials. The MAE is approximately 32% of the mean absolute observed feature change, so high correlation and sign agreement coexist with non-negligible magnitude error (Appendix D.8).
-
Our Response (Part III/V)
Response to Questions
-
Comment: "How does this ... be used?"
Response: We thank the reviewer for this question. We agree that preferring intermediate-difficulty prompts is established, and we do not claim it as our contribution. These approaches select prompts using success-based statistics of the current policy. For binary rewards, DAPO's dynamic sampling retains mixed-reward groups, online difficulty filtering prioritizes an intermediate success band, and learnability-based scores such as favor intermediate success probabilities.
Our distinction is between fixed reference difficulty and evolving signal availability (Section 3.1, Eqs. 5–6), followed by a local decomposition into signal, transfer, and readout (Section 3.3; Proposition 1, Eq. 9). Success-based selection affects the availability of group-relative signals, but a scalar success rate does not determine the full realized signal , the transfer , or the readout . We provide three complementary demonstrations of information beyond success-based selection.
1. Similar signal magnitude does not imply the same effect. In Appendix C.1, the frontier-hard and easy source groups have similar logit-space signal norms (0.00901 vs. 0.00914; Table 2) but opposite signed effective transfer onto the same frontier-hard response ( vs. ; Figure 6(b), with the definition in Eq. 45). Their local predicted effects therefore have opposite signs. This shows that signal magnitude alone does not determine transfer; it does not imply that these groups have identical success rates or that every success-based criterion would rank them identically. This comparison concerns local predictions on fixed responses, with finite-step calibration restricted to the hard and frontier-hard sources (Appendices C.1 and C.3; Figure 6).
2. Signal availability does not guarantee useful training. During hard-only training, the learning-signal availability plotted in Figure 1(b) rises from 0.255 to 0.479, while hard-only training still reduces accuracy on easier held-out strata (Section 3.1; Figure 1(d)). Similarly, the score assigns the same value to current success probabilities and ; this symmetry alone gives no information about transfer to a fixed observation. Reference-difficulty labels, however, are defined by the base policy and remain fixed as current success changes.
Across all ten settings, hard-only training is below its matched control on the held-out probe (Section 3.2, Figure 2; Appendix D.2, Table 5). The deficit persists under Dr. GRPO and under PPO with a learned critic, for which all-failure groups need not have zero advantages (Appendices D.1–D.2; Table 8). Training-time steering further provides intervention-based evidence that the selected feature directions affect downstream performance, without establishing that they uniquely mediate the hard-only deficit (Section 5.3; Table 1; Appendices D.3 and D.8). These findings motivate assessing training utility beyond signal availability; they do not establish that every hard prompt is harmful or that a particular online filter should reject every such prompt.
3. Within reference-difficulty bands, update directions are heterogeneous. Among the frontier prompts analyzed in Section 4.2, the sign of the mean sample–feature influence (Eq. 14) varies within each stratum: 57.0% negative and 43.0% positive for frontier-easy, and 45.3% negative, 51.6% positive, and 3.1% zero for frontier-hard. Meanwhile, 93.4% and 68.8% of the respective rollout groups have mixed rewards (Figure 5). Thus, a shared difficulty label or the presence of reward variation does not determine whether an update strengthens or suppresses a fixed feature. The local analysis distinguishes these directions conditional on the realized rollout group. The 48 fresh-rollout trials also show that accurate prediction along a realized update does not yet establish a consistent screening advantage over matched sources (Section 5.2; Appendix D.6, Tables 13–15).
Our Response (Part I/V)
We thank the reviewer so much for the insightful comments and appreciation of our work. We address your concerns and questions in the following point by point. We also make the corresponding changes in the revision following your suggestions.
Response to Weaknesses Part
Comment: "The empirical scope is ... regime."
Response: We thank the reviewer for this important concern. In the revision, we extended the experiments along each axis the reviewer raises: model scale and family, training data, benchmarks, and RLVR regime. All settings use the same difficulty-based comparison protocol, with difficulty recalibrated for each base model and each subset compared with a natural control matched in the number of distinct questions and prompt presentations. The results are summarized in Section 3.2 (Figure 2) and Appendix D.2 (Table 5), with the setup in Appendix D.1 and full results in Appendix D.2 (Tables 6–9). All entries below are accuracy differences in percentage points relative to the control.
(1) Model scale and family. We trained Qwen2.5-Math-7B and DeepSeek-Math-7B-Instruct, a different model family, on MATH with GRPO (Appendix D.2; Table 6).
(2) Training data. We trained the same three models on DAPO-Math, a second mathematical dataset whose integer answers allow reliable rule-based verification (Appendix D.2; Table 7).
(3) Beyond mathematical RLVR. We trained Qwen2.5-3B-Instruct on Sokoban, a multi-turn agentic planning task, and evaluated it on a held-out probe, the Sokoban test set, and FrozenLake, which is never used for training (Appendix D.2; Table 9).
On Sokoban, hard-only training leaves its own stratum essentially unchanged but lowers the easier strata, whereas frontier-only training improves every stratum and transfers to FrozenLake. The degradation on easier strata is consistent with the MATH pattern in Figure 1(d).
Effect sizes and consistency. On the held-out probe, frontier-only training is above its control and hard-only training is below it in all seven settings above, and each pattern holds in 43 of 45 evaluations (Appendix D.2; Tables 5–7 and 9). The four exceptions across the two regimes are all below one point. These counts describe consistency across the evaluated configurations; we do not treat the configurations or their shared benchmarks as independent statistical replications. The effects are also not uniformly small: hard-only training lowers Minerva Math for Qwen2.5-Math-7B by 5.52 and 4.05 points after MATH and DAPO-Math training, and on Sokoban it lowers frontier-easy success by 10.85 points. All differences are measured against natural controls matched in distinct questions and prompt presentations, providing controlled comparisons of difficulty-conditioned training under the evaluated settings.