- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2(+1)
- Soundness
- 1
Read review
Paper summary
This paper investigates delayed emergence of generalization in reinforcement learning with verifiable rewards (RLVR), where models initially memorize training examples and generalize to unseen tasks after a long training process. The authors study this phenomenon from a mechanistic perspective by analyzing the spectral properties of attention OV circuits and representation orthogonality during training. They provide theoretical analyses suggesting that early RL updates can be dominated by memorization-related directions and that insufficient reward variance can suppress the learning of generalizable features. Motivated by these observations, the authors propose Directed Feature Emergence (DFE), a spectral intervention that amplifies weak singular directions of attention output projections to accelerate generalization.
Critical strengths
- The research question on delayed emergence in RLVR is interesting to me. Understanding this phenomenon is relevant to the broader question of whether RL can induce transferable capabilities beyond memorization.
- This paper analyzed this question from a mechanistic perspective using both theoretical and empirical approaches.
- For application, this paper proposed DFE to accelerate generalization.
Critical weaknesses
I think the main weakness of this paper is that the proposed mechanistic explanation is not sufficiently supported.
As I understand, the central logic of the paper is that (1) early RL training is dominated by memorization-related directions, -> (2) insufficient reward variance suppresses the learning of generalizable features, leading to delayed emergence, and -> (3) these features correspond to weak singular directions; DFE can amplify them and accelerate generalization. While this is an interesting story, I find that the proposed logic chain largely relies on assumptions and hypotheses that are not sufficiently justified by either theoretical analysis or empirical evidence. More specifically:
First, the theoretical assumptions are not well justified or connected to the real models. In fact, I feel that the theories are decorative in this sense. For example, Theorem 1 follows directly from a standard variance decomposition under the assumed subspace and gradient statistics. The key challenge is not to establish such a result under simplified assumptions, but to justify why these assumptions capture the behavior of real models. I do not find sufficient evidence for this connection.
Second, the connection between weak singular directions and generalizable features remains unclear. The paper hypothesizes that generalizable features overlap with weak singular directions of , but provides little direct evidence for this assumption. In particular, why should the reward variance bottleneck lead to weak singular directions, and why should amplifying these directions specifically promote generalization? The changes in circuit rank and orthogonality shown in Figure 2 do not directly establish this connection.
Questions for authors
Q1: DFE's intuition seems somewhat similar to Muon's spectral balancing in the sense of amplifying the small singular value, although they operate on different matrices. Prior work [1-2] suggests that Muon may amplify noisy tail directions under low-SNR RLVR gradients. Could the authors discuss this connection? In particular, if some weak singular directions mainly contain noise rather than useful features, why wouldn't DFE suffer from similar issues?
Q2: Could the authors clarify how the standard deviations in the Avg column of Table 2 were computed, and whether Tables 2 and 6 use the same three paired runs and aggregation procedure?
For BouncingSim with PPO:
- Qwen3-1.7B: Table 2 reports SDs of 0.23 (vanilla) and 0.39 (DFE), whereas Table 6 reports a paired-gain SD of 0.13.
- Qwen3-4B: Table 2 reports SDs of 0.64 (vanilla) and 0.97 (DFE), whereas Table 6 reports a paired-gain SD of 0.26.
If these statistics describe the same paired observations, they must satisfy: $sd_{\mathrm{DFE}-\mathrm{vanilla}} \geq \left|sd_{\mathrm{DFE}}-sd_{\mathrm{vanilla}}\right|. $ This gives lower bounds of 0.16 and 0.33, respectively, which exceed the reported paired-gain SDs. Could the authors explain this inconsistency?
[1] Fan et al. (2026), Rethinking Muon Beyond Pretraining: Spectral Failures and High-Pass Remedies for VLA and RLVR. [2] Liu et al. (2026), Optimizer-Model Consistency: Full Finetuning with the Same Optimizer as Pretraining Forgets Less.
6 comments
Response to Reviewer c8r1 (Part 2/3)
W2 (continued): Matched-norm controls. To further address your concern, we show on real models that raising the weak singular values ends training above every equal-norm edit of the same matrices. We repeat the transfer protocol of Fig. 3 (App. I) with eight seeds at each size, training with GRPO on the Manufactoria Basic tier only and applying Eq. (4) once at step 5 to the final 16 o_proj matrices. One setting, and , serves both sizes. An inference-only search on half of the Has prompts chose it before these training runs. Each control changes the same matrices at the same step by the Frobenius norm of the Eq. (4) edit. The Gaussian control adds isotropic noise to the whole matrix, and the rotation control mixes the weak singular directions by a random orthogonal matrix, each from one fixed draw shared by all seeds. The table reports the twelve-sub-task mean, with the test-case pass rate of Fig. 3 on the nine sub-tasks outside the Basic tier and the binary pass rate on the three Basic sub-tasks.
| Twelve-sub-task mean (%) | Qwen3-1.7B GRPO (8 seeds) | Qwen3-4B GRPO (8 seeds) |
|---|---|---|
| RL only | 7.3 | 23.8 |
| + Gaussian, equal norm | 5.3 | 22.5 |
| + rotation of the weak directions, equal norm | 0.0 | 20.7 |
| + DFE, Eq. (4) | 10.2 | 29.6 |
| DFE minus RL only | +2.93 (8/8 seeds, sign test ) | +5.75 (8/8 seeds, ) |
| DFE minus Gaussian | +4.87 (8/8 seeds) | +7.04 (8/8 seeds) |
| Runs reaching the step-150 held-out Has score of RL only | DFE 8/8 by step 30, Gaussian 0/8, rotation 0/8 | DFE 8/8 by step 30, Gaussian 0/8, rotation 0/8 |
Raising the weak singular values helps in every seed, while the Gaussian and rotation controls score below RL only at both sizes. These runs also reproduce Sec. 6.4, where DFE reaches the step-150 held-out Has score of RL only by step 30.
W2 (continued): Which directions matter. You ask why amplifying these directions specifically should help. In every seed at both sizes, it helps more than spending the same norm on other directions of the same matrices. We change one part of the edit at a time, keeping the matrices, the step and, except for the removal, the norm of Eq. (4). RL only and DFE are rerun in the same batch with the same eight seeds.
| Twelve-sub-task mean (%), second batch | Qwen3-1.7B | DFE minus row | Qwen3-4B | DFE minus row |
|---|---|---|---|---|
| RL only | 7.2 | +2.81 (8/8) | 23.8 | +5.80 (8/8) |
| + raise protected singular values | 6.5 | +3.51 (8/8) | 21.5 | +8.07 (8/8) |
| + Eq. (4) increments, random directions | 8.3 | +1.70 (8/8) | 23.7 | +5.87 (8/8) |
| + remove weak components, not norm-matched | 1.2 | +8.82 (8/8) | 21.0 | +8.58 (8/8) |
| + DFE, Eq. (4) | 10.0 | 29.6 |
Spending the same norm on the protected singular values lowers the score below RL only in every seed at both sizes, and removing the weak components lowers the mean further, so raising the weak components is the helpful direction. Placing the same increments of Eq. (4) along random orthonormal directions falls short of DFE in every seed, so only raising the singular values of the existing weak directions of gives the full gain. The revision adds this study as Sec. 6.5, with every control in App. I.
Response to Reviewer c8r1 (Part 3/3)
Q1: Connection to Muon and noisy tail directions. Thank you for this connection. DFE differs from Muon in two ways, and raising the weak directions of does better than adding noise of the same size. First, Muon orthogonalizes every gradient update, which, as you note, may amplify noisy tail directions under low-SNR RLVR gradients (Fan et al., 2026, your reference [1]). DFE instead edits the weight once from its singular value decomposition and uses no gradient estimate. Second, DFE keeps the protected directions exactly and moves each weak singular value a fraction of the way toward the protected mean, which bounds the change in spectral norm by (Sec. 5).
Your question, whether some weak directions mainly carry noise, has a direct test. If the weak singular directions of carried only noise, raising them would do no better than equal-norm noise inside the weak subspace or the same increments along random directions. Both of these edits, each from one fixed draw, end below DFE in every seed at both sizes.
| Twelve-sub-task mean (%), second batch | Qwen3-1.7B | Qwen3-4B |
|---|---|---|
| RL only | 7.2 | 23.8 |
| + Gaussian in weak subspace | 4.6 | 23.9 |
| + Eq. (4) increments, random directions | 8.3 | 23.7 |
| + DFE, existing weak directions | 10.0 | 29.6 |
| DFE minus these two edits | +5.39 / +1.70 (8/8 each) | +5.65 / +5.87 (8/8 each) |
Raising the weak singular values also helps under three RL algorithms at Qwen3-4B, while the equal-norm Gaussian control lowers the score under each of them.
| Qwen3-4B, twelve-sub-task mean (%) | GRPO (8 seeds) | PPO with learned critic (3 seeds) | DAPO (3 seeds) |
|---|---|---|---|
| RL only | 23.8 | 21.4 | 26.5 |
| + Gaussian, equal norm | 22.5 | 17.9 | 23.3 |
| + DFE, Eq. (4) | 29.6 | 25.3 | 30.4 |
| DFE minus RL only | +5.75 (8/8 seeds) | +3.9 (3/3 seeds) | +3.9 (3/3 seeds) |
| DFE minus Gaussian | +7.04 (8/8 seeds) | +7.4 (3/3 seeds) | +7.1 (3/3 seeds) |
Each difference is the mean of the per-seed differences, computed before rounding. The revision discusses Muon in Sec. 2, citing Tveit et al. (2025), Fan et al. (2026) and Liu et al. (2026), with the comparison table in App. K.
Q2: Standard deviations in Tables 2 and 6. Thank you for this careful check. As you show, for the two BouncingSim PPO rows the Avg-column of Table 2 and the paired SD of Table 6 cannot both describe the same three paired runs. The revision therefore removes the Avg-column of Table 2 and Table 6 of the submission. App. I of the revision (Table 14) reports the per-seed gains of the controlled study. The table summarizes these paired gains over every seed.
| Twelve-sub-task mean, DFE minus RL only (points) | Seeds with a gain | Mean | Range over seeds | Sample SD |
|---|---|---|---|---|
| Qwen3-1.7B GRPO (8 seeds) | 8/8 | +2.93 | +2.49 to +3.20 | 0.23 |
| Qwen3-4B GRPO (8 seeds) | 8/8 | +5.75 | +4.13 to +6.60 | 0.88 |
We hope these results address your concerns, and we would be happy to provide any further clarification.
Response to Reviewer c8r1 (Part 1/3)
We thank Reviewer c8r1 for recognizing that delayed emergence in RLVR is relevant "to the broader question of whether RL can induce transferable capabilities beyond memorization", and that the paper analyzes it "from a mechanistic perspective". Below we answer each weakness and question, with matched-norm tests in real models.
W1: Connecting the theory to real models. Each theorem covers one link of your chain, and the controlled study in the new Sec. 6.5 tests in real models, Qwen3-1.7B and Qwen3-4B, what the third link predicts. Under Assumptions 1 and 2, Theorem 1 shows that at the initial weights, when ∣μ∣≪cm and the sampling terms are small, the expected gradient energy concentrates in memorization directions. This imbalance, associated with the memorization plateau, persists with more rollouts, since they reduce estimation noise but leave the population alignment μ unchanged. Theorem 2 uses neither assumption, since it is an exact identity for a softmax policy with binary reward. Theorem 3 rests on Assumption 3 and predicts that raising the singular value of an aligned direction shortens its emergence time. To make the argument easier to follow, Sec. 4 of the revision opens with a roadmap that maps each link of your chain to its theorem and states each assumption with its intuition.
To see how the edit acts in real models, we separate it from the RL that follows. We apply DFE and two equal-norm controls to the initial model and evaluate them with no RL update. The controls place the increments of Eq. (4) along random directions or add Gaussian noise inside the weak subspace, each from one fixed draw. We then compare them with a second batch of eight-seed runs of the same conditions, which apply the same edits at step 5 and train for 150 GRPO updates (Table 12 of the revision).
At Qwen3-1.7B, RL gains twice as much after DFE as without an edit, consistent with Theorem 3.
On the eight sub-tasks that play no part in training or tuning at Qwen3-4B, the edit alone already lifts the mean from 8.5 to 15.9. The same increments along random directions reach 12.4.
W2: Why weak directions, and why amplifying them helps. The reward-variance bottleneck holds back the weak singular directions the longest. Low reward variance slows the growth of every direction, since Eq. (3) scales the expected gradient of each prompt by p(1−p) under binary reward. The singular value then decides which directions are slowest, because under Assumption 3 a coordinate with singular value s grows at rate ηs2. A generalizing coordinate that lies in a weak direction of WO is therefore the last to emerge, and Theorem 3 predicts that raising its singular value shortens its emergence time. Zhu et al. (2025) find that RLVR leaves the weight spectrum nearly unchanged, so DFE raises these singular values directly. The matched-norm controls of the new Sec. 6.5 test what this link predicts, namely that raising the weak singular values helps more than equal-norm edits of the same matrices. Next to Fig. 2, the revision points to these controls as the functional test of the edit (App. D.1).