acceptodds
Under review as a conference paper at ICLR 2027

Demystifying and Accelerating Emergence to New Tasks in Reinforcement Learning

Abstract

Reinforcement learning with verifiable rewards (RLVR) has shown strong gains in complex reasoning for large language models, but a central question remains whether current post-training techniques truly incentivize models to generalize to new tasks, or whether they mainly reinforce patterns already present in the training data. In this work, we take a mechanistic perspective on this question by investigating a phenomenon called delayed emergence, where a policy first overfits observed rollouts through memorization and much later transitions to out-of-distribution generalization. By analyzing the circuits during training, we show that this capability transition is characterized by a non-monotonic trajectory in the proposed circuit rank and orthogonality index, reflecting a shift from memorization circuits to generalizing circuits. We further identify a critical bottleneck in the RLVR gradient, where low reward variance on training prompts can weaken the signal for transferable features and delay generalization. Based on this insight, we propose Directed Feature Emergence (DFE), a spectral intervention that provably encourages learning on the weak directions in the circuit while preserving the useful representation. Experiments on DELTA-Code, including Manufactoria and BouncingSim, show that DFE consistently accelerates delayed generalization under PPO and GRPO across Qwen3 models, especially in extreme out-of-distribution settings where the base model obtains nearly zero reward. Our results suggest that circuit-level dynamics provide both a diagnostic view of when RLVR moves beyond memorization and a practical target for accelerating task emergence.

Primary area:
reinforcement learning
Keywords:
RLVR, Task Emergence, LLMs

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_c8r1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2(+1)
Soundness
1
Read review
Paper summary

This paper investigates delayed emergence of generalization in reinforcement learning with verifiable rewards (RLVR), where models initially memorize training examples and generalize to unseen tasks after a long training process. The authors study this phenomenon from a mechanistic perspective by analyzing the spectral properties of attention OV circuits and representation orthogonality during training. They provide theoretical analyses suggesting that early RL updates can be dominated by memorization-related directions and that insufficient reward variance can suppress the learning of generalizable features. Motivated by these observations, the authors propose Directed Feature Emergence (DFE), a spectral intervention that amplifies weak singular directions of attention output projections to accelerate generalization.

Critical strengths
  1. The research question on delayed emergence in RLVR is interesting to me. Understanding this phenomenon is relevant to the broader question of whether RL can induce transferable capabilities beyond memorization.
  2. This paper analyzed this question from a mechanistic perspective using both theoretical and empirical approaches.
  3. For application, this paper proposed DFE to accelerate generalization.
Critical weaknesses

I think the main weakness of this paper is that the proposed mechanistic explanation is not sufficiently supported.

As I understand, the central logic of the paper is that (1) early RL training is dominated by memorization-related directions, -> (2) insufficient reward variance suppresses the learning of generalizable features, leading to delayed emergence, and -> (3) these features correspond to weak singular directions; DFE can amplify them and accelerate generalization. While this is an interesting story, I find that the proposed logic chain largely relies on assumptions and hypotheses that are not sufficiently justified by either theoretical analysis or empirical evidence. More specifically:

First, the theoretical assumptions are not well justified or connected to the real models. In fact, I feel that the theories are decorative in this sense. For example, Theorem 1 follows directly from a standard variance decomposition under the assumed subspace and gradient statistics. The key challenge is not to establish such a result under simplified assumptions, but to justify why these assumptions capture the behavior of real models. I do not find sufficient evidence for this connection.

Second, the connection between weak singular directions and generalizable features remains unclear. The paper hypothesizes that generalizable features overlap with weak singular directions of WOW_O, but provides little direct evidence for this assumption. In particular, why should the reward variance bottleneck lead to weak singular directions, and why should amplifying these directions specifically promote generalization? The changes in circuit rank and orthogonality shown in Figure 2 do not directly establish this connection.

Questions for authors

Q1: DFE's intuition seems somewhat similar to Muon's spectral balancing in the sense of amplifying the small singular value, although they operate on different matrices. Prior work [1-2] suggests that Muon may amplify noisy tail directions under low-SNR RLVR gradients. Could the authors discuss this connection? In particular, if some weak singular directions mainly contain noise rather than useful features, why wouldn't DFE suffer from similar issues?

Q2: Could the authors clarify how the standard deviations in the Avg column of Table 2 were computed, and whether Tables 2 and 6 use the same three paired runs and aggregation procedure?

For BouncingSim with PPO:

  • Qwen3-1.7B: Table 2 reports SDs of 0.23 (vanilla) and 0.39 (DFE), whereas Table 6 reports a paired-gain SD of 0.13.
  • Qwen3-4B: Table 2 reports SDs of 0.64 (vanilla) and 0.97 (DFE), whereas Table 6 reports a paired-gain SD of 0.26.

If these statistics describe the same paired observations, they must satisfy: $sd_{\mathrm{DFE}-\mathrm{vanilla}} \geq \left|sd_{\mathrm{DFE}}-sd_{\mathrm{vanilla}}\right|. $ This gives lower bounds of 0.16 and 0.33, respectively, which exceed the reported paired-gain SDs. Could the authors explain this inconsistency?

[1] Fan et al. (2026), Rethinking Muon Beyond Pretraining: Spectral Failures and High-Pass Remedies for VLA and RLVR. [2] Liu et al. (2026), Optimizer-Model Consistency: Full Finetuning with the Same Optimizer as Pretraining Forgets Less.

6 comments
AuthorsOpenReview ↗

Response to Reviewer c8r1 (Part 1/3)

We thank Reviewer c8r1 for recognizing that delayed emergence in RLVR is relevant "to the broader question of whether RL can induce transferable capabilities beyond memorization", and that the paper analyzes it "from a mechanistic perspective". Below we answer each weakness and question, with matched-norm tests in real models.

W1: Connecting the theory to real models. Each theorem covers one link of your chain, and the controlled study in the new Sec. 6.5 tests in real models, Qwen3-1.7B and Qwen3-4B, what the third link predicts. Under Assumptions 1 and 2, Theorem 1 shows that at the initial weights, when ∣μ∣≪cm|\mu| \ll c_m and the sampling terms are small, the expected gradient energy concentrates in memorization directions. This imbalance, associated with the memorization plateau, persists with more rollouts, since they reduce estimation noise but leave the population alignment μ\mu unchanged. Theorem 2 uses neither assumption, since it is an exact identity for a softmax policy with binary reward. Theorem 3 rests on Assumption 3 and predicts that raising the singular value of an aligned direction shortens its emergence time. To make the argument easier to follow, Sec. 4 of the revision opens with a roadmap that maps each link of your chain to its theorem and states each assumption with its intuition.

To see how the edit acts in real models, we separate it from the RL that follows. We apply DFE and two equal-norm controls to the initial model and evaluate them with no RL update. The controls place the increments of Eq. (4) along random directions or add Gaussian noise inside the weak subspace, each from one fixed draw. We then compare them with a second batch of eight-seed runs of the same conditions, which apply the same edits at step 5 and train for 150 GRPO updates (Table 12 of the revision).

Qwen3-1.7B, twelve-sub-task mean (%)After the edit and RLGain during RL
No edit7.2+2.8
+ Gaussian in weak subspace4.6+2.4
+ Eq. (4) increments, random directions8.3+3.2
+ DFE, Eq. (4)10.0+5.7

At Qwen3-1.7B, RL gains twice as much after DFE as without an edit, consistent with Theorem 3.

On the eight sub-tasks that play no part in training or tuning at Qwen3-4B, the edit alone already lifts the mean from 8.5 to 15.9. The same increments along random directions reach 12.4.

W2: Why weak directions, and why amplifying them helps. The reward-variance bottleneck holds back the weak singular directions the longest. Low reward variance slows the growth of every direction, since Eq. (3) scales the expected gradient of each prompt by p(1−p)p(1-p) under binary reward. The singular value then decides which directions are slowest, because under Assumption 3 a coordinate with singular value ss grows at rate ηs2\eta s^2. A generalizing coordinate that lies in a weak direction of WOW_O is therefore the last to emerge, and Theorem 3 predicts that raising its singular value shortens its emergence time. Zhu et al. (2025) find that RLVR leaves the weight spectrum nearly unchanged, so DFE raises these singular values directly. The matched-norm controls of the new Sec. 6.5 test what this link predicts, namely that raising the weak singular values helps more than equal-norm edits of the same matrices. Next to Fig. 2, the revision points to these controls as the functional test of the edit (App. D.1).

AuthorsOpenReview ↗

Response to Reviewer c8r1 (Part 2/3)

W2 (continued): Matched-norm controls. To further address your concern, we show on real models that raising the weak singular values ends training above every equal-norm edit of the same matrices. We repeat the transfer protocol of Fig. 3 (App. I) with eight seeds at each size, training with GRPO on the Manufactoria Basic tier only and applying Eq. (4) once at step 5 to the final 16 o_proj matrices. One setting, α=0.6\alpha = 0.6 and γ=0.90\gamma = 0.90, serves both sizes. An inference-only search on half of the Has prompts chose it before these training runs. Each control changes the same matrices at the same step by the Frobenius norm of the Eq. (4) edit. The Gaussian control adds isotropic noise to the whole matrix, and the rotation control mixes the weak singular directions by a random orthogonal matrix, each from one fixed draw shared by all seeds. The table reports the twelve-sub-task mean, with the test-case pass rate of Fig. 3 on the nine sub-tasks outside the Basic tier and the binary pass rate on the three Basic sub-tasks.

Twelve-sub-task mean (%)Qwen3-1.7B GRPO (8 seeds)Qwen3-4B GRPO (8 seeds)
RL only7.323.8
+ Gaussian, equal norm5.322.5
+ rotation of the weak directions, equal norm0.020.7
+ DFE, Eq. (4)10.229.6
DFE minus RL only+2.93 (8/8 seeds, sign test p=0.0078p = 0.0078)+5.75 (8/8 seeds, p=0.0078p = 0.0078)
DFE minus Gaussian+4.87 (8/8 seeds)+7.04 (8/8 seeds)
Runs reaching the step-150 held-out Has score of RL onlyDFE 8/8 by step 30, Gaussian 0/8, rotation 0/8DFE 8/8 by step 30, Gaussian 0/8, rotation 0/8

Raising the weak singular values helps in every seed, while the Gaussian and rotation controls score below RL only at both sizes. These runs also reproduce Sec. 6.4, where DFE reaches the step-150 held-out Has score of RL only by step 30.

W2 (continued): Which directions matter. You ask why amplifying these directions specifically should help. In every seed at both sizes, it helps more than spending the same norm on other directions of the same matrices. We change one part of the edit at a time, keeping the matrices, the step and, except for the removal, the norm of Eq. (4). RL only and DFE are rerun in the same batch with the same eight seeds.

Twelve-sub-task mean (%), second batchQwen3-1.7BDFE minus rowQwen3-4BDFE minus row
RL only7.2+2.81 (8/8)23.8+5.80 (8/8)
+ raise protected singular values6.5+3.51 (8/8)21.5+8.07 (8/8)
+ Eq. (4) increments, random directions8.3+1.70 (8/8)23.7+5.87 (8/8)
+ remove weak components, not norm-matched1.2+8.82 (8/8)21.0+8.58 (8/8)
+ DFE, Eq. (4)10.029.6

Spending the same norm on the protected singular values lowers the score below RL only in every seed at both sizes, and removing the weak components lowers the mean further, so raising the weak components is the helpful direction. Placing the same increments of Eq. (4) along random orthonormal directions falls short of DFE in every seed, so only raising the singular values of the existing weak directions of WOW_O gives the full gain. The revision adds this study as Sec. 6.5, with every control in App. I.

AuthorsOpenReview ↗

Response to Reviewer c8r1 (Part 3/3)

Q1: Connection to Muon and noisy tail directions. Thank you for this connection. DFE differs from Muon in two ways, and raising the weak directions of WOW_O does better than adding noise of the same size. First, Muon orthogonalizes every gradient update, which, as you note, may amplify noisy tail directions under low-SNR RLVR gradients (Fan et al., 2026, your reference [1]). DFE instead edits the weight WOW_O once from its singular value decomposition and uses no gradient estimate. Second, DFE keeps the protected directions exactly and moves each weak singular value a fraction α\alpha of the way toward the protected mean, which bounds the change in spectral norm by ασˉP\alpha\bar\sigma_P (Sec. 5).

Your question, whether some weak directions mainly carry noise, has a direct test. If the weak singular directions of WOW_O carried only noise, raising them would do no better than equal-norm noise inside the weak subspace or the same increments along random directions. Both of these edits, each from one fixed draw, end below DFE in every seed at both sizes.

Twelve-sub-task mean (%), second batchQwen3-1.7BQwen3-4B
RL only7.223.8
+ Gaussian in weak subspace4.623.9
+ Eq. (4) increments, random directions8.323.7
+ DFE, existing weak directions10.029.6
DFE minus these two edits+5.39 / +1.70 (8/8 each)+5.65 / +5.87 (8/8 each)

Raising the weak singular values also helps under three RL algorithms at Qwen3-4B, while the equal-norm Gaussian control lowers the score under each of them.

Qwen3-4B, twelve-sub-task mean (%)GRPO (8 seeds)PPO with learned critic (3 seeds)DAPO (3 seeds)
RL only23.821.426.5
+ Gaussian, equal norm22.517.923.3
+ DFE, Eq. (4)29.625.330.4
DFE minus RL only+5.75 (8/8 seeds)+3.9 (3/3 seeds)+3.9 (3/3 seeds)
DFE minus Gaussian+7.04 (8/8 seeds)+7.4 (3/3 seeds)+7.1 (3/3 seeds)

Each difference is the mean of the per-seed differences, computed before rounding. The revision discusses Muon in Sec. 2, citing Tveit et al. (2025), Fan et al. (2026) and Liu et al. (2026), with the comparison table in App. K.

Q2: Standard deviations in Tables 2 and 6. Thank you for this careful check. As you show, for the two BouncingSim PPO rows the Avg-column ±\pm of Table 2 and the paired SD of Table 6 cannot both describe the same three paired runs. The revision therefore removes the Avg-column ±\pm of Table 2 and Table 6 of the submission. App. I of the revision (Table 14) reports the per-seed gains of the controlled study. The table summarizes these paired gains over every seed.

Twelve-sub-task mean, DFE minus RL only (points)Seeds with a gainMeanRange over seedsSample SD
Qwen3-1.7B GRPO (8 seeds)8/8+2.93+2.49 to +3.200.23
Qwen3-4B GRPO (8 seeds)8/8+5.75+4.13 to +6.600.88

We hope these results address your concerns, and we would be happy to provide any further clarification.

Reviewer_FLpCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies delayed generalization emergence in RL with verifiable rewards, where training reward saturates long before performance on held-out or out-of-distribution tasks improves. From a toy modular-arithmetic setting, the authors argue that this delay reflects a slow shift from memorization to generalization in the attention circuit, which is amplified by low reward variance on saturated prompts. Building on this, they propose DFE, a one-shot spectral edit that strengthens the weak singular directions of late-layer attention output projections. On the DELTA-Code benchmarks, their method (DFE) improves PPO and GRPO across several Qwen3 model sizes.

Critical strengths

1 - The question this paper is addressing is important and fundamental: whether RLVR teaches transferable knowledge or only sharpens behavior the model already has.

2- The method is cheap and practical. It is a single edit that takes seconds and drops into existing PPO, GRPO or DAPO pipelines.

3- The gains are relatively consistent across model sizes, optimizers and benchmarks, and some are large (e.g., 18.6 →\rightarrow 33.8 average for Qwen3-4B GRPO on Manufactoria).

4- Ablations, compute overhead and limitations are reported openly.

5- The paper is clearly written and organized.

Critical weaknesses

1- Positioning and novelty.\textbf{Positioning and novelty.}

(a) DELTA-Code [1] already documented grokking-like transitions under RL, and this paper builds on its benchmark and reward schedule. The novelty is the spectral analysis and the intervention, and the framing should say so.

(b) There are prior works that accelerate delayed generalization in supervised settings (Grokfast [2], Muon [3], weight decay [4, 5]) that are neither discussed nor compared. The novelty and contribution of the paper without positioning it against those prior works are not clear.

(c) Theorem 2 is the standard policy-gradient identity for binary rewards, and nothing in its proof is specific to transformers or output-value circuits. The effect of low reward variance on the policy gradient was formalized by [6], which is not cited.

(d) Closely related spectral work is missing. [7] finds that RLVR mainly updates off the principal directions. [8] improves reasoning by removing exactly the small singular components that DFE amplifies. Again, without proper positioning against prior work, the paper is hard to judge.

2- Theory is based on unchecked assumptions.\textbf{Theory is based on unchecked assumptions.}

Based on my understanding, the employed assumptions are not empirically grounded or properly justified. Theorem 1 assumes a split into memorization and generalization subspaces with particular statistics, and its conclusion follows almost directly from those assumptions. None of these assumptions are checked empirically, and they appear only in the appendix without any hint or intuition for them.

3- Key hypotheses are not tested.\textbf{Key hypotheses are not tested.}

(a) The claim that useful directions overlap with the low-energy part of WOW_O ("We hypothesize that while the generalization subspace ...") is never tested directly. So, we basically trust a hypothesis (which is mainly induced by our own assumptions) and never really test it. Also, the higher circuit rank under DFE is partly produced by the edit itself, so it cannot serve as independent evidence of a generalizing circuit (this seems like circular evidence to me). In the Questions section, I provide another interpretation of the results alone (without the theoretical results, which are direct consequences of our own assumptions), which also seems reasonable to me.

(b) The paper does not explain how a weight edit relieves a bottleneck that comes from reward variance. Based on my understanding, we can still have this problem of degraded rank even if we strengthen the weak directions under extreme cases of zero reward variance.

For instance, the abstract says DFE helps most where the base model gets near-zero reward. Under Theorem 2, though, a prompt the model almost never solves (p≈0p \approx 0) contributes almost no gradient, so DFE has no training signal to amplify there. To me, the results suggestion a different conclusion. The largest BouncingSim gains are on families where the base model already succeeds sometimes (Gravity, MultiObj). The hardest Manufactoria sub-tasks (BitOp, Symm, Add) are evaluation-only, so any gains there come from transfer. The authors should say which they mean: does DFE amplify existing reward signal, or does it improve transfer to unseen tasks?

4- Overstated claims.\textbf{Overstated claims.}

It seems to me that some words imply more than what the method suggests. For instance, the term "directed" in the name of the method suggests a smart selection of the directions with small eigenvalues that actually contribute to generalizable knowledge, while the method strengthens all weak directions without distinguishing between them. Similarly, "circuits" are basically aggregated spectral statistics, while the term "circuits" implies neural pathways or some more explicit representation of the network.

[1] Y. Sun et al. DELTA-Code: How Does RL Unlock and Transfer New Programming Algorithms in LLMs? ICLR 2026

[2] J. Lee et al. Grokfast: Accelerated Grokking by Amplifying Slow Gradients. 2024

[3] A. Tveit, B. Remseth, A. Skogvold. Muon Optimizer Accelerates Grokking. arXiv:2504.16041, 2025.

[4] A. Power et al. Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets. arXiv:2201.02177, 2022.

[5] Z. Liu, E. J. Michaud, M. Tegmark. Omnigrok: Grokking Beyond Algorithmic Data. ICLR 2023.

[6] N. Razin et al. Vanishing Gradients in Reinforcement Finetuning of Language Models. ICLR 2024.

[7] H. Zhu et al. The Path Not Taken: RLVR Provably Learns Off the Principals. arXiv:2511.08567, 2025.

[8] P. Sharma, J. T. Ash, D. Misra. The Truth is in There: Improving Reasoning in Language Models with Layer-Selective Rank Reduction. ICLR 2024.

Questions for authors

1- Another interpretation:

In Fig. 2, the vanilla runs maintain a low circuit rank and learn slowly, while DFE raises the rank and learns faster. This resembles plasticity loss under non-stationary training, for which rank preservation is a known fix. The ablations point in the same direction: (i) editing qprojq_{\text{proj}}, which is not part of the OV circuit, still helps (Table 11); (ii) editing before the plateau captures most of the gain (Table 10); (iii) results are insensitive to α\alpha (Table 3).

Without a proper baselines and further analysis, this interpretation cannot be ruled out.

2- Please add a related-work discussion covering [6], Grokfast, Muon, weight decay, spectral regularization and plasticity work. Please also compare against the most relevant ones of these under the same or comparable compute budget. Theorem 2 should be credited as an instance of a known result.

3- Can you estimate the memorization and generalization subspaces empirically and check Theorem 1’s assumptions over training? In the toy model, the subspace spanned by the key Fourier frequencies would be a natural candidate for the generalization subspace. Please also bring the assumptions into the main text.

4- Does DFE work by moving prompts away from always-succeed or always-fail states? Please report per-prompt success rates, entropy, and KL divergence just before and after the edit.

5- In Fig. 2, the DFE and vanilla curves differ from the start, even though the edit is supposed to happen once at the plateau, and paired runs should be identical before it. Which schedule does Fig. 2 use?

6- Please use one DFE configuration across experiments, or provide a rule for choosing it. Also, specify plateau detection for the LLM benchmarks, and soften the terms “directed,” “circuit,” and “provably.”

10 comments
AuthorsOpenReview ↗

Response to Reviewer FLpC (Part 1/4)

We thank Reviewer FLpC for recognizing that the question is "important and fundamental", that DFE is "cheap and practical", and that the gains are "relatively consistent across model sizes, optimizers and benchmarks, and some are large". Below we test the low-energy hypothesis directly, compare DFE with the plasticity baselines you asked for, and answer each remaining point.

W1(a): Credit to DELTA-Code. Our contributions are the spectral account of the grokking-like transitions that DELTA-Code (Sun et al., 2026) documented under RL on the benchmark we build on, and the intervention that follows from this account. The revision credits DELTA-Code for these transitions in Sec. 1 and in the contributions.

W1(b) and Q2: Grokfast, Muon, weight decay, spectral and plasticity methods. Following your suggestion, we discuss all of these methods in Sec. 2 of the revision, with the comparison below as a table in App. K.

MethodWhat it changesWhen
Weight decay (Power et al., 2022; Liu et al., 2023)shrinks all weightsevery update
Grokfast (Lee et al., 2024)amplifies the slow component of the gradientevery update
Muon (Tveit et al., 2025; Fan et al., 2026)pushes all singular values of each update toward 1every update
Spectral regularization (Lewandowski et al., 2024)keeps the largest singular value of each layer near 1every update
LASER (Sharma et al., 2024)removes small singular components, no further trainingonce
Shrink-and-perturb, resets (Ash & Adams, 2020; Nikishin et al., 2022)shrinks and perturbs, or reinitializes, the weightsonce or periodically
DFE (ours)raises the weak singular values of WOW_O and keeps the protected directionsonce, then RL with the same optimizer

The first four methods act on every update, while DFE edits the weights once and keeps the optimizer type and its settings. In RLVR, Fan et al. (2026) report that Muon collapses under GRPO on Qwen3-1.7B and 4B.

W1(b) and Q2: Comparison under the same compute. DFE runs on the same RL budget as RL only, since the edit is a one-time step that adds no RL update. For the four WOW_O matrices of Tables 1 and 2, it takes 1.5 s at Qwen3-1.7B and 2.7 s at Qwen3-4B on one B200 (Table 19 of the revision). DFE and every intervention below therefore share the same budget of 150 GRPO updates under the transfer protocol of Fig. 3 (App. I).

We compare DFE under this budget with the interventions of the plasticity and spectral literature that act once, as DFE does. Shrink-and-perturb edits the same 16 o_proj matrices at step 5 with the norm of the Eq. (4) edit, as a shrink toward zero plus one fixed draw of Gaussian noise of equal norm. The AdamW restart starts a fresh AdamW state at step 5 without an edit, and since DFE and every control also restart AdamW at the edit, this row isolates the restart. The last control removes the weak components of the same matrices, the direction of LASER (Sharma et al., 2024). These runs come from a second batch of our controlled study, which reruns RL only and DFE with the same eight seeds.

Twelve-sub-task mean (%), second batchQwen3-1.7BDFE minus rowQwen3-4BDFE minus row
RL only7.2+2.81 (8/8)23.8+5.80 (8/8)
+ AdamW restart, no edit7.2+2.73 (8/8)24.2+5.37 (8/8)
+ shrink-and-perturb, equal norm0.4+9.62 (8/8)24.5+5.04 (8/8)
+ remove weak components, not norm-matched1.2+8.82 (8/8)21.0+8.58 (8/8)
+ DFE, Eq. (4)10.029.6

Under the same budget, DFE is higher than each of these interventions in all eight seeds at both sizes, and Table 10 of the revision reports them.

W1(c): Credit for Theorem 2. The revision credits Razin et al. (ICLR 2024) in the related work and presents Theorem 2 as an instance of their result in Sec. 4. Theorem 2 (Eq. (3)) names the factor p(1−p)p(1-p) that scales the learning signal of each training prompt. The part of the analysis specific to transformers is the edit of WOW_O and its effect on WOVW_{OV} (Sec. 5, and the paragraph on induced circuit amplification in App. C.4).

AuthorsOpenReview ↗

Response to Reviewer FLpC (Part 2/4)

W1(d): Spectral work. The revision discusses both works in Sec. 2. Zhu et al. (2025) find that RLVR updates lie off the principal directions and leave the weight spectrum nearly unchanged. DFE likewise keeps the principal directions fixed, and it raises the weak singular values that RLVR leaves nearly unchanged in their study. LASER removes small components and evaluates the edited model without further training, whereas DFE raises weak components and continues RL. Under our protocol, removing the weak components of the same 16 matrices at step 5 and continuing RL ends below RL only in every seed at both sizes. When RL continues after the edit, raising the weak components is therefore the helpful direction.

W2 and Q3: Assumptions. The revision brings the assumptions into the main text. Sec. 4 opens with a roadmap and states each assumption with its intuition, with the formal statements in App. C.1. Under these assumptions, Theorem 1 is a local result at the initial weights. It shows that, when ∣μ∣≪cm|\mu| \ll c_m, the gradient imbalance at the start of RL persists with more rollouts, since they reduce estimation noise but leave the population alignment unchanged. In the modular task, the subspace of the key Fourier frequencies (Nanda et al., 2023) is the natural generalization subspace, as you suggest, and the revision names it in Sec. 4. In LLMs these subspaces have no closed form, so we test the consequence the method relies on. DFE, which raises the weak singular values of WOW_O, ends training above every equal-norm edit of the same matrices at the same step, in every seed at both sizes.

W3(a): Testing the low-energy hypothesis. We now test the hypothesis directly, with equal-norm edits of the same matrices, so the evidence does not rest on circuit rank, a spectral statistic of the OV circuit (App. D.1). We repeat the transfer protocol of Fig. 3 (App. I) with eight seeds at each size, training with GRPO on the Manufactoria Basic tier only and applying Eq. (4) once at step 5 to the final 16 o_proj matrices. Each control changes the same matrices at the same step by the Frobenius norm of the Eq. (4) edit. The Gaussian control adds isotropic noise to the whole matrix, and the rotation control mixes the weak singular directions by a random orthogonal matrix, each from one fixed draw shared by all seeds. The table reports the twelve-sub-task mean, with the test-case pass rate of Fig. 3 on the nine sub-tasks outside the Basic tier and the binary pass rate on the three Basic sub-tasks.

Twelve-sub-task mean (%)Qwen3-1.7B GRPO (8 seeds)Qwen3-4B GRPO (8 seeds)
RL only7.323.8
+ Gaussian, equal norm5.322.5
+ rotation of the weak directions, equal norm0.020.7
+ DFE, Eq. (4)10.229.6
DFE minus RL only / Gaussian+2.93 / +4.87 (8/8 each)+5.75 / +7.04 (8/8 each)

Raising the weak singular values helps in every seed, and Gaussian edits at half and a quarter of this norm also end below DFE in every seed at both sizes. To test the low-energy hypothesis itself, we then change only which directions receive the change, with the same matrices, step and norm, in the second batch.

Twelve-sub-task mean (%), second batchQwen3-1.7BDFE minus rowQwen3-4BDFE minus row
RL only7.2+2.81 (8/8)23.8+5.80 (8/8)
+ raise high-energy singular values6.5+3.51 (8/8)21.5+8.07 (8/8)
+ Eq. (4) increments, random directions8.3+1.70 (8/8)23.7+5.87 (8/8)
+ DFE, raise low-energy singular values10.029.6

Spending the same norm on the high-energy directions lowers the score below RL only in every seed, and placing the same increments along random directions falls short of DFE in every seed. So the gain is largest when the increments go to the low-energy directions of WOW_O, as the hypothesis states. We have added this study to the revision as the new Sec. 6.5.

AuthorsOpenReview ↗

Response to Reviewer FLpC (Part 3/4)

Q1: The plasticity reading. Thank you for this reading. The plasticity baselines you asked for, the AdamW restart and random edits that raise the effective rank of WOW_O (one fixed draw per edit), all end below DFE in every seed. On your first observation, Table 11 of the submission shows that editing o_proj, which writes the output of the OV circuit, helps most, ahead of editing q_proj. Second, at Qwen3-4B a later edit keeps nearly the full gain. The same edit at step 60 improves on RL only in every seed and reaches 29.0 against 29.6 for the step-5 edit. Third, Table 3 changes the size of one directed edit, whereas the two readings differ on which directions are edited, and the controlled tests below vary exactly that.

To further address your concern, we test the plasticity reading with the standard plasticity interventions and with a rank-raising random edit, all at step 5 under the same protocol and the same eight seeds.

Qwen3-4B GRPO, second batch (8 seeds)Effective rank of WOW_O after the editTwelve-sub-task mean (%)DFE minus row
RL only156523.8+5.80 (8/8)
+ AdamW restart, no edit156524.2+5.37 (8/8)
+ shrink-and-perturb, equal norm161524.5+5.04 (8/8)
+ Gaussian in weak subspace, equal norm180523.9+5.65 (8/8)
+ DFE, Eq. (4)213429.6

Noise inside the weak subspace raises the effective rank by 240 and leaves the score at the level of RL only, and the AdamW restart and shrink-and-perturb also stay well below DFE. DFE is higher than each of these runs in every seed at both sizes, so among these edits, only raising the existing weak singular values gives the full gain. We have added the plasticity references to Sec. 2 and these effective ranks to Table 10 of the revision.

We also apply DFE and two of its equal-norm controls to the initial model and evaluate them with no RL update (Table 12 of the revision).

Qwen3-4B, eight never-trained sub-tasks (%)Edit only, no RL update
No edit8.5
+ Gaussian in weak subspace9.6
+ Eq. (4) increments, random directions12.4
+ DFE, Eq. (4)15.9
RL only after 150 updates13.6

At Qwen3-4B, DFE alone, with no RL update, already exceeds the final score of RL only on the sub-tasks the policy never trains on, and it is above the same increments along random directions. A change in trainability alone would not raise these scores before any training.

W3(b): Amplified signal or transfer. Both effects appear in our data. DFE acts through the existing reward contrast of the trained sub-tasks (Sec. 5 of the revision), and it raises the test-case pass rate on sub-tasks that contribute no gradient (Table 8 of the revision). As you mentioned, the largest BouncingSim gains are on Gravity and MultiObj, where the base policy already finds occasional perfect rollouts (Sec. 6.2), in line with Theorem 2. The "nearly zero reward" of the abstract refers to the base model's binary pass@1, and the revision rewrites it as transfer to sub-tasks never trained on. The table shows the sub-tasks that played no part in training or in choosing the configuration, at Qwen3-4B GRPO (test-case pass rate, %, eight seeds, †^\dagger evaluation-only). DFE improves on RL only in every seed on their mean and on each sub-task shown.

Never trained onReg.End.Com.Prep.BitOp$^\dagger$Symm$^\dagger$MnMxMean of all eight
RL only11.844.41.70.30.149.50.713.6
+ DFE, Eq. (4)28.862.43.60.80.866.21.920.6
Seeds with DFE higher8/88/88/88/88/88/88/88/8

The mean of all eight also includes Add. On Reg., End., Prep., BitOp, Symm and MnMx, the lowest DFE seed is above the highest RL-only seed.

Reviewer_zek8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper asks whether RLVR post-training induces genuinely new task capabilities or only reinforces existing patterns. It studies this question through "delayed emergence": a policy first fits its training rollouts by memorization and only much later generalizes out of distribution.

  • Phenomenon. On modular multiplication (p = 113) with a 2-layer transformer, the authors track the effective rank of the task-projected OV circuit ("circuit rank") and an orthogonality index (OI) of the task-token representations. They report that the memorization-to-generalization transition shows a non-monotonic circuit-rank trajectory (Fig. 2).
  • Theory. Three local results are given:
    • Theorem 1: under an assumed decomposition into memorization, generalization and noise subspaces, early gradient energy concentrates on the memorization directions.
    • Theorem 2: the RLVR gradient factorizes as E[p(1−p)Δφ], i.e. a "variance bottleneck".
    • Theorem 3: under assumed local linear dynamics, increasing a singular value from s to s′ shortens the hitting time roughly by a factor of s²/s′².
  • Method. Directed Feature Emergence (DFE) is a one-shot spectral edit of the attention output projection W_O in the last layers. It keeps the top singular directions holding 90% of the energy, moves the remaining singular values toward the mean of the protected ones (Eq. 4), and then resumes RL.
  • Validation. On DELTA-Code (Manufactoria and BouncingSim) with Qwen3-1.7B, 4B and 8B under PPO and GRPO, with 3 seeds, DFE improves benchmark-average pass@1. For example, Qwen3-4B with GRPO improves from 18.6 to 33.8 on Manufactoria and from 7.3 to 12.0 on BouncingSim. All 24 paired seed differences reported for the 1.7B and 4B models are positive; the paper does not give the corresponding 8B paired differences. It also reports ablations on α, pass@k, exploration settings, intervention timing and target layers, faster transfer curves (Fig. 3), and modular-arithmetic hitting times (Tables 8 and 10).
Critical strengths
  1. A simple intervention shows promising gains on a relevant problem. Understanding when RLVR moves beyond memorization is an important question, and a low-cost intervention that helps this transition would be useful. The reported DELTA-Code results are encouraging: gains appear across three model sizes, two reported optimizer variants and both task families, and all 24 paired seed differences reported for the 1.7B and 4B models are positive. Table 12 also suggests that the one-time edit is inexpensive, although its timing was measured on a B200 rather than the A100s used for the main experiments. These results provide a good reason to investigate the method further, provided the intervention responsible for them can be established.
Critical weaknesses
  1. It is difficult to tell whether the reported results evaluate the method that the theory analyzes.

    The most consequential discrepancy concerns the intervention itself. Eq. (4) and Algorithm 1 rescale singular values, but both released LLM training scripts call the additive mode in delta_code/dfe.py. That mode adds a random Gaussian matrix projected onto the complement of the leading singular subspaces and scaled to α‖W‖_F. It is a different operation, and the alternative inflate mode also differs from Eq. (4). The scripts apply the edit before RL starts, whereas Appendix F describes an intervention after detecting a memorization plateau; the imported plateau detector is not called. The supplement may be an outdated version, but clarification is needed before the gains can be attributed to the proposed mechanism.

    Other discrepancies make the experimental setup harder to reconstruct. Both optimizer variants use GRPOTrainer with different loss types and no critic. The LLM scripts default to a learning rate of 5×10⁻⁷ rather than the stated 5×10⁻⁶, and the Manufactoria script has no switch from pass-rate to binary reward. The released modular script implements REINFORCE without DFE, so it does not explain the reported PPO/GRPO experiments. Exact run configurations would resolve much of this uncertainty.

    Fig. 2 raises a related question about timing. The DFE and vanilla trajectories already differ substantially in the early stages, before some of the detector times in Table 8. Separate stochastic runs could diverge, so this is not by itself evidence of an error. Still, the seeds and intervention times should be identified, especially since both DFE curves are marked at 34.6k steps, which matches a listed PPO result but no listed GRPO result.

    These details matter scientifically. A random perturbation before training could improve exploration or reward contrast without selectively strengthening generalization directions. Theorem 3 analyzes the deterministic spectral edit, so it does not establish an explanation for the released additive mode. A comparison of these two edits with an equal-norm unstructured perturbation, applied at the same time, would help distinguish the proposed mechanism from a more general perturbation effect. Related work on shrink-and-perturb (Ash & Adams, arXiv:1910.08475), primacy bias (Nikishin et al., arXiv:2205.07802) and loss of plasticity (Dohare et al., doi:10.1038/s41586-024-07711-7) provides useful context for that distinction.

  2. The evidence does not yet connect the toy-model explanation to new capabilities in LLMs.

    The circuit analysis is an interesting starting point, but even the modular-arithmetic evidence needs care. The pronounced rise and fall in circuit rank appears mainly in the DFE runs; vanilla PPO's rank stays near 3 when accuracy improves around 120k steps. The curves have no seed variation or error bands, and the initial rank and orthogonality behavior could partly reflect random initialization. An initialization-matched reference would help establish what these diagnostics reveal about memorization.

    For the LLM experiments, the paper does not show training-reward and OOD trajectories that demonstrate a plateau followed by emergence. Fig. 4 gives final snapshots, where the modest rank increase cannot separate the effect of the edit from subsequent RL. Rank alone also does not identify a generalizing computation. The theory leaves this connection open: Theorem 1 relies on the assumed dominance of memorization directions, Theorem 2 expresses a reward-variance bottleneck related to prior vanishing-gradient analysis (Razin et al., arXiv:2310.20703), and Theorem 3 assumes that the amplified direction is a relevant generalization coordinate. Those assumptions have not been verified for the LLM intervention.

    The empirical scope further limits the claim of new-task generalization. The study uses DELTA-Code and Qwen3 with reasoning disabled, and most evaluation columns are held-out instances of families represented in training. On the evaluation-only sub-tasks BitOp, Symm and Add, changes are at most about one point; DFE is below vanilla GRPO on Symm and Add at both 4B and 8B. This does not negate the benchmark-average gains, but it makes the distinction between harder instances and entirely new task families important. A clearer account of that distinction, alongside evidence of the proposed transition in the LLMs themselves, would substantially strengthen the paper.

Questions for authors
  1. Intervention responsible for the gains. Could you clarify whether Eq. (4) or the released additive perturbation produced the main LLM results? These operations appear to differ in a way that matters for Theorem 3. A comparison at matched timing and perturbation norm, including an unstructured random perturbation, would help assess whether the gains specifically support the proposed spectral mechanism.

  2. Training setup and intervention timing. Was DFE applied before RL or after detecting a plateau in the reported LLM runs? Could you provide the corresponding configurations and clarify the optimizer implementation, checkpoint type, learning rate and reward schedule? In particular, it would help to explain how the reported PPO variant relates to the critic-free GRPOTrainer implementation in the supplement.

  3. Interpreting Fig. 2. Could you identify the plotted seeds and intervention times, and explain the early separation between DFE and vanilla trajectories? Separate stochastic runs may diverge, but the relationship to the detector times is unclear. An explanation of the shared 34.6k marker, together with the code and configurations for the modular experiments, would make this evidence easier to interpret.

  4. Evidence for the mechanism in LLMs. Do the LLM runs show a memorization plateau followed by improved generalization, as proposed in the toy analysis? Training-reward and OOD curves around the intervention would be helpful, ideally alongside a diagnostic of the relevant representation changes. This would clarify whether the final rank differences reflect the proposed transition or simply accompany improved training outcomes.

  5. Scope of new-task generalization. How should the gains on held-out instances of training families be distinguished from those on the evaluation-only BitOp, Symm and Add tasks? Since the latter changes are small, uncertainty estimates would help assess the evidence for entirely new capabilities. Results from another task domain or model family would be a useful further test of generality.

7 comments
AuthorsOpenReview ↗

Response to Reviewer zek8 (Part 1/4)

We thank Reviewer zek8 for recognizing the importance of understanding when RLVR moves beyond memorization, the simple and inexpensive intervention, and the promising gains across three model sizes, two reported optimizer variants and both task families. Below we identify the intervention responsible for these gains, as you asked, and answer each weakness and question.

W1 and Q1: Intervention responsible for the gains. Every DFE result in Tables 1 and 2 and in Fig. 3 uses Eq. (4). The main tables apply Eq. (4) with α=0.1\alpha = 0.1 to the final four o_proj matrices once after the memorization plateau (App. F). App. I of the revision specifies the protocol of every run below.

We ran the comparison you suggested at matched timing and perturbation norm, including an unstructured random perturbation. We repeat the transfer protocol of Fig. 3 (App. I) with eight seeds at each size, so the policy trains with GRPO on the Manufactoria Basic tier only. DFE edits the final 16 o_proj matrices once at step 5. Both sizes use α=0.6\alpha = 0.6 and γ=0.90\gamma = 0.90, the setting that an inference-only search on half of the Has prompts chose before these training runs. Each control changes the same matrices at the same step by the Frobenius norm of the Eq. (4) edit. The Gaussian control adds isotropic noise to the whole matrix, and the rotation control mixes the weak singular directions by a random orthogonal matrix, each from one fixed draw shared across training seeds. We report the mean over the twelve sub-tasks, using the test-case pass rate of Fig. 3 on the nine sub-tasks outside the Basic tier and the binary pass rate on the three Basic sub-tasks.

Twelve-sub-task mean (%)Qwen3-1.7B GRPO (8 seeds)Qwen3-4B GRPO (8 seeds)
RL only7.323.8
+ Gaussian, equal norm5.322.5
+ rotation of the weak directions, equal norm0.020.7
+ DFE, Eq. (4)10.229.6
DFE minus RL only+2.93 (8/8 seeds, sign test p=0.0078p = 0.0078)+5.75 (8/8 seeds, p=0.0078p = 0.0078)
DFE minus Gaussian+4.87 (8/8 seeds)+7.04 (8/8 seeds)

Each difference in this response is the mean of the per-seed differences, computed before rounding. DFE is higher than RL only and than both controls in every seed, while both controls score below RL only. These runs also reproduce Sec. 6.4, since every DFE seed reaches the final held-out Has score of RL only by step 30. We have added this comparison to Sec. 6.5 and Table 4 of the revision.

W1 and Q1: Random perturbations at the same norm and time. Following your suggestion to separate the spectral edit from a general perturbation effect, we compare Eq. (4) with further random edits and with an optimizer restart under the same protocol. One control adds Gaussian noise inside the weak singular subspace. Each control edits the same 16 matrices at step 5 with the norm of Eq. (4) unless the row says otherwise, and each random control uses one fixed draw shared by all seeds. RL only and DFE are rerun in the same batch with the same eight seeds.

Twelve-sub-task mean (%), second batchQwen3-1.7BDFE minus rowQwen3-4BDFE minus row
RL only7.2+2.81 (8/8)23.8+5.80 (8/8)
+ Gaussian in weak subspace4.6+5.39 (8/8)23.9+5.65 (8/8)
+ Gaussian, half norm5.7+4.32 (8/8)22.7+6.88 (8/8)
+ Gaussian, quarter norm6.6+3.33 (8/8)23.8+5.77 (8/8)
+ shrink-and-perturb0.4+9.62 (8/8)24.5+5.04 (8/8)
+ Eq. (4) increments, random directions8.3+1.70 (8/8)23.7+5.87 (8/8)
+ AdamW restart, no edit7.2+2.73 (8/8)24.2+5.37 (8/8)
+ DFE, Eq. (4)10.029.6

DFE is higher than each of these controls in all eight seeds at both sizes. Noise confined to the weak subspace, noise of a smaller norm and shrink-and-perturb (Ash & Adams, 2020) all stay well below DFE, and spending the increments of Eq. (4) on random directions also falls short of DFE in every seed. So only raising the singular values of the existing weak directions of WOW_O gives the full gain. An AdamW restart without an edit, which every edited run also performs, moves the score by less than half a point at either size. We have added shrink-and-perturb, primacy bias (Nikishin et al., 2022) and loss of plasticity (Dohare et al., 2024) to Sec. 2 of the revision, and these controls to Sec. 6.5 and Table 10 of the revision.

AuthorsOpenReview ↗

Response to Reviewer zek8 (Part 2/4)

W1: Exploration or reward contrast. The gain does not come from extra exploration. Right after the edit, DFE does not raise the policy entropy at either size, while the rotation and the random-direction control, each one fixed draw, raise it and end below DFE in every seed. At Qwen3-4B, DFE also does not raise the binary reward variance of the training groups (Table 13 of the revision). The table below shows Qwen3-4B (mean over eight seeds, second batch).

Qwen3-4B, steps 0 to 4 →\rightarrow 6 to 10Policy entropy (nats per token)KL to the initial policy
RL only0.058 →\rightarrow 0.0490.002 →\rightarrow 0.004
+ rotation of weak directions0.059 →\rightarrow 0.1120.001 →\rightarrow 0.150
+ Eq. (4) increments, random directions0.057 →\rightarrow 0.1010.001 →\rightarrow 0.123
+ DFE, Eq. (4)0.057 →\rightarrow 0.0570.001 →\rightarrow 0.145

The rotation moves the policy as far as DFE does (KL 0.150 against 0.145) and nearly doubles the entropy, yet it ends below RL only.

W1 and Q2: Training setup and intervention timing. In the reported LLM runs of Tables 1 and 2, DFE is applied once after the memorization plateau (App. F), not before RL. The table lists the settings of the reported runs against each point you raised.

Point raisedRuns of Tables 1 and 2 (App. F)
Edit and its timingEq. (4) with α=0.1\alpha = 0.1 on the final four o_proj matrices, once after the memorization plateau
CheckpointsQwen/Qwen3-1.7B, Qwen/Qwen3-4B and Qwen/Qwen3-8B, reasoning path disabled, bfloat16 rollouts, float32 optimizer states
Learning rate and sampling5×10−65 \times 10^{-6}, K=16K = 16 rollouts per prompt, temperature 1.0, no KL penalty
Reward scheduleManufactoria: per-test pass-rate warm-up, then the binary all-pass reward. BouncingSim: per-test pass rate at the 50-pixel tolerance
OptimizersPPO and GRPO

For your timing question, we also apply the same edit at step 60 at Qwen3-4B. There the training test-case pass rate of RL only stays flat between steps 60 and 90, so step 60 marks the start of that flat stretch.

Qwen3-4B GRPO, twelve-sub-task mean (%), 8 seedsMeanDFE at step 60 minus row
RL only23.8+5.27 (8/8)
+ Gaussian at step 60, equal norm22.4+6.60 (8/8)
+ DFE at step 6029.0
+ DFE at step 5, for reference29.6

The held-out Has score of every step-60 DFE run reaches the step-150 held-out Has score of RL only at the first evaluation after the edit. App. I of the revision reports this result.

W1 and Q2 (continued): Other RL algorithms. At Qwen3-4B the gain also holds under PPO with a learned critic, and at both sizes it holds under DAPO (three seeds each). In the controlled study, PPO uses a learned critic (a linear value head on the mean final hidden state), DAPO uses dynamic sampling, and every edited run, including every control, restarts AdamW at the edit.

Twelve-sub-task mean (%)RL only+ Gaussian, equal norm+ DFE, Eq. (4)DFE minus RL onlyDFE minus Gaussian
Qwen3-4B, PPO with learned critic21.417.925.3+3.9 (3/3)+7.4 (3/3)
Qwen3-4B, DAPO26.523.330.4+3.9 (3/3)+7.1 (3/3)
Qwen3-1.7B, DAPO8.56.912.4+3.8 (3/3)+5.5 (3/3)

DFE is higher than RL only and the Gaussian control in every seed, and each of these DFE runs reaches the step-150 held-out Has score of RL only by step 30. We have added these runs to Table 7 of the revision.

AuthorsOpenReview ↗

Response to Reviewer zek8 (Part 3/4)

W1 and Q3: Interpreting Fig. 2. The DFE and vanilla curves in Fig. 2 come from separate single runs that share no training prefix, which is why they differ from the start. The shared 34.6k marker is the step at which both DFE runs first reach 90% held-out accuracy, and the revised caption states these facts. For seed-level evidence on timing, every DFE seed in the eight-seed comparison of Table 4 of the revision reaches the step-150 held-out Has score of RL only by step 30. No run of the Gaussian or rotation control, each one fixed draw, reaches it.

W2: Circuit diagnostics. The controlled study tests the function of the edit in the LLM, with each equal-norm control sharing the checkpoint, training seed, step and norm of DFE. Next to Fig. 2, the revision states that Sec. 6.5 tests the edit functionally.

W2 and Q4: Evidence for the mechanism in LLMs. At Qwen3-4B, the fit to the training prompts under RL only improves much faster than transfer to the held-out sub-task, while DFE lifts the held-out score early in training. Over the 150 steps, the training test-case pass rate of RL only rises from 0.56 to 0.84, while its held-out Has score moves from 0.161 to 0.221. The table shows these curves next to the held-out curve of DFE, as means over eight seeds. Each training value is the mean of the ten updates up to that step (updates 0 to 4 at step 0).

Qwen3-4B GRPOStep 0306090120150
RL only, training test-case pass rate0.560.710.770.760.810.84
RL only, held-out Has0.1610.1790.1890.1890.2050.221
DFE at step 5, held-out Has0.1630.2950.2940.3000.3050.309

From step 30 on, the held-out score of DFE exceeds the step-150 score of RL only. At Qwen3-1.7B, the held-out score of DFE keeps rising during RL, and it is higher at step 150 than at step 30 in every seed.

Qwen3-1.7B GRPO, held-out HasStep 0306090120150
RL only0.0180.0190.0220.0240.0330.035
DFE at step 50.0190.0800.0960.1100.1130.154

To separate the edit from the learning that follows, we also apply DFE and two of its equal-norm controls to the initial model with no RL update (Table 12 of the revision). At Qwen3-1.7B, the edit alone leaves the twelve-sub-task mean unchanged, and RL then gains 5.7 points after DFE against 2.8 without an edit. At Qwen3-4B, the edit alone already lifts the eight never-trained sub-tasks from 8.5 to 15.9, above the 13.6 that RL only reaches on them after 150 updates.

Held-out Has, GRPOQwen3-1.7B (8 seeds)Qwen3-4B (8 seeds)
Runs reaching RL only's step-150 scoreDFE 8/8 by step 30, Gaussian 0/8, rotation 0/8DFE 8/8 by step 30, Gaussian 0/8, rotation 0/8
Area under curve / 150, DFE minus RL only+0.072 (8/8 positive)+0.096 (8/8 positive)
Same, Gaussian−0.019-0.019 (0/8 positive)−0.037-0.037 (0/8 positive)
Same, rotation−0.023-0.023 (0/8 positive)−0.041-0.041 (0/8 positive)

In the second batch, DFE is above RL only and each control of Table 10 of the revision at every evaluation from step 30 to step 150, in every seed.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

Dear Area Chair and Reviewers,

Thank you so much for your time and efforts in evaluating our paper. We are grateful for your insightful comments and suggestions, which have helped us further strengthen the submission. We would like to summarize the key contributions highlighted by the reviewers and the new results and revisions we have made to address the initial concerns. Table, figure and section numbers refer to the submitted PDF unless the text names the revision.

Key Contributions:

Important question: We study whether RLVR teaches transferable capabilities or only sharpens existing behavior, a question the reviewers found "important and fundamental" (Reviewer FLpC) and relevant to whether RL can induce capabilities "beyond memorization" (Reviewer c8r1).

Mechanistic account and a simple method: We analyze delayed emergence from a mechanistic perspective with both theory and experiments (Reviewer c8r1). DFE turns this account into a "simple intervention" (Reviewer zek8), a one-time spectral edit that is "cheap and practical" (Reviewer FLpC).

Consistent gains: DFE improves PPO and GRPO across three Qwen3 sizes on Manufactoria and BouncingSim (Reviewers zek8, FLpC), and it raises the average pass@1 in all 12 settings of Tables 1 and 2.

Avg pass@1 (%), mean of three seeds, RL →\rightarrow RL + DFEQwen3-1.7BQwen3-4BQwen3-8B
Manufactoria, PPO6.6 →\rightarrow 11.514.9 →\rightarrow 26.620.6 →\rightarrow 34.7
Manufactoria, GRPO8.5 →\rightarrow 15.118.6 →\rightarrow 33.825.5 →\rightarrow 41.5
BouncingSim, PPO1.3 →\rightarrow 2.65.1 →\rightarrow 9.37.5 →\rightarrow 14.5
BouncingSim, GRPO1.9 →\rightarrow 3.67.3 →\rightarrow 12.011.8 →\rightarrow 18.3

New Results and Revisions:

Matched-norm controlled study: All three reviewers asked for evidence that raising the weak singular values, as Eq. (4) does, is what produces the gains. Following the comparison Reviewer zek8 proposed, we repeat the transfer protocol of Fig. 3 (App. I) with eight seeds at Qwen3-1.7B and Qwen3-4B. We compare Eq. (4) with two edits of the same matrices at the same step and of the same Frobenius norm. One adds an isotropic Gaussian perturbation and the other rotates the weak singular directions, each from one fixed draw shared by all seeds. DFE uses one setting at both sizes, α=0.6\alpha = 0.6 and γ=0.90\gamma = 0.90, which an inference-only search on half of the Has prompts chose before these training runs, and the held-out Has curves use the other half. The table reports the twelve-sub-task mean, with the test-case pass rate of Fig. 3 on the nine sub-tasks outside the Basic tier and the binary pass rate on the three Basic sub-tasks.

Twelve-sub-task mean (%)Qwen3-1.7B GRPO (8 seeds)Qwen3-4B GRPO (8 seeds)
RL only7.323.8
+ Gaussian, equal norm5.322.5
+ rotation of the weak directions, equal norm0.020.7
+ DFE, Eq. (4)10.229.6

Eq. (4) improves on RL only in every seed at both sizes (sign test p=0.0078p = 0.0078), while both controls score below RL only. DFE also leads RL only in every seed when Has, whose prompts served the configuration search, is left out.

DFE leads RL only in every seed on three measures:

RL only →\rightarrow DFE (seeds with DFE higher)Twelve-sub-task mean (%)Held-out Has, step 150Held-out Has, area under the curve / 150
Qwen3-1.7B, GRPO7.3 →\rightarrow 10.2 (8/8)0.035 →\rightarrow 0.154 (8/8)0.025 →\rightarrow 0.097 (8/8)
Qwen3-1.7B, DAPO8.5 →\rightarrow 12.4 (3/3)0.037 →\rightarrow 0.203 (3/3)0.027 →\rightarrow 0.114 (3/3)
Qwen3-4B, GRPO23.8 →\rightarrow 29.6 (8/8)0.221 →\rightarrow 0.309 (8/8)0.191 →\rightarrow 0.286 (8/8)
Qwen3-4B, PPO with learned critic21.4 →\rightarrow 25.3 (3/3)0.200 →\rightarrow 0.303 (3/3)0.176 →\rightarrow 0.273 (3/3)
Qwen3-4B, DAPO26.5 →\rightarrow 30.4 (3/3)0.254 →\rightarrow 0.311 (3/3)0.207 →\rightarrow 0.287 (3/3)

All 25 DFE runs reach the final held-out Has score of RL only by step 30, which reproduces Sec. 6.4.

Never-trained sub-tasks: At Qwen3-4B, the mean over the eight sub-tasks that play no part in training or in choosing the configuration rises under each algorithm in every seed.

Qwen3-4B, eight never-trained sub-tasks (%)GRPO (8 seeds)PPO with learned critic (3 seeds)DAPO (3 seeds)
RL only13.612.715.6
+ DFE, Eq. (4)20.6 (8/8)18.8 (3/3)21.0 (3/3)

The edit alone, with no RL update, lifts them from 8.5 to 15.9, above the final RL-only score (Table 12 of the revision).

What in Eq. (4) produces the gain: Only raising the singular values of the existing weak directions of WOW_O, the quantity that Theorem 3 links to faster emergence, gives the full gain. In a second batch with eight seeds per size, each control changes one part of the edit and keeps the matrices, the step and, unless the row says otherwise, the norm of Eq. (4). Every random control uses one fixed draw shared by all seeds.

DFE minus control, twelve-sub-task mean (points)Qwen3-1.7BQwen3-4B
RL only+2.81+5.80
AdamW restart at the edit step, no edit+2.73+5.37
Raise the protected singular values instead+3.51+8.07
Increments of Eq. (4) on random directions+1.70+5.87
Gaussian inside the weak subspace+5.39+5.65
Gaussian at half / a quarter of the norm+4.32 / +3.33+6.88 / +5.77
Shrink-and-perturb+9.62+5.04
Remove the weak components (the direction of LASER, not norm-matched)+8.82+8.58

DFE is higher than each control in all eight seeds at both sizes. On held-out Has, DFE is above RL only and each of these controls at every evaluation from step 30 to step 150, in every seed.

Edit alone versus edit plus RL: At Qwen3-1.7B, RL gains twice as much after DFE as without an edit. We apply DFE and two of its equal-norm controls to the initial model with no RL update (Table 12 of the revision). RL then gains 5.7 points after DFE, against 2.8 without an edit and 3.2 after the random-direction control.

Timing of the edit and policy statistics: At Qwen3-4B, applying the same edit at step 60 raises the twelve-sub-task mean from 23.8 to 29.0 (8/8 seeds). It also lifts the held-out Has score above the step-150 score of RL only at the first evaluation after the edit in every seed. Right after the edit, DFE does not raise the policy entropy at either size, while the rotation control raises it and ends below RL only.

Intervention responsible for the gains: Every DFE result in Tables 1 and 2 uses Eq. (4) with the settings of App. F, and Fig. 3 uses Eq. (4) with the settings of App. I. App. I of the revision specifies the protocol of the controlled study and reports the per-seed gains.

Positioning, theory and terms: The revision credits DELTA-Code for the transitions it documented under RL and credits Razin et al. (2024) for the reward-variance effect, which Theorem 2 states in closed form. Its Sec. 2 discusses Zhu et al., LASER, Grokfast, Muon, weight decay and plasticity methods, with a comparison table in the new App. K. Its Sec. 4 opens with a roadmap that maps each link of the argument to its theorem and names the controlled study of the new Sec. 6.5 as the test of what the third link predicts in LLMs. It states each assumption with its intuition, with the formal statements in App. C.1. Its Sec. 5 states that low reward contrast weakens the learning signal of each prompt (Eq. (3)), while DFE raises the squared singular value that sets the growth rate of the weak directions (Theorem 3). The revision removes "provably", defines "directed" and uses "OV circuit" only for WOV=WVWOW_{OV} = W_V W_O.

We believe these revisions have improved our paper and hope they will strengthen your confidence in our submission.

Thank you again for your time and consideration.

Sincerely,

Authors of Submission 28858

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.