acceptodds
Under review as a conference paper at ICLR 2027

PUSH: Sparse Representation Dynamics Enable Cost-Efficient Reinforcement Learning in LLMs

Abstract

Reinforcement learning (RL) for large language models relies on costly on-policy sampling. Exploiting the representational trajectory traversed during RL is a promising direction for improving training efficiency and model performance. We use sparse autoencoders (SAEs) to decompose an LLM's residual-stream activations into high-dimensional feature coordinates to interpret the model state during the RL process. The token-aligned change in SAE feature activations from the pre-RL model to the current model forms the SAE signal. We find that the direction along which the model improves during RL is low-dimensional in SAE coordinates, with five features at the final transformer layer accounting for 96.83% of the aggregate squared norm in code generation. These directions persist throughout training so that the signal predicts where RL is heading despite non-monotonic training dynamics. We introduce Progress Utilization via Sparse Hidden Representations (PUSH). On a fixed set of guidance texts, PUSH resolves this signal into its dynamic features and extends their per-token targets along the directions these features have established over training. Each PUSH application fits these targets in a minute-scale activation-matching update outside the RL reward channel and standard RL then resumes under its original objective. Experiments in code generation and mathematical reasoning show immediate performance gains from PUSH without additional RL rollouts, and these gains persist during continued RL. Applying PUSH repeatedly reduces the cumulative sampling cost of RL by 30% on average at matched accuracy. Our results show that PUSH uses interpretability to explain prior learning and direct subsequent training.

Primary area:
interpretability and explainable AI
Keywords:
reinforcement learning, sparse autoencoders, LLM

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_XxVkOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how reinforcement learning shapes internal LLM representations through the lens of frozen Top-KK sparse autoencoders (K=192K=192, 8×8\times expansion) and turns that observation into a lightweight mid-training intervention called PUSH (Sections 3–4, Figures 2–3). By tracking token-aligned feature shifts Δzk,t=zk,t−z0,t\Delta z _ {k,t} = z _ {k,t} - z _ {0,t} (Eqs. 1–2) between the pre-RL base model and intermediate RL checkpoints, the authors find that representation movement is remarkably low-dimensional and directionally persistent: at block H28, the same five SAE features (F3380, F20383, F17379, F26279, F17031) rank Top-5 across all six GRPO and DAPO runs and capture 93.4%–96.8% of the squared mean shift ∥Δz‾k∥22\Vert \overline{\Delta z} _ k \Vert _ 2^2 (Section 6.1, Appendix Table 2), while roughly 81% of features moving at step 60 continue in the same direction at step 1047 (Figure 2b).

To exploit this regularity, PUSH pauses RL at an intermediate checkpoint, selects the top D=K/2=96D = K/2 = 96 changed SAE features per answer token on a tiny fixed guidance set G\mathcal{G} (ng=64n _ g = 64 AceCode or 128 GSM8K problems), splits them into a suppression set AtA _ t (features that dropped out of the pre-RL Top-KK support, targeted to 0) and a continuation set BtB _ t (features in current support, extrapolated via yB,t,i=[znow,t,i+γ(znow,t,i−zpre,t,i)]+y _ {B,t,i} = [z _ {\mathrm{now},t,i} + \gamma(z _ {\mathrm{now},t,i} - z _ {\mathrm{pre},t,i})] _ + with γ=1\gamma=1, Eq. 3), fits two short rank-32 LoRA branches over 8–16 minibatches (Eq. 4, taking 53–77 seconds on one H200 GPU, Appendix Table 23), merges ΔWPUSH=s(αΔWA+βΔWB)\Delta W _ {\mathrm{PUSH}} = s(\alpha \Delta W _ A + \beta \Delta W _ B) with default (α,β)=(2,1),s=1(\alpha,\beta)=(2,1), s=1 into the base weights (Eq. 5), and resumes RL. Tested on Qwen2.5-Coder-7B-Instruct (GRPO and DAPO on AceCode across 3 seeds, Tables 1–2) and single-run GSM8K experiments on Qwen2.5-Math-7B and DeepSeekMath-Instruct-7B, PUSH yields immediate accuracy jumps in 15 of 16 code settings (up to +4.13±0.82+4.13 \pm 0.82 points in Table 1; +5.96±0.19+5.96 \pm 0.19 at step 210 in Appendix Table 18) and cuts total rollouts to reach matched accuracy by roughly 30% (29.09% ±\pm 5.67% under GRPO and 31.61% ±\pm 2.82% under DAPO, Table 2).

Critical strengths

1. A novel, well-substantiated bridge from SAE representation dynamics to RL rollout efficiency.
Turning early historical feature changes (znow,t−zpre,tz _ {\mathrm{now},t} - z _ {\mathrm{pre},t}) on 64 unlabeled guidance prompts into token-level suppression and continuation targets is a creative idea, and the ablations in Section 6.3 (Table 4) prove that the specific SAE structure is doing the work: shuffling token alignment (w/o align.) or replacing Top-DD sparse selection with random D=96D=96 features (w/o Top-D) wipes out the +2.96 to +5.96 point gains almost completely (dropping to -0.50 to +0.29 points under w/o Top-D), and PUSH also cleanly outperforms weight-space model extrapolation (ExPO), even when ExPO is scaled to match ΔWPUSH\Delta W _ {\mathrm{PUSH}} in Frobenius norm across all 196 modules (winning 11 of 12 comparisons in Appendix Table 20).

2. Thorough, well-controlled empirical evaluation on the primary code-generation suite.
In the main AceCode setting (Qwen2.5-Coder-7B-Instruct under GRPO and DAPO), the authors evaluate across 8 starting checkpoints and 3 random seeds, compare against four natural baselines (SFT, ADL, RepE, and ExPO, Table 1), track 118 continuation trajectories (Figure 4, Appendix Figures 20–21), and translate the accuracy jump into a concrete systems metric: saving roughly 30% of total rollouts (about 52% of post-PUSH rollouts, Table 2) at the cost of a single 53-second LoRA fit (Appendix Table 23).

Critical weaknesses

1. Narrow empirical breadth beyond 7B code generation (single-run GSM8K math validation).
While the paper frames PUSH as a general RL post-training accelerator, the rigorous multi-seed statistical evidence is confined to a single model (Qwen2.5-Coder-7B-Instruct) and dataset (AceCode, with transfer checks on MBPP/MBPP+ in Appendix D.1.1). The mathematical reasoning experiments on Qwen2.5-Math-7B and DeepSeekMath-Instruct-7B are conducted on GSM8K with only one training seed per model (Appendix C.4 Table 17 and Appendix D.1 Figure 14), and Single PUSH on DeepSeekMath yields only a 1.54% total rollout reduction in Appendix Table 21 (8.65k rollouts saved, though Multiple PUSH reaches 24.87%). Expanding multi-seed validation to the math setting (or a harder math benchmark than GSM8K) is needed to match the generality of the claims.

2. Lack of a self-contained, validation-free stopping rule to avoid late-stage performance drops.
As shown in Figure 6b, Appendix Table 18 (step 810: -1.21 pp on GRPO, -3.38 pp on DAPO), and Appendix D.3.2 (where late Multiple PUSH schedules 270/600/870 and 600/720/840 underperform Single PUSH by -2.00 and -1.13 pp at step 1020), linear extrapolation along Δz\Delta z breaks down once RL representations approach saturation. While Appendix E.2 (Table 25, Figure 22) sketches an "exploratory stage-wise policy" (Early s=2s=2, Middle s=1s=1, Late No PUSH), the stage boundaries are chosen by inspecting validation-split scans rather than triggered automatically from internal ΔSAE\Delta\text{SAE} statistics (such as feature velocity ∥Δzk,t∥2\Vert \Delta z _ {k,t} \Vert _ 2 or directional consistency). For a practitioner deploying PUSH in a new run, a concrete heuristic for when to stop applying PUSH without running validation scans at every checkpoint is missing.

Questions for authors
  1. Clarification affecting soundness (Optimizer state after base-weight write-back): When PUSH merges ΔWPUSH\Delta W _ {\mathrm{PUSH}} into WbaseW _ {\mathrm{base}} (Appendix B.3 Eqs. 11–12) and resumes RL on the existing LoRA adapter U0U _ 0, you restore the pre-PUSH AdamW momentum and variance buffers unchanged (Appendix C.1, line 1509). Did you test resetting AdamW moments (or running a few warmup steps) after the weight jump, and does stale momentum contribute to the brief post-PUSH dip or the late-stage instability seen at step 810?

  2. Clarification affecting empirical strength (Math variance): Can you report 3-seed results (mean ±\pm std) for at least one of the mathematical reasoning models (Qwen2.5-Math-7B or DeepSeekMath-Instruct-7B), so we can tell whether the GSM8K immediate and continuation gains (Appendix Figure 14 and Table 21) are as consistent across seeds as the AceCode results?

  3. Clarification affecting practical deployment (Automatic stopping criterion): Looking at your ΔSAE\Delta\text{SAE} trajectories in Section 3 / Appendix A, is there an intrinsic representation signal computable purely on the 64-problem guidance set G\mathcal{G}—for example, a drop in step-to-step cosine similarity of Δz‾\overline{\Delta z} or a plateau in ∣Δzt,i∣\lvert \Delta z _ {t,i} \rvert between the last two checkpoints—that reliably predicts when PUSH will turn negative (around step 600–810) without requiring external validation-set evaluations?

6 comments

Thank you for your positive feedback and insightful questions! We address them below.

Q1

When PUSH merges ΔWPUSH\Delta W_{\mathrm{PUSH}} into WbaseW_{\mathrm{base}} (Appendix B.3 Eqs. 11–12) and resumes RL on the existing LoRA adapter U0U_0, you restore the pre-PUSH AdamW momentum and variance buffers unchanged (Appendix C.1, line 1509). Did you test resetting AdamW moments (or running a few warmup steps) after the weight jump, and does stale momentum contribute to the brief post-PUSH dip or the late-stage instability seen at step 810?

  1. Preserving optimizer state is part of the pluggable design. Our original design intent was to make PUSH an adjustable and pluggable weight intervention. We therefore restore the pre-PUSH AdamW state and training settings when RL resumes, preserving continuity of the original training procedure without introducing an additional optimizer restart (Appendix C.1).

  2. Preliminary comparisons of preserving versus resetting AdamW state. Thank you for your careful analysis! We conducted experiments to examine this: starting from the same model after one PUSH, we resume RL with AdamW state preserved or its first and second moments zeroed, keeping all other settings unchanged; the table reports gains after RL continuation over the baseline at the same training step (pp), with immediate gains in parentheses. Each cell lists preserved / reset.

    PUSH step+30 RL steps+60 RL steps+90 RL steps+120 RL steps+150 RL steps
    GRPO @ 420 (+3.63)+2.88 / +3.50+0.88 / +0.75+0.63 / +1.25+1.88 / +1.75+3.63 / +3.75
    GRPO @ 210 (+5.63)+4.25 / +4.38+3.88 / +4.38+3.00 / +2.75+1.38 / +2.13+1.25 / +1.38
    DAPO @ 420 (+3.50)+4.38 / +5.25+4.13 / +4.380.00 / +0.50+2.38 / +3.13+0.38 / +0.13
    DAPO @ 210 (+4.63)+4.63 / +5.13+2.63 / +2.50+2.88 / +3.63+1.88 / +1.75+2.38 / +3.50

    In these experiments, continuation gains still fluctuate after resetting optimizer state. Stale momentum alone does not explain these fluctuations in the tested trajectories.

  3. The relationship between step 810 and optimizer state. The immediate accuracy changes at step 810 (the training saturation stage) for both GRPO and DAPO were measured after PUSH but before RL updates resumed. Stale momentum has not yet been used for an update and therefore cannot explain this immediate drop (Table D.1).

Q2 / W1

Can you report 3-seed results (mean ± std) for at least one of the mathematical reasoning models (Qwen2.5-Math-7B or DeepSeekMath-Instruct-7B), so we can tell whether the GSM8K immediate and continuation gains (Appendix Figure 14 and Table 21) are as consistent across seeds as the AceCode results?

  1. Three-seed statistics for immediate and continuation gains on mathematical reasoning. We added three-seed experiments with Qwen2.5-Math-7B Base. The model has positive immediate gains in all runs at the six checkpoints below and saves rollouts during continuation. Mean immediate gains (pp) are: step 32: +4.27 (±0.70); step 64: +4.27 (±1.51); step 80: +2.13 (±1.03); step 128: +3.67 (±1.14); step 192: +2.07 (±1.40); step 304: +2.07 (±1.10).

    On GSM8K, mean cumulative rollout savings are 18.99% (±3.96) for Single PUSH and 21.59% (±2.66) for Multiple PUSH. These means are computed over the respective branches with eligible shared accuracy targets; the branch sets differ. Across runs, both generally maintain accuracy gains over baseline during continuation. The revised manuscript includes their continuation accuracy-gain curves in Appendices D.3.1 (Figure D.8(c)) and D.3.2 (Figure D.9(b–c)), respectively, and per-run rollout statistics in Appendix D.4.1 (Table D.6).

  2. Generalization to harder math benchmarks and other tasks. We added GRPO training experiments on MATH [1] with evaluation on MATH500, alongside digital hardware design experiments at 1.5B and 7B, also using GRPO. Single PUSH reduces cumulative rollouts to shared accuracy targets by 22.76%, 18.25%, and 38.37%, respectively; hardware results are averaged over the VerilogEval V1 Human and Machine test sets.

  3. Differences in cumulative rollout savings between Single and Multiple PUSH. For DeepSeekMath-Instruct-7B, the 1.54% and 24.87% measure cumulative rollout savings to shared accuracy targets, not immediate accuracy gains. Single PUSH saves only post-intervention rollouts, so pre-intervention costs dilute the cumulative savings ratio. Multiple PUSH uses newly accumulated representation changes to repeatedly construct targets and sustain its lead, yielding greater cumulative savings.

[1] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. arXiv preprint arXiv:2103.03874, 2021.

Q3 / W2

Looking at your ΔSAE\Delta\mathrm{SAE} trajectories in Section 3 / Appendix A, is there an intrinsic representation signal computable purely on the 64-problem guidance set G\mathcal{G}—for example, a drop in step-to-step cosine similarity of Δz‾\overline{\Delta z} or a plateau in ∣Δzt,i∣|\Delta z_{t,i}| between the last two checkpoints—that reliably predicts when PUSH will turn negative (around step 600–810) without requiring external validation-set evaluations?

Our work focuses on revealing the sparsity of feature changes during RL by analyzing the ΔSAE signal and successfully applying this property to accelerate training efficiently. Future work will build on these findings to develop mature adaptive intervention and stopping policies.

  1. Fixed settings in the main experiments and stage-wise analysis on validation. The main experiments in this work (including coding, mathematical reasoning, and digital hardware design) use fixed intervention schedules and strengths. Validation scores are not used to decide these interventions. Single PUSH intervenes once at the scheduled checkpoint, then resumes RL; Multiple PUSH intervenes repeatedly along the same trajectory. Coding uses s=1s=1 and the same schedules across training seeds within each algorithm. Immediate gains, continuation performance, and rollout savings are evaluated on the test sets.

    The external validation-set evaluations you mention belong to a separate exploratory study in this work. It examines the sensitivity of PUSH gains to intervention strength and training stage, and explores whether adapting strength to each stage improves results. At each checkpoint, we compare A/B coefficients and measure immediate gains over the same-step baseline on the validation split. Each starting checkpoint is evaluated independently. Automatically identifying these stages, adjusting interventions, and deciding when to stop from intrinsic representation signals is precisely the further question you raise.

  2. Intrinsic representation signals: directional consistency and change magnitude. Thank you very much for your perceptive and insightful ideas! We have already observed the phenomena you describe during our experiments:

    On the fixed 64 guidance problems from one DAPO trajectory, the cosine similarity between consecutive change directions on PUSH-selected features falls from 0.598 at step 690 to 0.041 at step 720 and becomes negative at step 750 (−0.509). The RMS magnitude of cumulative changes increases from 0.0210 at step 540 to 0.0289 at step 690, then remains within 0.0289–0.0300 over steps 690–840, forming a local plateau. These observations are consistent with your discussion of late-stage saturation and provide key information for guiding future adaptive interventions.

    We will investigate combining directional consistency and change magnitude to adjust PUSH timing and strength, reducing, skipping, or stopping interventions when the historical direction is no longer suitable. We will pair signals with intervention gains along the same trajectories to assess their ability to detect saturation in gains and representations.

Thank you again for your thoughtful review and insightful questions about PUSH’s reliability and practical applicability! If our responses have resolved your concerns, we would appreciate your reconsideration of the evaluation.

Thank you again for your careful review and thoughtful questions!

The discussion closes in less than one day. Your questions have been very helpful to us. We value your views and would welcome your assessment of the additional experiments and analysis.

Thank you again for taking the time to review our work!

Reviewer_z1bcOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to reuse representation changes already produced by RL to reduce subsequent on-policy sampling. The authors train and freeze a sparse autoencoder (SAE) on the pre-RL model's residual stream, and compare feature activations before and after RL on the same texts and token positions to obtain delta SAE. The code experiments show substantial directional persistence in position-averaged activation changes, with five final-layer features accounting for most of the squared norm of the mean change.

PUSH selects the features with the largest historical changes at each token on a small, fixed set of guidance texts. It separately fits suppression targets for features leaving the sparse support and extrapolation targets for the remaining selected features. Two temporary LoRA branches produce a parameter update, after which the original RL process resumes. Experiments cover GRPO/DAPO for code generation and two mathematical models, examining immediate performance, continued training, parameter/representation extrapolation, and several ablations. Across three training seeds in the code experiments, Multiple PUSH reduces cumulative rollout counts by 29.09% and 31.61% for the two algorithms, respectively, measured at accuracy thresholds reached by both methods.

Critical strengths
  1. The method turns historical representation changes into testable training gains. In Table 1, PUSH outperforms SFT, dense activation extrapolation, PCA extrapolation, and weight extrapolation at most starting points under both RL algorithms. In Table 4, disrupting token alignment, replacing token-specific directions with their mean, or randomly selecting features substantially reduces the gains. These comparisons support the value of selecting historical changes aligned with the current token, beyond simply performing extra updates on guidance texts. Appendix D.2.2's ExPO comparison matched in parameter-displacement norm also reduces the plausibility of an explanation based solely on update magnitude.
  2. The continued-training experiments and statistical design are substantial. The code experiments include three independent runs for each of GRPO and DAPO, multiple starting points, and single and repeated interventions, with a clear distinction between continuation branches and independent training seeds. Table 2 reports reductions in total rollouts, not only in rollouts remaining after the intervention. The appendix states that DAPO uses actual generation counts, including responses before filtering. Guidance data are separate from test data, and the paper additionally evaluates different guidance sets and MBPP/MBPP+. Together, these results provide stronger evidence of training value than a single immediate improvement would.
Critical weaknesses
  1. The interpretation that a few sparse features carry RL progress is not yet directly linked to capability gains. The concentration statistics in Sections 3 and 6.1 and Appendix A.3 average over positions before computing the squared norm of feature changes. Changes with different directions at different tokens can cancel. Consequently, five features explaining approximately 97% of the mean change does not mean that they explain approximately 97% of token-level changes or functional updates. The close agreement between Top5-decoded changes and the overall mean direction in the appendix still primarily validates this aggregate statistic. Actual PUSH can select 96 features per token, and Appendix A.3.3 does not provide stable individual semantic interpretations for three of the Top5 features. The existing ablations support the practical intervention, but do not establish that these five global features are the main causal source of capability improvements. This distinction affects the strength of the interpretability contribution and calls for narrower claims or direct capability interventions.
  2. Overall efficiency and some training comparisons have important unresolved boundaries. Appendix B.3 writes the effective LoRA increments from both branches into the originally frozen base weights, then retains the original adapter for continued RL. A sum of low-rank increments need not satisfy the original rank-32 constraint. PUSH may therefore have a larger effective update space during continuation than the original RL baseline, while the single-branch updates in SFT/ADL/RepE do not match this aspect. The displacement-matched ExPO comparison controls magnitude, but not branch merging, base-weight write-back, or cumulative effective rank. Furthermore, Table 23's approximately one-minute timing starts with an existing SAE/cache and excludes SAE training, preparatory hidden-state extraction, and guidance-response preparation. Table 2's approximately 30% reduction averages rollout savings across retrospectively shared thresholds and multiple starting points; it is not directly equivalent to a reduction in total GPU time under one fixed intervention schedule. These boundaries need clarification to fully assess the cost advantage.
Questions for authors
  1. Functional meaning of the low-dimensional change: How much change energy do the five features cover when squaring before averaging over tokens, when measuring changes per problem, and after excluding formatting-related tokens? Can extending or suppressing only these five features approach the accuracy gains of full PUSH? This would directly affect my assessment of the claim that sparse representations explain and drive progress.
  2. Fairness of parameterization: Please report the numerical rank or singular-value spectrum of the effective updates before and after each PUSH, and compare against historical activation extrapolation or RL controls with the same base-weight write-back, branch count, and update count. Do the main gains persist if the PUSH update is compressed back to the original rank-32 constraint? This could distinguish target quality from an expanded update space.
  3. Overall training efficiency: What are the GPU time and peak memory costs including SAE training, data/hidden-state preparation, all PUSH interventions, and RL? Can a fixed schedule and target accuracy be selected on an independent validation set, followed by a report of total cost reductions for each training seed? I recognize the current rollout-saving statistics, but this clarification would determine whether they support an overall cost claim.
8 comments

Thank you for your comprehensive review! We are glad that you appreciate our approach and experimental evaluation. Your questions give us the opportunity to examine the concentration and persistence of the Top5 features in ΔSAE more comprehensively and rigorously, and to clarify the connection between ΔSAE's representation properties and their practical application. We explain below how these findings motivate PUSH and present the supporting evidence.

W1 / Q1: Properties of the Top5 features in ΔSAE

The Top5 features in ΔSAE are consistently recovered

Across the settings you describe, the same five features account for most of the change energy and retain their rankings across seeds and algorithms:

(1) Changing the aggregation order: We compute ΔSAE over pre-RL → step 1047 using the full set of 1,000 problems. After changing the aggregation from “average over tokens, then square” to “square each token's change, then average,” we rerank all 28,672 SAE features by their contributions to ΔSAE change energy. In all three runs of both GRPO and DAPO, the same five features remain in the top five, in exactly the same order.

Their coverage of ΔSAE change energy decreases from 93.16%–96.74% under averaging before squaring to 66.14%–77.81% under squaring before averaging. These five features constitute only approximately 0.0174% of all features, so this result still demonstrates the sparsity of feature dynamics during RL.

(2) After excluding formatting-related tokens from the 1,000 problems, we extract representations again and recompute the rankings. The top five features and their order remain unchanged in all six runs. Under both aggregation orders, their coverage of ΔSAE change energy is higher after exclusion, without a single exception across the three runs of either RL algorithm. Table 1 gives the results.

One possible explanation is that features associated with basic formatting are already relatively stable before RL, whereas the dominant representation changes reinforced by RL occur more in reasoning-related features. Removing formatting positions therefore makes these dominant changes more prominent, increasing the coverage of the Top5 features in ΔSAE.

Table 1. Coverage of the Top5 features in ΔSAE under two aggregation orders, before and after excluding formatting-related tokens (%)

AlgorithmRunAverage then square: before exclusionAverage then square: after exclusionSquare then average: before exclusionSquare then average: after exclusion
GRPO196.5796.7377.8178.68
GRPO295.0195.2672.4373.42
GRPO393.1693.5666.1467.46
DAPO196.7496.8776.9677.69
DAPO296.4696.5873.6374.32
DAPO396.6796.8177.7178.43

We exclude whitespace-only tokens, tokenizer-defined special tokens, and tokens corresponding to the opening and closing lines of Markdown code fences, including language labels. Mixed tokens that contain actual code are retained.

(3) Replication across problems: We compute the coverage of the Top5 features in ΔSAE separately for each of the 1,000 problems under both aggregation orders.

When averaging before squaring, the median per-problem Top5 coverage across the six runs is 90.75%–95.11%, and 0%–0.8% of problems have coverage of at most 50%. When squaring each token's feature changes before averaging within each problem, the median coverage is 65.11%–76.41%, and 0%–4.7% of problems have coverage of at most 50%. Thus, even after removing the effect of cancellation between token-level changes within a problem, the Top5 features still account for most of the change energy on most problems.

After reranking under the alternative aggregation order, these five features remain the top five in the same order. Their trajectories across checkpoints also show persistence in the directions of these dominant feature changes (§3, Figure 5). Coverage under both aggregation orders and with and without formatting-related tokens, together with the per-problem distributions under both orders, has been added to Appendix A.3.1 of the revised manuscript (Table A.3; Figures A.5–A.6).

Our analysis of the Top5 features shared across tokens and their training trajectories reveals sparsity and directional persistence in RL-induced representation changes in SAE space. It also reveals how historical activation differences, ΔSAE, provide insight into subsequent directions of change (§3). Existing research shows that activation magnitude alone is insufficient to capture a feature's influence on model outputs [1, 3]; in our work, ΔSAE extracts the changes induced by RL.

After identifying these sparse dynamics during RL, we set out to use them to guide training, both to validate the property we had observed and to investigate whether it could help reduce the high cost of RL.

However, the point we particularly want to emphasize is that, for many discoveries, moving from an observed property to a practical application is not a one-dimensional, idealized process; it calls for reasoning and experiments that take the structure and properties of the tools themselves into account.

SAEs are widely used in LLM interpretability. Their ability to address neuron polysemanticity has opened up ways to analyze model internals. Most existing work studies feature interpretation and intervention in static, already-trained models (see [1–3]). Yet a substantial gap remains between SAE explanations and downstream interventions: analyzing feature activations can explain the function or meaning of certain features, while intervening on it can produce behavior inconsistent with that interpretation [1, 2], or cause side effects that interfere with other outputs [1, Fig. 2], [4, Sec. 4.4], [5].

We see the side effects of static interventions as a major reason why interpretability research has yet to establish a complete loop between feature explanation and intervention.

Here, by static intervention, we mean interpreting a feature through extensive analysis and then applying the same intervention policy at every output token, without selecting features dynamically for each token. Prior work shows that this can substantially disrupt model reasoning; experiments showing performance degradation under strong all-token steering are reported in [4, Sec. 4.4]. Different tokens carry different contextual and reasoning information and therefore activate different features in different circumstances.

This is why, when translating our aggregate observation into a training intervention, we did not simply restrict interventions to the Top5 features in ΔSAE.

We aim to use dynamic interventions to guide the model at low cost by naturally making use of the information it already contains. Our finding about the Top5 features in ΔSAE describes changes shared across tokens. It reveals a meaningful aggregate property. However, this aggregate finding does not mean that it can be directly translated into an intervention. Below, we explain the rationale and details of dynamic intervention.

[1] Arad, D., Mueller, A., & Belinkov, Y. (2025). SAEs are good for steering—if you select the right features. EMNLP, 2025.
[2] Paulo, G. S., Mallen, A. T., Juang, C., & Belrose, N. (2025). Automatically interpreting millions of features in large language models. ICML, 2025.
[3] Shu, D., Wu, X., Zhao, H., Du, M., & Liu, N. (2025). Beyond input activations: Identifying influential latents by gradient sparse autoencoders. EMNLP, 2025.
[4] Cho, I., & Hockenmaier, J. (2025). Toward efficient sparse autoencoder-guided steering for improved in-context learning in large language models. EMNLP, 2025.
[5] Siu, V., Crispino, N., Park, D., Henry, N. W., Wang, Z., Liu, Y., Song, D., & Wang, C. (2026). SteeringSafety: Benchmarking representation steering in LLMs across safety perspectives. ICML, 2026.

To understand the underlying reason, let us return to the SAE architecture itself. It expands the model's hidden states from 3,584 → 28,672 dimensions, an 8-fold expansion intended to address overlap and polysemanticity in the neurons activated at each token. We are familiar with how SAEs are trained: each token is constrained to activate only a small number of features. Different tokens therefore activate different features from the same SAE dictionary, with these activations collectively covering the 28,672-dimensional feature space. The training objective also includes an AuxK auxiliary loss to discourage dead features. This means that applications of SAE interpretability should use dynamic interventions, with features aligned to each token, to make effective use of information already present in the model, rather than assuming that interpreting features shared across many tokens justifies applying the same intervention policy at every output position.

PUSH implements dynamic intervention at two levels:

  1. Adapting to the training state. Before each application of PUSH, we reselect intervention features and construct fitting targets from the current model's representation changes relative to the pre-RL model on the guidance problems (Δzt\Delta z_t). The selected features and targets remain fixed during that PUSH fitting stage.
  2. Constructing targets adaptively for each token. Within each PUSH intervention, we select up to 96 features with the largest nonzero absolute changes separately for each token. We construct fitting targets using the direction and magnitude of change at that position, using token-level alignment to provide a more precise signal for the auxiliary update.

In our ablations, averaging these directions or shuffling token correspondence—the static interventions discussed above—removes most of the gains (Table 4). When this alignment is preserved, post-PUSH representations approach those at later states along the natural RL trajectory, accompanied by accuracy improvements (Table 3).

The SAE already enforces token-level sparsity with K=192K = 192. Restricting targets to five fixed features uses only approximately 2.60% of this sparsity budget (5/1925/192), while applying the same feature set at every position. This restriction preserves each token's direction of change and target magnitude on these five features, but discards changes outside the Top5 at each position.

The overlap between the global Top5 and token-level intervention targets varies with training stage, supporting dynamic target selection based on current representation changes. These statistics have been added to Appendix E.3 of the revised manuscript.

We must still determine which features to adjust at each position, in which direction, and by how much. The global Top5 summarize changes shared across tokens, while the activations needed at individual tokens differ, and our experiments show that these differences matter. PUSH therefore reselects features from the current model at each intervention and preserves each token's own direction and magnitude of change to construct targets that can be fitted directly (§4). This design converts the observed sparsity and persistence into a concrete training signal. The fixed-Top5 controls below test whether using only the globally dominant coordinates can achieve the same gains.

Reviewer_bGCDOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies reducing the costly on-policy sampling of RL for LLMs. It proposes PUSH: on a fixed set of guidance texts, SAEs is used to resolve historical representational change into dynamic features, and per-token targets are extrapolated along the directions these features have established over training. After PUSH update, standard RL resumes under its original objective. Experiments show that PUSH yields larger gains than related alternatives and, at matched accuracy, reduces the cumulative sampling cost of RL by about 30% on average.

Critical strengths
  1. The motivation is clear and the technical route is well defined. The authors argue that costly on-policy rollouts are underutilized after each update, and that change already obtained from past training can be reused to advance training without additional sampling, thereby reducing sampling cost. The pluggable and adjustable nature of the PUSH update is also attractive for practical use.

  2. The experiments compare multiple baselines and include fairly complete ablations. The results are used to analyze and explain where the advantages of PUSH come from.

Critical weaknesses
  1. Improvements in final task performance are modest; the main reported advantage is the reduction in cumulative RL sampling cost at matched accuracy.
  2. Experiments show that applying PUSH late in training can yield negative gains, so the timing of application is critical. The paper analyzes this stage sensitivity but does not provide a method for choosing when to apply PUSH.
  3. Generality remains to be established: effectiveness across other models and task domains beyond the evaluated settings still needs validation.
Questions for authors

Figure 4 shows that the advantage from a single PUSH application can be caught up by the RL baseline, and that repeatedly applying PUSH is needed to sustain the advantage. Can the authors explain why the PUSH advantage is closed under continued RL?

4 comments

Thank you very much for your feedback and follow-up questions! We address these questions using our representation analyses and continued-training results to explain the characteristics of PUSH interventions, and report new digital hardware design experiments that test its generalization across tasks.

W1 / Q1: Final performance and the changing advantage of PUSH during continued RL

RL requires extensive sampling for gradient updates, but the representation changes accumulated during this process contain valuable information about what the model has learned. We use ΔSAE to identify sparse and persistent directions of representation change and guide training accordingly. Across experiments, this approach consistently reduces the sampling cost of reaching the same performance.

For training performance, existing approaches accelerate RL primarily by optimizing system scheduling [1], reducing unproductive sampling [2], or lowering the numerical precision of rollouts [3]. These acceleration strategies can sometimes even come at the cost of model performance [3]. PUSH accelerates training while preserving final performance (Table 2). Across multiple task domains, applying PUSH once at certain checkpoints yields accuracy above the highest level attained by the corresponding baseline over the full recorded training run, without any subsequent RL training (Appendix D.3.1, Figure D.8). Our initial exploration of strategies for adjusting PUSH strength suggests that it could raise the achievable performance further (Figure 6; Appendix E.2; Appendix D.3, Figure D.9).

From an interpretability perspective, our representation analyses show that after PUSH, the model's representations in both the raw hidden-state space and the SAE feature space become close to those at later checkpoints of the RL baseline, while accuracy also improves (Table 3). For example, after PUSH at GRPO step 240, the nearest baseline checkpoints in these two spaces are steps 390 and 420, respectively. PUSH takes approximately one minute and immediately improves accuracy to a level that the RL baseline first reaches at step 480 (Appendix E.1, Table E.1).

Our interpretation is that PUSH captures directions of feature change already learned during RL and uses ΔSAE to filter out redundant noise, helping the model acquire, at an earlier stage, capabilities that would otherwise require more sampling. Specifically, it converts information accumulated from sparse rewards into fine-grained token-level targets that can be fitted directly. PUSH is a brief, pluggable optimization procedure that introduces no new rewards and leaves the subsequent RL objective unchanged. As task performance approaches saturation, further updates along these directions yield smaller gains. Meanwhile, further costly RL training gradually brings the baseline to the performance already achieved by PUSH, narrowing the gap.

Multiple PUSH sustains the advantage through the same mechanism. Each period of resumed RL produces new representation changes. Subsequent PUSH interventions dynamically reselect features and construct fitting targets from these changes, turning the information learned during subsequent RL into further training gains and even improving final task performance (Appendix D.3, Figure D.9).

[1] Fu, W., Gao, J., Shen, X., Zhu, C., Mei, Z., He, C., Xu, S., Wei, G., Mei, J., Wang, J., Yang, T., Yuan, B., & Wu, Y. (2025). AReaL: A large-scale asynchronous reinforcement learning system for language reasoning. NeurIPS, 2025.
[2] Zheng, H., Zhou, Y., Bartoldson, B., Kailkhura, B., Lai, F., Zhao, J., & Chen, B. (2025). Act only when it pays: Efficient reinforcement learning for LLM reasoning via selective rollouts. NeurIPS, 2025.
[3] Li, Y., Elangovan, R., Dong, X., Panda, P., & Khailany, B. (2026). QuRL: Low-precision reinforcement learning for efficient reasoning. ICLR, 2026.

W2: Timing of PUSH interventions

In our main coding experiments, we use a fixed update strength and apply PUSH at checkpoints spaced approximately evenly over the course of training. After each intervention, we resume the original RL training and track subsequent performance. To assess the consistency of the effects across runs, we use the same PUSH starting steps across training seeds within each algorithm. We do not specifically optimize intervention timing or update strength for performance gains. Even with this simple initial intervention schedule, PUSH consistently reduces the sampling cost of reaching matched accuracy, and repeated interventions further improve training performance (Table 2; Figure 4).

PUSH selects features and constructs fitting targets based on the cumulative change relative to the pre-RL model, Δzt=znow,t−zpre,t\Delta z_t=z_{\mathrm{now},t}-z_{\mathrm{pre},t}. As task performance reaches a plateau, some features may still be selected because their cumulative changes are large, even if they have already been sufficiently adjusted. Further amplifying or suppressing these features may disrupt the activations needed for the current task, leading to smaller or negative gains late in training (Figure 6; Appendix D.1, Table D.1).

Our work focuses on using ΔSAE to identify the sparsity and strong persistence of representation changes during RL, and thoroughly validates the feasibility of using ΔSAE as an interpretability-based approach to guide training. We agree with your observation that the timing of application is critical. We will therefore pursue stage-adaptive strategies as an important direction for future work: using representation changes in SAE feature space at different training stages to automatically determine whether the update direction remains effective, and then selecting intervention times, adjusting update strength, or stopping interventions accordingly. Such policies could better adapt PUSH to the model's optimization needs as performance approaches saturation.

W3: Generalization across tasks

Our original experiments cover two task domains, three models, and multiple datasets. The coding experiments use both GRPO and DAPO and include transfer evaluation on MBPP/MBPP+ (Appendix D.1.1). These results support the consistency and effectiveness of PUSH as an interpretability-guided training method.

Following your valuable suggestion, we extended our evaluation to digital hardware design with 1.5B and 7B models. The models generate circuit designs from functional specifications, and correctness is evaluated through functional simulation. The experiments cover both VerilogEval V1 test sets (Human and Machine).

Both models show immediate accuracy gains after PUSH on both test sets. During continued RL, the 1.5B and 7B models reach matched accuracy with approximately 18.25% and 38.37% fewer rollouts, respectively, averaged across the two test sets.

The settings, Single PUSH continuation curves, and rollout savings for the new hardware design experiments have been added to Table C.1, Figure D.8, and Table D.6 of the revised manuscript, respectively. These results further validate the generalization of PUSH across tasks.

The authors have addressed my concerns. I will maintain my positive rating.

Thank you again for your detailed review and insightful suggestions! They have helped us improve the paper.
We are glad our additional experiments and analysis have addressed your concerns. We have uploaded the revised manuscript with these additions.
Given the overall contributions of this work, we hope you might consider a more favorable overall assessment. If you would like to discuss any points further, we would be very glad to answer.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Global response to reviewers

We thank all reviewers for their thoughtful feedback and positive assessment of this work’s contributions. Reviewers highlighted PUSH’s use of historical representation changes to improve training, the substantial continued-training experiments, and the comparisons with multiple baselines and ablation analyses. We also appreciate the recognition of this work’s value in connecting interpretability with training.

Below, we address the main concerns, the evidence supporting our conclusions, and the additional experimental results.

G1 - Generalization and immediate gains

Reviewers (bGCD, XxVk) asked about generalization and consistent immediate gains. On harder math and digital hardware design tasks, Single PUSH reduces cumulative rollouts to shared accuracy targets.

Table G1a. Single PUSH rollout savings on additional tasks

TaskModelSingle PUSH rollout savings
Mathematical reasoningQwen2.5-Math-7B Base22.76%
Digital hardware designQwen2.5-Coder-1.5B-Instruct18.25%
Digital hardware designQwen2.5-Coder-7B-Instruct38.37%

Math is evaluated on MATH500; hardware results average the VerilogEval V1 Human and Machine test sets. We extended Qwen2.5-Math-7B Base experiments on GSM8K to three training seeds: cumulative rollout savings are 18.99% ± 3.96% (Single PUSH) and 21.59% ± 2.66% (Multiple PUSH), as mean ± sample SD over each method’s eligible branches with shared accuracy targets.

Table G1b. Immediate gains on GSM8K (Qwen2.5-Math-7B Base; pp, three-seed mean ± sample SD), relative to the same-seed, same-step baseline.

Step326480128192304432
Single PUSH+4.27±0.70+4.27±1.51+2.13±1.03+3.67±1.14+2.07±1.40+2.07±1.10−0.07±0.23

Earlier schedules improve accuracy, while step 432 is nearly unchanged, consistent with diminishing gains as training approaches saturation. Reviewer bGCD confirmed that their concerns were addressed.

G2 - Feature analysis and parameterization controls (z1bc W1/Q1 & W2/Q2)

Additional analyses address three questions:

(1) Feature concentration. The Top 5 features constitute only 0.0174% of all 28,672 features, yet account for 66.14%–77.81% of feature-change energy under the alternative aggregation order, with concentration increasing to 67.46%–78.68% after excluding formatting tokens (Appendix A.3.1).

(2) Token-level feature selection. Across eight conditions, mean immediate gains are +2.92 to +5.96 pp for Full PUSH versus −0.04 to +1.50 for Norm-matched Top5. Dynamic token-level feature selection is important; matching update norms alone does not close this gap.

(3) Parameterization fairness. We matched checkpoints, data, LoRA branches, and update settings across SFT, ADL, RepE, and PUSH. PUSH achieves the highest mean gain in all 16 comparisons. Most immediate gains also remain after compressing the cumulative effective update back to the original rank of 32. This further supports our central claim that representation changes identified by ΔSAE can effectively guide training; PUSH’s advantage is not explained solely by a larger parameterization space.

G3 - Continued-training gains and sampling efficiency (bGCD W1/Q1; XxVk Q2/W1)

Reviewers asked whether immediate gains persist during continued training. In coding, Single PUSH and Multiple PUSH reduce cumulative rollouts to shared accuracy targets by 22.29% and 29.09% under GRPO, and by 18.97% and 31.61% under DAPO, respectively.

On GSM8K, across three seeds, 19 of the 21 Single PUSH branches have mean continuation accuracy at least as high as baseline, as do all 21 Multiple PUSH branches. Averages run from the first (Single PUSH) or last (Multiple PUSH) intervention to training end. XxVk confirmed that Q1 was resolved and the main concern in Q2 regarding immediate gains was addressed.

G4 - From representation properties to training improvements (bGCD W1/Q1 & W2; XxVk Q3/W2)

After PUSH at GRPO step 240, the nearest baseline checkpoints in raw representation space and SAE feature space are steps 390 and 420, while accuracy reaches a level first attained by the baseline at step 480. PUSH thus advances both representations and task performance.

We introduce ΔSAE to identify sparsity and directional persistence in representation changes, and use PUSH to turn these properties into training improvements. These results demonstrate a feasible bridge between interpretability and training. Developing adaptive intervention and stopping policies from intrinsic representation signals is a natural extension of this direction.

AuthorsOpenReview ↗

PDF revision

We have uploaded the revised manuscript with the additional experiments and analyses in the appendix. Feature-concentration analyses are in Appendix A.3.1, Top5 and token-level target controls in Appendix E.3, and rank-32 compression and matched-parameterization controls in Appendix D.2.3. Immediate gains, continuation curves, and rollout savings for the extended tasks and repeated runs are in Appendices D.1, D.3, and D.4.1, respectively. We thank the reviewers again for their helpful feedback!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.