- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates how reinforcement learning shapes internal LLM representations through the lens of frozen Top- sparse autoencoders (, expansion) and turns that observation into a lightweight mid-training intervention called PUSH (Sections 3–4, Figures 2–3). By tracking token-aligned feature shifts (Eqs. 1–2) between the pre-RL base model and intermediate RL checkpoints, the authors find that representation movement is remarkably low-dimensional and directionally persistent: at block H28, the same five SAE features (F3380, F20383, F17379, F26279, F17031) rank Top-5 across all six GRPO and DAPO runs and capture 93.4%–96.8% of the squared mean shift (Section 6.1, Appendix Table 2), while roughly 81% of features moving at step 60 continue in the same direction at step 1047 (Figure 2b).
To exploit this regularity, PUSH pauses RL at an intermediate checkpoint, selects the top changed SAE features per answer token on a tiny fixed guidance set ( AceCode or 128 GSM8K problems), splits them into a suppression set (features that dropped out of the pre-RL Top- support, targeted to 0) and a continuation set (features in current support, extrapolated via with , Eq. 3), fits two short rank-32 LoRA branches over 8–16 minibatches (Eq. 4, taking 53–77 seconds on one H200 GPU, Appendix Table 23), merges with default into the base weights (Eq. 5), and resumes RL. Tested on Qwen2.5-Coder-7B-Instruct (GRPO and DAPO on AceCode across 3 seeds, Tables 1–2) and single-run GSM8K experiments on Qwen2.5-Math-7B and DeepSeekMath-Instruct-7B, PUSH yields immediate accuracy jumps in 15 of 16 code settings (up to points in Table 1; at step 210 in Appendix Table 18) and cuts total rollouts to reach matched accuracy by roughly 30% (29.09% 5.67% under GRPO and 31.61% 2.82% under DAPO, Table 2).
Critical strengths
1. A novel, well-substantiated bridge from SAE representation dynamics to RL rollout efficiency.
Turning early historical feature changes () on 64 unlabeled guidance prompts into token-level suppression and continuation targets is a creative idea, and the ablations in Section 6.3 (Table 4) prove that the specific SAE structure is doing the work: shuffling token alignment (w/o align.) or replacing Top- sparse selection with random features (w/o Top-D) wipes out the +2.96 to +5.96 point gains almost completely (dropping to -0.50 to +0.29 points under w/o Top-D), and PUSH also cleanly outperforms weight-space model extrapolation (ExPO), even when ExPO is scaled to match in Frobenius norm across all 196 modules (winning 11 of 12 comparisons in Appendix Table 20).
2. Thorough, well-controlled empirical evaluation on the primary code-generation suite.
In the main AceCode setting (Qwen2.5-Coder-7B-Instruct under GRPO and DAPO), the authors evaluate across 8 starting checkpoints and 3 random seeds, compare against four natural baselines (SFT, ADL, RepE, and ExPO, Table 1), track 118 continuation trajectories (Figure 4, Appendix Figures 20–21), and translate the accuracy jump into a concrete systems metric: saving roughly 30% of total rollouts (about 52% of post-PUSH rollouts, Table 2) at the cost of a single 53-second LoRA fit (Appendix Table 23).
Critical weaknesses
1. Narrow empirical breadth beyond 7B code generation (single-run GSM8K math validation).
While the paper frames PUSH as a general RL post-training accelerator, the rigorous multi-seed statistical evidence is confined to a single model (Qwen2.5-Coder-7B-Instruct) and dataset (AceCode, with transfer checks on MBPP/MBPP+ in Appendix D.1.1). The mathematical reasoning experiments on Qwen2.5-Math-7B and DeepSeekMath-Instruct-7B are conducted on GSM8K with only one training seed per model (Appendix C.4 Table 17 and Appendix D.1 Figure 14), and Single PUSH on DeepSeekMath yields only a 1.54% total rollout reduction in Appendix Table 21 (8.65k rollouts saved, though Multiple PUSH reaches 24.87%). Expanding multi-seed validation to the math setting (or a harder math benchmark than GSM8K) is needed to match the generality of the claims.
2. Lack of a self-contained, validation-free stopping rule to avoid late-stage performance drops.
As shown in Figure 6b, Appendix Table 18 (step 810: -1.21 pp on GRPO, -3.38 pp on DAPO), and Appendix D.3.2 (where late Multiple PUSH schedules 270/600/870 and 600/720/840 underperform Single PUSH by -2.00 and -1.13 pp at step 1020), linear extrapolation along breaks down once RL representations approach saturation. While Appendix E.2 (Table 25, Figure 22) sketches an "exploratory stage-wise policy" (Early , Middle , Late No PUSH), the stage boundaries are chosen by inspecting validation-split scans rather than triggered automatically from internal statistics (such as feature velocity or directional consistency). For a practitioner deploying PUSH in a new run, a concrete heuristic for when to stop applying PUSH without running validation scans at every checkpoint is missing.
Questions for authors
-
Clarification affecting soundness (Optimizer state after base-weight write-back): When PUSH merges into (Appendix B.3 Eqs. 11–12) and resumes RL on the existing LoRA adapter , you restore the pre-PUSH AdamW momentum and variance buffers unchanged (Appendix C.1, line 1509). Did you test resetting AdamW moments (or running a few warmup steps) after the weight jump, and does stale momentum contribute to the brief post-PUSH dip or the late-stage instability seen at step 810?
-
Clarification affecting empirical strength (Math variance): Can you report 3-seed results (mean std) for at least one of the mathematical reasoning models (
Qwen2.5-Math-7BorDeepSeekMath-Instruct-7B), so we can tell whether the GSM8K immediate and continuation gains (Appendix Figure 14 and Table 21) are as consistent across seeds as the AceCode results? -
Clarification affecting practical deployment (Automatic stopping criterion): Looking at your trajectories in Section 3 / Appendix A, is there an intrinsic representation signal computable purely on the 64-problem guidance set —for example, a drop in step-to-step cosine similarity of or a plateau in between the last two checkpoints—that reliably predicts when PUSH will turn negative (around step 600–810) without requiring external validation-set evaluations?
6 comments
Q3 / W2
Looking at your trajectories in Section 3 / Appendix A, is there an intrinsic representation signal computable purely on the 64-problem guidance set —for example, a drop in step-to-step cosine similarity of or a plateau in between the last two checkpoints—that reliably predicts when PUSH will turn negative (around step 600–810) without requiring external validation-set evaluations?
Our work focuses on revealing the sparsity of feature changes during RL by analyzing the ΔSAE signal and successfully applying this property to accelerate training efficiently. Future work will build on these findings to develop mature adaptive intervention and stopping policies.
-
Fixed settings in the main experiments and stage-wise analysis on validation. The main experiments in this work (including coding, mathematical reasoning, and digital hardware design) use fixed intervention schedules and strengths. Validation scores are not used to decide these interventions. Single PUSH intervenes once at the scheduled checkpoint, then resumes RL; Multiple PUSH intervenes repeatedly along the same trajectory. Coding uses and the same schedules across training seeds within each algorithm. Immediate gains, continuation performance, and rollout savings are evaluated on the test sets.
The external validation-set evaluations you mention belong to a separate exploratory study in this work. It examines the sensitivity of PUSH gains to intervention strength and training stage, and explores whether adapting strength to each stage improves results. At each checkpoint, we compare A/B coefficients and measure immediate gains over the same-step baseline on the validation split. Each starting checkpoint is evaluated independently. Automatically identifying these stages, adjusting interventions, and deciding when to stop from intrinsic representation signals is precisely the further question you raise.
-
Intrinsic representation signals: directional consistency and change magnitude. Thank you very much for your perceptive and insightful ideas! We have already observed the phenomena you describe during our experiments:
On the fixed 64 guidance problems from one DAPO trajectory, the cosine similarity between consecutive change directions on PUSH-selected features falls from 0.598 at step 690 to 0.041 at step 720 and becomes negative at step 750 (−0.509). The RMS magnitude of cumulative changes increases from 0.0210 at step 540 to 0.0289 at step 690, then remains within 0.0289–0.0300 over steps 690–840, forming a local plateau. These observations are consistent with your discussion of late-stage saturation and provide key information for guiding future adaptive interventions.
We will investigate combining directional consistency and change magnitude to adjust PUSH timing and strength, reducing, skipping, or stopping interventions when the historical direction is no longer suitable. We will pair signals with intervention gains along the same trajectories to assess their ability to detect saturation in gains and representations.
Thank you again for your thoughtful review and insightful questions about PUSH’s reliability and practical applicability! If our responses have resolved your concerns, we would appreciate your reconsideration of the evaluation.
Thank you again for your careful review and thoughtful questions!
The discussion closes in less than one day. Your questions have been very helpful to us. We value your views and would welcome your assessment of the additional experiments and analysis.
Thank you again for taking the time to review our work!
Thank you for your positive feedback and insightful questions! We address them below.
Q1
Preserving optimizer state is part of the pluggable design. Our original design intent was to make PUSH an adjustable and pluggable weight intervention. We therefore restore the pre-PUSH AdamW state and training settings when RL resumes, preserving continuity of the original training procedure without introducing an additional optimizer restart (Appendix C.1).
Preliminary comparisons of preserving versus resetting AdamW state. Thank you for your careful analysis! We conducted experiments to examine this: starting from the same model after one PUSH, we resume RL with AdamW state preserved or its first and second moments zeroed, keeping all other settings unchanged; the table reports gains after RL continuation over the baseline at the same training step (pp), with immediate gains in parentheses. Each cell lists preserved / reset.
In these experiments, continuation gains still fluctuate after resetting optimizer state. Stale momentum alone does not explain these fluctuations in the tested trajectories.
The relationship between step 810 and optimizer state. The immediate accuracy changes at step 810 (the training saturation stage) for both GRPO and DAPO were measured after PUSH but before RL updates resumed. Stale momentum has not yet been used for an update and therefore cannot explain this immediate drop (Table D.1).
Q2 / W1
Three-seed statistics for immediate and continuation gains on mathematical reasoning. We added three-seed experiments with Qwen2.5-Math-7B Base. The model has positive immediate gains in all runs at the six checkpoints below and saves rollouts during continuation. Mean immediate gains (pp) are: step 32: +4.27 (±0.70); step 64: +4.27 (±1.51); step 80: +2.13 (±1.03); step 128: +3.67 (±1.14); step 192: +2.07 (±1.40); step 304: +2.07 (±1.10).
On GSM8K, mean cumulative rollout savings are 18.99% (±3.96) for Single PUSH and 21.59% (±2.66) for Multiple PUSH. These means are computed over the respective branches with eligible shared accuracy targets; the branch sets differ. Across runs, both generally maintain accuracy gains over baseline during continuation. The revised manuscript includes their continuation accuracy-gain curves in Appendices D.3.1 (Figure D.8(c)) and D.3.2 (Figure D.9(b–c)), respectively, and per-run rollout statistics in Appendix D.4.1 (Table D.6).
Generalization to harder math benchmarks and other tasks. We added GRPO training experiments on MATH [1] with evaluation on MATH500, alongside digital hardware design experiments at 1.5B and 7B, also using GRPO. Single PUSH reduces cumulative rollouts to shared accuracy targets by 22.76%, 18.25%, and 38.37%, respectively; hardware results are averaged over the VerilogEval V1 Human and Machine test sets.
Differences in cumulative rollout savings between Single and Multiple PUSH. For DeepSeekMath-Instruct-7B, the 1.54% and 24.87% measure cumulative rollout savings to shared accuracy targets, not immediate accuracy gains. Single PUSH saves only post-intervention rollouts, so pre-intervention costs dilute the cumulative savings ratio. Multiple PUSH uses newly accumulated representation changes to repeatedly construct targets and sustain its lead, yielding greater cumulative savings.
[1] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. arXiv preprint arXiv:2103.03874, 2021.