- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Sparse Autoencoder Reinforcement Learning (SAERL) uses SAE representations to organize data for RL post training. Projected gradients guide selection of strong and nonredundant samples, with zero reward samples retained separately. SAE activations then support clustering, difficulty ordering, and batch construction with moderate mixing. The schedule is built offline using small models and reused for larger models. Mathematical reasoning experiments report improvements in accuracy and convergence, while ablations support complementarity between ordering and batching and show additional gains over corresponding hidden state representations.
Critical strengths
The practical gains are meaningful. Table 1 improves average accuracy by 3.1 percentage points on the 1.5B model and reduces mean steps to target by approximately 42%. Reusing the schedule for 7B training further increases its relevance. I see this as a useful demonstration of interpretability tools contributing to an actual training workflow.
Critical weaknesses
-
There is a gap between the interpretation of nonredundant updates and the representation actually used. Equation (7) in Appendix A.1 sets , so opposing projected gradients can produce identical selection vectors. This representation describes where gradient magnitude is concentrated, but does not establish that two samples provide interchangeable updates. Taking absolute values may be useful empirically; my concern is the explanation attached to it. A signed representation control, or a measurement of actual update alignment among pairs classified as redundant, would directly test this assumption.
-
The cost boundary of the efficiency claim remains incomplete. Appendix C.1 reports 468 A100 GPU hours for training the two SAEs, while selection requires another 32.6 A800 GPU hours in Appendix C.4. SAE training is excluded from the main preprocessing comparison. Amortization is reasonable when suitable SAEs already exist or are reused, but total savings for a first application remain unestablished. Since efficiency is a central contribution, the paper should distinguish first use from reuse and estimate when the preparation cost is recovered.
Questions for authors
Of the final 44,510 training samples, how many come from the zero reward set in Equation (3), and how many come from the gradient selected core? A focused control would also help: keep , total subset size, and the curriculum construction procedure fixed, but replace the gradient selected core with an equally sized random sample from its candidate pool. This would distinguish the contribution of gradient selection from the contribution of retaining problems the small model initially cannot solve.
3 comments
Response to Reviewer QEZu (1/2)
We sincerely thank the reviewer for a detailed and thoughtful review. We are also glad the reviewer sees SAERL as "a useful demonstration of interpretability tools contributing to an actual training workflow", which is our central goal. All changes in the revised PDF are highlighted in blue.
A1 (W1): Taking absolute values in Eq. 7 shows where gradient magnitude concentrates, which does not establish that redundant samples give interchangeable updates.
We thank the reviewer for this insightful observation. The reviewer accurately characterizes as describing where gradient magnitude concentrates across SAE features, which is precisely the sense in which our selection treats two samples as redundant, namely when their updates engage largely the same features. We have made this meaning explicit in the introduction, Section 2.3, and Appendix A.1 of the revision so that it is not read as a claim about interchangeable updates.
As the reviewer suggests, taking absolute values was an empirical choice. On the 1.5B pool, signed cosines between samples are centered at zero (0.0005 ± 0.13), so signed gradient profiles are nearly orthogonal across samples, as is typical in high-dimensional sparse spaces, and a signed threshold could hardly separate pairs.
We also see a natural interpretation of this choice. With absolute values, the offline step records which SAE features a sample engages and how strongly, and leaves the direction of each update to online training, where it is recomputed at every RL step from fresh on-policy rollouts. The selection criterion therefore targets feature coverage.
The experiments support this design. Selection alone improves GRPO from 49.5 to 52.4 (Table 2) and outperforms selection with hidden-state gradients (50.8, Table 3), and the full pipeline built on the 1.5B subset also improves the 7B model (Table 1). We have added this interpretation and the signed-cosine statistics to Appendix A.1.
A2 (W2): The efficiency claim omits the SAE training cost and should distinguish first use from reuse.
We agree that the efficiency claim should distinguish first use from reuse. We listed SAE training separately because the SAEs are a reusable asset trained on general text (FineWeb-Edu and Wikipedia), and public Qwen SAEs now exist via Qwen-Scope. Below is the full accounting, counting A100 and A800 hours as roughly comparable and pricing an RL step at 0.084 / 0.221 A800 GPU hours at 1.5B / 7B.
| GPU hours | LearnAlign | CROPI | SAERL |
|---|---|---|---|
| SAE training (once, reusable) | – | – | 468 |
| Selection + encoding, 1.5B | 61.7 | 277.4 (per run) | 33.6 |
| Selection, 7B | 433.8 | 382.0 | 0 (reused) |
| Steps to target, 1.5B / 7B (GRPO: 537 / 313) | 517 / 363 | 590 / 510 | 310 / 223 |
| Total incl. RL to target, 1.5B + 7B (GRPO: 114.2) | 619.0 | 821.5 | 576.8 |
- First use against selection baselines is recovered within a single application at 1.5B and 7B, since SAERL's total including SAE training (576.8) is below LearnAlign (619.0) and CROPI (821.5).
- First use against plain GRPO is recovered after about 12 further pairs of 1.5B and 7B runs that reuse the subset, each saving about 39 GPU hours. Training SAEs from scratch thus pays off under reuse or with public SAEs, in which case SAERL's total is already comparable to GRPO (108.8 vs. 114.2), and SAERL also improves accuracy over GRPO by 3.1 / 1.9 points at 1.5B / 7B.
We have added this accounting to Section 3.3 and Appendix C.4 of the revision, and the efficiency claim now states both cases.
Gentle Reminder
Dear Reviewer,
With the discussion period nearing its end, we wanted to follow up on our previous responses. We hope we have addressed your questions satisfactorily. If you have any remaining concerns or questions regarding our response, we would be happy to address them promptly.
Thank you again for your time and valuable feedback.
Best regards,
The Authors
Response to Reviewer QEZu (2/2)
A3 (Q1): The contributions of the zero-reward set and the gradient-selected core should be separated, e.g., with a random-core control.
Both parts contribute, and they play different roles. H0 keeps hard problems that gradient selection cannot see by construction, while the gradient-selected core contributes beyond H0. Of the 44,510 samples, 16,500 come from H0 and 28,010 from the core. Zero-reward samples have zero GRPO advantage and hence zero gradient representations. They are problems the policy should eventually learn, and the curriculum places them late. Table 3 already holds H0 fixed, and switching the core from hidden-state to SAE gradients raises accuracy from 50.8 to 52.4. The requested control keeps H0, the subset size, and the training order fixed, using random order as in SAERL$_{\text{Sel}}$ to isolate selection from curriculum effects, and draws the core randomly from the same candidate pool, namely the samples whose rollouts receive mixed rewards.
The random core keeps part of the accuracy gain and converges even more slowly than full-data GRPO, whereas the gradient-selected core reaches 52.4 in 453 steps. Gradient selection thus accounts for the faster convergence and adds 1.3 points beyond retaining zero-reward problems. We have added this control to Section 4.1.
Final remark
Thank you again. These comments helped us clarify what our gradient representation captures and the scope of the efficiency claim, and we are happy to discuss further.