acceptodds
Under review as a conference paper at ICLR 2027

Teach the Way It Thinks: Guiding LLM Post-training Data Engineering with Sparse Autoencoders

Abstract

Data engineering for large language model (LLM) reinforcement learning (RL) decides which samples to train on, when to present them, and which to batch together. Existing methods mostly rely on external feedback, such as pass rates or difficulty labels, which observes only the outcome on each sample. We argue that model internals offer richer and more model-intrinsic feedback that can guide all three decisions. We propose SAERL which organizes RL training data in the feature space of Sparse Autoencoders (SAEs), a mechanistic interpretability tool that decomposes model internals into sparse, interpretable features. SAE features let us view the model as a set of functional regions, each formed by features that related samples activate together. For each sample, its SAE activations indicate which regions it engages, and its gradients projected onto SAE features indicate how training on it would change these regions. Guided by this view, SAERL keeps samples with strong and non-redundant gradient signals, trains each functional region from easy to hard, and trains one functional region at a time, drawing each batch from a single activation cluster with moderate cross-cluster mixing. The resulting curriculum is constructed once with small models and transfers to larger ones. Experiments on mathematical reasoning show consistent gains across model scales and RL algorithms. For instance, on Qwen2.5-Math-1.5B, SAERL improves the average accuracy of GRPO by 3.1% and reaches the target accuracy with 42% fewer training steps. Further analyses reveal that ordering and batching are complementary. These results demonstrate that model internals are a powerful and practical foundation for post-training data engineering.

Primary area:
foundation or frontier models, including LLMs
Keywords:
data-efficient training, reinforcement learning, data engineering, Sparse Autoencoder

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_QEZuOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Sparse Autoencoder Reinforcement Learning (SAERL) uses SAE representations to organize data for RL post training. Projected gradients guide selection of strong and nonredundant samples, with zero reward samples retained separately. SAE activations then support clustering, difficulty ordering, and batch construction with moderate mixing. The schedule is built offline using small models and reused for larger models. Mathematical reasoning experiments report improvements in accuracy and convergence, while ablations support complementarity between ordering and batching and show additional gains over corresponding hidden state representations.

Critical strengths

The practical gains are meaningful. Table 1 improves average accuracy by 3.1 percentage points on the 1.5B model and reduces mean steps to target by approximately 42%. Reusing the schedule for 7B training further increases its relevance. I see this as a useful demonstration of interpretability tools contributing to an actual training workflow.

Critical weaknesses
  1. There is a gap between the interpretation of nonredundant updates and the representation actually used. Equation (7) in Appendix A.1 sets gi=∣qi[G]∣g_i=|q_i[G]|, so opposing projected gradients can produce identical selection vectors. This representation describes where gradient magnitude is concentrated, but does not establish that two samples provide interchangeable updates. Taking absolute values may be useful empirically; my concern is the explanation attached to it. A signed representation control, or a measurement of actual update alignment among pairs classified as redundant, would directly test this assumption.

  2. The cost boundary of the efficiency claim remains incomplete. Appendix C.1 reports 468 A100 GPU hours for training the two SAEs, while selection requires another 32.6 A800 GPU hours in Appendix C.4. SAE training is excluded from the main preprocessing comparison. Amortization is reasonable when suitable SAEs already exist or are reused, but total savings for a first application remain unestablished. Since efficiency is a central contribution, the paper should distinguish first use from reuse and estimate when the preparation cost is recovered.

Questions for authors

Of the final 44,510 training samples, how many come from the zero reward set H0H_0 in Equation (3), and how many come from the gradient selected core? A focused control would also help: keep H0H_0, total subset size, and the curriculum construction procedure fixed, but replace the gradient selected core with an equally sized random sample from its candidate pool. This would distinguish the contribution of gradient selection from the contribution of retaining problems the small model initially cannot solve.

3 comments
AuthorsOpenReview ↗

Response to Reviewer QEZu (2/2)

A3 (Q1): The contributions of the zero-reward set and the gradient-selected core should be separated, e.g., with a random-core control.

Both parts contribute, and they play different roles. H0H_0 keeps hard problems that gradient selection cannot see by construction, while the gradient-selected core contributes beyond H0H_0. Of the 44,510 samples, 16,500 come from H0H_0 and 28,010 from the core. Zero-reward samples have zero GRPO advantage and hence zero gradient representations. They are problems the policy should eventually learn, and the curriculum places them late. Table 3 already holds H0H_0 fixed, and switching the core from hidden-state to SAE gradients raises accuracy from 50.8 to 52.4. The requested control keeps H0H_0, the subset size, and the training order fixed, using random order as in SAERL$_{\text{Sel}}$ to isolate selection from curriculum effects, and draws the core randomly from the same candidate pool, namely the samples whose rollouts receive mixed rewards.

CoreAvg accuracySteps to target
GRPO (full data)49.5653
H0H_0 + random core51.1680
H0H_0 + gradient-selected core (SAERL$_{\text{Sel}}$)52.4453

The random core keeps part of the accuracy gain and converges even more slowly than full-data GRPO, whereas the gradient-selected core reaches 52.4 in 453 steps. Gradient selection thus accounts for the faster convergence and adds 1.3 points beyond retaining zero-reward problems. We have added this control to Section 4.1.

Final remark

Thank you again. These comments helped us clarify what our gradient representation captures and the scope of the efficiency claim, and we are happy to discuss further.

AuthorsOpenReview ↗

Response to Reviewer QEZu (1/2)

We sincerely thank the reviewer for a detailed and thoughtful review. We are also glad the reviewer sees SAERL as "a useful demonstration of interpretability tools contributing to an actual training workflow", which is our central goal. All changes in the revised PDF are highlighted in blue.

A1 (W1): Taking absolute values in Eq. 7 shows where gradient magnitude concentrates, which does not establish that redundant samples give interchangeable updates.

We thank the reviewer for this insightful observation. The reviewer accurately characterizes ∣qi[G]∣|q_i[G]| as describing where gradient magnitude concentrates across SAE features, which is precisely the sense in which our selection treats two samples as redundant, namely when their updates engage largely the same features. We have made this meaning explicit in the introduction, Section 2.3, and Appendix A.1 of the revision so that it is not read as a claim about interchangeable updates.

As the reviewer suggests, taking absolute values was an empirical choice. On the 1.5B pool, signed cosines between samples are centered at zero (0.0005 ± 0.13), so signed gradient profiles are nearly orthogonal across samples, as is typical in high-dimensional sparse spaces, and a signed threshold could hardly separate pairs.

We also see a natural interpretation of this choice. With absolute values, the offline step records which SAE features a sample engages and how strongly, and leaves the direction of each update to online training, where it is recomputed at every RL step from fresh on-policy rollouts. The selection criterion therefore targets feature coverage.

The experiments support this design. Selection alone improves GRPO from 49.5 to 52.4 (Table 2) and outperforms selection with hidden-state gradients (50.8, Table 3), and the full pipeline built on the 1.5B subset also improves the 7B model (Table 1). We have added this interpretation and the signed-cosine statistics to Appendix A.1.

A2 (W2): The efficiency claim omits the SAE training cost and should distinguish first use from reuse.

We agree that the efficiency claim should distinguish first use from reuse. We listed SAE training separately because the SAEs are a reusable asset trained on general text (FineWeb-Edu and Wikipedia), and public Qwen SAEs now exist via Qwen-Scope. Below is the full accounting, counting A100 and A800 hours as roughly comparable and pricing an RL step at 0.084 / 0.221 A800 GPU hours at 1.5B / 7B.

GPU hoursLearnAlignCROPISAERL
SAE training (once, reusable)––468
Selection + encoding, 1.5B61.7277.4 (per run)33.6
Selection, 7B433.8382.00 (reused)
Steps to target, 1.5B / 7B (GRPO: 537 / 313)517 / 363590 / 510310 / 223
Total incl. RL to target, 1.5B + 7B (GRPO: 114.2)619.0821.5576.8
  • First use against selection baselines is recovered within a single application at 1.5B and 7B, since SAERL's total including SAE training (576.8) is below LearnAlign (619.0) and CROPI (821.5).
  • First use against plain GRPO is recovered after about 12 further pairs of 1.5B and 7B runs that reuse the subset, each saving about 39 GPU hours. Training SAEs from scratch thus pays off under reuse or with public SAEs, in which case SAERL's total is already comparable to GRPO (108.8 vs. 114.2), and SAERL also improves accuracy over GRPO by 3.1 / 1.9 points at 1.5B / 7B.

We have added this accounting to Section 3.3 and Appendix C.4 of the revision, and the efficiency claim now states both cases.

AuthorsOpenReview ↗

Gentle Reminder

Dear Reviewer,

With the discussion period nearing its end, we wanted to follow up on our previous responses. We hope we have addressed your questions satisfactorily. If you have any remaining concerns or questions regarding our response, we would be happy to address them promptly.

Thank you again for your time and valuable feedback.

Best regards,

The Authors

Reviewer_Kd5HOpenReview ↗
Rating
2
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SAE RL, a framework that uses sparse autoencoders (SAEs) to improve data selection and training for LLM reinforcement learning. The method uses SAE activations and gradient information to select useful training examples, arrange them from easy to hard, and group similar examples into batches.

Experiments on math reasoning tasks show that SAE RL improves both model accuracy and training efficiency compared with existing methods. The authors also study the effects of different components and show that training curricula designed using smaller models can also work for larger models.

Critical strengths
  1. The paper explores how SAE features can improve data selection, ordering, and batching in RL training. This offers a useful alternative to methods that mainly rely on external feedback or difficulty scores.

  2. The method combines gradient-based data selection with activation-based curriculum learning. The design is clear, and the results show that these components work well together.

  3. The paper compares SAE RL with several baselines across different model sizes and RL settings. The results show better accuracy and faster convergence. The ablation studies also help explain the benefits of the proposed method.

Critical weaknesses
  1. The experiments focus on math tasks using Qwen models and a single training dataset. It is unclear whether the method generalizes to other tasks, such as coding, instruction following, or general reasoning.

  2. The paper reports faster convergence and lower data selection costs but does not include the cost of SAE training. A more complete cost analysis is needed to support the efficiency claims.

  3. Although the paper compares SAE features with hidden states, the improvements may also come from clustering and batching strategies. Comparisons with simpler representations and clustering methods would help clarify the benefits of using SAEs.

Questions for authors
  1. Does SAE RL provide similar improvements on other tasks, such as coding or instruction following?

  2. How does the total cost compare with existing methods when SAE training and RL training are both included? How many training runs are needed to recover the extra cost?

  3. How does SAE-based clustering compare with simpler methods, such as PCA or embedding-based clustering, under the same curriculum and batching settings?

7 comments
AuthorsOpenReview ↗

Response to Reviewer Kd5H

We thank the reviewer for the constructive comments and for recognizing the clear design, the complementary components, and the thorough comparisons. All changes in the revised PDF are highlighted in blue.

A1 (W1, Q1): The experiments cover only math with Qwen models and one dataset, so generalization to other tasks is unclear.

We agree that this is a limitation of our experimental scope, as acknowledged in Section 6. We chose this scope deliberately, because math RLVR offers reliable verifiable rewards and difficulty-graded benchmarks that make differences attributable to data engineering, and because it is the standard testbed of the closest methods (e.g., LearnAlign, ADARFT). Within this scope, we vary the model scale (Table 1), the RL algorithm (Table 7), the batch size and metric (Table 6), the SAE source family including Llama-3.1-8B (Table 9), and the dataset providing difficulty supervision (Table 10). The SAEs are trained on general text, selection only needs verifiable rewards, and the difficulty labels can come from another dataset, so code RLVR with unit-test rewards meets the same requirements and is our natural next step.

A2 (W2, Q2): The efficiency claim omits the SAE training cost, and the number of runs needed to recover it is unclear.

We kindly remind the reviewer that the SAE training cost is reported in Appendix C.1 (468 A100 GPU hours), and we agree that it should be made part of the efficiency claim. We listed it separately in Appendix C.4 because the SAEs are a reusable asset trained on general text (FineWeb-Edu and Wikipedia), and public Qwen SAEs (Qwen-Scope) are now available. Below is the full accounting, counting A100 and A800 hours as roughly comparable and pricing an RL step at 0.084 / 0.221 A800 GPU hours at 1.5B / 7B.

GPU hoursLearnAlignCROPISAERL
SAE training (once, reusable)––468
Selection + encoding, 1.5B61.7277.4 (per run)33.6
Selection, 7B433.8382.00 (reused)
Steps to target, 1.5B / 7B (GRPO: 537 / 313)517 / 363590 / 510310 / 223
Total incl. RL to target, 1.5B + 7B (GRPO: 114.2)619.0821.5576.8
  • Against selection baselines, a single application at 1.5B and 7B already recovers the SAE cost, since SAERL's total including SAE training (576.8) is below LearnAlign (619.0) and CROPI (821.5).
  • Against plain GRPO, each further pair of 1.5B and 7B runs that reuses the subset saves about 39 GPU hours, so the SAE cost is recovered after about 12 such pairs. Training SAEs from scratch thus pays off when they are reused or replaced by public SAEs, in which case SAERL's total is already comparable to GRPO (108.8 vs. 114.2), and SAERL also improves accuracy over GRPO by 3.1 / 1.9 points at 1.5B / 7B.

We have added this accounting to Section 3.3 and Appendix C.4 of the revision.

A3 (W3, Q3): The gains may come from clustering and batching, and SAEs should be compared with simpler representations under the same pipeline.

We agree that part of the improvement comes from the clustering and batching strategy, and Table 3 allows us to separate the two contributions, as it holds the clustering, ordering, and batching fixed and changes only the representation. In curriculum construction under the same strategy, hidden-state activations, i.e., pooled embeddings from the same model, gain 0.8 points over GRPO, whereas SAE features gain 3.0, so most of the curriculum gain is attributable to the representation itself. As suggested, we further run the full pipeline with raw hidden states and with their dimension-matched PCA projections (Qwen2.5-Math-1.5B, step 900).

Representation (full pipeline)AIMEAMCGSM8KMATHMNVOLPDAvg
GRPO30.053.481.171.426.134.849.5
Hidden-state representation36.753.884.972.128.035.551.8
Hidden-state PCA projection36.753.884.171.727.135.051.4
SAE representation (SAERL)36.755.385.472.729.436.052.6

SAERL achieves the highest average accuracy (52.6) and matches or exceeds both alternatives on all six benchmarks, while PCA projections bring no gain over hidden states (51.4 vs. 51.8). Together with Table 3, this shows that the SAE representation consistently outperforms hidden states in selection, curriculum construction, and the full pipeline. Qualitatively, SAE clusters combine topic, problem format, and solution profile, and align only weakly with the dataset's topic labels (NMI 0.088, Appendix E.2). We have added the full-pipeline comparison to Table 3 and Section 4.2.

Final remark

We hope these clarifications and results address the concerns, and we would be grateful if the reviewer could reconsider the assessment.

AuthorsOpenReview ↗

Gentle Reminder

Dear Reviewer,

As the discussion period is drawing to a close, we wanted to check in and see whether our responses have sufficiently addressed your concerns. If there are any remaining points you would like us to clarify, please feel free to let us know. We would be glad to provide further explanations.

Thank you again for your time and valuable feedback.

Best regards,

The Authors

Thank you for the detailed rebuttal and additional experiments. I appreciate the new analyses, which help address some of my concerns. However, the improvement over the strongest 7B baseline remains marginal (+0.4% accuracy, with nearly identical convergence), and the SAE-specific benefits are still not fully isolated. Given the added complexity of the framework, I remain unconvinced that the empirical gains are sufficiently substantial. I will therefore maintain my score of 2.

Reviewer_CDHJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SAE RL, which uses sparse autoencoder features to guide data selection, curriculum ordering, and batch construction for LLM reinforcement learning. SAE-projected gradients are used to select high-signal, non-redundant samples, while SAE activations support clustering and difficulty-based scheduling. Experiments on mathematical reasoning benchmarks report improvements in accuracy and convergence efficiency across model scales and RL algorithms.

Critical strengths

Using model-internal features for both data selection and curriculum construction is an interesting direction, particularly when the resulting data schedule can transfer from smaller to larger models.

Critical weaknesses
  1. The full framework provides negligible additional accuracy gains over its individual components. In Table 2, selection-only and curriculum-only achieve 52.4% and 52.5% average accuracy, respectively, compared with 52.6% for the full method. Moreover, the improvement over the strongest competing baseline on 7B is only 0.5 percentage points, without sufficient uncertainty estimates to establish its robustness.

  2. The SAE-specific contribution and mechanistic justification are insufficiently established. The proposed pipeline largely applies conventional techniques (e.g., gradient-based filtering, K-means clustering, difficulty sorting, and batch mixing) in an SAE feature space. While Table 3 shows improvements over raw hidden-state representations, it remains unclear whether these gains arise from SAE-specific structure or simply from a different representation space.

Questions for authors
  1. Can the authors compare SAE representations with alternative representations (e.g., PCA, dense autoencoders, or random projections)?
  2. How does the method work for larger-scale models?
3 comments
AuthorsOpenReview ↗

Response to Reviewer CDHJ (2/2)

A3 (W2, Q1): It is unclear whether the gains are specific to SAEs, and SAEs should be compared with alternatives such as PCA.

We would like to clarify a possible misunderstanding about Table 3, which already isolates the SAE-specific structure. The SAE representation is a deterministic function of the very hidden states used in the hidden-state baseline (same model, same layer), so it carries no additional information, and the decomposition into sparse features is the only difference between the two. The gains in Table 3 (+1.6 for selection, +2.2 for curriculum, all else fixed) can therefore only come from this decomposition, which is what we refer to as SAE-specific structure.

Following the reviewer's suggestion, and to further rule out that any re-encoding would help, we run the full pipeline with raw hidden states and with their PCA projections, dimension-matched to the SAE representation. PCA is equivalent to an optimal linear autoencoder and thus serves as the dense, non-sparse counterpart of the SAE. Random projections approximately preserve the hidden-state geometry (Johnson–Lindenstrauss) and are already approximated by the hidden-state row.

Representation (full pipeline)AIMEAMCGSM8KMATHMNVOLPDAvg
GRPO30.053.481.171.426.134.849.5
Hidden-state representation36.753.884.972.128.035.551.8
Hidden-state PCA projection36.753.884.171.727.135.051.4
SAE representation (SAERL)36.755.385.472.729.436.052.6

SAERL outperforms both raw hidden states (51.8) and their dimension-matched PCA projections (51.4), and matches or exceeds them on all six benchmarks. As a dense re-encoding of the same dimension brings no gain, SAERL's advantage over hidden states stems from the sparse decomposition itself. We have added these results to Table 3 and Section 4.2.

We use standard operators (cosine filtering, K-means, sorting, tail swapping) deliberately, so that the gains can be attributed to the representation itself. As mechanistic support, Appendix B links SAE-space proximity to gradient similarity, and Appendix E shows that SAE clusters combine topic, problem format, and solution profile.

Final remark

We hope these clarifications and results address the concerns, and we would be grateful if the reviewer could reconsider the assessment.

AuthorsOpenReview ↗

Response to Reviewer CDHJ (1/2)

We thank the reviewer for the feedback and for finding our direction interesting. We address each concern below and report the requested experiments. All changes in the revised PDF are highlighted in blue.

A1 (W1): The full method adds little accuracy over its components (52.6 vs. 52.4 / 52.5 in Table 2).

We agree that the final accuracy of the full method is close to that of each component, and its main benefit lies in faster convergence. As stated in Section 4.1, combining selection and curriculum reduces the average steps to target from 427 and 437 to 333 (22–24% fewer) while slightly improving accuracy. Each component alone already gains about 3 points over GRPO and beats all baselines in its category (52.4 / 52.5 vs. best 50.1 / 50.6), and the full method matches the curriculum-only accuracy with only 43% of the data. The complementarity highlighted in our abstract concerns ordering and batching, which is clearly visible in accuracy (50.9 and 49.8 alone vs. 52.5 combined). We have made this distinction explicit in Section 4.1 of the revision.

A2 (W1, Q2): The 0.5-point gain at 7B lacks uncertainty estimates, and larger models are not tested.

We would like to add context to the 0.5-point margin at 7B and provide further uncertainty estimates.

The margin is obtained in an asymmetric setting, in which SAERL reuses the subset and curriculum built with small models and requires no computation on the 7B model, whereas LearnAlign, CROPI, and GAINRL all compute their signals on it. Even so, SAERL has the best accuracy (61.9) and the largest gain over GRPO (+1.9). It is also the only method ranking first at both scales, while GAINRL, the strongest 7B baseline, gives no gain at 1.5B.

Regarding uncertainty, Figure 1 reports three seeds for both methods at 1.5B (peak gap +2.1 points), and at 7B we add three seeds for SAERL and compare them with the baselines in Table 1.

Method (7B)Avg accuracy (%)Steps to target
GRPO (Table 1)60.0313
GAINRL (Table 1)61.4250
SAERL (3 seeds)61.8 ± 0.1249 ± 24

SAERL's mean stays above GAINRL with a standard deviation of only 0.1, so the 7B gain is robust to the seed, and its average convergence matches GAINRL while staying well ahead of GRPO. We have added these results to Sections 3.2 and 3.3.

Regarding larger models, we agree that results beyond 7B would further strengthen the paper. A single RL run at 14B–32B costs several times our 7B runs, which exceeds our academic compute budget, particularly within the limited window of the rebuttal period. Our 1.5B–7B range also follows closely related work (e.g., LearnAlign, ADARFT, GAINRL). Notably, SAERL's preparation cost stays constant as the target model grows, whereas baselines that compute signals on it become more expensive, so its practical advantage should widen at larger scales. We have stated this limitation in Section 6 of the revision and will add larger-scale results if resources permit.

AuthorsOpenReview ↗

Gentle Reminder

Dear Reviewer,

We wanted to briefly follow up as the discussion period is approaching its end. We hope our responses have adequately addressed your concerns. If there are any remaining questions or points that require further clarification, we would be happy to address them.

Thank you again for your time and valuable feedback.

Best regards,

The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Author Responses

Dear Area Chair,

We thank you and the reviewers for the constructive discussion. We briefly summarize the reviews, the revisions we made, and the current status of the discussion.

Strengths noted by reviewers. Reviewers find that organizing RL training data with SAE features is an interesting and useful direction (CDHJ, Kd5H), with "meaningful" practical gains and "a useful demonstration of interpretability tools contributing to an actual training workflow" (QEZu). They also note the clear design and complementary components (Kd5H), and the transfer of the data schedule from small to large models (CDHJ, QEZu).

Main concerns and revisions (highlighted in blue in the revised PDF):

  1. SAE-specific contribution (CDHJ, Kd5H). We added a full-pipeline comparison with raw hidden states and dimension-matched PCA projections, in which SAE features perform best (Table 3, Section 4.2).
  2. Cost of SAE training (Kd5H, QEZu). We added a full cost accounting that separates first use from reuse. Including SAE training, SAERL's total cost is below that of the selection baselines after a single application (Section 3.3, Appendix C.4).
  3. Robustness at 7B (CDHJ). We added three-seed results at 7B, where SAERL remains above the strongest baseline with low variance (Sections 3.2 and 3.3).
  4. Rationale of gradient selection (QEZu). We clarified the meaning of the selection representation and added the requested random-core control (Appendix A.1, Section 4.1).
  5. Task scope (Kd5H). We acknowledged this limitation and clarified the robustness already covered within math RLVR (Section 6).

Discussion status. QEZu maintains a positive assessment, and Kd5H has not raised further concerns after our response. CDHJ maintains the score, citing the 7B margin and the isolation of the SAE contribution. We respectfully believe the latter reflects a misunderstanding: Table 3 changes only the representation, with the model, layer, and pipeline fixed, so the sparse decomposition is the only difference. Regarding the 7B margin, SAERL is the only method ranking first at both scales, and it achieves this at 7B without any computation on the target model, unlike all baselines.

We would be grateful if these responses could be taken into account.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.