- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes SAEPO, a Sparse Autoencoder (SAE)-guided token-level credit assignment method for outcome-supervised policy optimization in large language model reasoning. The method addresses the coarse credit assignment problem in conventional RLVR algorithms, where all tokens within a rollout share the same outcome-level advantage. Specifically, SAEPO employs a pretrained and frozen SAE to extract sparse semantic representations from token hidden states, computes token-level semantic scores by contrasting correct and incorrect rollouts of the same question, and incorporates these scores into the original outcome advantages. The resulting module can be integrated into both GRPO and GSPO without modifying their underlying reward verifiers or core policy optimization mechanisms.
The authors evaluate SAEPO on three language model backbones across six mathematical reasoning benchmarks and two code generation benchmarks. The experimental results demonstrate consistent improvements in average accuracy over vanilla GRPO/GSPO and several recent token-level credit assignment methods. Additional ablation studies and interpretability analyses are provided to investigate the effectiveness of sparse representations and the resulting token-level credit signals.
Critical strengths
Well-motivated and practically relevant problem. The paper addresses an important limitation of outcome-supervised reinforcement learning, where sequence-level rewards cannot distinguish the contributions of individual reasoning tokens. Improving fine-grained credit assignment is particularly relevant for long-chain reasoning tasks, making the research problem meaningful and timely.
Interesting integration of sparse representations and policy optimization. SAEPO introduces a question-local semantic contrast mechanism that uses sparse token representations to construct fine-grained advantages. Unlike approaches requiring separately trained process reward models or additional process annotations, the proposed method derives its training signals directly from sampled rollouts and outcome correctness. This provides an interesting perspective on representation-guided credit assignment.
Flexible and compatible optimization framework. The proposed advantage-shaping mechanism can be integrated into both GRPO and GSPO while preserving their original reward verification and policy optimization procedures. Moreover, the frozen SAE is only used during training, avoiding additional inference-time computation. This modular design improves the practical applicability of the approach.
Broad experimental evaluation with encouraging results. The experiments cover three model backbones, two policy optimization algorithms, six mathematical reasoning benchmarks, and two code generation benchmarks. SAEPO consistently improves average accuracy over the corresponding vanilla algorithms and generally outperforms the compared token-level credit assignment methods. The additional ablation studies and feature visualizations provide useful supporting evidence.
Critical weaknesses
Insufficient justification for the necessity and superiority of SAE representations. The main ablation compares SAE features against raw hidden states, but this comparison alone cannot establish that SAE-specific sparsity or semantic decomposition is responsible for the improvements. Other representation transformations, such as standard autoencoders, trained MLP encoders, or dimensionality transformations, might provide similar benefits. Since SAE is a central component of the proposed method, the absence of these comparisons weakens the evidence for its necessity and distinctive contribution.
Limited validation of the relationship between semantic similarity and reasoning correctness. The proposed token-level credit assignment assumes that tokens exhibiting greater semantic similarity to correct rollouts should receive higher advantages. However, semantic similarity does not necessarily imply that a token contributes positively to the correctness of a reasoning process. Incorrect reasoning steps may contain mathematical concepts similar to those in correct solutions. The current qualitative visualizations mainly demonstrate semantic relevance rather than the faithfulness of the assigned credit. More direct evidence is needed to validate this central assumption.
Additional training overhead is not adequately evaluated. SAEPO requires separately pretraining an SAE for each backbone, extracting token-level features during RL training, and performing cross-rollout semantic alignment. These operations introduce additional computational costs, particularly for long reasoning sequences. Although the paper reports faster improvements in training accuracy, it does not sufficiently quantify the additional SAE pretraining cost, training time, or GPU memory consumption compared with vanilla GRPO/GSPO. Therefore, the practical efficiency benefits remain unclear.
Unclear transferability, stability, and interpretability of pretrained SAE features. The authors train separate SAEs for different backbones and freeze them throughout policy optimization. However, the model's internal representations may change during RL training, potentially affecting the reliability of the pretrained SAE features. The reconstruction quality is mainly evaluated on FineWeb-Edu, and the paper does not sufficiently demonstrate feature stability throughout RL training or generalization across different model backbones. Furthermore, showing representative high-activation tokens provides qualitative interpretability evidence but does not establish that the learned features are consistently interpretable or faithfully associated with reasoning correctness.
Incomplete ablation studies and limited statistical reliability. Although the paper examines raw representations, reversed semantic scores, SAE sparsity, and layer selection, several important design choices remain insufficiently investigated, including the semantic-score coefficient, alignment temperature, correctness-balance weighting, and batch-level normalization. In addition, the results appear to be reported using a single random seed without standard deviations or confidence intervals. This is particularly concerning for relatively small improvements and benchmarks with limited evaluation samples, such as AIME25.
Questions for authors
Necessity of SAE representations: What evidence demonstrates that SAE representations are more effective than other learned feature transformations for token-level credit assignment? Could the authors compare SAEPO against standard autoencoders or MLP-based encoders under comparable representation dimensions and training budgets? Such comparisons would help establish whether the improvements originate specifically from sparse semantic decomposition rather than general representation transformation.
Faithfulness of token-level credit assignment: How can the authors verify that higher SAE-derived semantic scores correspond to genuinely correct or useful intermediate reasoning steps rather than merely question-relevant vocabulary? Could the authors provide a quantitative evaluation using reasoning trajectories containing identifiable correct and incorrect intermediate steps? This would help validate the central assumption underlying the proposed method.
Computational overhead and practical efficiency: What are the additional training time, peak GPU memory consumption, and computational costs introduced by SAE pretraining, token feature extraction, and cross-rollout semantic alignment? How does SAEPO compare with vanilla GRPO/GSPO under matched wall-clock time or computational budgets, rather than only matched training steps?
Transferability and stability of pretrained SAE features: Can a pretrained SAE provide consistent semantic representations across different tasks, such as mathematical reasoning and code generation, without task-specific adaptation? More importantly, does its reconstruction quality or feature activation distribution change significantly as the policy model evolves during RL training? Additional evidence regarding feature stability and transferability would strengthen the justification for using a separately pretrained and frozen SAE.
Hyperparameter sensitivity and statistical significance: How sensitive is SAEPO to the semantic-score coefficient α, alignment temperature τ, and correctness-balance weighting ρ? In addition, can the authors provide results from multiple random seeds for representative settings to establish that the reported improvements are statistically reliable? These analyses would help clarify the robustness of the proposed method.
13 comments
Author Response (2/5)
W2: "[...] semantic similarity does not necessarily imply that a token contributes positively to the correctness of a reasoning process. [...] semantic relevance rather than the faithfulness of the assigned credit."
W4.3: "[...] showing representative high-activation tokens [...] does not establish that the learned features are consistently interpretable or faithfully associated with reasoning correctness."
Q2: "How can the authors verify that higher SAE-derived semantic scores correspond to genuinely correct or useful intermediate reasoning steps rather than merely question-relevant vocabulary? [...] quantitative evaluation [...] identifiable correct and incorrect intermediate steps?"
This is a good suggestion. Inspired by your comments, to test whether SAEPO’s token scores can (1) distinguish correct from incorrect reasoning steps and (2) reflect final answer correctness, we ran an additional paired minimal-edit experiment. Specifically, we will establish the connection: SAEPO's token-level semantic score correctness of a local step prefix-continuation success rates (final accuracy).
Constructing a Controlled Dataset. We start by constructing a controlled dataset. We first sampled 976 problems from the training set and selected 1,864 Qwen3-1.7B-Base rollouts on them (904 correct, 960 incorrect). We removed rollouts where step boundaries are ambiguous or steps are difficult to edit, leading to a final controlled dataset with 403 correct and 507 incorrect rollouts. For each rollout, we used Qwen3-32B to pick one step and propose a minimal edit of it to construct a controlled dataset:
- (1) Error Edits: In a correct step, we changed a few tokens with LLM so that the step contains a single verifiable error introduced through a minimal edit (e.g., ).
- (2) Fixing Edits: In a wrong step, we applied a minimal edit that fixes it with LLM.
The LLM judge (Qwen3-32B), checking each version of the step blindly, must confirm that the original step is correct and the edited one is not for Case (1) above (and the reverse for Case (2) above), leaving 304 error edits and 120 fixing edits confirmed and included in the final evaluation. All tokens outside the edited span were kept unchanged.
Results on the Controlled Dataset: Connection of "SAEPO's Token-Level Semantic Score" "Correctness of a Local Step". We then fed these edited rollouts back into the model, took its hidden states, and recomputed (i.e., the token-level semantic score). is the mean of over the edited tokens minus the mean of over the original tokens they replace, divided by the standard deviation of over all tokens of the original rollout, so that edits from different rollouts are on a common scale. The table below shows the results.
| Edit | Step (judge): original → edited | Number of samples | Example | at edited tokens [95% CI] |
|---|---|---|---|---|
| Introduce an error | correct → incorrect | 304 | −0.58 [−0.70, −0.47] | |
| Fix an error | incorrect → correct | 120 | (Cylinder volume) | +0.16 [+0.06, +0.25] |
We can see that drops (i.e., negative ) when a correct step is broken and rises (i.e., positive ) when a wrong step is fixed. This verifies that our token-level score could distinguish correct from incorrect reasoning steps.
(To be continued.)
Author Response (3/5)
(Continued from the previous post.)
Effect on Continuation Success: Connection of "Correctness of a Local Step" "Prefix-Continuation Success Rates". The experiment above establishes the connection between our SAEPO's generated token score and the correctness of a reasoning step. Next, we ran an additional experiment to verify the connection between the correctness of a reasoning step and the correctness of the final answer. Specifically, we cut each rollout right after the edited tokens and let the policy continue reasoning 32 times from the original prefix and 32 times from the edited one. The table below shows the results. ( here is the fraction of continuations that reach the correct answer.)
| Edit | number of samples | : original → edited | [95% CI] | Edits with lowered / raised |
|---|---|---|---|---|
| Introduce an error | 304 | 0.74 → 0.17 | −0.57 [−0.61, −0.54] | 95% / 4% |
| Fix an error | 120 | 0.15 → 0.32 | +0.17 [+0.12, +0.22] | 23% / 62% |
We can see that a single broken step lowers the policy's own success rate in 95% of the edits, lowering the accuracy from 0.74 to 0.17 on average. Meanwhile, a single fix raises the policy's own success rate in 62% of the fixes, improving the accuracy from 0.15 to 0.32. These results verify the connection between the correctness of a reasoning step and the correctness of the final answer in our dataset, establishing its high quality.
Combining all additional results above, we can therefore establish the two-hop connection: SAEPO's token-level semantic score correctness of a local step prefix-continuation success rates (final accuracy).
We have revised Sec 4.2 (Table 3 and Table 4) in our paper to include the results and discussion above as suggested.
W3: "Additional training overhead is not adequately evaluated. [...] SAE pretraining cost, training time, or GPU memory consumption compared with vanilla GRPO/GSPO."
Q3: "[...] computational costs introduced by SAE pretraining, token feature extraction, and cross-rollout semantic alignment? [...] matched wall-clock time or computational budgets, rather than only matched training steps?"
Thank you for mentioning this. Following your suggestion we ran more experiments to document the time & memory consumption of all computation steps in the SAEPO and Vanilla GRPO/GSPO runs on RL training with Qwen3-1.7B-Base. Results are summarized in the table below.
| Per step | Vanilla GRPO | SAEPO (GRPO) | Vanilla GSPO | SAEPO (GSPO) |
|---|---|---|---|---|
| Rollout generation | 33.1 s | 37.8 s | 32.9 s | 34.5 s |
| Reward grading | 3.5 s | 2.7 s | 3.8 s | 2.7 s |
| Old/reference log-probs | 12.3 s | 13.6 s | 12.3 s | 13.1 s |
| Policy update | 32.4 s | 36.6 s | 32.7 s | 34.7 s |
| SAE token feature extraction | – | 0.77 s | – | 0.68 s |
| Cross-rollout semantic alignment | – | 0.34 s | – | 0.29 s |
| Total | 81.4 s | 92.0 s | 81.8 s | 86.0 s |
| Wall-clock, 250 steps | 5.66 h | 6.39 h | 5.68 h | 5.97 h |
| Peak GPU memory | 17.3 GB | 18.4 GB | 17.3 GB | 18.4 GB |
We can see that when training, the extra time introduced by the SAE consists of SAE token feature extraction and cross-rollout semantic alignment, which is 1.11 s per step for GRPO, i.e., 1.4% of a vanilla GRPO step (81.4 s), and 0.97 s per step for GSPO, i.e., 1.2% of a vanilla GSPO step (81.8 s). The rest of the difference in the total comes from SAEPO policies generating longer responses, not from the SAE computation. Therefore our SAEPO's computational overhead is minimal.
Note that SAE pre-training is one-off, which is trained once per backbone and reused by every run. Our SAE used by Qwen3-1.7B-Base remains unchanged for both math and code, takes 8.1 GPU-hours for pre-training.
We have included the results and discussion in the revision (Table 5 and Appendix C.5) as suggested.
Next we compare SAEPO with vanilla GRPO/GSPO under matched wall-clock time at the same training time and report the training set accuracies across different methods:
| Wall-clock | Vanilla GRPO | SAEPO (GRPO) | Vanilla GSPO | SAEPO (GSPO) |
|---|---|---|---|---|
| 1 h | 12.9 | 15.8 | 11.8 | 13.3 |
| 2 h | 16.9 | 25.8 | 14.6 | 21.2 |
| 3 h | 20.0 | 28.1 | 17.3 | 28.0 |
| 4 h | 20.0 | 28.8 | 20.1 | 24.9 |
| 5 h | 20.7 | 28.2 | 21.6 | 27.4 |
| 5.66 h (vanilla's 250 steps) | 22.2 | 28.7 | 20.7 | 25.5 |
These results show that SAEPO reaches vanilla's final training accuracies after 1.35 h (vanilla GRPO) and 1.75 h (vanilla GSPO), while vanilla GRPO and vanilla GSPO need 5.66 h and 5.68 h, respectively. This highlights our SAEPO's advantage in terms of accelerating convergence in RL training.
We have included the results and discussion in the revision (Table 6 and Appendix C.5) as suggested.
Author Response (1/5)
Thanks for your constructive and encouraging comments. We are glad you found our paper
"well-motivated", our proposed method"interesting"and empirical results"encouraging". Below we address your questions in detail, one by one. We also revised the paper accordingly, with all changes marked in blue.W1:
"Insufficient justification for the necessity and superiority of SAE representations. [...] cannot establish that SAE-specific sparsity or semantic decomposition is responsible for the improvements."Q1:
"Could the authors compare SAEPO against standard autoencoders or MLP-based encoders under comparable representation dimensions and training budgets?"This is a good question. In our paper, we compared credit assignment using SAE representation with raw hidden states from the large language model (LLM), and SAE representations showed better performance (Table 2 of the paper).
To further address your comment, we ran additional experiments to compare SAE, autoencoder (AE), and MLP-based encoders under latent dimensions (65,536 latents). For training budgets, we trained our SAE, AE and MLP on the policy's own math rollouts (Qwen3-1.7B-Base with 32,275 trajectories sampled from our RL training set, approximately 43.8M tokens). We trained the SAE under both reconstruction and sparsity losses, the AE under the reconstruction loss, and trained the MLP to predict the outcome of math rollouts with the cross-entropy loss through a linear readout. We then froze all of them and ran RL training with the SAEPO pipeline, with only the latent code differing. We report the accuracies on MATH benchmarks below:
The comparison above showed that the SAEPO with sparse semantic decomposition achieves higher average accuracy than the dense AE and MLP representations.
Moreover, SAE's latent code makes the credit signal interpretable. Each token activates only a few sparse, largely monosemantic latents, so the similarity that determines its credit decomposes into contributions from the concepts it shares with aligned tokens in correct and incorrect rollouts. This lets us identify which concepts are rewarded or penalized and track them across problems and over training. Dense AE or MLP codes spread the same information over many entangled coordinates with no individual meaning; therefore their credit cannot be traced back to human-interpretable concepts.
We have included the results and discussion above in the revision (Table 9 and Appendix F.1) as suggested.