acceptodds
Under review as a conference paper at ICLR 2027

SAEPO: Sparse-Autoencoder-Guided Credit Assignment for Policy Optimization

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) has demonstrated strong performance in improving large language model (LLM) reasoning, yet its credit-assignment signal remains coarse. Outcome verifiers provide scalable supervision by judging whether a completed response is correct, but they do not directly indicate which intermediate reasoning tokens contribute to success or failure. As a result, policy optimization can be inefficient, especially when training base models directly from sparse outcome feedback. In this paper, we introduce SAEPO, a Sparse AutoEncoder-guided token-level credit assignment method for outcome-supervised Policy Optimization. Rather than treating correctness only as a rollout-level (sequence-level) signal, SAEPO uses a pretrained sparse autoencoder (SAE) to build token-level semantic representations within reasoning trajectories; it scores each token based on the SAE concepts (e.g., penalizing wrong “final-answer” tokens but rewarding correct “reasoning-process” tokens), learned in an unsupervised manner, to achieve fine-grained credit assignment. SAEPO is a plug-in advantage-shaping module compatible with various policy optimization algorithms. We instantiate SAEPO with both GRPO and GSPO, covering token-ratio and sequence-ratio policy-gradient objectives, and evaluate SAEPO across three LLM backbones on six math and two code benchmarks. Empirical results show that SAEPO improves both the math and code benchmark averages over the matched baselines, with gains of up to 6.63% and 5.57%, respectively. On Qwen3-1.7B-Base, training curves also show faster early accuracy growth. Feature and token-level analyses suggest that the SAE-derived signal favors question-relevant concepts and reasoning tokens.

Primary area:
interpretability and explainable AI
Keywords:
Reinforcement Learning, Interpretability, Sparse Autoencoders

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_9YKKOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SAEPO, a Sparse Autoencoder (SAE)-guided token-level credit assignment method for outcome-supervised policy optimization in large language model reasoning. The method addresses the coarse credit assignment problem in conventional RLVR algorithms, where all tokens within a rollout share the same outcome-level advantage. Specifically, SAEPO employs a pretrained and frozen SAE to extract sparse semantic representations from token hidden states, computes token-level semantic scores by contrasting correct and incorrect rollouts of the same question, and incorporates these scores into the original outcome advantages. The resulting module can be integrated into both GRPO and GSPO without modifying their underlying reward verifiers or core policy optimization mechanisms.

The authors evaluate SAEPO on three language model backbones across six mathematical reasoning benchmarks and two code generation benchmarks. The experimental results demonstrate consistent improvements in average accuracy over vanilla GRPO/GSPO and several recent token-level credit assignment methods. Additional ablation studies and interpretability analyses are provided to investigate the effectiveness of sparse representations and the resulting token-level credit signals.

Critical strengths

Well-motivated and practically relevant problem. The paper addresses an important limitation of outcome-supervised reinforcement learning, where sequence-level rewards cannot distinguish the contributions of individual reasoning tokens. Improving fine-grained credit assignment is particularly relevant for long-chain reasoning tasks, making the research problem meaningful and timely.

Interesting integration of sparse representations and policy optimization. SAEPO introduces a question-local semantic contrast mechanism that uses sparse token representations to construct fine-grained advantages. Unlike approaches requiring separately trained process reward models or additional process annotations, the proposed method derives its training signals directly from sampled rollouts and outcome correctness. This provides an interesting perspective on representation-guided credit assignment.

Flexible and compatible optimization framework. The proposed advantage-shaping mechanism can be integrated into both GRPO and GSPO while preserving their original reward verification and policy optimization procedures. Moreover, the frozen SAE is only used during training, avoiding additional inference-time computation. This modular design improves the practical applicability of the approach.

Broad experimental evaluation with encouraging results. The experiments cover three model backbones, two policy optimization algorithms, six mathematical reasoning benchmarks, and two code generation benchmarks. SAEPO consistently improves average accuracy over the corresponding vanilla algorithms and generally outperforms the compared token-level credit assignment methods. The additional ablation studies and feature visualizations provide useful supporting evidence.

Critical weaknesses

Insufficient justification for the necessity and superiority of SAE representations. The main ablation compares SAE features against raw hidden states, but this comparison alone cannot establish that SAE-specific sparsity or semantic decomposition is responsible for the improvements. Other representation transformations, such as standard autoencoders, trained MLP encoders, or dimensionality transformations, might provide similar benefits. Since SAE is a central component of the proposed method, the absence of these comparisons weakens the evidence for its necessity and distinctive contribution.

Limited validation of the relationship between semantic similarity and reasoning correctness. The proposed token-level credit assignment assumes that tokens exhibiting greater semantic similarity to correct rollouts should receive higher advantages. However, semantic similarity does not necessarily imply that a token contributes positively to the correctness of a reasoning process. Incorrect reasoning steps may contain mathematical concepts similar to those in correct solutions. The current qualitative visualizations mainly demonstrate semantic relevance rather than the faithfulness of the assigned credit. More direct evidence is needed to validate this central assumption.

Additional training overhead is not adequately evaluated. SAEPO requires separately pretraining an SAE for each backbone, extracting token-level features during RL training, and performing cross-rollout semantic alignment. These operations introduce additional computational costs, particularly for long reasoning sequences. Although the paper reports faster improvements in training accuracy, it does not sufficiently quantify the additional SAE pretraining cost, training time, or GPU memory consumption compared with vanilla GRPO/GSPO. Therefore, the practical efficiency benefits remain unclear.

Unclear transferability, stability, and interpretability of pretrained SAE features. The authors train separate SAEs for different backbones and freeze them throughout policy optimization. However, the model's internal representations may change during RL training, potentially affecting the reliability of the pretrained SAE features. The reconstruction quality is mainly evaluated on FineWeb-Edu, and the paper does not sufficiently demonstrate feature stability throughout RL training or generalization across different model backbones. Furthermore, showing representative high-activation tokens provides qualitative interpretability evidence but does not establish that the learned features are consistently interpretable or faithfully associated with reasoning correctness.

Incomplete ablation studies and limited statistical reliability. Although the paper examines raw representations, reversed semantic scores, SAE sparsity, and layer selection, several important design choices remain insufficiently investigated, including the semantic-score coefficient, alignment temperature, correctness-balance weighting, and batch-level normalization. In addition, the results appear to be reported using a single random seed without standard deviations or confidence intervals. This is particularly concerning for relatively small improvements and benchmarks with limited evaluation samples, such as AIME25.

Questions for authors

Necessity of SAE representations: What evidence demonstrates that SAE representations are more effective than other learned feature transformations for token-level credit assignment? Could the authors compare SAEPO against standard autoencoders or MLP-based encoders under comparable representation dimensions and training budgets? Such comparisons would help establish whether the improvements originate specifically from sparse semantic decomposition rather than general representation transformation.

Faithfulness of token-level credit assignment: How can the authors verify that higher SAE-derived semantic scores correspond to genuinely correct or useful intermediate reasoning steps rather than merely question-relevant vocabulary? Could the authors provide a quantitative evaluation using reasoning trajectories containing identifiable correct and incorrect intermediate steps? This would help validate the central assumption underlying the proposed method.

Computational overhead and practical efficiency: What are the additional training time, peak GPU memory consumption, and computational costs introduced by SAE pretraining, token feature extraction, and cross-rollout semantic alignment? How does SAEPO compare with vanilla GRPO/GSPO under matched wall-clock time or computational budgets, rather than only matched training steps?

Transferability and stability of pretrained SAE features: Can a pretrained SAE provide consistent semantic representations across different tasks, such as mathematical reasoning and code generation, without task-specific adaptation? More importantly, does its reconstruction quality or feature activation distribution change significantly as the policy model evolves during RL training? Additional evidence regarding feature stability and transferability would strengthen the justification for using a separately pretrained and frozen SAE.

Hyperparameter sensitivity and statistical significance: How sensitive is SAEPO to the semantic-score coefficient α, alignment temperature τ, and correctness-balance weighting ρ? In addition, can the authors provide results from multiple random seeds for representative settings to establish that the reported improvements are statistically reliable? These analyses would help clarify the robustness of the proposed method.

13 comments
AuthorsOpenReview ↗

Author Response (1/5)

Thanks for your constructive and encouraging comments. We are glad you found our paper "well-motivated", our proposed method "interesting" and empirical results "encouraging". Below we address your questions in detail, one by one. We also revised the paper accordingly, with all changes marked in blue.

W1: "Insufficient justification for the necessity and superiority of SAE representations. [...] cannot establish that SAE-specific sparsity or semantic decomposition is responsible for the improvements."

Q1: "Could the authors compare SAEPO against standard autoencoders or MLP-based encoders under comparable representation dimensions and training budgets?"

This is a good question. In our paper, we compared credit assignment using SAE representation with raw hidden states from the large language model (LLM), and SAE representations showed better performance (Table 2 of the paper).

To further address your comment, we ran additional experiments to compare SAE, autoencoder (AE), and MLP-based encoders under latent dimensions (65,536 latents). For training budgets, we trained our SAE, AE and MLP on the policy's own math rollouts (Qwen3-1.7B-Base with 32,275 trajectories sampled from our RL training set, approximately 43.8M tokens). We trained the SAE under both reconstruction and sparsity losses, the AE under the reconstruction loss, and trained the MLP to predict the outcome of math rollouts with the cross-entropy loss through a linear readout. We then froze all of them and ran RL training with the SAEPO pipeline, with only the latent code differing. We report the accuracies on MATH benchmarks below:

RepresentationMATH-500GSM8KAIME25AMCMinervaOlympiadBenchAvg.
SAE latent code65.978.06.734.533.322.340.12
MLP latent code59.077.53.326.526.519.635.40
AE latent code63.874.13.334.533.121.838.43

The comparison above showed that the SAEPO with sparse semantic decomposition achieves higher average accuracy than the dense AE and MLP representations.

Moreover, SAE's latent code makes the credit signal interpretable. Each token activates only a few sparse, largely monosemantic latents, so the similarity that determines its credit decomposes into contributions from the concepts it shares with aligned tokens in correct and incorrect rollouts. This lets us identify which concepts are rewarded or penalized and track them across problems and over training. Dense AE or MLP codes spread the same information over many entangled coordinates with no individual meaning; therefore their credit cannot be traced back to human-interpretable concepts.

We have included the results and discussion above in the revision (Table 9 and Appendix F.1) as suggested.

AuthorsOpenReview ↗

Author Response (2/5)

W2: "[...] semantic similarity does not necessarily imply that a token contributes positively to the correctness of a reasoning process. [...] semantic relevance rather than the faithfulness of the assigned credit."

W4.3: "[...] showing representative high-activation tokens [...] does not establish that the learned features are consistently interpretable or faithfully associated with reasoning correctness."

Q2: "How can the authors verify that higher SAE-derived semantic scores correspond to genuinely correct or useful intermediate reasoning steps rather than merely question-relevant vocabulary? [...] quantitative evaluation [...] identifiable correct and incorrect intermediate steps?"

This is a good suggestion. Inspired by your comments, to test whether SAEPO’s token scores can (1) distinguish correct from incorrect reasoning steps and (2) reflect final answer correctness, we ran an additional paired minimal-edit experiment. Specifically, we will establish the connection: SAEPO's token-level semantic score →\rightarrow correctness of a local step →\rightarrow prefix-continuation success rates (final accuracy).

Constructing a Controlled Dataset. We start by constructing a controlled dataset. We first sampled 976 problems from the training set and selected 1,864 Qwen3-1.7B-Base rollouts on them (904 correct, 960 incorrect). We removed rollouts where step boundaries are ambiguous or steps are difficult to edit, leading to a final controlled dataset with 403 correct and 507 incorrect rollouts. For each rollout, we used Qwen3-32B to pick one step and propose a minimal edit of it to construct a controlled dataset:

  • (1) Error Edits: In a correct step, we changed a few tokens with LLM so that the step contains a single verifiable error introduced through a minimal edit (e.g., 24→2524 \to 25).
  • (2) Fixing Edits: In a wrong step, we applied a minimal edit that fixes it with LLM.

The LLM judge (Qwen3-32B), checking each version of the step blindly, must confirm that the original step is correct and the edited one is not for Case (1) above (and the reverse for Case (2) above), leaving 304 error edits and 120 fixing edits confirmed and included in the final evaluation. All tokens outside the edited span were kept unchanged.

Results on the Controlled Dataset: Connection of "SAEPO's Token-Level Semantic Score" →\rightarrow "Correctness of a Local Step". We then fed these edited rollouts back into the model, took its hidden states, and recomputed SS (i.e., the token-level semantic score). ΔS\Delta S is the mean of SS over the edited tokens minus the mean of SS over the original tokens they replace, divided by the standard deviation of SS over all tokens of the original rollout, so that edits from different rollouts are on a common scale. The table below shows the results.

EditStep (judge): original → editedNumber of samplesExampleΔS\Delta S at edited tokens [95% CI]
Introduce an errorcorrect → incorrect3048×3=24  →  8×3=258\times3=24\;\to\;8\times3=25−0.58 [−0.70, −0.47]
Fix an errorincorrect → correct120(Cylinder volume) V=13πr2h  →  V=πr2hV=\frac{1}{3}\pi r^2h\;\to\;V=\pi r^2h+0.16 [+0.06, +0.25]

We can see that SS drops (i.e., negative ΔS\Delta S) when a correct step is broken and rises (i.e., positive ΔS\Delta S) when a wrong step is fixed. This verifies that our token-level score SS could distinguish correct from incorrect reasoning steps.

(To be continued.)

AuthorsOpenReview ↗

Author Response (3/5)

(Continued from the previous post.)

Effect on Continuation Success: Connection of "Correctness of a Local Step" →\rightarrow "Prefix-Continuation Success Rates". The experiment above establishes the connection between our SAEPO's generated token score and the correctness of a reasoning step. Next, we ran an additional experiment to verify the connection between the correctness of a reasoning step and the correctness of the final answer. Specifically, we cut each rollout right after the edited tokens and let the policy continue reasoning 32 times from the original prefix and 32 times from the edited one. The table below shows the results. (PP here is the fraction of continuations that reach the correct answer.)

Editnumber of samplesPP: original → editedΔP\Delta P [95% CI]Edits with PP lowered / raised
Introduce an error3040.74 → 0.17−0.57 [−0.61, −0.54]95% / 4%
Fix an error1200.15 → 0.32+0.17 [+0.12, +0.22]23% / 62%

We can see that a single broken step lowers the policy's own success rate in 95% of the edits, lowering the accuracy from 0.74 to 0.17 on average. Meanwhile, a single fix raises the policy's own success rate in 62% of the fixes, improving the accuracy from 0.15 to 0.32. These results verify the connection between the correctness of a reasoning step and the correctness of the final answer in our dataset, establishing its high quality.

Combining all additional results above, we can therefore establish the two-hop connection: SAEPO's token-level semantic score →\rightarrow correctness of a local step →\rightarrow prefix-continuation success rates (final accuracy).

We have revised Sec 4.2 (Table 3 and Table 4) in our paper to include the results and discussion above as suggested.

W3: "Additional training overhead is not adequately evaluated. [...] SAE pretraining cost, training time, or GPU memory consumption compared with vanilla GRPO/GSPO."

Q3: "[...] computational costs introduced by SAE pretraining, token feature extraction, and cross-rollout semantic alignment? [...] matched wall-clock time or computational budgets, rather than only matched training steps?"

Thank you for mentioning this. Following your suggestion we ran more experiments to document the time & memory consumption of all computation steps in the SAEPO and Vanilla GRPO/GSPO runs on RL training with Qwen3-1.7B-Base. Results are summarized in the table below.

Per stepVanilla GRPOSAEPO (GRPO)Vanilla GSPOSAEPO (GSPO)
Rollout generation33.1 s37.8 s32.9 s34.5 s
Reward grading3.5 s2.7 s3.8 s2.7 s
Old/reference log-probs12.3 s13.6 s12.3 s13.1 s
Policy update32.4 s36.6 s32.7 s34.7 s
SAE token feature extraction–0.77 s–0.68 s
Cross-rollout semantic alignment–0.34 s–0.29 s
Total81.4 s92.0 s81.8 s86.0 s
Wall-clock, 250 steps5.66 h6.39 h5.68 h5.97 h
Peak GPU memory17.3 GB18.4 GB17.3 GB18.4 GB

We can see that when training, the extra time introduced by the SAE consists of SAE token feature extraction and cross-rollout semantic alignment, which is 1.11 s per step for GRPO, i.e., 1.4% of a vanilla GRPO step (81.4 s), and 0.97 s per step for GSPO, i.e., 1.2% of a vanilla GSPO step (81.8 s). The rest of the difference in the total comes from SAEPO policies generating longer responses, not from the SAE computation. Therefore our SAEPO's computational overhead is minimal.

Note that SAE pre-training is one-off, which is trained once per backbone and reused by every run. Our SAE used by Qwen3-1.7B-Base remains unchanged for both math and code, takes 8.1 GPU-hours for pre-training.

We have included the results and discussion in the revision (Table 5 and Appendix C.5) as suggested.

Next we compare SAEPO with vanilla GRPO/GSPO under matched wall-clock time at the same training time and report the training set accuracies across different methods:

Wall-clockVanilla GRPOSAEPO (GRPO)Vanilla GSPOSAEPO (GSPO)
1 h12.915.811.813.3
2 h16.925.814.621.2
3 h20.028.117.328.0
4 h20.028.820.124.9
5 h20.728.221.627.4
5.66 h (vanilla's 250 steps)22.228.720.725.5

These results show that SAEPO reaches vanilla's final training accuracies after 1.35 h (vanilla GRPO) and 1.75 h (vanilla GSPO), while vanilla GRPO and vanilla GSPO need 5.66 h and 5.68 h, respectively. This highlights our SAEPO's advantage in terms of accelerating convergence in RL training.

We have included the results and discussion in the revision (Table 6 and Appendix C.5) as suggested.

Reviewer_TgQdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This work focuses on constructing finer-grained token-level credit from verifiable outcome rewards. It uses sparse autoencoder representations to compare the semantic features of correct and incorrect rollouts of the same question and adjusts token-level advantages accordingly for GRPO and GSPO training. Experiments are conducted on mathematical reasoning and code generation tasks.

Critical strengths

The method introduces question local contrasts between correct and incorrect rollouts in a sparse representation space, without relying on step by step correctness annotations or additional strong teachers, and adds no SAE computation at inference time. The experiments cover multiple model backbones and two task domains. The results provide evidence of the module’s effectiveness in the evaluated settings.

Critical weaknesses
  1. There is insufficient justification for the link between representation similarity and local contribution. The SAE is trained to reconstruct activations while maintaining sparsity. Cosine similarity between representations from correct and incorrect rollouts does not, by itself, establish token correctness or a token’s contribution to the probability of success; the paper does not establish a theoretical connection between this similarity and true local credit.
  2. The advantage shaping used in this work lacks a corresponding analysis of the optimization objective. Adding action-dependent standardized scores to the advantage changes the policy-update estimate; preserving the form of the optimizer does not, by itself, establish consistency with the original expected-return objective.
  3. The ablation studies are incomplete. The current experiments do not independently examine the choice of cosine similarity versus other metrics or the effects of different alignment rules, nor do they report sensitivity analyses for α\alpha and τ\tau, which are fixed at 22 and 0.050.05, respectively. Moreover, the interpretability demonstrations do not validate credit assignments against ground truth. A topic-relevant word receiving a high semantic score does not establish that the corresponding reasoning operation is beneficial. The semantic scores shown in Figure 5 do not necessarily have the same sign as the final advantages, and there is no comparison with independently estimated prefix-continuation success rates or local step evaluations.
Questions for authors

See weaknesses.

8 comments
AuthorsOpenReview ↗

Author Response (1/7)

Thank you for the careful and constructive review. Below we address your concerns in detail. We have also revised the paper accordingly, with all changes marked in blue.

W1.1: "There is insufficient justification for the link between representation similarity and local contribution. [...] Cosine similarity [...] does not, by itself, establish token correctness or a token's contribution to the probability of success..."

Thank you for mentioning this. Following your suggestion, we ran additional experiments to validate credit assignments against ground truth. Specifically, we will establish the connection: SAEPO's token-level semantic score →\rightarrow correctness of a local step →\rightarrow prefix-continuation success rates.

Constructing a Controlled Dataset. We start by constructing a controlled dataset. We first sampled 976 problems from the training set and selected 1,864 Qwen3-1.7B-Base rollouts on them (904 correct, 960 incorrect). We then removed rollouts where step boundaries are ambiguous or steps are difficult to edit, leading to a final controlled dataset with 403 correct and 507 incorrect rollouts. For each rollout, we used Qwen3-32B to pick one step and propose a minimal edit of it to construct a controlled dataset:

  • (1) Error Edits: In a correct step, we changed a few tokens with LLM so that the step contains a single verifiable error introduced through a minimal edit (e.g., 24→2524 \to 25).
  • (2) Fixing Edits: In a wrong step, we applied a minimal edit that fixes it with LLM.

The LLM judge (Qwen3-32B), checking each version of the step blindly, must confirm that the original step is correct and the edited one is not for Case (1) above (and the reverse for Case (2) above), leaving 304 error edits and 120 fixing edits confirmed and included in the final evaluation. All tokens outside the edited span were kept unchanged.

Results on the Controlled Dataset: Connection of "SAEPO's Token-Level Semantic Score" →\rightarrow "Correctness of a Local Step". We then fed these edited rollouts back into the model, took its hidden states, and recomputed the token-level semantic score SS. ΔS\Delta S is the mean of SS over the edited tokens minus the mean of SS over the original tokens they replace, divided by the standard deviation of SS over all tokens of the original rollout, so that edits from different rollouts are on a common scale. The table below shows the results.

EditStep (judge): original → editedNumber of samplesExampleΔS\Delta S at edited tokens [95% CI]
Introduce an errorcorrect → incorrect3048×3=24  →  8×3=258\times3=24\;\to\;8\times3=25−0.58 [−0.70, −0.47]
Fix an errorincorrect → correct120(Cylinder volume) V=13πr2h  →  V=πr2hV=\frac{1}{3}\pi r^2h\;\to\;V=\pi r^2h+0.16 [+0.06, +0.25]

We can see that SS drops (i.e., negative ΔS\Delta S) when a correct step is broken and rises (i.e., positive ΔS\Delta S) when a wrong step is fixed. This verifies that our token-level score SS could distinguish correct from incorrect reasoning steps. In other words, it establishes the connection between our SAEPO's generated token score and the correctness of a reasoning step.

Effect on Continuation Success: Connection of "Correctness of a Local Step" →\rightarrow "Prefix-Continuation Success Rates". Next, we ran an additional experiment to verify the connection between the correctness of a reasoning step and the correctness of the final answer, i.e., "prefix-continuation success rates" you mentioned. Specifically, we cut each rollout right after the edited tokens and let the policy continue reasoning 32 times from the original prefix and 32 times from the edited one. The table below shows the results. (PP here is the fraction of continuations that reach the correct answer.)

Editnumber of samplesPP: original → editedΔP\Delta P [95% CI]Edits with PP lowered / raised
Introduce an error3040.74 → 0.17−0.57 [−0.61, −0.54]95% / 4%
Fix an error1200.15 → 0.32+0.17 [+0.12, +0.22]23% / 62%

We can see that a single broken step lowers the policy's own success rate in 95% of the edits, lowering the accuracy from 0.74 to 0.17 on average. Meanwhile, a single fix raises the policy's own success rate in 62% of the fixes, improving the accuracy from 0.15 to 0.32. These results verify the connection of "Correctness of a Local Step →\rightarrow Prefix-Continuation Success Rates" mentioned above.

Combining all additional results above, we can therefore establish the two-hop connection: SAEPO's token-level semantic score →\rightarrow correctness of a local step →\rightarrow prefix-continuation success rates (final accuracy).

AuthorsOpenReview ↗

Author Response (2/7)

W1.2: ...establish a theoretical connection between this similarity and true local credit."

Thank you for mentioning this. Below we start with an example showing the advantage of the token-level reward, discuss more technical details on how this is possible, and then provide more theoretical analysis.

Example. Figure 5 (bottom) shows an example where the rollout for a geometry problem ends with a wrong answer, yet the relevant geometry concepts "side length", "area", and "volume" still successfully receive positive credits (rewards). This is desirable because such rewards could extract meaningful training signals even in incorrect rollouts. Similar examples are available in Appendix H's Figure 6 (e.g., the last example).

Technical Detail. This mechanism is made possible via Equations (10–12) in the paper. Specifically, for each question qq and each token tt in a rollout gg, SAEPO computes its semantic similarity to tokens in each of the other rollouts of the same question h≠gh \neq g, using SAE features and soft semantic alignment.

Now, consider computing the score for a token at position tt in an incorrect rollout gg where the token tt happens to represent a "relevant mathematical concept". If a similar concept also appears at position uu in a correct rollout hh, the two tokens will have a high cosine similarity cos⁡(ϕqg,t,ϕqh,u)\cos(\phi_{qg,t}, \phi_{qh,u}) in Equation (10). This high similarity will receive a larger softmax alignment weight ωqg,t,u(h)\omega_{qg,t,u}^{(h)}, thereby contributing a large positive value to the correct-rollout alignment score sqg,t+s^+_{qg,t} in Equation (11).

Consequently, even though the token belongs to an incorrect rollout, it can still receive a positive token-level semantic score when its alignment with correct rollouts exceeds that with incorrect rollouts, as defined in Equation (12).

This is precisely a key motivation of our SAEPO: rewarding useful reasoning components even within incorrect rollouts, rather than uniformly penalizing all tokens simply because the final answer is incorrect.

Below we provide more theoretical analysis on our method SAEPO.

Summary of Theoretical Analysis. Let Pπ(s)P^\pi(s) be the probability that policy π\pi, continuing from prefix ss, reaches a correct answer. A token's local contribution is the change it produces in this probability. We first relate the raw score to similarity-weighted success (Lemma 1), then show how score changes enter the expected token-level update (Proposition 1). The connection to true local contribution depends on the representation, so we also test it through controlled edits and sampled continuations (i.e., results mentioned in our response to W1.1 above).

Notation. For question qq, policy π=πθ\pi=\pi_\theta independently samples GG solutions. Let r(y)∈0,1r(y)\in\\{0,1\\} indicate correctness of the solution yy and pq∈(0,1)p_q\in(0,1) denote the policy's success probability. For scored solution yy, let sts_t contain qq and its first tt tokens, with s0=qs_0=q. The similarity k(s,y′)≥0k(s,y')\ge0 is the softmax-weighted mean (temperature τ\tau) of cosine similarities between the nonnegative SAE code at the last token of ss and those at all positions of another solution y′y' (more details in Appendix B of the revised paper). The raw score mtm_t is the mean similarity to other correct solutions minus that to other incorrect solutions (more details in Appendix B of the revised paper). We exclude yy and leave the token unscored if either reference set is empty. The SAE score is St=ρmtS_t=\rho m_t, where ρ=2min⁡(n+,n−)/G\rho=2\min(n_+,n_-)/G and n+,n−n_+,n_- count correct and incorrect solutions in the group.

Lemma 1 (What the Score Measures). To interpret the raw score mtm_t, i.e., the mean similarity of sts_t to the other correct solutions minus that to the other incorrect ones, define the average similarity kˉ(s)=Ey′[k(s,y′)]>0\bar{k}(s)=\mathbb{E}_{y'}[k(s,y')]>0 and the similarity-weighted success rate Pk(s)=Ey′[k(s,y′)r(y′)]/kˉ(s)P_k(s)=\mathbb{E}_{y'}[k(s,y')r(y')]/\bar{k}(s), with y′∼π(⋅∣q)y'\sim\pi(\cdot\mid q). Then

E[mt∣st=s,token scored]=kˉ(s)pq(1−pq)(Pk(s)−pq).\mathbb{E}[m_t\mid s_t=s,\\ \text{token scored}]=\frac{\bar{k}(s)}{p_q(1-p_q)}\big(P_k(s)-p_q\big).

Proof Sketch. Conditional on the reference labels, the expected difference of sample means is E[k∣r=1]−E[k∣r=0]\mathbb{E}[k\mid r=1]-\mathbb{E}[k\mid r=0]. Substituting the definitions of kˉ\bar{k} and PkP_k gives the identity.

Intuition and Remark on Lemma 1. Lemma 1 shows that the expected raw score is positive when similarity-weighted success exceeds the question's average. However, PkP_k weights other solutions by similarity, whereas PπP^\pi conditions on the actual prefix. Using the former as a proxy requires the representation assumption Pk≈PπP_k\approx P^\pi; reconstruction and sparsity alone do not establish it.

(To be continued.)

AuthorsOpenReview ↗

Author Response (3/7)

(Continued from the previous post.)

This explains the prefix score, but not yet how it credits the token just added. We therefore examine its role in the update using Proposition 1 below.

Proposition 1 (Credit in the Expected Update). In expectation, SAEPO credits each token by the change that the token causes in the expected score. SAEPO adds αS^t\alpha\hat S_t (α>0\alpha>0) to GRPO's outcome advantage of token yty_t, where S^t=(St−μB)/(σB+ϵscore)\hat S_t=(S_t-\mu_B)/(\sigma_B+\epsilon_{\mathrm{score}}); μB\mu_B and σB\sigma_B are the mean and standard deviation of StS_t over all scored tokens of the current training batch, ϵscore\epsilon_{\mathrm{score}} is a small constant, and S^t=0\hat S_t=0 for unscored tokens. Let f(s)=E[S^t∣st=s]f(s)=\mathbb{E}[\hat S_t\mid s_t=s] be the expected score of prefix ss (any constant for s0s_0), and let gt=∇θlog⁡πθ(yt∣st−1)g_t=\nabla_\theta\log\pi_\theta(y_t\mid s_{t-1}), the direction that increases the probability of token yty_t. Then

E[∑tS^tgt]=E[∑t(f(st)−f(st−1))gt],\mathbb{E}\left[\sum_t\hat{S}_t g_t\right]=\mathbb{E}\left[\sum_t\big(f(s_t)-f(s_{t-1})\big)g_t\right],

with any fixed baseline value at s0s_0.

Intuition and Remark on Proposition 1. In other words, in expectation SAEPO credits each token by how much it raises or lowers the expected score, i.e., f(st)−f(st−1)f(s_t)-f(s_{t-1}), just as the exact policy gradient of the success rate credits each token by how much it raises or lowers the success probability PπP^\pi. Lemma 1 links this credit to PkP_k: with ρ>0\rho>0, μB\mu_B and σB\sigma_B held fixed, a scored token has S^t=(ρmt−μB)/(σB+ϵscore)\hat{S}_t=(\rho m_t-\mu_B)/(\sigma_B+\epsilon_{\mathrm{score}}) and whether it is scored depends only on the other solutions, so f(s)f(s) is a positive multiple of the expected raw score E[mt∣st=s,token scored]\mathbb{E}[m_t\mid s_t=s,\\ \text{token scored}] in Lemma 1 plus a constant. Hence, for t≥2t\ge2 with kˉ(st)=kˉ(st−1)\bar{k}(s_t)=\bar{k}(s_{t-1}), these score differences f(st)−f(st−1)f(s_t)-f(s_{t-1}) are a positive multiple of Pk(st)−Pk(st−1)P_k(s_t)-P_k(s_{t-1}) (Corollary 1 in Appendix B).

In our response to W1.1 above, we also provide empirical results from our additional experiments (inspired by your comments) to support the theoretical analysis.

These theoretical and empirical results establish the connection between (1) the SAEPO's token level scores and (2) local reasoning quality as well as continuation success (i.e., "true local credit" as you mentioned).

More empirical results are included in our response to W1.1 above and our response to W3.3 below.

We have included all theoretical analysis with complete proofs in Appendix B, and added empirical results and discussions above in Sec 4.2 of our revision.

Reviewer_FvjiOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors utilize a sparse autoencoder to represent the hidden states of tokens and compare the differences between correct and incorrect answers for the same question, incorporating the resulting token-level semantic scores into the advantage values of GRPO or GSPO. A major contribution is that token-level fine-grained training signals can be obtained without process annotations.

Critical strengths
  1. SAEPO converts a frozen SAE features into token-level training signals by contrasting correct and incorrect solutions to the same question. This directly addresses the coarse supervision in outcome-based policy optimization without requiring process labels or a separate reward model.
  2. The method can be incorporated into GRPO and GSPO without adding inference-time SAE computation. It improves acc over the corresponding base methods across three model backbones and both math and code tasks.
Critical weaknesses

The paper does not demonstrate that its token-level scores actually identify the correctness or contribution of individual reasoning steps.

The scores measure how similar a token is to correct vs incorrect solutions to the same question, but an incorrect derivation can also use entirely relevant mathematical concepts.

Questions for authors

can the authors use reasoning trace with step-level annotations, or pairs of solutions that differ only by a single verifiable error introduced through minimal edits, to test whether SAEPO’s token scores can distinguish correct from incorrect reasoning steps?

4 comments
AuthorsOpenReview ↗

Author Response (1/2)

Thank you for your constructive and encouraging comments. We are glad that you found that SAEPO "directly addresses the coarse supervision in outcome-based policy optimization." and "improves acc over the corresponding base methods". Below, we address your questions in detail. We have also revised the paper accordingly, with all changes marked in blue.

W1: "The paper does not demonstrate that its token-level scores actually identify the correctness or contribution of individual reasoning steps."

Q1: "Can the authors use reasoning trace with step-level annotations, or pairs of solutions that differ only by a single verifiable error introduced through minimal edits, to test whether SAEPO’s token scores can distinguish correct from incorrect reasoning steps?"

This is a good suggestion. Inspired by your comments, to test whether SAEPO’s token scores can distinguish correct from incorrect reasoning steps, we ran an additional paired minimal-edit experiment. Specifically, we will establish the connection: SAEPO's token-level semantic score →\rightarrow correctness of a local step →\rightarrow prefix-continuation success rates (final accuracy).

Constructing a Controlled Dataset. We start by constructing a controlled dataset. We first sampled 976 problems from the training set and selected 1,864 Qwen3-1.7B-Base rollouts on them (904 correct, 960 incorrect). We removed rollouts where step boundaries are ambiguous or steps are difficult to edit, leading to a final controlled dataset with 403 correct and 507 incorrect rollouts. For each rollout, we used Qwen3-32B to pick one step and propose a minimal edit of it to construct a controlled dataset:

  • (1) Error Edits: In a correct step, we changed a few tokens with LLM so that the step contains a single verifiable error introduced through a minimal edit (e.g., 24→2524 \to 25).
  • (2) Fixing Edits: In a wrong step, we applied a minimal edit that fixes it with LLM.

The LLM judge (Qwen3-32B), checking each version of the step blindly, must confirm that the original step is correct and the edited one is not for Case (1) above (and the reverse for Case (2) above), leaving 304 error edits and 120 fixing edits confirmed and included in the final evaluation. All tokens outside the edited span were kept unchanged.

Results on the Controlled Dataset: Connection of "SAEPO's Token-Level Semantic Score" →\rightarrow "Correctness of a Local Step". We then fed these edited rollouts back into the model, took its hidden states, and recomputed SS (i.e., the token-level semantic score). ΔS\Delta S is the mean of SS over the edited tokens minus the mean of SS over the original tokens they replace, divided by the standard deviation of SS over all tokens of the original rollout, so that edits from different rollouts are on a common scale. The table below shows the results.

EditStep (judge): original → editedNumber of samplesExampleΔS\Delta S at edited tokens [95% CI]
Introduce an errorcorrect → incorrect3048×3=24  →  8×3=258\times3=24\;\to\;8\times3=25−0.58 [−0.70, −0.47]
Fix an errorincorrect → correct120(Cylinder volume) V=13πr2h  →  V=πr2hV=\frac{1}{3}\pi r^2h\;\to\;V=\pi r^2h+0.16 [+0.06, +0.25]

We can see that SS drops (i.e., negative ΔS\Delta S) when a correct step is broken and rises (i.e., positive ΔS\Delta S) when a wrong step is fixed. This verifies that our token-level score SS could distinguish correct from incorrect reasoning steps.

(To be continued.)

AuthorsOpenReview ↗

Author Response (2/2)

(Continued from the previous post.)

Effect on Continuation Success: Connection of "Correctness of a Local Step" →\rightarrow "Prefix-Continuation Success Rates". The experiment above establishes the connection between our SAEPO's generated token score and the correctness of a reasoning step. Next, we ran an additional experiment to verify the connection between the correctness of a reasoning step and the correctness of the final answer. Specifically, we cut each rollout right after the edited tokens and let the policy continue reasoning 32 times from the original prefix and 32 times from the edited one. The table below shows the results. (PP here is the fraction of continuations that reach the correct answer.)

Editnumber of samplesPP: original → editedΔP\Delta P [95% CI]Edits with PP lowered / raised
Introduce an error3040.74 → 0.17−0.57 [−0.61, −0.54]95% / 4%
Fix an error1200.15 → 0.32+0.17 [+0.12, +0.22]23% / 62%

We can see that a single broken step lowers the policy's own success rate in 95% of the edits, lowering the accuracy from 0.74 to 0.17 on average. Meanwhile, a single fix raises the policy's own success rate in 62% of the fixes, improving the accuracy from 0.15 to 0.32. These results verify the connection between the correctness of a reasoning step and the correctness of the final answer in our dataset, establishing its high quality.

Combining all additional results above, we can therefore establish the two-hop connection: SAEPO's token-level semantic score →\rightarrow correctness of a local step →\rightarrow prefix-continuation success rates (final accuracy).

We have revised Sec 4.2 (Table 3 and Table 4) in our paper to include the results and discussion above as suggested.

W2: "The scores measure how similar a token is to correct vs incorrect solutions to the same question, but an incorrect derivation can also use entirely relevant mathematical concepts."

This is an insightful point. We agree that our scores measure how similar a token is to correct vs incorrect solutions to the same question and that an incorrect derivation can also use entirely relevant mathematical concepts.

In fact, our SAEPO does produce positive (high) score for "an incorrect derivation" that "use entirely relevant mathematical concepts". The intuition is that an "incorrect derivation" in an incorrect rollout may actually have a high token-level score because it is similar to certain "derivation" (i.e., reasoning steps) in a certain correct rollout. (This intuition is actually a key motivation of our token-level method.)

For example, in Figure 5 (bottom), the rollout for a geometry problem ends with a wrong answer, yet the geometry concepts "side length", "area", "volume", "cube" and "pyramid" receive positive credit, because they are "relevant mathematical concepts" to the problem. Similar examples are available in Appendix H's Figure 6 (e.g., the last example).

Technical Detail. This mechanism is made possible via Equations (10–12) in the paper. Specifically, for each question qq and each token tt in a rollout gg, SAEPO computes its semantic similarity to tokens in each of the other rollouts of the same question h≠gh \neq g, using SAE features and soft semantic alignment.

Now, consider computing the score for a token at position tt in an incorrect rollout gg where the token tt happens to represent a "relevant mathematical concept". If a similar concept also appears at position uu in a correct rollout hh, the two tokens will have a high cosine similarity cos⁡(ϕqg,t,ϕqh,u)\cos(\phi_{qg,t}, \phi_{qh,u}) in Equation (10). This high similarity will receive a larger softmax alignment weight ωqg,t,u(h)\omega_{qg,t,u}^{(h)}, thereby contributing a large positive value to the correct-rollout alignment score sqg,t+s^+_{qg,t} in Equation (11).

Consequently, even though the token belongs to an incorrect rollout, it can still receive a positive token-level semantic score when its alignment with correct rollouts exceeds that with incorrect rollouts, as defined in Equation (12).

This is precisely a key motivation of our SAEPO: rewarding useful reasoning components even within incorrect rollouts, rather than uniformly penalizing all tokens simply because the final answer is incorrect.

AuthorsOpenReview ↗

Follow-Up on the Author Response (Discussion Deadline (Nov. 18) Approaching)

Dear Reviewer Fvji,

With the author-reviewer discussion deadline approaching (Nov. 18, AoE), we would like to follow up on our responses to your questions.

Your comments on whether token-level semantic scores reflect the correctness and contribution of individual reasoning steps have been particularly helpful in improving the completeness of our paper. We would be grateful if you could consider our rebuttal and the added controlled step-edit experiments and prefix-continuation evaluations when finalizing your assessment, and reconsider your evaluation where appropriate.

Thank you again for your time and constructive feedback.

Best regards,

SAEPO Authors

AuthorsOpenReview ↗

Follow-Up Before the Discussion Deadline (Nov. 18, AoE)

Dear Reviewer Fvji,

With less than 24 hours remaining before the discussion deadline (Nov. 18, AoE), we would like to follow up on our responses to your questions.

Your comments on the correctness of token-level semantic scores and contribution of individual reasoning steps are particularly helpful for improving our paper. We therefore would greatly appreciate your further feedback on our added experiments and clarifications.

Thank you again for your careful review.

Best regards,

SAEPO Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Remarks by Authors

We sincerely thank all reviewers for their time and constructive feedback. We are glad that the reviewers highlighted:

  • The relevance of the problem, describing it as "meaningful and timely" [9YKK] and noting that SAEPO "directly addresses the coarse supervision in outcome-based policy optimization" [Fvji];
  • The design and practicality of our approach, recognizing its "question-local semantic contrast mechanism" [9YKK], its ability to derive training signals "without requiring process labels or a separate reward model" [Fvji], and that it "adds no SAE computation at inference time" [TgQd];
  • The experimental evaluation, describing it as "Broad experimental evaluation with encouraging results" [9YKK] and finding "evidence of the module’s effectiveness in the evaluated settings" [TgQd];
  • The presentation, rated "Clear and easy to follow" [9YKK].

Following the reviewers’ suggestions, we have strengthened the paper with new analyses and evidence:

  • Reviewer Fvji: Added additional controlled experiments to evaluate token-level credit.

    • Our results provide quantitative evidence linking semantic scores, local-step correctness, and continuation success (final accuracy).
    • We also clarified how SAEPO can assign positive credit to useful reasoning components within an incorrect rollout.
  • Reviewer TgQd: Added ablation experiments and theoretical analysis of token-level credit and the optimization objective.

    • Our analysis connects raw semantic scores to similarity-weighted success (final accuracy) and shows how score changes enter the expected token-level update.
    • We added experiments comparing similarity metrics and alignment rules, sensitivity analyses for the semantic-score weight and alignment temperature, and clarified that Figure 5 displays the final shaped advantages used in policy optimization.
  • Reviewer 9YKK: Added experiments on representation comparisons, efficiency measurements, SAE stability analyses, and robustness evaluations.

    • Under matched training data and latent dimensions, SAE representations achieve 40.12% average accuracy, compared with 38.43% for a standard autoencoder and 35.40% for an MLP encoder.
    • We quantified one-time SAE pretraining cost, RL runtime, and peak GPU memory, and demonstrated improved training accuracy under matched RL wall-clock budgets.
    • We also evaluated cross-task transfer and feature stability during RL, added correctness-balance and normalization ablations, and reported three-seed results, with SAEPO outperforming baselines at every seed. The controlled-edit experiments described above also address this reviewer’s concerns about the faithfulness of the credit signal.

We have incorporated these analyses, experiments, and clarifications into the revised manuscript, with changes marked in blue. We also thank the AC and SAC for their guidance and oversight throughout the review process.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.