acceptodds
Under review as a conference paper at ICLR 2027

Guided Sparse Autoencoders for Interpreting Reasoning Features

Abstract

Large Language Models (LLMs) increasingly solve complex tasks through multi-step Chain-of-Thought (CoT) reasoning, yet the internal representations supporting these trajectories remain difficult to interpret. Sparse Autoencoders (SAEs) decompose dense activations into sparse latent features, but standard reconstruction-sparsity only training often misses subtle reasoning-related features because they contribute negligibly to overall activation reconstruction variance. Consequently, important reasoning primitives do not naturally occupy identifiable latent directions. We propose the Guided Sparse Autoencoder (G-SAE), a partitioned latent-space architecture that injects supervision directly into sparse representation learning. A subset of latent dimensions is jointly supervised for step reasoning correctness, while the remaining dimensions retain an unsupervised role in reconstructing broader activation information. By combining labeled ProcessBench activations with unlabeled activations, G-SAE merges targeted correctness pressure with broad reconstruction coverage. Evaluations across different model scales demonstrate that supervised latent partitioning improves correctness prediction while maintaining reconstruction fidelity.

Primary area:
interpretability and explainable AI
Keywords:
Mechanistic Interpretability, Sparse Autoencoders, Large Language Models, Reasoning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_4iVYOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

G-SAE is an SAE with a split latent space. It consists of (a) a small supervised partition (0.4–2.5% of latents, sigmoid-weighted ReLU, no Top-K) with a BCE-trained linear head on ProcessBench step-correctness labels, along with a diversity penalty, and (b) a Top-K unsupervised partition trained only for reconstruction. Each reasoning step is represented by the last token activation of its prefix at the middle layer. Training combines 1138 labeled steps with roughly 263k unlabeled ones. Across six Qwen2.5-based models, a post-hoc linear probe on G-SAE latents improves OOD AUC over both a reconstruction-only SAE (+0.035–0.119) as well as raw activations with a small reconstruction cost. In single-feature steering on Qwen2.5-7B scored by a PRM, only the 3 tested supervised features show significant effects.

Critical strengths
  • The full-supervision ablation (A1): similar OOD AUC (within 0.030) but NMSE worsens by 0.142–0.756 vs. at most 0.037 for G-SAE. This is strong evidence for the partitioned trade-off. The method is also lightweight.

  • Careful data protocol (problem-level splits, position/difficulty balancing, held-out harder tiers) and acknowledges PRM failure cases and post-hoc example selection.

Critical weaknesses

1. The method overlaps with uncited prior work, and the improvements are not shown to be robust or specific to the SAE.

My biggest concern is that Härle et al. [1] already published a method (with the same name), G-SAE, using normalized-MSE reconstruction plus a BCE loss that ties designated latents to concept labels, with steering through the conditioned decoder directions. An earlier version of that work (arXiv 2411.07122) does the same thing with a single latent. The differences from this work, such as the multi-latent block with a linear head, dropping Top-K on the supervised units, the step-level setting, the unlabeled stream, and the diversity penalty, are incremental. Moreover, the diversity penalty doesn’t consistently help (in Fig. 4, λdiv=0.5λ_{div} = 0.5 performs worse than 0 for 2 of 6 models). At minimum, the paper needs to cite this work and explain what is genuinely new.

I also don’t think the main result is established yet. Table 1 reports single runs with no error bars, while the paper’s own sweeps move by up to approx. 0.09 AUC between neighboring settings of ksupk_{sup} and λdivλ_{div}, which is as large as the claimed improvement. The defaults are also odd, since 5 models use a 0.4% supervised ratio but only one uses 2.5%, which happens to be the best OOD value. No selection criterion or split is ever stated in the paper. On 2 of the 3 7B models, the gain over raw activations is only +0.016/0.026. The supervised encoder + head is itself a small classifier trained on the same 1138 labels, so beating a linear probe on raw activations might simply be from supervised capacity. Baseline results on raw activations with a tuned logistic probe and a capacity-matched MLP are missing.

2. Steering results don’t support the causal claim.

In Table 3, significance is tested against the unsteered baseline rather than random direction control, which is about +0.008 in every row. The effect of feature 6 (+0.0100) is just +0.0018 above the random one, and that contrast is never tested, so superiority over random directions has never been shown in the paper. For feature 13 with the highest AUC (0.809, which is above the OOD AUC of the full probe at 0.722), steers in the wrong direction. Moreover, the opposite steer/negate signs also show up for 5 of 6 unsupervised features. The effects are at most 0.0284, measured on one model with 3 features per partition and judged by a single PRM the authors themselves show can be gamed (A8.1). There is no final-answer accuracy and no difference-of-means or probe-direction baseline, so it’s unclear what the SAE improves over a standard steering vector.

Minor points. The FSup range in the intro/A.1 doesn’t match Table 4. Inconsistent decimals in tables.

[1] Härle, Ruben, et al. "Measuring and guiding monosemanticity." NeurIPS, 2025.

Questions for authors
  • Q1: How does your method differ from Härle et al.’s G-SAE, and do those differences matter empirically? Would their single-latent Sigmoid(TopK) reach similar OOD AUC on your data?

  • Q2: Could you document the probe and report which latents it uses, what it is trained on, and how it is regularized? What AUC does your trained head reach and a probe on the unsupervised partition alone?

  • Q3: How many seeds are behind Table 1? Please report mean ± std. On which split and by what criterion were the variables chosen? And why does KsK_s differ between the two 7B models (215 for Deepseek vs 430 for Qwen)?

  • Q4: Did the authors consider a tuned logistic probe and a capacity-matched MLP on raw activations, using the same labels and splits? Similarly, why were steering results not compared against matched random directions, difference-of-means, and probe-direction baselines? Could the authors provide these comparisons with paired confidence intervals and report final answer accuracy?

  • Q5: How dense is the supervised partition in practice (mean L0, per-unit activation frequency), given that the baseline is matched only on the maximum count? Please also report dead latents, the EV definition, and spliced-in LM loss if feasible.

5 comments
AuthorsOpenReview ↗

Response to Reviewer 4iVY

We thank the reviewer; see also the general response.

1. Härle et al. (W1, Q1). We now cite it.

Difference from Härle et al. (NeurIPS 2025). Although both methods introduce supervision into a partitioned SAE latent space, they differ in the supervision target, architectural mechanism, and intended feature organization.

Härle et al. associate designated latent dimensions with predefined semantic concepts (e.g., toxicity, style, and privacy attributes), using concept-specific objectives on designated latents. Their objective is to localize known concepts into designated features and evaluate their monosemanticity. In contrast, our method does not prescribe the semantic identity of individual latents. Instead, it uses a shared behavioral signal, step-level correctness, to guide a multi-dimensional supervised subspace through a linear prediction head. The internal features are learned from the activation data rather than assigned to predefined concepts. Thus, supervision acts as an inductive bias on a learned representation rather than as a direct concept-to-neuron assignment.

This distinction is particularly relevant to reasoning evaluation. Step correctness is a single binary outcome, but incorrect reasoning can arise from heterogeneous failure modes, including algebraic errors, invalid deductions, and incomplete reasoning. Our approach does not assume a one-to-one correspondence between the correctness label and a latent dimension. Instead, it allows multiple learned features to contribute to a shared behavioral prediction objective.

Architectural distinction. Härle et al. incorporate supervised latents into a Top-K-constrained representation, whereas our method decouples the supervised partition from Top-K competition. The supervised partition uses ReLU-gated activations and is trained through a linear prediction head, while the reconstruction-oriented partition retains Top-K sparsity. This separation allows behavioral supervision to guide the supervised subspace without requiring its features to compete directly with reconstruction-oriented features for a global Top-K activation budget.

Therefore, our contribution is not simply the introduction of supervision into a partitioned SAE, but a different formulation of supervised feature learning: using a shared behavioral objective to guide a learned latent subspace, and it targets step-level correctness in chain-of-thought reasoning specifically.

2. Seeds and selection (W2, Q3). Table 1 now reports mean ± std over 5 SAE seeds, with per-seed results in App. A.10 (Table 9). G-SAE has the higher mean AUC on 5 of 6 models and wins on 21 of 30 model–seed pairs; its AUC also varies less across seeds than the Top-KK SAE's on 5 of 6 models. The probe is fixed and identical for all SAEs (Sec. 4.2), trained on GSM8K+MATH and tested on held-out OOD steps; the advantage holds on the same five models for every CC from 10−310^{-3} to 1010 (App. A.10, Table 10).

3. Tuned probe and MLP (W3, Q4, Q2). Both are added (Sec. 4.3; App. A.10, Table 11): a logistic probe on standardized raw activations with CC selected on validation, and a capacity-matched MLP (hidden width ksupk_{\text{sup}}, 5 seeds), on the same labels and splits. They reach comparable or higher AUC than G-SAE (e.g. Qwen2.5-7B: probe 0.740, MLP 0.735, G-SAE 0.723; Qwen2.5-Math-7B: probe 0.741, MLP 0.725, G-SAE 0.735). Unlike these classifiers, G-SAE keeps the correctness information inside a dictionary that reconstructs the activations, and supervision makes its supervised units up to 2–3× more selective than in the label-free Recon-G-SAE (on the 7B models) at no AUC cost (App. A.9). The probe is specified in Sec. 4.2.

4. Density, dead latents, EV (Q5). These are reported in App. A.9 (Table 8): supervised units are active on 28–49% of steps vs. 52–82% without labels. EV is defined in Sec. 4.2; spliced-in LM loss is left to future work.

5. Steering (W4, Q4). Redone with random-direction, DoM and probe controls, a problem-level bootstrap, Holm correction and pass@1 (Sec. 4.6, Table 3; App. A.12; details in our reply to D9Qn). Supervised units shift the PRM beyond random directions with larger effects than the baselines, two with both signs, and one unit changes final-answer accuracy in both directions; neither baseline does either.

6. Minor (W5). We corrected the FSup range and unified decimals.

Thank you for the detailed response and experiments. However, I still have these concerns:

  1. I could not locate the citation or discussion of Härle et al. in the revised manuscript. While the label differs, the method is nearly identical, with NMSE + BCE on a designated latent block and decoder-column steering. The modifications are ksup>1k_{sup} > 1, a linear head, and no Top-K on the supervised block. A ksup=1k_{sup} = 1 run with your protocol would show whether these matter. If the gap is small, the paper is an application of guided SAEs to step correctness, which should be stated explicitly.
  2. I am still missing an explanation for why KsK_s differs between the two 7B models.
  3. Could you report paired CIs comparing steer/negate effects directly against DoM and probe-direction controls? A single latent outperforming a DoM direction would be an important result, but this comparison is not yet statistically established. Also, Unit 63 still steers in the opposite direction from its predicted sign.

I appreciate the additional work, but the methodological novelty and the interpretation of the steering advantage remain my main concerns.

Thank you for the follow-up. We address the three points in turn.

1. Relation to Härle et al. (2025)

You are right that the citation was missing; we apologize. It is now discussed in Related Work, where we state explicitly that the core idea (NMSE + BCE on a designated latent block, steering along decoder columns) comes from Härle et al.'s Guided SAE, and that we extend this paradigm to step-level reasoning correctness. Our design differs in using several gated units (sigmoid · ReLU, outside Top-K) read by a linear head, a diversity loss, and step-level activations with an unlabelled reconstruction stream.

We ran the closest variant: G-SAE with a single supervised unit (k_sup = 1). With one unit, the head only rescales that unit, so, as in Härle et al., the correctness loss acts on one latent. Unsup and G-SAE: mean over 5 seeds (Table 1); k_sup = 1: single seed. Δ = G-SAE − (k_sup = 1).

LLMk_supAUC UnsupAUC k_sup = 1AUC G-SAEΔAUC
Qwen2.5-0.5B220.6870.7210.713−0.008
Qwen2.5-Math-1.5B370.6690.6670.677+0.010
DS-R1-Qwen-1.5B2300.6840.7010.704+0.003
DS-R1-Qwen-7B860.6890.6630.667+0.004
Qwen2.5-7B860.7070.6860.723+0.037
Qwen2.5-Math-7B860.7050.7210.735+0.014

With one unit, AUC is mixed relative to Unsup (higher on 3 models, lower on 3). The full block is higher on 5 of 6 models at the same reconstruction, but clearly beyond seed variability (std 0.008–0.038) only on Qwen2.5-7B. Together with Recon-G-SAE (no labels, similar AUC to G-SAE), this suggests that the small gain comes from having several gated units rather than from the labels. The remaining difference (a sigmoid latent inside Top-K) is untested. The revised Related Work now positions G-SAE as a step-level extension of the guided SAE approach.

2. Why K_s differs between the 7B models

K_s was selected per model on the GSM8K+MATH validation split, from the same grid for all models. For DeepSeek-R1-Distill-Qwen-7B the validation AUC favoured K_s = 215 (1.0% of the Top-K latents active), while the two Qwen2.5-7B models favoured K_s = 430 (2.0%).

3. Steering against DoM and probe directions, and unit 63

We ran the comparison you asked for on the existing generations. Every unit, DoM and probe condition uses the same problems and sampling seeds, so we report Δ_unit − Δ_control with a paired problem bootstrap (10,000 draws) and Holm correction over the conditions, per metric and control (two new tables in Appendix A.12).

On Qwen2.5-7B, against the probe, 4 of the 15 units are stronger, 3 weaker and 8 not different. Against DoM, 6 are stronger and 1 weaker, but this mostly reflects DoM being a weak direction: against random, neither control changes pass@1 significantly. Unit 67 exceeds both on pass@1 (+4.8 pp vs. the probe; +6.2 / −6.5 pp vs. DoM). Single units therefore steer about as well as a supervised probe direction, and the paper now states exactly this, with no general superiority claim.

Unit 63 is reported as steering opposite to its head-weight sign, and it is also weaker than both controls on the PRM. Across the five G-SAEs on Qwen2.5-7B, 2 of the 10 significant unit conditions go against the sign. This is why the revision describes the sign as predictive only: "+" is the direction the correctness head associates with correct steps, and whether steering along it helps is tested, not assumed.

We thank you again for the careful reading, which has made the paper's scope and claims more precise.

Reviewer_B9XnOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Guided SAE (G-SAE), which partitions an SAE's latent space into a small supervised subset trained with step-level correctness labels and a larger unsupervised subset for reconstruction. Evaluated on six models (0.5B–7B), G-SAE improves OOD correctness probing by 0.035–0.119 AUC over standard SAEs at minimal reconstruction cost. Causal steering shows only supervised features produce significant PRM score changes, while unsupervised features with comparable AUC do not — demonstrating that predictive association and causal relevance can dissociate.

Critical strengths
  • The motivation is concrete: the paper shows standard SAEs lose correctness signal (Table 1), not just asserts it. The partitioned design is a simple, sensible compromise that avoids tanking reconstruction.

  • The causal validation is interesting. Showing predictive-but-not-causal unsupervised features vs. predictive-and-causal supervised features is a meaningful finding for the interpretability community.

  • Thorough evaluation across six models at three scales with systematic ablations on partition ratio.

Critical weaknesses
  • The scope is math-only. I do not see a single non-math domain attempted. Whether "step-level correctness" transfers to outher tasks is unknown.

  • The diversity regularizer doesn't consistently help (Appendix A.5) and is sometimes harmful. This weakens the case that the full recipe is well-motivated.

  • I think the method depends too much on ProcessBench label quality. No analysis of robustness to noisy or biased labels.

  • No comparison with other supervised or semi-supervised representation learning methods that could achieve similar tradeoffs.

Questions for authors
  • How sensitive is the supervised partition to label noise? Have you tried training with partially incorrect step labels?

  • What happens if you increase the supervised partition beyond 2.5%? The non-monotonic ablation in Appendix A.4 needs explanation.

  • Would you consider dropping the diversity regularizer from the core method given its inconsistent effect?

2 comments
AuthorsOpenReview ↗

Response to Reviewer B9Xn

We thank the reviewer; see also the general response.

1. Label noise and scarcity (W3, Q1). G-SAE was retrained (6 LLMs × 3 seeds) with 2–30% flipped labels or with 90% down to 10% of the labels, and compared with a capacity-matched MLP trained on the same labels (App. A.11, Table 12). Under noise, it never degrades significantly faster than the MLP, and at 30% flipped labels it degrades significantly less (−0.048 vs. −0.069 AUC; interaction +0.021, 95% CI [0.007, 0.037]). Under scarcity, it never degrades significantly more from 90% down to 20% of the labels, and it degrades significantly less with 40% and 30% (interaction +0.012 and +0.015). Only with 10% of the labels (114 steps) does it degrade more (−0.019, CI [−0.040, 0.001]), mainly on Qwen2.5-7B.

2. Diversity regularizer (W2, Q3). We agree that its effect on AUC is inconsistent (App. A.5). We added it because several supervised units can otherwise learn redundant patterns: it penalizes units that are active on the same steps, encouraging them to capture complementary aspects of correctness. Its purpose is therefore to diversify the supervised units, not to raise AUC. Its weight λdiv\lambda_{\text{div}} is a hyperparameter: all reported models use λdiv=0.5\lambda_{\text{div}}=0.5, and for a new model it can be tuned on the validation split or set to 0, which App. A.5 also evaluates.

3. Other approaches (W4). We now compare G-SAE with a capacity-matched MLP and a tuned probe on raw activations, a fully supervised SAE (FSup, App. A.1), a standard Top-KK SAE (Table 1), and the label-free gate-matched Recon-G-SAE (App. A.10, Table 11). The raw-activation classifiers reach comparable or higher AUC but do not reconstruct the activations; G-SAE outperforms the Top-KK SAE on 5 of 6 models and matches Recon-G-SAE in AUC while making the supervised units more selective (App. A.9).

4. Supervised partition beyond 2.5% (Q2). In App. A.4 (Fig. 3), ratios of 1.2% and above generally outperform 0.2%, the worst or near-worst setting on five of six models. Beyond this, differences between neighbouring ratios are small relative to the fluctuations within each curve, so we read the non-monotonicity as variability rather than a systematic effect; the best ratio is model-dependent (1.2–3.0%).

5. Math-only (W1). We agree; the OOD evaluation only tests harder math. Non-math step-labelled data is future work, stated in the limitations (Sec. 4.8).

Thank you for your review. As a brief update, we have also revised Contribution 2 so that the diversity regularizer is described as an optional component with model-dependent effects (App. A.5), following your question.

Reviewer_D9QnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work introduces the Guided Sparse Autoencoder (G-SAE), designed to extract step-level reasoning-correctness features from LLMs while preserving activation reconstruction quality. Standard unsupervised SAEs lag behind raw-activation probes, and fully supervised SAEs badly hurt reconstruction. G-SAE splits its encoder into a small supervised branch (Swish/SiLU) and a larger unsupervised Top-K ReLU branch, sharing one linear decoder. It is trained on a mix of unlabeled Math-Shepherd and labeled ProcessBench data, with binary classification and cosine diversity losses for the supervised subset. Evaluated on six LLMs across out-of-distribution reasoning benchmarks, G-SAE achieves 0.035–0.119 higher OOD ROC-AUC for step correctness than unsupervised SAEs and outperforms raw-activation probes, with only minor reconstruction degradation. Further analyses show the supervised subspace encodes reasoning features, and single-feature steering can alter PRM scores.

Critical strengths
  1. The paper identifies a key limitation: unsupervised SAE probes lag behind raw activation probes, while full supervision severely damages reconstruction. By allocating only 0.4%–2.5% of the dictionary to the supervised branch with a shared decoder, G-SAE retains over 97% explained variance and brings OOD AUC gains of 0.035–0.119 with minor reconstruction overhead. This controlled tradeoff is the core contribution.
  2. Labels are balanced across step positions and difficulty tiers to avoid positional shortcuts. Harder tiers are excluded from supervision. Steering tests use random control directions, hyperparameters are tuned on a separate pilot set, and bootstrap uncertainty intervals are provided.
Critical weaknesses
  1. The supervised partition is dense, without Top‑K or L1 constraints, which makes it essentially a nonlinear MLP bottleneck.
  2. The baseline is only a linear raw probe; the performance gains may come from nonlinearity rather than guided dictionary learning.
  3. L0 statistics for the supervised latent zsupz^\text{sup} are missing, so calling these “sparse monosemantic features” is questionable.
  4. Causal steering experiments have limited scope: only one model, one PRM, three features and 100 prompts.
  5. Steering is only applied at the prompt and first 1–2 tokens, mismatching the step‑final activation training setup.
  6. PRM score changes from steering are small and close to the effect of random control directions.
  7. The highest‑AUC feature shows opposite steering behavior, which may detect errors instead of correctness.
  8. One steering case achieves high PRM reward via reward hacking (plan without answer), with no end‑to‑end Pass@1 reported.
  9. Multiple comparison correction is absent; the tests may produce false positives.
  10. The reconstruction-only baseline does not match the gating and partition structure of G‑SAE, confounding attribution.
  11. Details of post‑hoc probe fitting and feature selection are underspecified.
  12. All labels are math-only; the causal steering results cannot be generalized to other reasoning tasks.
  13. The evaluation set is naturally imbalanced, while training is rebalanced; calibration effects are not reported.
  14. The diversity penalty LdivL_\text{div} is unstable and hurts OOD AUC on some 7B models.
  15. There is no comparison against the closely related Step‑Level SAE (SSAE).
Questions for authors
  1. Can blinded human evaluation or an independent step evaluator verify the steering outputs instead of only one PRM? How often do high PRM scores reflect valid reasoning instead of generic plans?
  2. Compare G-SAE against (a) a standalone nonlinear MLP probe with width ksupk_\text{sup} on raw activations, and (b) a partition- and gate-matched reconstruction-only ablation without correctness labels. Report L0L0 statistics and seed-level uncertainty for AUC and reconstruction metrics.
  3. Provide confidence intervals for the effect differences between supervised features and random/unsupervised directions, set the bootstrap unit per problem, and apply multiple-comparison correction (Holm/Bonferroni) for the 27 tests. Are findings consistent with another PRM and at larger model scales?
  4. Report final answer Pass@1 on the 100 steering prompts under unsteered, positive, negated and random steering settings. How frequently do reward-hacking trajectories (e.g., Feature 2) appear?
  5. Is LdivL_\text{div} beneficial, comparing λdiv=0\lambda_\text{div}=0 and the default value? How is ksupk_\text{sup} selected for a new model? Which data splits are used for probe training and Table 3 feature selection? Are conclusions robust under natural class imbalance?
5 comments
AuthorsOpenReview ↗

Response to Reviewer D9Qn (1/2)

We thank the reviewer; see also the general response.

1. Dense MLP bottleneck? (W1–W3, Q a). The capacity-matched MLP is a pure classifier: it reads the full activation and is trained only to predict the label. G-SAE's supervised units must predict correctness while also helping reconstruct the residual stream (EV 0.69–0.83), and each has a decoder direction that can be added back to the model, which we use for steering (Sec. 4.6). Despite these constraints, G-SAE is within 0.01 of the MLP on Qwen2.5-Math-1.5B and DeepSeek-R1-1.5B and higher on Qwen2.5-Math-7B (0.735 vs. 0.725); the MLP is higher on the other three models (App. A.10, Table 11). The supervised units also do not behave like an always-on bottleneck: without labels, the corresponding gated units are active on 52–82% of OOD steps; with supervision, on 28–49%, with 2–3× fewer active units per step on the 7B models (App. A.9, Table 8). Since they remain more frequent than Top-KK latents, we describe them as selective.

2. Matched reconstruction-only baseline and seeds (W10, Q b). G-SAE combines two components: a partitioned dictionary whose supervised block uses gated units outside the Top-KK budget, and semi-supervised training of that block with step labels alongside unlabeled reconstruction data. Recon-G-SAE is the ablation that keeps the first and removes the second: it matches G-SAE's gating, partition, data and sampling, without labels; all SAEs use 5 seeds (Sec. 4.3). The ablation separates the two contributions:

  • The partitioned design accounts for most of the AUC gain over a standard Top-KK SAE: both gated SAEs outperform it, and they reach similar AUC to each other (with the Table 1 probe, G-SAE is higher on 3 of 6 models, most clearly Qwen2.5-Math-7B, +0.025, and lower by at most 0.011 elsewhere; with MaxAbs scaling and a tuned CC, G-SAE is higher on the two Qwen2.5 7B models, +0.021 and +0.018; App. A.10, Table 11).
  • The labels change what the supervised units do: without them the gated units are active on 52–82% of OOD steps, with them on 28–49%, with 2–3× fewer active units per step on the 7B models (App. A.9). Only the supervised units are tied to a correctness head, which gives each one a signed direction we can steer (Sec. 4.6).

3. Probe and splits (W11, Q). Sec. 4.2 now specifies the probe: L2 logistic regression on standardized latents (all latents of each SAE), C=10−3C=10^{-3}, identical for all SAEs; results for CC from 10−310^{-3} to 1010 are in App. A.10 (Table 10). It is trained on GSM8K+MATH train and tested on OlympiadBench + Omni-MATH (Sec. 4.1). The raw-activation probe uses the same setup, with CC selected on GSM8K+MATH validation.

4. Imbalance and calibration (W13). This revealed an inaccuracy in our description, which we corrected (Sec. 4.1, Fig. 2). All results in the main text use the OlympiadBench and Omni-MATH steps of the validation split (292 steps), balanced 50/50 within each step position and difficulty exactly like the training data, so training and test share the same class balance. To address the question directly, we now also evaluate the same probes, unchanged, on the held-out evaluation split at its natural error rate (about 13% errors; ID 1,202 and OOD 2,118 steps) and report AUPRC and calibration (App. A.13, Tables 15–16). In short, the conclusions hold at the natural error rate: G-SAE has a higher AUC than the Top-KK SAE on five of six models on both natural sets. Probes trained on balanced data underestimate P(correct)P(\text{correct}) (ECE 0.31–0.40), but a standard prior-shift correction using only the ID error rate brings ECE to 0.02–0.07 for every representation.

AuthorsOpenReview ↗

Response to Reviewer D9Qn (2/2)

(continued)

5. Steering (W4–W9, Q). The study is redone on Qwen2.5-7B (Sec. 4.6, Table 3) and Qwen2.5-0.5B (App. A.12) with 300 MATH problems. 15 units are steered per model, the top 3 from each of 5 independently trained G-SAEs (different seeds), with both signs. Every effect is measured against random directions (W6) and compared with difference-of-means and probe baselines, with a problem-level bootstrap and Holm correction (W9), and we report final-answer pass@1 (W8).

  • On Qwen2.5-7B, 8 of 30 unit conditions shift the PRM significantly beyond random in the expected direction and 2 in the opposite one, with effects up to 0.022, larger than the baselines (≤ 0.010). Two units are significant with both signs; neither baseline is.
  • One unit changes final-answer accuracy in both directions (+3.6 / −3.6 pp, Holm-significant), and its gain is not driven by unanswered samples (Table 3).
  • Reward hacking (W8): the PRM score is averaged over samples that give a final answer, so a direction that makes the model stop without answering can raise it by removing low-scoring samples. We therefore count samples that receive a high PRM score but give no final answer, and recompute the PRM with unanswered samples scored 0. One PRM gain is of this kind: it produces 25 more such samples than random (71 vs. 46), and scoring them as 0 removes the gain (+0.017 to +0.001; App. A.12). The other significant gains do not show this pattern.
  • Directions are now signed by their head weight, so "+" always points toward correct steps and a unit with AUC < 0.5 acts as an error detector (W7).
  • On Qwen2.5-0.5B, effects are less consistent: 7 of 12 significant conditions go in the expected direction, and accuracy (7%) is too low to show changes (App. A.12).
  • Prompt-centred steering (W5), a single PRM and no human evaluation are stated as limitations (Sec. 4.8); pass@1 is our PRM-independent check.

6. Diversity (W14, Q). We added the regularizer because several supervised units can otherwise learn redundant patterns; it penalizes units active on the same steps. Its purpose is to diversify the supervised units, not to raise AUC, and its effect on AUC is model-dependent (App. A.5). λdiv\lambda_{\text{div}} is a hyperparameter: all reported models use λdiv=0.5\lambda_{\text{div}}=0.5, and for a new model it can be tuned on the validation split or set to 0, which App. A.5 also evaluates.

7. SSAE (W15). SSAE learns step-level features without supervision, uses correctness only post hoc (final-answer verification or an LLM judge) on different data, and applies its features to answer voting, whereas we target step-level correctness and test our features by steering. A direct comparison would mix differences in data, labels and objective, so its role is played by our reconstruction-only baselines trained on the same data: the Top-KK SAE (Table 1) and Recon-G-SAE (App. A.10).

8. Math-only (W12). We make no claim beyond math; non-math step-labelled data is future work (Sec. 4.8).

Reviewer_D9QnOpenReview ↗

Response to the authors’ rebuttal

Thank you for the additional experiments and revised claims. Recon-G-SAE, the capacity-matched MLP, five SAE seeds, and expanded steering evaluations address my core concerns. The problem-level bootstrap, Holm correction, checks for unanswered samples, and bidirectional pass@1 results for one unit strengthen the intervention evidence. I lean toward weak acceptance, pending clarified evaluation independence and updated results.

Three clarifications remain:

  • Evaluation protocol. Thanks for updating the main-text evaluation description and adding natural-prevalence held-out results with fixed probes. I am still unclear about the split: the 292-step balanced OOD validation subset, 2,118-step held-out OOD evaluation set, and 300 MATH steering problems. Please spell out which data fed model/hyperparameter selection, probe tuning, feature ranking, and steering setup. Confirm clean separation from SAE training and selection data, including steering pilots. Calling a split “validation” alone does not rule out leakage; a concise split-and-selection summary would resolve this. Also, please report or flag G-SAE, Recon-G-SAE and raw-classifier results on the natural-prevalence OOD set with seed-level uncertainty.
  • Contribution attribution. The new results shift the interpretation of predictive gains: the gated architecture drives most improvements over Top-K, while correctness supervision mainly improves selectivity. I appreciate you removing the claim of beating tuned raw classifiers. Ensure main conclusions match the multi-seed findings: higher mean AUC on five of six models, gains of 0.007–0.030, and NMSE overhead of 0.027–0.049. Lower activation frequency supports selectivity but does not prove semantic or causal specificity. Similarly, a correctness-head weight only gives a predictive sign and cannot guarantee the intervention’s causal direction. If the paper keeps arguing supervision yields more controllable features than a reconstruction-only dictionary, direct comparison is required. A narrower claim limited to the demonstrated interventions is also acceptable.
  • Steering evidence scope and reporting. The ±3.6 pp pass@1 shifts add useful evidence beyond PRM, and it is helpful to see how gains vanish when unanswered outputs are scored zero. Explicitly state the reference condition and uncertainty for these accuracy changes, the family of tests covered by Holm correction, and how top-three units per seed were selected independently of steering evaluation. The interventions work for some units: on 7B, 8/30 conditions shift PRM as expected and 2 shift it oppositely; 0.5B results are far less consistent. The bidirectional accuracy effect on one unit should be framed as a local success. Testing candidates across five seeds does not show the effect replicates across seeds. Any claim of superiority over DoM/probe baselines needs direct head-to-head comparison; significance for G-SAE paired with non-significance for a baseline is insufficient. The final-answer results still do not settle the stronger claim of precise step-level correctness control.

Overall, these additions resolve most of my original concerns. The work makes a solid, carefully bounded contribution combining reconstruction, correctness-linked selective features, and concrete steering behavior. With the protocol clarifications and appropriately scoped conclusions above, I can support weak accept.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers

We thank all reviewers. Shared concerns were supervised capacity vs. dictionary, single runs, density of the supervised partition, label robustness, and steering controls. The revision adds:

  1. Table 1, revised (Sec. 4.4). G-SAE vs. the reconstruction-only SAE over 5 training seeds (mean ± std; per-seed results in App. A.10, Table 9). The probe is now fully specified (Sec. 4.2): L2 logistic regression on standardized SAE latents, C=10−3C=10^{-3}, identical for all SAEs, trained on GSM8K+MATH and tested on OlympiadBench + Omni-MATH. Results for CC from 10−310^{-3} to 1010 are in App. A.10 (Table 10).
  2. Supervised baselines and a gate-matched SAE (Sec. 4.3; App. A.10, Table 11): a tuned logistic probe and a capacity-matched MLP on raw activations (same labels and splits), and Recon-G-SAE, which has G-SAE's exact gated architecture, partition, data and sampling but no labels.
  3. Sparsity and unit activity (App. A.9, Table 8).
  4. Label noise and scarcity (App. A.11, Table 12).
  5. Controlled steering (Sec. 4.6, Table 3; App. A.12, Tables 13–16): random-direction null, DoM and probe baselines, problem-level bootstrap, Holm correction, pass@1.
  6. Natural error rate and calibration (App. A.13, Tables 17–18).
  7. Clarifications: evaluation set (Sec. 4.1, Fig. 2), probe and metrics (Sec. 4.2), limitations (Sec. 4.8). We also revised the abstract, contributions, discussion and conclusion to match the new results.

Main findings.

  • G-SAE has a higher mean OOD AUC than the Top-KK SAE on 5 of 6 models (+0.007 to +0.030; 21 of 30 model–seed pairs) and varies less across seeds on 5 of 6, at a reconstruction cost of +0.027–0.049 NMSE.
  • The label-free Recon-G-SAE reaches similar AUC, so this gain comes mostly from the gated architecture; supervision mainly makes the supervised units selective (active on 28–49% of OOD steps vs. 52–82% without labels) at no cost in AUC.
  • Tuned classifiers trained directly on raw activations reach comparable or higher AUC; we no longer claim a predictive gain over them.
  • Under label noise and scarcity, G-SAE never degrades significantly faster than an MLP trained on the same labels, and it degrades significantly less at 30% flipped labels and with 40% or 30% of the labels; only with 10% of the labels does it fall behind.
  • In steering, supervised units shift the PRM beyond random directions, two with both signs, and one unit changes final-answer accuracy in both directions. In a direct paired comparison, units steer about as well as a probe direction and more strongly than a DoM baseline (App. A.12).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.