- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes Intrinsic Sparse-state Alignment (ISA), a framework that supplements response-level safety fine-tuning with supervision of sparse autoencoder (SAE) features at the final prompt token. ISA first trains a linear probe to rank predictive SAE features, then screens candidate features through interventions that assess harmful-request safety gains and benign-refusal costs. A compact score is subsequently fitted to approximate the broad probe using the retained features. During LoRA adaptation, the method combines response-level training with broad-probe and compact-score margin losses, together with bounded feature-activation guidance. Experiments on Llama-3.1-8B, Qwen3-8B, and Gemma-2-9B show lower harmful-response and benign-refusal rates than SFT trained on the same response targets. Comparisons with DPO reveal a safety–refusal trade-off rather than uniform superiority. The paper’s central contribution is the use of behaviorally screened sparse features as internal training targets, particularly screening that explicitly accounts for unnecessary refusal.
Critical strengths
1. The matched SFT comparison provides meaningful evidence for the proposed supervision.
SFT and ISA share alignment prompts, response targets, and adapter settings. On Llama, ISA reduces HarmBench ASR from 5.50% to 0.50% and XSTest-Unsafe ASR from 27.00% to 16.50%, while also reducing benign refusal. The direction of improvement is consistent across the three evaluated model families. Ordinary ISA also requires no inference-time activation editing.
2. The construction controls go beyond simply comparing the full method against SFT.
Table 2 includes dense auxiliary supervision, broad-probe-only supervision, and predictive-only feature selection. In particular, behavioral screening reduces XSTest-Unsafe ASR from 36.50% to 16.50% relative to predictive-only selection, while also lowering benign refusal. This provides useful evidence that feature construction matters at the evaluated settings.
3. The paper generally interprets its evidence carefully.
The authors acknowledge that DPO achieves lower harmful-response rates on two benchmarks, that reduced refusal does not establish helpfulness, and that improved separation under a directly supervised score is not an independent mechanistic explanation. The second-evaluator analysis and limited human refusal audit also strengthen the evaluation, within their stated scope.
Critical weaknesses
1. The novelty beyond existing SAE-based alignment is not sufficiently isolated.
ISA combines established techniques, including probing, SAE interventions, LoRA, and auxiliary representation supervision. Its most distinctive contribution is behavioral screening under a benign-refusal constraint. However, the predictive-only comparison tests screening as a whole. A harmful-only screening baseline is needed to establish the specific value of the benign-refusal constraint.
2. The connection between behaviorally validated interventions and the actual training objective is incomplete. Stage II identifies intervention directions and interaction-adjusted weights , but neither directly enters the compact score or the activation-guidance objective. Instead, the scoring weights are obtained by regressing onto the original predictive probe. Thus, behavioral validation determines which features are retained, but the directions encouraged during training are not explicitly constrained to agree with the validated edits.
3. Baseline comparisons do not establish a superior safety–refusal frontier. DPO and steering baselines are evaluated at individual configurations rather than across comparable tuning sweeps. The SAILS implementation omits its optional subspace-preservation loss, while circuit breakers and persona steering are local adaptations. These comparisons are informative but insufficient to establish an advantage over well-tuned alternatives.
4. Statistical evidence is limited, particularly for benign-refusal improvements.
Results use one reported seed without confidence intervals. Some gains correspond to only one or two responses—for example, Llama’s XSTest-Safe refusals decrease from five to three. The clearest ablation benefits also concentrate on XSTest, with mixed results on HarmBench and AdvBench. Multi-seed results and paired uncertainty estimates would strengthen the conclusions.
5. Helpfulness preservation and the proposed mechanism remain insufficiently validated.
Lower refusal rates do not establish better answers, and the human audit assesses refusal detection rather than response quality. Likewise, increased separation under a directly supervised score does not explain the behavioral gains. Helpfulness evaluation and post-alignment feature interventions are needed to substantiate these aspects.
6. Key implementation details are missing.
The appendix omits essential settings, including screening gain values, the benign-degradation budget, precise ablation and interaction-check rules, and several loss and guidance hyperparameters. Data split sizes, total training budgets, and computational overhead are also insufficiently specified, limiting reproducibility of the central contribution.
Questions for authors
Referring to the Critical Weaknesses.
3 comments
W4 and W5 — Statistics, answer quality, and mechanism
Across matched Llama seeds 41, 42, and 43, XSTest-Unsafe ASR decreases by 22.0, 10.5, and 15.5 percentage points. Mean ± sample standard deviation falls from 31.17 ± 3.82% for SFT to 15.17 ± 2.31% for ISA.
All five benchmark metrics improve within every seed. Mean refusal decreases from 2.76 ± 0.43% to 1.14 ± 0.50% on OKTest and from 2.27 ± 0.83% to 1.47 ± 0.46% on XSTest-Safe. These results address alignment-training variability. The benign changes remain small, and prompt-level paired counts and intervals will be reported separately.
The human audit in Appendix E.3 assesses refusal labels. Figure 3’s fiction-writing example illustrates task-relevant assistance, with the retrospective case-selection scope stated in Appendix F. Systematic answer quality and capability preservation require separate measurements.
Figure 5 and Table 10 provide reconstruction-substitution diagnostics. ISA’s aggregate reconstruction-induced next-token loss increase is 0.977, compared with 1.250 for SFT and 1.236 for Base. Its feature overlap with Base is 0.510, compared with 0.316 for SFT.
These measurements characterize the adapted representations under SAE substitution. As §4.4 and Appendix D.2 acknowledge, the larger gap under the directly supervised score is a diagnostic of the training target. Establishing whether individual selected features retain their behavioral effects requires post-alignment intervention replay. We will retain this distinction in the mechanistic interpretation.
W6 — Reproducibility
We will expand the appendix with the executed primary Llama settings and complete feature-level records. Screening used gains of −1 and +1, a benign penalty of 0.5, and 64 harmful plus 64 benign prompts. It retained 16 features, with at most eight per layer. Zero-ablation outcomes were recorded without an admission threshold.
Probe fitting used 12,000 training prompts and 2,000 unique validation prompts. The Stage II compact fit reused the probe-training examples, with a ridge penalty of 10⁻³ and an unregularized intercept.
Alignment used 36,978 examples and 100 optimizer updates. The harmful and benign components of probe and compact-score supervision, together with guidance, each had unit coefficients. Broad-score harmful margin/benign cap were 1.0/−0.2, and compact-score settings were 1.0/−1.0.
The revision will also document the full controller and guidance settings, data-split roles, and separate screening, training, and inference costs, including generated tokens, training tokens, GPU-hours, and latency under a fixed decoding budget.
Dear Reviewer,
Please reply to the authors' rebuttal.
Thanks!
AC.
Thank you for distinguishing evidence for a screened feature set from evidence for a particular intervention mechanism. We have added the harmful-only screening comparison and two matched Llama training seeds, and clarified the primary experiment’s implementation.
W1 and W3 — Contribution of benign-aware screening and baseline comparisons
ISA’s contribution concerns selecting SAE coordinates using intervention outcomes and then using those coordinates for prompt-conditioned internal supervision. The original predictive-only comparison tests the screening procedure as a whole. We agree that a harmful-only control is needed to examine the contribution of incorporating benign-refusal costs.
We have now added this comparison. All entries below are percentages, and lower is better.
Including the benign penalty reduces OKTest refusal by 1.43 percentage points and XSTest-Safe refusal by 6.00 percentage points. Harmful-only screening achieves lower ASR on the three harmful benchmarks. This supports the penalty’s role in reducing unnecessary refusal at the evaluated setting, while identifying the associated safety–refusal trade-off.
Section 3.2 and Appendix B.1 express the screening score as
q(f,α)=Δharm(f,α)−λbenΔben(f,α).
In the primary Llama run, the benign penalty coefficient was 0.5. Screening operationalized harmful-side gains through keyword-detected refusal changes and applied the benign penalty only to positive increases in benign refusal. This run used a soft penalty without a hard benign-refusal cap. We will make these operational details explicit in the revision.
We also agree that the baseline results describe operating points. On Llama XSTest, DPO achieves 13.50% unsafe ASR and 15.20% benign refusal, while ISA achieves 16.50% and 1.20%, respectively. Establishing a frontier requires comparable development sweeps and held-out comparisons at comparable operating levels.
Our SAILS arm uses subspace initialization and response-loss training, omitting its optional preservation loss. Circuit breakers and persona steering are local adaptations. Our local reproduction of Li and Kim remained partial, so its comparison will be clearly separated from matched-backbone results.
W2 — Selection directions and training directions
Section 3.2 and Appendix B.2 distinguish the Safety Influence Set, which records intervention information, from the Safety Weight Set, which defines the compact score.
In Stage II, interventions on the frozen backbone determine the retained coordinates V. We then fit the compact readout by ridge regression from the retained backbone activations to the broad probe’s logits.
In Stage III, the SAE encoders, normalization statistics, broad probe, and compact scoring parameters remain fixed. Both scores are evaluated on the current model’s activations, and only LoRA parameters are updated. The margin objectives encourage higher prompt-risk scores for harmful prompts and lower scores for benign prompts, while response supervision teaches the desired answers.
Thus, behavioral screening selects coordinates, while the fitted readout supplies the training signal on those coordinates. For the primary run, the saved quantities were
\qquad c_f=\lvert\omega_f\rvert.$$ Pairwise checks were disabled in this run, so these magnitudes were not interaction-adjusted. We will clarify that interpretation. The objective does not explicitly enforce agreement between training directions and the screened gate settings, and such agreement remains unverified. We will make this distinction clearer in Figure 1 and the method description.