acceptodds
Under review as a conference paper at ICLR 2027

Safety Alignment of Large Language Models via Intrinsic Sparse States

Abstract

Safety alignment requires models to reject harmful requests without refusing legitimate requests on sensitive topics. Response-level supervision specifies desired answers but leaves the internal distinction between these requests to be learned indirectly. We introduce ISA, an **I**ntrinsic **S**parse-state **A**lignment framework that complements response-level training with explicit supervision of sparse autoencoder (SAE) feature activations at the final prompt token. A trained probe identifies predictive features, and intervention tests retain those that improve harmful-request safety within a benign-refusal budget. The selected features then support margin losses on prompt-risk scores and bounded activation guidance during LoRA adaptation. Across Llama-3.1-8B, Qwen3-8B, and Gemma-2-9B, ISA reduces unsafe-response rates on three harmful-request benchmarks and refusal rates on two benign-sensitive benchmarks relative to SFT trained on the same response targets. ISA substantially reduces benign refusal compared with SFT+DPO, although DPO achieves lower unsafe-response rates on two benchmarks. Construction controls further show that behavioral screening reduces both harmful compliance and benign refusal on XSTest relative to predictive selection alone. Loss ablations distinguish how the internal objectives contribute to this safety–refusal balance.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
large language models, safety alignment, sparse autoencoders, internal-state supervision, over-refusal

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_4KjeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes Intrinsic Sparse-state Alignment (ISA), a framework that supplements response-level safety fine-tuning with supervision of sparse autoencoder (SAE) features at the final prompt token. ISA first trains a linear probe to rank predictive SAE features, then screens candidate features through interventions that assess harmful-request safety gains and benign-refusal costs. A compact score is subsequently fitted to approximate the broad probe using the retained features. During LoRA adaptation, the method combines response-level training with broad-probe and compact-score margin losses, together with bounded feature-activation guidance. Experiments on Llama-3.1-8B, Qwen3-8B, and Gemma-2-9B show lower harmful-response and benign-refusal rates than SFT trained on the same response targets. Comparisons with DPO reveal a safety–refusal trade-off rather than uniform superiority. The paper’s central contribution is the use of behaviorally screened sparse features as internal training targets, particularly screening that explicitly accounts for unnecessary refusal.

Critical strengths

1. The matched SFT comparison provides meaningful evidence for the proposed supervision.
SFT and ISA share alignment prompts, response targets, and adapter settings. On Llama, ISA reduces HarmBench ASR from 5.50% to 0.50% and XSTest-Unsafe ASR from 27.00% to 16.50%, while also reducing benign refusal. The direction of improvement is consistent across the three evaluated model families. Ordinary ISA also requires no inference-time activation editing.

2. The construction controls go beyond simply comparing the full method against SFT.
Table 2 includes dense auxiliary supervision, broad-probe-only supervision, and predictive-only feature selection. In particular, behavioral screening reduces XSTest-Unsafe ASR from 36.50% to 16.50% relative to predictive-only selection, while also lowering benign refusal. This provides useful evidence that feature construction matters at the evaluated settings.

3. The paper generally interprets its evidence carefully.
The authors acknowledge that DPO achieves lower harmful-response rates on two benchmarks, that reduced refusal does not establish helpfulness, and that improved separation under a directly supervised score is not an independent mechanistic explanation. The second-evaluator analysis and limited human refusal audit also strengthen the evaluation, within their stated scope.

Critical weaknesses

1. The novelty beyond existing SAE-based alignment is not sufficiently isolated.
ISA combines established techniques, including probing, SAE interventions, LoRA, and auxiliary representation supervision. Its most distinctive contribution is behavioral screening under a benign-refusal constraint. However, the predictive-only comparison tests screening as a whole. A harmful-only screening baseline is needed to establish the specific value of the benign-refusal constraint.

2. The connection between behaviorally validated interventions and the actual training objective is incomplete. Stage II identifies intervention directions dfd_f and interaction-adjusted weights cfc_f, but neither directly enters the compact score or the activation-guidance objective. Instead, the scoring weights ωf\omega_f are obtained by regressing onto the original predictive probe. Thus, behavioral validation determines which features are retained, but the directions encouraged during training are not explicitly constrained to agree with the validated edits.

3. Baseline comparisons do not establish a superior safety–refusal frontier. DPO and steering baselines are evaluated at individual configurations rather than across comparable tuning sweeps. The SAILS implementation omits its optional subspace-preservation loss, while circuit breakers and persona steering are local adaptations. These comparisons are informative but insufficient to establish an advantage over well-tuned alternatives.

4. Statistical evidence is limited, particularly for benign-refusal improvements.
Results use one reported seed without confidence intervals. Some gains correspond to only one or two responses—for example, Llama’s XSTest-Safe refusals decrease from five to three. The clearest ablation benefits also concentrate on XSTest, with mixed results on HarmBench and AdvBench. Multi-seed results and paired uncertainty estimates would strengthen the conclusions.

5. Helpfulness preservation and the proposed mechanism remain insufficiently validated.
Lower refusal rates do not establish better answers, and the human audit assesses refusal detection rather than response quality. Likewise, increased separation under a directly supervised score does not explain the behavioral gains. Helpfulness evaluation and post-alignment feature interventions are needed to substantiate these aspects.

6. Key implementation details are missing.
The appendix omits essential settings, including screening gain values, the benign-degradation budget, precise ablation and interaction-check rules, and several loss and guidance hyperparameters. Data split sizes, total training budgets, and computational overhead are also insufficiently specified, limiting reproducibility of the central contribution.

Questions for authors

Referring to the Critical Weaknesses.

3 comments

Thank you for distinguishing evidence for a screened feature set from evidence for a particular intervention mechanism. We have added the harmful-only screening comparison and two matched Llama training seeds, and clarified the primary experiment’s implementation.

W1 and W3 — Contribution of benign-aware screening and baseline comparisons

ISA’s contribution concerns selecting SAE coordinates using intervention outcomes and then using those coordinates for prompt-conditioned internal supervision. The original predictive-only comparison tests the screening procedure as a whole. We agree that a harmful-only control is needed to examine the contribution of incorporating benign-refusal costs.

We have now added this comparison. All entries below are percentages, and lower is better.

Screening methodHarmBench ASRAdvBench ASRXSTest-Unsafe ASROKTest refusalXSTest-Safe refusal
Harmful-only0.000.5812.502.007.20
ISA with benign penalty0.500.7716.500.571.20

Including the benign penalty reduces OKTest refusal by 1.43 percentage points and XSTest-Safe refusal by 6.00 percentage points. Harmful-only screening achieves lower ASR on the three harmful benchmarks. This supports the penalty’s role in reducing unnecessary refusal at the evaluated setting, while identifying the associated safety–refusal trade-off.

Section 3.2 and Appendix B.1 express the screening score as

q(f,α)=Δharm(f,α)−λbenΔben(f,α).q(f,\alpha)=\Delta_{\mathrm{harm}}(f,\alpha)-\lambda_{\mathrm{ben}}\Delta_{\mathrm{ben}}(f,\alpha).

In the primary Llama run, the benign penalty coefficient was 0.5. Screening operationalized harmful-side gains through keyword-detected refusal changes and applied the benign penalty only to positive increases in benign refusal. This run used a soft penalty without a hard benign-refusal cap. We will make these operational details explicit in the revision.

We also agree that the baseline results describe operating points. On Llama XSTest, DPO achieves 13.50% unsafe ASR and 15.20% benign refusal, while ISA achieves 16.50% and 1.20%, respectively. Establishing a frontier requires comparable development sweeps and held-out comparisons at comparable operating levels.

Our SAILS arm uses subspace initialization and response-loss training, omitting its optional preservation loss. Circuit breakers and persona steering are local adaptations. Our local reproduction of Li and Kim remained partial, so its comparison will be clearly separated from matched-backbone results.

W2 — Selection directions and training directions

Section 3.2 and Appendix B.2 distinguish the Safety Influence Set, which records intervention information, from the Safety Weight Set, which defines the compact score.

In Stage II, interventions on the frozen backbone determine the retained coordinates V. We then fit the compact readout by ridge regression from the retained backbone activations to the broad probe’s logits.

In Stage III, the SAE encoders, normalization statistics, broad probe, and compact scoring parameters remain fixed. Both scores are evaluated on the current model’s activations, and only LoRA parameters are updated. The margin objectives encourage higher prompt-risk scores for harmful prompts and lower scores for benign prompts, while response supervision teaches the desired answers.

Thus, behavioral screening selects coordinates, while the fitted readout supplies the training signal on those coordinates. For the primary run, the saved quantities were

\qquad c_f=\lvert\omega_f\rvert.$$ Pairwise checks were disabled in this run, so these magnitudes were not interaction-adjusted. We will clarify that interpretation. The objective does not explicitly enforce agreement between training directions and the screened gate settings, and such agreement remains unverified. We will make this distinction clearer in Figure 1 and the method description.

W4 and W5 — Statistics, answer quality, and mechanism

Across matched Llama seeds 41, 42, and 43, XSTest-Unsafe ASR decreases by 22.0, 10.5, and 15.5 percentage points. Mean ± sample standard deviation falls from 31.17 ± 3.82% for SFT to 15.17 ± 2.31% for ISA.

All five benchmark metrics improve within every seed. Mean refusal decreases from 2.76 ± 0.43% to 1.14 ± 0.50% on OKTest and from 2.27 ± 0.83% to 1.47 ± 0.46% on XSTest-Safe. These results address alignment-training variability. The benign changes remain small, and prompt-level paired counts and intervals will be reported separately.

The human audit in Appendix E.3 assesses refusal labels. Figure 3’s fiction-writing example illustrates task-relevant assistance, with the retrospective case-selection scope stated in Appendix F. Systematic answer quality and capability preservation require separate measurements.

Figure 5 and Table 10 provide reconstruction-substitution diagnostics. ISA’s aggregate reconstruction-induced next-token loss increase is 0.977, compared with 1.250 for SFT and 1.236 for Base. Its feature overlap with Base is 0.510, compared with 0.316 for SFT.

These measurements characterize the adapted representations under SAE substitution. As §4.4 and Appendix D.2 acknowledge, the larger gap under the directly supervised score is a diagnostic of the training target. Establishing whether individual selected features retain their behavioral effects requires post-alignment intervention replay. We will retain this distinction in the mechanistic interpretation.

W6 — Reproducibility

We will expand the appendix with the executed primary Llama settings and complete feature-level records. Screening used gains of −1 and +1, a benign penalty of 0.5, and 64 harmful plus 64 benign prompts. It retained 16 features, with at most eight per layer. Zero-ablation outcomes were recorded without an admission threshold.

Probe fitting used 12,000 training prompts and 2,000 unique validation prompts. The Stage II compact fit reused the probe-training examples, with a ridge penalty of 10⁻³ and an unregularized intercept.

Alignment used 36,978 examples and 100 optimizer updates. The harmful and benign components of probe and compact-score supervision, together with guidance, each had unit coefficients. Broad-score harmful margin/benign cap were 1.0/−0.2, and compact-score settings were 1.0/−1.0.

The revision will also document the full controller and guidance settings, data-split roles, and separate screening, training, and inference costs, including generated tokens, training tokens, GPU-hours, and latency under a fixed decoding budget.

Dear Reviewer,

Please reply to the authors' rebuttal.

Thanks!

AC.

Reviewer_UQoQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ISA adds supervision of final-prompt-token SAE feature activations to LoRA safety fine-tuning. A WildJailbreak-trained probe ranks features, interventions on the base model screen them, and hinge and activation-guidance losses push a fixed score over the retained features apart for harmful and non-harmful prompts. ISA is compared with response-matched SFT, SFT+DPO, SAILS, circuit breakers and persona steering on three harmful-request and two over-refusal benchmarks across three 8–9B backbones, with selection controls and loss ablations.

Critical strengths
  1. Over response-matched SFT, XSTest-Unsafe ASR drops by 9.50–13.50 points in all three families (Table 3, p.9), and the gap holds under a second judge (Table 11, p.20). This is the main evidence that internal supervision adds value beyond response targets.
  2. Removing behavioral screening (Probe-Only Selection) raises XSTest-Unsafe ASR from 16.50% to 36.50% (Table 2, p.7). This supports the paper’s distinctive component.
Critical weaknesses
  1. The benign-refusal claim is not statistically supported. All results are single seed-42 runs (Table 5, p.14). Gains over SFT are 5→3 of 250 responses on XSTest-Safe and 8→2 of 350 on OKTest (Table 1, p.6); by my calculation these give Fisher p = 0.72 and 0.11. On Gemma the gains are one response each (Table 3, p.9). Five of six ablated variants are worse than plain SFT on XSTest-Unsafe (Table 2, p.7), which is consistent with seed noise. Helpfulness and capability are not measured.
  2. The comparisons are unmatched. Only ISA uses the balanced WildJailbreak prompts (App. A.1, p.13), whose benign half mimics XSTest categories (Jiang et al., NeurIPS 2024). Loss weights and training steps are unreported, and baselines use single configurations; DPO’s 15.20% XSTest-Safe refusal suggests under-tuning (Table 1, p.6). The closest prior method, Li & Kim (ICML 2025), appears only as external numbers (Table 8, p.17). RepBend (ACL 2025) and ReFAT (ICLR 2025) are not discussed.
Questions for authors

Clarifications:

  1. How were the loss weights, margins and training steps tuned, and did the baselines receive a comparable search?
  2. Figure 4 shows 11 retained features, but |V| = 16 (Table 7, p.15). How were the others chosen?

Suggestions:

  1. Results over at least three seeds, with paired tests, for SFT, ISA, Probe-Only Selection and the loss ablations.
  2. Results for SFT trained with the WildJailbreak probe prompts added, and for Li & Kim on the same backbone.
5 comments

Thank you for your careful reading and constructive suggestions. We have added two matched Llama training seeds and clarified the primary experiment’s settings.

W1 and Suggestion 3 — Statistical support

We agree that the seed-42 benign-refusal changes are small. On XSTest-Safe, refusals decrease from 5 to 3 out of 250 prompts, and on OKTest from 8 to 2 out of 350. Your Fisher calculations highlight the limited event counts. Because the checkpoints answer the same prompts, prompt-level paired inference requires the discordant outcomes. We will report these counts and paired intervals separately.

We have now completed matched SFT and ISA runs at seeds 41, 42, and 43. The following values are percentages, reported as mean ± sample standard deviation across the three seeds.

MetricSFTISA
HarmBench ASR4.67 ± 0.760.83 ± 0.58
AdvBench ASR1.22 ± 0.400.58 ± 0.33
XSTest-Unsafe ASR31.17 ± 3.8215.17 ± 2.31
OKTest refusal2.76 ± 0.431.14 ± 0.50
XSTest-Safe refusal2.27 ± 0.831.47 ± 0.46

ISA improves all five metrics within every seed. The XSTest-Unsafe reductions are 22.0, 10.5, and 15.5 percentage points, averaging 16.00 points. These results strengthen the evidence for consistency across alignment-training runs. The benign differences remain modest, and Gemma’s reported gains remain one response per benign benchmark.

Regarding five of six ablated variants exceeding SFT’s XSTest-Unsafe ASR, those rows compare different construction or loss configurations. Their differences combine component effects with training variability. The new matched runs establish recurrence of the full ISA–SFT improvement. The construction and loss ablations retain their single-seed evidence scope.

We also distinguish refusal frequency from answer quality. The human audit validates sampled refusal labels. Systematic helpfulness and capability preservation require separate measurements.

Clarification 1 — Training settings and tuning

The primary Llama run uses 100 optimizer updates. The three internal-loss weights are 1.0. The broad-score harmful margin/benign cap are 1.0/−0.2, and the compact-score settings are 1.0/−1.0.

As reported in Appendix C.1, DPO uses β = 0.1, a learning rate of 5 × 10⁻⁶, 14,152 training pairs, and 500 validation pairs. Its evaluated configuration has lower ASR on AdvBench and XSTest-Unsafe and higher refusal on the two benign benchmarks than ISA. We interpret this as an observed operating-point trade-off. We will document the actual configurations tried, search budgets, and checkpoint-selection rules, including methods evaluated at a single setting.

Clarification 2 — Figure 4

Figure 4 displays 32 of the 64 candidates, including 11 of the 16 retained Llama features. The remaining five were selected from the same candidate pool using the same procedure and fall outside the displayed subset.

For the primary run, selection ranked intervention scores with a benign-refusal penalty and retained at most eight features per layer. Zero-ablation outcomes were recorded without an admission threshold, and no hard benign-refusal cap was imposed. We will reconcile these run-specific details with the procedural description, clarify the displayed counts, and provide the complete selected-feature records.

W2 and Suggestion 4 — WildJailbreak and baseline comparability

As Appendix A.1 states, SFT and ISA share alignment prompts and response targets, including a WildJailbreak subset. In the primary Llama comparison, both use the same 36,978 response-training examples, with 12,326 in each utility, harmful, and benign stratum.

ISA additionally uses prompt-label supervision for interface construction. The primary probe-training set contains 12,000 prompts, comprising 6,037 harmful and 5,963 non-harmful examples, with 2,000 unique validation prompts. The compact ridge fit reuses the probe-training examples. We will update the exactly balanced and 95%/5% description to these realized counts.

Thus, response training is matched, while probe construction remains an additional supervision resource for ISA. Your concern about this resource and XSTest-inspired categories remains valid. The reported protocol reserves benchmark prompts for evaluation.

Regarding Li and Kim (2025), we inspected the official implementation and attempted a local reproduction. Their full pipeline introduces a dedicated [CLS] token and classification head, performs additional pretraining on Wikipedia with Llama-Guard-generated safety labels, and subsequently applies supervised fine-tuning (§§3.1 and 4.1 of their paper).

In the public materials we inspected, we could not verify the exact safety-labeled pretraining corpus or a checkpoint corresponding to that initialization. We acknowledge that §4.3 also reports a competitive alignment-only ablation, so the missing pretraining resources specifically limit reconstruction of the full reported pipeline.

Adapting the implementation to Llama-3.1 additionally requires validating its tokenizer and vocabulary handling and its custom attention, cache, and generation logic. The official repository pins a Transformers commit and requires source modifications for dynamic safety re-evaluation.

In our attempt, we obtained a local Llama-2 SFT checkpoint, converted it to Hugging Face format, and ran generation on four local evaluation sets. We have not reconstructed the reported safety-pretraining stage or validated the implementation on Llama-3.1. We will therefore describe this as a partial local reproduction and distinguish its outputs from author-reported reference results. The requested faithful, matched-backbone comparison remains open, and our comparative claims will reflect that scope.

Li and Kim integrate explicit safety classification into attention and decoding. RepBend incorporates activation steering into fine-tuning losses, and ReFAT uses refusal-feature ablation for adversarial training. ISA uses intervention-selected SAE coordinates for prompt-conditioned internal supervision. We will expand the related-work discussion to clarify these relationships.

Dear Reviewer,

Please reply to the authors' rebuttal.

Thanks!

AC.

Reviewer_SWn9OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ISA adds supervision of SAE feature activations to response-level safety training. A linear probe ranks candidate features at the final prompt token. Behavioral interventions retain features that improve safety within a benign-refusal budget. The selected features support risk scores and activation targets during LoRA training. Experiments cover Llama, Qwen, and Gemma. They compare ISA with supervised fine-tuning (SFT), other alignment methods, and feature-selection and loss ablations.

Critical strengths
  • Behavioral feature selection is well motivated. A feature that predicts harmfulness need not improve safety when edited. Testing safety gains together with benign-refusal costs is a useful selection strategy. The screening ablation provides encouraging evidence on XSTest.

  • The comparisons provide useful evidence. SFT and ISA share response targets and adapter settings. This supports assessing the added internal supervision. Tests cover three model families and include simpler supervision controls. A second safety judge and a human refusal-label audit strengthen evaluation. The paper also acknowledges the trade-off with DPO.

Critical weaknesses
  • W1. Some ISA-specific settings need clarification. Table 5 reports general training settings, feature preselection, and the 64-feature screening pool. However, I could not find the tested intervention strengths, the penalty and allowed increase in benign refusal, or the weights of the three internal losses. The numbers of prompts used for screening and calibration are also unclear. Please provide these values and explain how interaction checks change the feature weights.

  • W2. The stability of the gains needs clearer evidence. The paper reports one training seed. Some benign-refusal gains are small. On Gemma, each improvement on OKTest and XSTest-Safe is a net difference of about one response. The safety gains are larger, but uncertainty is still needed for the main SFT–ISA comparison. Please report paired outcome counts and variation across training seeds.

  • W3. The broader safety–utility trade-off needs evaluation. Lower benign refusal does not establish that useful capabilities are preserved. Please report task performance relative to matched SFT and evaluate jailbreaks optimized against each defended model. A calibrated guardrail on the same SFT checkpoint would also provide a useful practical baseline.

Questions for authors

Q1. Please provide the complete selection and training settings identified in W1. Include the selection rule, the full selected feature sets, and their measured intervention effects.

Q2. How stable is the main SFT–ISA comparison across training seeds? Please report paired outcome counts and uncertainty, especially for the small benign-refusal improvements.

C1. The ablations already provide useful evidence, but the benefit is benchmark-dependent. Behavioral screening improves both XSTest metrics over predictive-only selection. Predictive-only selection has lower harmful-response rates on HarmBench and AdvBench, as the paper acknowledges. The added complexity is therefore more convincing for XSTest than for the other benchmarks. An optional comparison with and without the benign-refusal constraint would help isolate its contribution.

C2. If I understand the procedure correctly, the features are screened on the original model. Their behavioral effects are not retested after alignment. The larger score separation is expected because that score is directly supervised. The authors acknowledge this limitation. Repeating feature interventions on ISA and matched SFT models would strengthen the interpretation. I regard this as an additional mechanism check, not a prerequisite for the reported behavioral gains.

C3. The human audit validates refusal labels. It does not assess the quality of answers classified as non-refusals. A small blinded check of relevance and helpfulness would strengthen the practical interpretation of lower refusal rates.

5 comments

Thank you for recognizing the motivation for behavioral feature selection and the value of the response-matched comparisons. We have added two matched Llama training seeds and clarified the primary experiment’s implementation.

W1 and Q1 — Selection and reproducibility

For the primary Llama experiment, we screened 64 candidates on the same 64 harmful and 64 benign prompts. Screening used keyword-detected refusal changes, greedy generation with at most 64 new tokens, and gated-controller gains of −1 and +1.

Section 3.2 and Appendix B.1 express the screening score as

q(f,α)=Δharm(f,α)−λbenΔben(f,α).q(f,\alpha)=\Delta_{\mathrm{harm}}(f,\alpha)-\lambda_{\mathrm{ben}}\Delta_{\mathrm{ben}}(f,\alpha).

The two terms measure harmful-request safety gains and benign-refusal costs relative to the unedited backbone. In the primary Llama run, the benign penalty coefficient was 0.5. We will make these implementation details explicit in the revision.

We retained the 16 highest-scoring features, with at most eight per layer. This run used a soft benign-refusal penalty, with no hard refusal cap or minimum positive-gain requirement. Zero-ablation outcomes were recorded without an admission threshold. Pairwise checks were disabled, so no interaction-based reweighting occurred in this run.

Probe fitting used 12,000 training prompts and 2,000 unique validation prompts. The Stage II compact fit reused the probe-training examples. All internal-loss components had unit coefficients.

We will incorporate these run-specific clarifications into the relevant method and appendix descriptions and provide the complete feature IDs and measured intervention effects. The backbone-specific retained counts reported in Table 7 are 16, 16, and 24 for Llama, Qwen, and Gemma, respectively.

W2 and Q2 — Stability and uncertainty

The additional matched Llama runs give the following results. All entries are percentages, and lower is better.

SeedMethodHarmBench ASRAdvBench ASRXSTest-Unsafe ASROKTest RRXSTest-Safe RR
41SFT4.000.7734.503.143.20
41ISA1.500.1912.501.432.00
42SFT5.501.3527.002.292.00
42ISA0.500.7716.500.571.20
43SFT4.501.5432.002.861.60
43ISA0.500.7716.501.431.20

Mean ± sample standard deviation across these seeds is

MetricSFTISA
HarmBench ASR4.67 ± 0.760.83 ± 0.58
AdvBench ASR1.22 ± 0.400.58 ± 0.33
XSTest-Unsafe ASR31.17 ± 3.8215.17 ± 2.31
OKTest refusal2.76 ± 0.431.14 ± 0.50
XSTest-Safe refusal2.27 ± 0.831.47 ± 0.46

ISA improves all five metrics within every seed. The mean XSTest-Unsafe reduction is 16.00 percentage points. These runs address alignment-training variability and complement the original cross-family comparison and second-evaluator check.

We agree that the benign changes remain small. XSTest-Safe reductions are 1.2, 0.8, and 0.4 percentage points across the three seeds, and Gemma’s reported benign gains remain one net response per benchmark. We report these at their actual scale. The standard deviations describe training-run variation. We will report prompt-level discordant counts and paired intervals separately.

W3 and C3 — Utility and robustness

We agree that lower refusal alone does not establish preserved utility. Figure 3(c) provides a qualitative example of relevant narrative advice from ISA where matched SFT refuses. However, this retrospectively selected case does not establish overall answer quality. The 100-response human audit in Appendix E.3 assesses binary refusal labels.

Our completed quantitative evaluation concerns benchmark unsafe-response and refusal rates. General capability, systematic blinded answer-quality assessment, model-specific adaptive attacks, and a calibrated guardrail comparison remain outside the completed evaluation. We will make this scope explicit in the revision.

C1 — Contribution of benign-aware screening

We have added the requested harmful-only screening comparison. Incorporating the benign penalty reduces OKTest refusal from 2.00% to 0.57% and XSTest-Safe refusal from 7.20% to 1.20%, corresponding to reductions of 1.43 and 6.00 percentage points.

This improvement accompanies a safety–refusal trade-off. Harmful-only screening achieves lower ASR on HarmBench (0.00% versus 0.50%), AdvBench (0.58% versus 0.77%), and XSTest-Unsafe (12.50% versus 16.50%).

These results support the benign penalty’s role in reducing unnecessary refusal at the evaluated setting, while showing the associated cost in harmful-response rates. We will report this trade-off explicitly and scope the screening claim accordingly.

Screening methodHarmBench ASRAdvBench ASRXSTest-Unsafe ASROKTest refusalXSTest-Safe refusal
Harmful-only0.000.5812.502.007.20
ISA with benign penalty0.500.7716.500.571.20

C2 — Post-alignment interpretation

Figure 5 and Table 10 examine reconstruction substitution using fixed SAEs. The aggregate reconstruction-induced next-token loss increase is 0.977 for ISA, compared with 1.250 for SFT and 1.236 for Base. ISA also has greater feature overlap with Base than SFT does, 0.510 versus 0.316.

These measurements provide representation-level diagnostics under SAE substitution. Establishing whether individual selected features retain their behavioral effects after alignment requires intervention replay. We agree that this would provide an additional mechanism check.

Thank you for the additional experiments and clarifications.

  1. Mostly addressed. Please include the complete selected-feature IDs, their measured intervention effects, and the remaining calibration details.

  2. The new seeds resolve my concern about training variability. For Llama’s five-to-three refusal reduction on XSTest-Safe, how many prompts changed from refusal to answer, and how many changed the other way?

  3. Please provide a systematic evaluation of answer quality beyond refusal rates. I find the paper promising and think the current average score is too harsh, but I need stronger evidence before recommending acceptance at ICLR.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers and the Area Chair

We sincerely thank the Area Chair and all the reviewers for their time, careful reading, and constructive feedback. We are particularly grateful to the reviewers who engaged in the follow-up discussion. We also express our regret that we could not receive follow-up feedback from all reviewers before the discussion deadline, but we deeply appreciate the thoroughness of the initial reviews, which have substantially guided our revisions.

The reviews identified several weaknesses in our initial submission. We have conducted additional experiments and analyses to address these concerns. Below is a summary of the core issues raised and how we have resolved them:

1. Statistical Robustness & The Over-Refusal Claim

  • Concern: Results were based on a single seed; benign-refusal improvements were small and lacked paired statistical evidence.
  • Resolution: We added two matched SFT-ISA training seeds (seeds 41, 42, 43). ISA consistently improves all five metrics within every seed (e.g., XSTest-Unsafe ASR reduction averages 16.00 points). For the small benign gains, we provided exact prompt-level paired transitions (e.g., 5 refusal→non-refusal and 3 non-refusal→refusal). We clarified that the matched SFT baseline already sits near the zero-refusal boundary (~2.0%), creating a floor effect. Figure 2(a) visualizes this: ISA pushes refusal rates further down to ~1.2%, achieving a joint improvement without the heavy refusal tax seen in methods like DPO.

2. Answer Quality & Mechanistic Validation

  • Concern: Lower refusal does not mean better answers; behavioral gains need content-level and mechanistic validation.
  • Resolution: We conducted a systematic, blinded LLM-as-a-judge evaluation (gpt-6.1-sol) on all 242 common non-refusal answers for XSTest-Safe. ISA achieved higher helpfulness ratings than SFT (+0.24) with a negligible relevance drop (-0.07). For mechanism checks, we provided SAE reconstruction-substitution diagnostics and acknowledged the distinction between selected intervention directions and the fitted training targets.

3. Reproducibility & Calibration Details

  • Concern: Missing implementation details (loss weights, margins, screening hyperparameters).
  • Resolution: We provided the complete selected-feature IDs, their readout coefficients (ωf\omega_f), and measured intervention effects. We clarified the exact screening scores (qrunq_{\text{run}}), the prompt split, and the calibration process (including the 2,000 non-harmful held-out prompts, calibrated thresholds τp\tau_p and τs\tau_s, and tolerances).

We hope the AC and the reviewers find these additions satisfactory.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.