acceptodds
Withdrawn from ICLR 2027

The Safety Operator: Modulating the Expression of Safety Instructions via Spectral Optimization

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Context tokens in a transformer-based language model can be absorbed into the model's weights as a multiplicative operator. We study this operator in the setting of safety instructions and show that influencing its dominant eigenvalue modulates how strongly the instruction shapes generation. We derive a Contrastive Safety Loss with a suppression weight that controls the tradeoff between emphasizing the safety instruction on harmful queries while suppressing it on harmless queries. Varying maps a relationship between the attack success rate and the over-refusal rate, supporting the hypothesis that the operator's eigenvalue acts as a continuous dial for the instruction's influence. Moreover, this relationship holds relatively independently of how the Safety Loss is parameterized, yielding Pareto-improved safety instructions for appropriate values of .

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM safety, LLM defence, system instruction optimization, implicit updates, prompt optimization, GCG, AutoDan

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_9skLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a "contrastive" loss based on the eigenvalue of a rank-one operator representing safety-prompt-induced activation changes within the transformer’s MLP weights in target large language models. Besides, the loss optimizes prompts to amplify their influence on harmful queries while suppressing it on harmless queries, which can be adjusted with a suppression weight ρ. Experiments across four prompt searching space optimization settings on Gemma models show controllable safety–over-refusal trade-offs and "Pareto improvements" of the proposed method in some configurations.

Critical strengths
  1. I think the proposed eigenvalue-based loss is somewhat novel and intuitively motivated, balancing safety on harmful queries against over-refusal on benign ones. And this loss can be applied to different prompt searching settings (e.g., soft prompt, hardGCG).
  2. The experiments demonstrate promising results, with some configurations improving both attack success and over-refusal rates ("Pareto improvements") over the original safety prompt.
  3. The paper cites/discusses relevant prior works and connects its formulation to existing theoretical and prompt optimization methods properly.
Critical weaknesses
  1. I think the comparisons with Original SI and No-SI in Sections 5.1–5.2 demonstrate the usefulness of the method, but it doesn't sufficiently isolate the benefit of the eigenvalue-based objective.It is not that surprising that a loss balances between safety refusal and harmless question instruction-following can bring about "Pareto-improvement", you experiment results on DRO just confirm this. In addition, the DRO comparison in Appendix B.5 (Table 1 and Figure 4) does not clearly establish superior performance: the authors acknowledge better overall ASR/ORR results for DRO while emphasizing their method’s higher improvement-retention rate. Controlled comparisons with alternative representation-based losses, including the norm-based objectives mentioned in Remark 3.1, would strengthen the contribution.

  2. I am not fully convinced by the interpretation of the eigenvalue as instruction strength in Sections 3.2–3.3. Equation (5) captures only the projection of the safe activation onto the clean activation. As Equation (7) in Appendix A shows, driving the eigenvalue toward 1 leaves an angle-dependent residual. I think Corollary A.4.1 should distinguish approximate closeness from strict convergence and provide stronger empirical validation of the alignment assumption.

  3. I suggest the layer ranges reported in Appendices C–F would benefit from systematic ablations, including different depths, single-layer versus multi-layer objectives, and feedforward-input versus block-output activations. I also find the 90.9% versus 64.2% retention comparison in Appendix B.5 insufficient to establish stronger generalization, since the candidate pools differ and checkpoints from the same trajectory are correlated. A common validation-based selection rule and independent training seeds would make this comparison more convincing.

  4. It would be better to fix some presenations in the paper. For example, Figure 1's overlap between words and arrows (to make it more prefessional) and switch Figure 2 from png/jpeg image into some vector images like pdf image (it is blurred when I want to check the image in detail). Another example is the author claims "Mixed-token optimization: 1B model with the Short SI produces no strictly Pareto improving", but claims "*We instantiate the loss across four parameterization landscapes: continuous embedding optimization (Soft), unconstrained discrete vocabulary search (Hard GCG), readabilityconstrained discrete search (HardR), and hybrid suffix adaptation (Mixed), showing that eigenvalue modulation operates consistently across all four." in the contributions. It seems that in "Mixed-token optimization" settings, the performance is not that consisitent to other three settings.

Questions for authors

See weakness.

In addition, why you characterize the proposed loss as "contrastive"? Eq (6) applies different objectives to harmful and harmless inputs, but does not explicitly contrast their representations through an attraction–repulsion or relative-comparison mechanism. I might be a litte confusing to other authors without good explanation on this.

6 comments

We thank the reviewer for finding our eigenvalue-based loss "somewhat novel" and "intuitively motivated", for noting the value in that it "can be applied to different prompt searching settings", that our "experiments demonstrate promising results with Pareto improvements”, and that the paper connects its formulation "to existing theoretical and prompt optimization methods properly".

The review rates its soundness concerns as minor or fixable. We address each of them below with new analyses, and we summarize the additions to the revision in our general response; if our answers resolve these concerns, we hope the reviewer will consider raising the score.

Critical Weaknesses

The comparisons with Original SI and No-SI show that the method is useful but do not isolate the benefit of the eigenvalue objective; Pareto improvements from a loss that balances refusal and helpfulness are not surprising, as the DRO results confirm. The DRO comparison (Appendix B.5) does not establish superior performance. Controlled comparisons with alternative representation-based losses, including the norm-based objectives of Remark 3.1, would strengthen the contribution.

Goal. Our goal is to introduce the safety operators and to show, with the safety eigenvalue as their simplest spectral quantity, that they give a useful control on instruction expression; we do not claim a state-of-the-art algorithm. We nevertheless ran the two controlled comparisons that the reviewer asks for.

Non-spectral baselines. We added vector steering and safety-instruction boosting, run under our protocol (Fig. L1). Neither lowers both rates significantly: none of the 62 steering configurations lowers ASR significantly without raising ORR, and none of the 50 boosting configurations lowers ASR significantly. Our best candidates lower both significantly (Soft: −3.9 pp ASR and −4.0 pp ORR; Hard GCG: −2.8 pp ASR and −3.6 pp ORR). We also ran RPO (Zhou et al., 2024) with a penalty on harmless queries that we add, an output-based loss that balances refusal and helpfulness on discrete suffixes (Appendix B.6, Fig. B1). Of its 64 candidates, 16 lower both test rates by point estimate, none of them significantly, and neither rate decreases significantly on average.

Alternative operator losses. We also compared six losses built on the operator, on Soft and under the same protocol, including the Frobenius-norm suppression of Remark 3.1 (Fig. M1). All six trace the trade-off and produce Pareto improvements, consistent with our claim that the control comes from the operator rather than from one particular loss. The best loss combines a hinged eigenvalue term on harmful queries with the Frobenius suppression on harmless ones (−3.8 pp ASR and −4.6 pp ORR, both significant), whereas the Frobenius suppression paired with the squared-eigenvalue term of our loss ranks last. The revision reports this comparison in Appendix M.

DRO. We agree that DRO's best candidate is stronger than ours and that our comparison does not establish superiority, and the revision tones down our language accordingly (Section 5.2 and Appendix B.5). The DRO result is consistent with our message: an objective defined on the operator alone, without an external probe, also produces Pareto improvements, and it applies to discrete and readable instructions, where DRO does not.

The interpretation of the eigenvalue as instruction strength is not fully convincing: Eq. (5) captures only a projection, and Eq. (7) leaves an angle-dependent residual when λ→1\lambda \to 1. Corollary A.4.1 should distinguish approximate closeness from strict convergence and validate the alignment assumption empirically.

We agree and have made the statement precise. Equation (7) is exact: as λsafe→1\lambda_{\mathrm{safe}} \to 1, the distance ∥S−I∥F\Vert S - I \Vert_F tends to ∣tan⁡θ∣|\tan\theta|, so SS reaches the identity only if θ\theta also tends to zero. The revision states the corollary this way (Appendix A.3). Sections 3.2 and 3.3 now state the alignment condition where they interpret the eigenvalue.

Alignment in practice. The median cos⁡θ\cos\theta over all training checkpoints ranges from 0.97 to 1.00 across the four parameterizations, and it drops to 0.78–0.94 in the 5% least aligned checkpoints (Fig. A1). The activations are thus roughly aligned, but not enough to equate the distance with the eigenvalue term: the angle term of Eq. (7) is the larger part of the distance at every checkpoint (Fig. N4), and on harmless queries the operator stays close to the identity without reaching it (median ∥S−I∥F\Vert S - I \Vert_F between 0.08 and 0.27). A penalty on the full distance is therefore needed, as Remark 3.1 suggests, and the best-performing of the loss variants above uses one (Fig. M1). The revision reports these values (Appendix A.3).

Eigenvalue and refusal. To test the interpretation directly, we correlated the harmful-query eigenvalue with ASR over all training checkpoints (Fig. N1). A larger eigenvalue goes with a lower ASR in seven of the eight settings (short and long SI; Pearson rr from −0.17 to −0.70). The exception is HardR on the short SI, where the readable suffix rotates the activation at almost constant norm, so the eigenvalue decreases as the operator moves away from the identity. Splitting the eigenvalue into its norm and angle factors supports the reviewer's point: for Soft, Mixed and Hard GCG on the short SI, stretching along the clean direction and rotating away from it both lower ASR, so the two effects partly cancel inside the eigenvalue. The distance ∥S−I∥F\Vert S - I \Vert_F accounts for both and tracks ASR in all four parameterizations on the short SI. The eigenvalue also tracks refusal for DRO, which never optimizes it (Fig. N3). The revision adds these plots (Appendix N), with the correlations of ASR and ORR with ρ\rho (Fig. O1).

The layer ranges need systematic ablations (depths, single- versus multi-layer objectives, feedforward-input versus block-output activations). The 90.9% versus 64.2% retention comparison is insufficient: candidate pools differ, checkpoints from the same trajectory are correlated, and a common selection rule and independent seeds would help.

Layers. We tuned the layer range of each parameterization on the eval split before the ρ\rho sweep, and the revision reports this sweep in Appendix P (Figs. P1 and P2). With the short SI, every range that we tried, down to three layers, reduced ASR; the range mostly changed over-refusal, in a direction that depends on the parameterization, which is why each parameterization has its own range. A layer-by-layer profile of the trained Soft instructions (Fig. P3) shows that the operator changes little before the first layer of the loss and that the change persists in the last layers, which the loss does not cover. We also ran a controlled ablation on Soft that varies only the layers of the loss (depth of the range, single versus multiple layers), with five repeated runs to measure the variation between runs (Fig. P4). Every setting, including each single layer, lowers the in-training ASR by an amount within the range of the repeated runs, and the change in ORR is within their range for 8 of the 11 settings. In this ablation, we find no effect of the layer choice that we can distinguish from the variation between runs, making the layer range a relatively robust hyper-parameter. We keep the activation site at the input of the first feedforward matrix, the only place where the operator corresponds to a weight update, in line with the implicit update theory of Dherin et al. (2025) and Goldwaser et al. (2026).

Retention. All methods, DRO included, use the same rule: every eval-split Pareto improvement is re-evaluated on the test split, and DRO's pool is larger only because it finds more improvements (137 vs. 11). Counting each training run once, 80% of Soft's improvements hold on test, against 41% for DRO. Both methods still keep at least one test improvement in most of their runs (4 of 5 for Soft, 6 of 8 for DRO), so we agree that the comparison is suggestive rather than conclusive without independent seeds, and the revision says so (Appendix B.5). To account for the correlation between checkpoints of the same run, the revision also compares the methods by the test effect averaged over all their candidates, with intervals from a bootstrap over training runs (Appendix I.5, Fig. I2): on average, DRO lowers ORR far more than Soft, and ASR less.

Presentation: Figure 1 has overlapping words and arrows, Figure 2 should be a vector image, and the contribution claiming that eigenvalue modulation operates consistently across all four parameterizations conflicts with Mixed producing no strict Pareto improvement.

The revision fixes both figures. By "consistently", we meant that ρ\rho controls the ASR–ORR trade-off in all four parameterizations, which holds for Mixed as for the others (correlation with log⁡(1+ρ)\log(1+\rho): +0.67 for ASR and −0.62 for ORR; Fig. O1). Pareto improvements are a separate matter: Mixed reaches near-Pareto candidates with the short SI and one Pareto improvement with the long SI. On Gemma 3 4B, 8 of its 16 candidates are Pareto improvements on the test split by point estimate (Table 15). The revision rephrases the contribution to make this distinction.

Questions For Authors

Why call the loss "contrastive"? Eq. (6) applies different objectives to harmful and harmless inputs but does not contrast their representations through attraction–repulsion or relative comparison.

We call it contrastive because it applies opposite objectives to two contrasting sets of queries, with the identity operator (the instruction having no effect) as their common reference: on harmful queries the loss pushes the operator away from the identity (repulsion), and on harmless queries it pulls the operator toward it (attraction). Section 3.3 of the revision states this explicitly.

Ethics flag: the paper includes malicious examples, such as the user prompt "how to make a bomb".

Our harmful examples are of the kind found in standard benchmarks, and the paper shows no harmful completions. The revision adds a content warning to the abstract.

Reviewer_Aju9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper derives a safety-instruction operator and optimizes its non-unit eigenvalue contrastively on harmful and harmless queries. The loss drives soft instructions, continuous suffixes, GCG and readable suffix search. Gemma 3 1B experiments use separate train/eval/test splits; several candidates improve direct-request compliance and over-refusal, while Mixed fails to improve both in the main setting.

Critical strengths

S1. One activation-level objective supports four prompt parameterizations without a separately trained refusal probe or target output strings. This could be useful even with a narrower mechanistic interpretation.

S2. The evaluation separates optimization, screening and test measurement, and reports negative results, stronger DRO points and transfer limitations rather than only successful checkpoints.

Critical weaknesses

W1. The proposed mechanism lacks direct validation. Results do not show the eigenvalue, norm-ratio and angle diagnostics needed to connect optimization with refusal. The eigenvalue combines scale and alignment; it does not alone identify safety-instruction strength. Appendix A also gives ∥S−I∥F=∣tan⁡θ∣\|S-I\|_F=|\tan\theta| at λsafe=1\lambda_{\mathrm{safe}}=1, so returning the eigenvalue to one requires an alignment condition to imply a near-identity operator. Matched nonspectral objectives and unoptimized suffix controls are needed to separate the mechanism from generic prompt-optimization effects.

W2. The strongest Pareto/generalization claims exceed the reported uncertainty. "Confirmed" means two negative point estimates, not simultaneous significance (App. I.4); the HardR winner's ORR interval crosses zero. Intervals cover generation variability on fixed prompts, not prompt-sampling uncertainty. Testing all eval-selected candidates is legitimate, but emphasizing the best test outcomes across correlated checkpoints needs selection-aware analysis or exploratory wording. DRO's stronger points limit claims of a performance advantage, while increased XSTest ORR for all tested Soft/Mixed candidates limits generalization. Neither finding invalidates a new objective by itself.

Questions for authors
  1. Mechanism: Please report eigenvalue, norm ratio and angle before/after optimization, their relationship to refusal, and the activation's normalization location.
  2. Specificity: How does a matched nonspectral contrastive objective perform under the same search budget? Include unoptimized suffixes on the test split.
  3. Uncertainty: Please distinguish point-estimate improvement from significance and give prompt-/selection-aware uncertainty for a fixed eval-selected candidate or defined family. Apply the same selection rule to DRO.
4 comments

We thank the reviewer for this careful review. We are glad that it sees two strengths in our work: (1) that "one activation-level objective supports four prompt parameterizations without a separately trained refusal probe or target output strings", which "could be useful even with a narrower mechanistic interpretation", and (2) that our "evaluation separates optimization, screening and test measurement, and reports negative results, stronger DRO points and transfer limitations rather than only successful checkpoints".

The review asks for direct evidence of the safety-operator mechanism, rather than our indirect trade-off curves, and for claims calibrated to the reported uncertainty. We agree with both requests: the revision adds the direct evidence and recalibrates our Pareto claims, and the general response summarizes its additions. Please let us know if a soundness concern remains beyond these points; if our answers resolve them, we hope the reviewer will consider raising the score.

We now answer your concerns point by point.

Critical Weaknesses

W1. The mechanism lacks direct validation: the eigenvalue, norm-ratio and angle diagnostics are not shown; the eigenvalue mixes scale and alignment, so it does not alone identify instruction strength; and since ∥S−I∥F=∣tan⁡θ∣\Vert S - I \Vert_F = |\tan\theta| at λsafe=1\lambda_{\mathrm{safe}} = 1 (Appendix A), a near-identity operator also requires alignment. Matched non-spectral objectives and unoptimized suffix controls are needed to rule out generic prompt-optimization effects.

What the paper shows. We agree that the paper shows the mechanism only indirectly, through the ASR–ORR trade-off curve by varying ρ\rho. Our argument is the following: our contrastive safety loss acts on the model only through the safety eigenvalue (as this is the only quantity intervening in the loss expression), so if the eigenvalue had no effect on refusal, varying the suppression weight would produce noise, and not a clear trade-off curve in each parameterization.

Direct evidence. To your point, we can plot these quantities directly, and the revision now does (Appendices N and O, summarized in Section 5.2). These plots revealed aspects of the operator's dynamics that we had missed, in particular about alignment, and we thank the reviewer for the suggestion. They confirm our prediction in all cases but one:

  • Suppression weight (Fig. O1). In all four parameterizations, ASR rises and ORR falls with log⁡(1+ρ)\log(1+\rho), and every correlation is significant.
  • Operator diagnostics (Figs. N1 and N2). For every checkpoint of every run of the suppression-weight sweep, we plot ASR against (1) the safety eigenvalue, (2) the norm ratio, (3) the cosine of the angle and (4) the Frobenius distance between the safety operator and the identity. A larger eigenvalue goes with a lower ASR in seven of the eight settings (four parameterizations, short and long SI; the long-SI relations are weaker, because ASR is already close to zero there). The exception is HardR with the short SI: its eigenvalue and its norm ratio correlate with ASR in the opposite direction, while the cosine and the distance behave as in the other parameterizations. The reason is geometric. The readable suffix rotates the activation at almost constant norm, so the eigenvalue decreases as the operator moves away from the identity, whereas the distance still tracks refusal.
  • A non-spectral objective (Fig. N3). The DRO loss is not spectral, yet a higher safety eigenvalue also goes with a higher refusal of harmful queries across the DRO checkpoints (Pearson r=−0.82r = -0.82 between the eigenvalue and ASR).

Alignment. These results support our theory, but the alignment data qualify it. The activations are roughly aligned, as we posited: the median cos⁡θ\cos\theta over all checkpoints lies between 0.97 and 1.00 (Fig. A1). Even with this alignment, the angle term of Eq. (7) dominates the distance to the identity at every checkpoint (Fig. N4). Rotation therefore plays a role in the dynamics that our spectral loss does not account for. Among the six operator losses that we now compare, the best-performing one penalizes the full distance (Fig. M1; exploratory), which suggests that better losses built on the safety operator are within reach. The revision states the alignment condition where it interprets the eigenvalue (Sections 3.2 and 3.3) and restates Corollary A.4.1 accordingly (Appendix A.3).

We answer the request for matched objectives and unoptimized suffixes under Specificity below.

W2. The Pareto claims exceed the reported uncertainty: "confirmed" means two negative point estimates, not joint significance, and the HardR winner's ORR interval crosses zero; the intervals cover generation variability on fixed prompts, not prompt sampling; and emphasizing the best of many correlated checkpoints needs selection-aware analysis or exploratory wording. DRO's stronger points and the higher XSTest ORR of Soft and Mixed limit claims of advantage and generalization, although neither invalidates a new objective.

We agree with most of these points, and the revision tones down our claims.

Significance. The revision says "by point estimate" wherever a Pareto improvement is not significant; the tables already mark significant differences with asterisks. Appendix I.4 now defines "confirmed" as an improvement by point estimate and calls a Pareto improvement significant only when both reductions are. For the HardR candidate that the review mentions, Section 5.2 now reports a significant ASR reduction with no significant change in ORR. Many of our intervals cross zero, but they are not centered at zero, as they would be were there no effect (right panels of Figures 2 and 3). For Soft and for Hard GCG, at least 10 of the 11 test candidates lower each rate by point estimate, and no candidate is significantly worse on either axis. For HardR, 49 of the 65 candidates lower ORR, while its ASR is not clearly shifted. The averages over all candidates make this precise (see Uncertainty below).

Selection. Our protocol fixes the candidates on the eval split before any test evaluation, so their test results carry no selection bias; only the emphasis on the best of them does. To avoid this emphasis, our answer to the "Uncertainty" question averages the test results over all candidates of each method, and the revision presents the best candidates only as examples (see Appendix I.5).

Fixed test prompts. Like standard jailbreak benchmarks, we report rates on a fixed set of prompts, and our intervals measure the variability of the model's generations on this held-out test split, which is what changes when an evaluation is repeated. We test new prompts directly on AdvBench, which played no role in training or checkpoint selection (Appendix K, summarized in Section 5.2). Appendix I.4 states that our intervals are conditional on the test prompts, and the revision repeats it in the main text (Section 5.1).

DRO. We agree that DRO's best candidates are stronger, and we do not claim a performance advantage. Our contribution is to show that an intrinsic, probe-free spectral objective also lowers both rates, and that it applies to discrete and readable instructions, where DRO does not. Under the same selection rule, DRO's candidates lower ORR far more than Soft's on average, but ASR less (see Uncertainty below).

XSTest. The revision states this limitation in the main text (Section 5.2): the continuous methods raise XSTest ORR, whereas the discrete methods lower it by up to 12 pp.

Questions For Authors

Mechanism: report the eigenvalue, norm ratio and angle before and after optimization, their relation to refusal, and where the activation is normalized.

The revision adds scatter plots of the safety eigenvalue, the norm ratio, the angle and the distance to the identity against ASR, and of the distance against ORR, at every checkpoint of training, with a table of all the correlations (Appendix N; see W1).

The activation is the output of the pre-feedforward RMSNorm, that is, the direct input of the MLP, where the implicit weight-update equivalence holds (Dherin et al., 2025; Goldwaser et al., 2026). The norm ratio therefore compares activations after normalization.

Specificity: how does a matched non-spectral contrastive objective perform under the same budget? Include unoptimized suffixes on the test split.

Matched objective. For Soft, DRO already serves as a non-spectral contrastive control. In Appendix B.5, we run DRO on the same training pairs, with the same 250-step budget, the same 16 values of ρ\rho and the same selection protocol as Soft; it replaces λsafe\lambda_{\mathrm{safe}} by a probe-based contrast of representations (results under W2 and Uncertainty). The revision also includes two inference-time baselines, vector steering and safety-instruction boosting (Fig. L1): none of the 62 steering configurations lowers ASR significantly without raising ORR, and none of the 50 boosting configurations lowers ASR significantly. For the discrete parameterizations, we ran RPO (Zhou et al., 2024) under the same protocol (Appendix B.6, Fig. B1). RPO searches a discrete suffix as Hard GCG does, but it minimizes an output-likelihood loss on refusal strings. As published, it has no term on harmless queries, and it lowers ASR to 2.1% while raising ORR to 48.8%. With a penalty on harmless queries that we add, whose weight plays the role of ρ\rho, 16 of its 64 candidates lower both test rates by point estimate, and neither rate decreases significantly on average (see Uncertainty below). RPO ran for the same number of steps, with more than twice the compute of our discrete methods. Each of these comparisons rests on a single sweep, and the revision states this limitation (Section 6).

Unoptimized suffix. We evaluated HardR's unoptimized 26-token seed on the held-out test split (Appendix J.3, Fig. J1). It changes ASR by −0.9 pp and ORR by +1.5 pp, neither significantly, so the seed alone is not a Pareto improvement. The other parameterizations start from the original instruction itself (Soft) or from a placeholder suffix of ten "!" tokens (Mixed, Hard GCG), so HardR is the only one whose initialization carries a new instruction.

Uncertainty: distinguish point-estimate improvement from significance, and give prompt- and selection-aware uncertainty for a fixed eval-selected candidate or a defined family, with the same rule for DRO.

As explained under W2, the revision distinguishes point estimates from significance. To remove the emphasis on the best candidates, we average the test effect over all the candidates of each method (Appendix I.5, Fig. I2). Section 5.2 of the revision now leads with these averages. Our protocol fixes these candidates on the eval split, before any test evaluation, and we apply the same selection rule to DRO and to RPO. We find:

  • Soft (11 candidates from 5 runs): ASR −2.0 pp, ORR −1.8 pp.
  • Hard GCG (11 candidates from 7 runs): ASR −1.4 pp, ORR −1.7 pp.
  • HardR (65 candidates from 13 runs): ASR −0.4 pp, ORR −1.6 pp.
  • Mixed (10 near-Pareto candidates from 4 runs, because Mixed has no Pareto improvement on the eval split): ASR −4.0 pp, ORR +1.5 pp.
  • DRO (137 candidates from 8 runs): ASR −0.9 pp, ORR −10.4 pp.
  • RPO with a harmless-query penalty (64 candidates from 13 runs): ASR −0.4 pp, ORR −0.2 pp.

Checkpoints of the same run are correlated, so we compute 95% intervals with a bootstrap over training runs. They exclude zero for the six effects of Soft, Hard GCG and HardR (HardR's ASR only barely). DRO lowers ORR much more than our methods, but its ASR interval contains zero ([−1.8, +0.9] pp). Both intervals of RPO contain zero, its ASR interval only barely.

Thank you again for your helpful feedback on our work. As the discussion period is ending, we would greatly appreciate it if you could take a moment to review our response, as we put a lot of effort into responding to your concerns as fully as possible. Many thanks!

Reviewer_BN6UOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This work introduces a spectral framework for optimizing safety instructions in language models. Building on prior work on context-induced weight updates represented as multiplicative operators, the authors propose a rank 1 Safety Operator and use its eigenvalue to characterize the influence of safety instructions on model activations. They introduce a safety objective with a suppression weight ((\rho)) that aims to balance amplifying the influence of safety instructions on harmful queries and suppressing it on harmless ones. The method is evaluated using four optimization strategies (Soft, Hard GCG, HardR, and Mixed), primarily on Gemma 3 1B, with additional experiments on Gemma 3 4B. The results suggest that spectral optimization can improve the safety utility tradeoff in some settings, although its effectiveness varies across parameterizations and evaluation conditions.

Critical strengths

The paper has the following positive points:

  • The paper connects implicit context-induced weight updates to a spectral objective. The formulation is clear, and gives a useful starting point for studying how instructions affect internal representations.
  • The objective targets presented by the paper is a relevant trade-off which is a resisting harmful requests without over-refusing harmless ones.
  • The evaluation consistes of four parameterizations test of the objective depends on one search algorithm.
  • The paper reports results that limit its claims such as no strict Pareto improvement for Mixed, overfitting with the long instruction, over-refusal increases on XSTest, and a DRO comparison favoring DRO.
  • All selected candidates are re-evaluated on AdvBench and XSTest without retraining, a useful transfer check. However, reported only in the appendix.
  • The full protocol is repeated on Gemma 3 4B, giving an initial test at larger scale. Reported only in Appendix!
Critical weaknesses

I have the following feedback on weaknesses:

  • The operator builds on prior implicit-update work, and the search methods adapt GCG and AutoDAN. No comparison is made to simpler non-spectral losses, and DRO outperforms the method.
  • The 16 non-uniform ρ values (0–5000) are unjustified, and results are not monotonic (e.g., Soft has no Pareto-improving candidates at ρ = 20–350; Table 2). Pareto improvements even occur at ρ = 0, where suppression is absent (Table 6, HardR, ρ = 0, step 30).
  • Layer ranges vary across methods without ablation (12–23 for Soft vs. 8–23 for the others on 1B variant).
  • Activations are taken before or after pre-MLP normalization, which can change what λ_safe measures, was not clear.
  • All experiments use Gemma 3 1B in the main paper, 4B only shown in the Appendix which is another model family is needed to establish generality.
Questions for authors

In addition to my comments in the weaknesses secrtion, I have the following questions:

  • Optimization needs paired forward passes, up to 128 candidates per step, and 16 ρ runs per method. No training time, GPU hours, or memory usage is reported.
  • Long-SI Soft raises ORR by up to +45.48 pp (Table 19), Long-SI GCG raises ASR in 15 of 18 candidates (Table 21), and XSTest ORR rises for all continuous candidates (Table 26). These should be summarized in the main paper. Why the author's choses to draft it in the appendix!
  • 200 of the 300 pairs are synthetic, and the main text calls, the generator only “a frontier model” (named only in Appendix H.1). The autorater is never named, and no human validation is reported.
  • The introduction gives the operator’s equation before its intuition, and the scope statement appears only after the contributions. The conclusion ends with a future-work list rather than a discussion of implications.
  • The paper is extermly long (i.e., 59 pages). This is unusal for technical papers.
5 comments

We thank the reviewer for finding that our "formulation is clear", that it "gives a useful starting point for studying how instructions affect internal representations", and that our paper addresses "a relevant trade-off" between resisting harmful requests and over-refusing harmless ones.

None of the weaknesses raised in this review questions the correctness of our theory, of our statistical protocol, or of any reported result. They concern presentation, naming, hyperparameter choices and scope. Several points are inaccurate, and all are addressable. We answer them below with references to the paper, and we summarize the additions to the revision in our general response.

Please consider raising your score if you find our answers to your concerns satisfactory.

Critical Weaknesses

The operator builds on prior implicit-update work, and the search methods adapt GCG and AutoDAN. No comparison is made to simpler non-spectral losses, and DRO outperforms the method.

Contribution. Our formulation builds on the implicit weight-update equivalence of Dherin et al. (2025) and Goldwaser et al. (2026). Our contribution is the Safety Operator, its rank-1 spectral geometry (a single eigenvalue λsafe\lambda_{\textrm{safe}} that measures the dilation along the safety direction), and its use for safety steering through the probe-free Contrastive Safety Loss, all new. We use standard search methods (GCG, AutoDAN) deliberately, as an experimental control that isolates the objective.

Simpler baselines. We added two non-spectral baselines, vector steering and safety-instruction boosting, under our protocol (Fig. L1). Neither lowers both rates significantly: none of the 62 steering configurations lowers ASR significantly without raising ORR, and none of the 50 boosting configurations lowers ASR significantly. In contrast, our best candidates lower both significantly (Soft: −3.9 pp ASR and −4.0 pp ORR; Hard GCG: −2.8 pp and −3.6 pp). We also ran RPO (Zhou et al., 2024), which uses the same kind of discrete search as Hard GCG, with an output-likelihood loss in place of our spectral loss (Appendix B.6, Fig. B1). As published, it lowers ASR to 2.1% but raises ORR to 48.8%. With a penalty on harmless queries that we add, 16 of its 64 candidates lower both test rates by point estimate, none of them significantly.

DRO. DRO applies only to continuous Soft embeddings, relies on an external PCA and probes, and cannot produce discrete suffixes. Its best Soft point is stronger than ours, but fewer of its eval candidates hold on test (64% vs. 91% for Soft), and for ρ≥500\rho \geq 500 its ASR degrades to 22–33%, against 14% for the original instruction (Appendix B.5). A single comparison in one setting does not show that DRO outperforms our approach, nor the converse, and this is not our goal. We show that the Safety Operator gives a useful dial on instruction expression across four parameterizations (Section 1), and our new comparison of six operator losses (Fig. M1) shows that the results can be improved by changing the loss within this framework.

The 16 non-uniform ρ\rho values (0–5000) are unjustified, and results are not monotonic (e.g., Soft has no Pareto-improving candidates at ρ\rho = 20–350; Table 2). Pareto improvements even occur at ρ\rho = 0, where suppression is absent (Table 6, HardR, ρ\rho = 0, step 30).

Grid. In Eq. (6), ρ\rho weights the suppression term (harmless queries) against the expression term (harmful queries), so we space our 16 values roughly logarithmically (as Section 5.1 of the revision now states), within a fixed compute budget. The two ends bracket the regimes of aggressive refusal (low ρ\rho) and strong suppression (ρ≥1000\rho \geq 1000).

Monotonicity. We predict an expression–suppression continuum (Section 5), not a curve that is monotonic point by point: each run is a stochastic optimization (batch size 1), screened with a single generation per query (R=1R = 1, Table 13). The revision adds plots that test this relation directly (Fig. O1): over all eval checkpoints, log⁡(1+ρ)\log(1+\rho) correlates significantly with ASR (Pearson r∈[+0.25,+0.72]r \in [+0.25, +0.72]) and inversely with ORR (r∈[−0.70,−0.18]r \in [-0.70, -0.18]) in all four parameterizations.

Tables 2 and 6. For Soft at ρ\rho = 20–350, ORR falls with ρ\rho as predicted but stays above the baseline (lowest eval ORR 20.0% vs. 18.8%), so no checkpoint qualifies as a Pareto improvement. The HardR candidate at ρ\rho = 0 is an early checkpoint (step 30) of a run that starts from a hand-written 26-token seed and optimizes the safety loss together with a fluency term. Neither of its test differences is significant, and the same run later reaches 97% ORR, as a low ρ\rho predicts.

Layer ranges vary across methods without ablation (12–23 for Soft vs. 8–23 for the others on 1B variant).

We understand the requested ablation as a sensitivity analysis with respect to the layer range, and we take this opportunity to clarify how we chose the layers, which the paper does not explain enough.

Tuning. For each parameterization, we tuned the hyperparameters, including the start layer of the range, on the eval split in a preliminary sweep, before the suppression-weight sweep that the paper reports in full. The revision now reports this preliminary sweep (Figs. P1 and P2 in Appendix P). With the short SI, every start layer that we tried lowers ASR in all four parameterizations, and the start layer mostly shifts ORR, in a direction that depends on the parameterization. With the long SI the effect is less clear, because the original instruction already has an ASR close to zero. These differences are small compared with the spread between runs, and for each parameterization we kept the setting that reduced ASR the most at an acceptable ORR, which is why each has its own range.

New analyses. A layer-by-layer profile of the trained Soft instructions (Fig. P3) shows that the operator changes little before the first layer of the loss, that the change builds up inside the range, and that it persists in the last layers, which the loss does not cover. We also ran a controlled ablation on Soft that changes only the layers of the loss (start layer, end of the range, single layers), with five repeated runs of the paper's range to measure the variation between runs (Fig. P4). Every setting, down to a single layer, lowers the in-training ASR by 4.7 to 6.1 pp, within the range of the five repeated runs (4.0 to 6.3 pp). The change in ORR is within their range for 8 of the 11 settings. In this ablation, we find no effect of the layer choice that we can distinguish from the variation between runs.

Activations are taken before or after pre-MLP normalization, which can change what λsafe\lambda_{\mathrm{safe}} measures, was not clear.

The activations are taken after the pre-MLP normalization: they are the vectors multiplied by the first feedforward matrix, as in Dherin et al. (2025) and Goldwaser et al. (2026). This is the quantity that the weight-update equivalence requires, and the revision states it explicitly (Section 3.1).

All experiments use Gemma 3 1B in the main paper, 4B only shown in the Appendix which is another model family is needed to establish generality.

We agree that a second model family would strengthen the paper, and the revision repeats the Soft experiment on Qwen3-4B (Appendix J.4). The original instruction already blocks every harmful test request on this model, so the experiment tests over-refusal. The operator relates to over-refusal as on Gemma 3 (Fig. J3), and five candidates lower test over-refusal significantly, by up to 4.8 pp, with no significant change in attack success (Fig. J2).

Questions For Authors

Optimization needs paired forward passes, up to 128 candidates per step, and 16 ρ\rho runs per method. No training time, GPU hours, or memory usage is reported.

The revision reports training time and compute in Appendix Q. Each training run fits on a single TPU v4 chip (32 GB of memory) and takes 3.5 to 7 hours, so a full 16-value sweep costs 58 to 114 chip-hours, depending on the parameterization.

Let us also clarify three points. The clean pass, which computes AcleanA_{\mathrm{clean}}, needs no backward pass, because it does not depend on the optimized parameters. Only the two discrete methods evaluate candidates at each step (128, four times fewer than standard GCG), while Soft and Mixed take plain gradient steps. Finally, the 16 values of ρ\rho trace the full trade-off curve to demonstrate the phenomenon, and they include extreme values that one would not use in practice.

Long-SI Soft raises ORR by up to +45.48 pp (Table 19), Long-SI GCG raises ASR in 15 of 18 candidates (Table 21), and XSTest ORR rises for all continuous candidates (Table 26). These should be summarized in the main paper. Why the author's choses to draft it in the appendix!

We agree that the results of Appendices J and K deserve more visibility. The Limitations paragraph of Section 6 already notes that optimizing all the embeddings of the long SI (about 380) overfits the eval split, which inflates test ORR, and that OOD transfer varies across parameterizations. The revision adds to Section 5.2 a paragraph with a numerical summary of these results. The main text also summarizes the new analyses: the baselines, the response to ρ\rho and the mechanism evidence in Section 5.2, and the layer analysis in Section 5.1.

We see these results as findings rather than failures. Our goal (Section 1) is to characterize how spectral modulation behaves across parameterizations, and we leave state-of-the-art defense engineering to future work. From this perspective, the long-SI results show where an unconstrained search space overfits, and that restricting the search to a short suffix (Mixed) or adding a fluency prior (HardR) acts as a regularizer.

200 of the 300 pairs are synthetic, and the main text calls, the generator only "a frontier model" (named only in Appendix H.1). The autorater is never named, and no human validation is reported.

Thank you for flagging this omission. The revision names these models in Section 5.1: Claude Opus 4.6 generated the 200 additional pairs (with Gemini 3.1 Pro for analysis and resampling), and the autorater is Gemini 3.0 Flash. As Appendix I.2 reports, the five autorater passes agree unanimously on 99.79% of responses, and the autorater never grades its own generations: it rates the outputs of Gemma 3.

We also ran a blind human audit of the autorater on a stratified sample of 100 test responses (Fig. I1). Two of the authors agree with the autorater on 91% and 93% of the responses that they labeled, and with each other on 94% (Cohen's κ between 0.81 and 0.87); the revision reports the audit in Appendix I.2.

The introduction gives the operator's equation before its intuition, and the scope statement appears only after the contributions. The conclusion ends with a future-work list rather than a discussion of implications.

We thank the reviewer for these structural suggestions. In the revision, we expand the geometric intuition before introducing the operator equation S(Y)=I+Δq(Y)S(Y) = I + \Delta_q(Y), and we move the scope statement before the list of contributions. In Section 6 ("From steering vectors to steering operators"), we separate the broader conceptual and practical implications from the numbered future-work directions.

The paper is extremely long (i.e., 59 pages). This is unusual for technical papers.

Our main text adheres to the ICLR page limit, and we made an effort to keep it self-contained. The appendices hold the full proofs, the pseudocode of the algorithms, the exact optimized prompts, the per-checkpoint confidence intervals, and the extended scaling and OOD evaluations. To make the paper easier to read without them, the revision brings summary statistics from Appendices J and K into Section 5.2 and adds a short roadmap at the start of the appendix.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Consolidated Response to Reviewers

We thank the reviewers for their careful reading and constructive suggestions, which have made our revised version significantly stronger.

Our goal is to introduce safety operators and to show that the safety eigenvalue, their simplest spectral quantity, gives a dial on how strongly a safety instruction is expressed; we do not propose a state-of-the-art safety defense for LLMs. The revision makes five main additions.

  • Simpler alternatives (Appendices L and M). We compare our objective, under the same protocol, with two non-spectral methods, vector steering and safety-instruction boosting, and, on Soft, with five other losses built on the operator. Neither non-spectral method reduces attack success significantly without raising over-refusal, while all six operator losses find candidates that lower both ASR and ORR by point estimate, and the best of them does so significantly. We also run RPO, the closest existing method for discrete suffixes (Appendix B.6): as published it raises over-refusal sharply, and with a penalty on harmless queries that we add, neither rate decreases significantly on average over its candidates.

  • Direct evidence for the mechanism (Appendices N and O). We relate the safety eigenvalue, its norm ratio and angle factors, and the operator's distance from the identity to attack success and over-refusal at every training checkpoint. In all four parameterizations of the main experiments, refusal grows as the operator moves away from the identity, consistent with our theory, and attack success rises while over-refusal falls as the suppression weight ρ\rho increases. We also measure the alignment that our theory assumes and restate Corollary A.4.1 exactly (Appendix A.3): the activations with and without the instruction are roughly aligned, but rotation still accounts for most of the operator's distance from the identity.

  • Calibrated claims (Appendix I.5). Our Pareto claims no longer rest on the best candidates. We average the test effect over all the candidates that each method selects on the eval split, with intervals from a bootstrap over training runs and the same rule for DRO and RPO: both rates decrease for Soft, Hard GCG and HardR, and their intervals exclude zero, HardR's ASR interval only barely (Fig. I2). We say "by point estimate" wherever an improvement is not significant, and we claim no performance advantage over DRO.

  • A second model family (Appendix J.4). We repeat the Soft experiment on Qwen3-4B, in addition to Gemma 3 1B and 4B, to test whether the mechanism carries over to another architecture. The original safety instruction already blocks every harmful test request on this model (ASR = 0.0%), so the experiment tests over-refusal. The operator relates to over-refusal as on Gemma 3: the farther it is from the identity on harmless queries, the higher the over-refusal (Fig. J3). On the test split, 21 of the 26 candidates whose attack success does not differ significantly from the original instruction lower over-refusal, and 5 do so significantly, by up to 4.8 pp (Fig. J2).

  • Methodology Details. We bring the key experimental details and limitations from the appendix into the main text, and we add the compute cost and the hyperparameter sweeps, including the layer range, needed to reproduce our results (Appendices P and Q). A controlled layer ablation on Soft, requested by two reviewers, finds no effect of the layer choice that we can distinguish from the variation between runs (Fig. P4).

We answer the remaining points, including the presentation remarks, in the individual responses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.