acceptodds
Under review as a conference paper at ICLR 2027

When Autoregressive Consistency Hurts Safety Alignment

Abstract

Safety alignment in large language models (LLMs) is fragile in part because it is often shallow: fine-tuning mainly reshapes the model's behavior near the first few output tokens. We argue that this phenomenon can be understood through **autoregressive consistency**, the tendency of next-token prediction to preserve and extend the current response trajectory consistently. By analyzing the learning dynamics of safety alignment in an idealized strong-consistency regime, we show that late-position alignment gradients are suppressed and late conditional distributions remain stable over a controlled optimization path. The same mechanism also predicts a broader class of attacks on LLMs: attacks that induce harmful continuation states at arbitrary positions in the output trajectory. As a concrete example, we introduce random insertion attack, which inserts a short harmful span into an otherwise safe refusal trajectory and exploits autoregressive consistency to sustain the resulting harmful branch, thereby bypassing safety alignment. Notably, a short harmful span can redirect the generation to be harmful even after a long refusal prefix, highlighting autoregressive consistency as a potential broader failure mechanism. This suggests that safety alignment should also **break harmful autoregressive consistency** throughout the output trajectory. We therefore propose adversarial safety alignment, an initial framework based on worst-case harmful continuation states, and instantiate it with random worst-insertion training. Overall, our results suggest that autoregressive consistency should be treated as a central consideration in both safety alignment and attack design.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
safety alignment, autoregressive consistency, random insertion attack, shallow alignment

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_1dQPOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper attempts to investigate shallow safety alignment from an interesting perspective on autoregressive consistency. The authors provide a theoretical analysis of the learning dynamics of SFT and DPO, suggesting that autoregressive consistency can suppress alignment gradients at late-position. Based on this, the authors first introduce a random insertion attack to investigate LLM's vulnerability to such attacks, then they further propose an adversarial safety alignment framework and instantiate it with worst-insertion training. Experimental results demonstrate the effectiveness of the proposed attack and show that the defense substantially improves robustness against harmful continuations across multiple LLMs.

Critical strengths
  1. An interesting perspective on shallow safety alignment. This paper provides an interesting perspective on shallow safety alignment by connecting it to autoregressive consistency. The authors attempt to explain this phenomenon through the learning dynamics of SFT and DPO, offering a theoretical understanding of why safety alignment tends to concentrate on early output tokens.

  2. The theoretical analysis, attack method and defense have a well-designed connection. The authors present the theoretical analysis of the autoregressive consistency. Building on the analysis of autoregressive consistency, the authors introduce the random insertion attack and a corresponding adversarial safety alignment method. The experimental results provide empirical support for the proposed perspective and demonstrate the effectiveness of both the attack and defense methods.

Critical weaknesses
  1. The mechanism of adversarial safety alignment is insufficiently justified. In Section 3, the adversarial safety alignment method combines training on harmful continuation states with worst-case insertion selection. However, it remains unclear whether the improvements primarily arise from exposure to harmful continuation states or from the proposed worst-case optimization.

  2. Non-negligible impact on utility performance. As shown in Table 4, the authors report that the ``Insertion'' method achieves an MMLU score of 63.3 on Llama-3.1-8B-Instruct, which is a -5.1 decrease compared to the initial model. Table 4 reveals a noticeable increase in over-refusal rates for several models.

Questions for authors

Please see the weaknesses above and the following questions:

  1. The authors should discuss more about the selection of some hyperparameters, e.g., k, m, and the harm span length.

  2. The authors should discuss more about the trade-off between utility performance and safety alignment.

  3. Although Figure 3 provides some empirical evidence of autoregressive consistency, it remains unclear whether the observed token probabilities are sufficiently high to satisfy the strong consistency assumption underlying Theorem 2.1. I'm concerned that the maximum probability shown in Figure 3 is lower than 0.6, and the tolerance is non-negligible.

4 comments
AuthorsOpenReview ↗

Rebuttal, Part I/II

Thank you for recognizing the perspective on shallow alignment and the connection between the theory, attack, and defense. We address each weakness and question below.


Response to Weakness 1: Exposure to harmful states versus worst-case selection

We thank the reviewer for this interesting question.

Exposure to harmful continuation states accounts for most of the improvement; worst-case selection provides an additional benefit by accelerating recovery learning. We separate these effects using three Llama-3.1-8B-Instruct training conditions. Uniform-one chooses one candidate insertion position uniformly at random for each training example and supervises one refusal recovery response. Worst-one uses the same recovery target but selects the position with the largest harmful-minus-refusal margin, targeting sampled histories where the model most strongly favors continuing the harmful branch over restarting refusal. This focuses recovery supervision on the model’s current weaknesses in interrupting harmful autoregressive consistency (Section 3.1).

The held-out harmful-continuation risk learning curves for Clean, Uniform-one, and Worst-one on Llama-3.1-8B-Instruct, measured over the first safety-data epoch, show a large recovery gain from Clean to Uniform-one. Worst-one further reduces risk relative to Uniform-one by 10.40, 13.86, and 1.49 percentage points at 0.25, 0.5, and 1 epoch, respectively. These results support faster recovery learning through worst-case selection under limited training budgets. The curves appear in Appendix D.8, Figure 6: panel (a) compares all three conditions, while panel (b) enlarges the Uniform-one and Worst-one curves.

For this comparison, Clean, Uniform-one, and Worst-one share initialization, safety and utility examples, targets, example order, optimizer, and update budgets. The two recovery conditions also use the same candidate-position procedure, differing only in the selection rule. Each held-out query is evaluated at eight common insertion positions after at least 32 refusal tokens, with a 50-token harmful span; risk is the fraction of queries with a harmful newly generated continuation at any tested position. These controls separate exposure to harmful continuation states from the additional effect of selection.


Response to Weakness 2: utility trade-off

Thank you for raising this concern. Insertion achieves substantial safety gains, while we acknowledge the utility costs relative to Initial. These costs vary across models and metrics, and Insertion achieves higher MMLU scores and lower benign-refusal rates than Clean under the same training recipes. Our objective combines recovery training with utility supervision (Section 3.2). We assess the resulting safety–utility trade-off below using Initial to measure absolute changes and Clean to compare with ordinary safety SFT.

The following pairs show Initial → Insertion. ASR is measured under random insertion with a 50-token harmful span; ASR, MMLU, and XSTest refusal are percentages.

ModelInsertion ASR ↓MMLU ↑XSTest refusal ↓MT-Bench ↑
Llama-3.1-8B-Instruct97.4 → 2.868.4 → 63.36.4 → 11.67.78 → 7.35
Gemma-3-12B-it87.9 → 2.161.0 → 60.83.6 → 13.68.36 → 8.17

Both models gain substantial robustness, but their utility costs differ: Llama loses 5.1 MMLU points, whereas Gemma loses 0.2 points; benign refusal increases by 5.2 and 10.0 points, respectively. The safety and utility measurements appear in Tables 3 and 4, respectively. DeepSeek-R1-Distill-Qwen-7B provides a complementary example: Insertion reduces random-insertion ASR at a 50-token harmful span from 92.9% to 2.1%, while MATH-500 performance remains essentially unchanged (93.4% → 93.6%) and XSTest refusal stays at 1.6%. These results demonstrate large recovery gains with preserved performance on the evaluated reasoning and benign-refusal metrics (Table 3; Section 3.2, utility evaluation immediately before Table 4).

The added Clean controls provide a complementary comparison: under the same training recipes, Insertion retains 3.2/3.3 more MMLU points and produces 28.8/53.6 fewer benign-refusal points than Clean on Llama-3.1/Gemma. MMLU uses all 14,042 test questions, and XSTest uses all 250 safe prompts with the Table 4 protocol (Appendix D.5, Table 12; Appendix D.6, Table 13). This advantage establishes lower measured utility costs than ordinary safety SFT under these recipes.

ModelMetricInitialCleanInsertion
Llama-3.1-8B-InstructMMLU68.4%60.1%63.3%
Llama-3.1-8B-InstructXSTest refusal6.4%40.4%11.6%
Gemma-3-12B-itMMLU61.0%57.5%60.8%
Gemma-3-12B-itXSTest refusal3.6%67.2%13.6%
AuthorsOpenReview ↗

Rebuttal, Part II/II

Taken together, these results show that Insertion achieves large recovery gains while incurring lower measured utility costs than ordinary safety SFT under the same recipes. This supports the practical significance of our central contribution: a mechanism-level account connecting autoregressive consistency, alignment learning dynamics, and continuation vulnerability, from which we derive an attack diagnosis and an alignment principle. Building on this principle, we hope future work can further improve capability preservation and reduce over-refusal while retaining these safety gains.


Response to Question 1: Selection of kk, mm, and harmful-span length

Thank you for the suggestion. These hyperparameters control the coverage and cost of the practical approximation; we now explain their roles and report transfer across harmful-span lengths.

Here, kk controls insertion-position coverage and scoring cost, mm controls the candidate refusal restarts used to score each position, and the harmful-span length determines the continuation state used in training. The added discussion appears in Appendix D.9, “Hyperparameter roles,” and the implementation and refusal bank are specified in Section 3.2 and Appendix C.4. Recovery transfers beyond the 50-token training span: Insertion achieves random-insertion ASRs of 2.7%, 3.4%, and 4.8% at test spans of 25, 50, and 75 tokens, respectively, covering both shorter and longer harmful spans (Table 1).


Response to Question 2: Trade-off between utility and safety alignment

Thank you for the suggestion. Please refer to our response to Weakness 2 for the safety–utility trade-off, including the absolute changes relative to Initial and the comparison with Clean.


Response to Question 3: Strong consistency and the finite-confidence evidence

Figure 3 shows increased late-token predictability, and we agree that it does not justify treating the consistency tolerance as negligible. We address the concern by clarifying the guarantee's scope and directly testing the predicted learning pattern at finite confidence.

In particular, Theorem 2.1 gives finite-ϵ\epsilon bounds whose implications are clearest in the strong-consistency limit: late gradient contributions and changes in late conditional distributions vanish as ϵ→0\epsilon \to 0 under the stated conditions. This establishes a sufficient mechanism for shallow alignment. The paragraph immediately after Theorem 2.1 in Section 2.1 states this interpretation; Figure 3 presents the finite-confidence counterpart.

To test this prediction during actual training, we measure gradients of the mean refusal-token loss with respect to all model parameters after each of the first three optimizer steps of Llama-2-7B safety fine-tuning. We compare token positions 1–5 (early) with 46–50 (late), averaging example gradients before taking each norm. Late/early gradient-norm ratios are only 0.070–0.108, even though mean late-token probabilities are around 0.7. Thus, the predicted late-gradient suppression occurs at finite confidence (Section 2.1, Figure 4; Appendix D.1).

This finite-confidence evidence supports the autoregressive-consistency perspective that motivates the subsequent attack and defense: learning predictable refusals need not ensure recovery after harmful history changes. Random insertion exposes this vulnerability, and Insertion significantly reduces ASR to 2.7–4.8% (Sections 2.2 and 3.1; Table 1). Together, the sufficient theory, finite-confidence learning evidence, and attack/defense results support a connected account of the failure mechanism and a practically useful alignment principle.

I thank the authors for their responses. Most of my concerns have been addressed, although I still have some reservations regarding the over-refusal rates and the utility degradation on Llama.

I appreciate the authors' efforts in the rebuttal and I will raise my score.

Thank you for your thoughtful follow-up and for reconsidering your assessment.

We appreciate your remaining concerns about over-refusal and utility degradation on Llama, and will ensure these trade-offs remain clearly presented in the revised manuscript. Your constructive feedback has helped us strengthen the evaluation and clarify the paper.

Thank you again for your time and engagement throughout the discussion.

Reviewer_d9jsOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2(+1)
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper studies why safety alignment in LLMs is shallow, meaning it mainly affects the first few output tokens. The authors explain this through autoregressive consistency: once a response starts on a refusal path, the model already predicts the later refusal tokens with high confidence. Because of this, later tokens give little learning signal, and alignment updates concentrate on the early tokens. They formalize this for SFT and DPO. Based on the same idea, they introduce a random insertion attack, which places a short harmful span inside an otherwise safe refusal and lets the model continue from there. To defend against it, they propose adversarial safety alignment, which trains the model to recover a refusal from the most harmful inserted state. Experiments show that this training reduces the success of both insertion and prefill attacks.

Critical strengths
  1. The question is meaningful. Asking why shallow alignment happens, not only that it happens, is a useful direction.
  2. The random insertion attack is simple and makes a clear point. Even a model that has already started refusing can be redirected by a short harmful span.
  3. The defense is easy to implement and is tested on several model families, including a reasoning model.
  4. The paper checks its keyword-based metric against an LLM judge and human annotation.
Critical weaknesses
  1. The explanation of shallow alignment is not new. The original work on shallow alignment already showed that alignment mainly changes the early tokens, and proposed training on harmful prefixes to make it deeper [1]. Prior work has also explained shallow alignment through the weakening of the learning signal at later token positions [2]. The theory here follows a basic property of training: tokens that are already predicted with high confidence produce small gradients. It is unclear what this paper explains beyond [1] and [2].

  2. The theory's key assumption is not supported. The analysis needs later tokens to be predicted almost deterministically. The paper's own measurements show only moderate confidence at those positions. In that regime the bounds are loose, so the theory says little about real models.

  3. The attack and defense are close to prior work. Training a model to switch to a refusal at any position of a harmful response has already been proposed [3]. So has letting a model recover from a harmful state in the middle of generation [4]. Neither is discussed or compared. The random insertion attack also uses the same access as a prefill attack and gives similar success rates, so it adds little as a new attack.

References

[1] Safety Alignment Should Be Made More Than Just a Few Tokens Deep. ICLR 2025.

[2] Rethinking Deep Alignment Through the Lens of Incomplete Safety Learning. AAAI 2026.

[3] Refuse Whenever You Feel Unsafe: Improving Safety in LLMs via Decoupled Refusal Training. ACL 2025.

[4] Backtracking Improves Generation Safety. ICLR 2025.

Questions for authors

Please refer to the Weaknesses.

7 comments
AuthorsOpenReview ↗

Rebuttal, Part I/III

Thank you for your careful review and for recognizing the importance of understanding why shallow alignment arises. We would like to first clarify that our central contribution is a novel theoretical and empirical account of autoregressive consistency as both a mechanism that can limit safety learning and a source of vulnerability when generation must change course. We establish a sufficient mechanism for shallow alignment, and directly test its predicted learning pattern at finite confidence.

This account also explains why the continuation tendency that sustains refusal can sustain a harmful branch after the history changes—even while the earlier refusal remains. Our attack diagnosis and adversarial alignment objective investigate these consequences and demonstrate their significance for safety evaluation and training. Below we distinguish both the theory and its broader explanatory consequences from the cited works (Sections 2–3; Appendix A).


Response to Weakness 1: Novelty beyond prior explanations

We respectfully disagree that our explanation is already established by the cited works or reduces to the familiar confidence–gradient fact.

Our core contribution identifies and formalizes autoregressive consistency as a sufficient mechanism for shallow alignment, and develops its consequences for safety beyond refusal initiation. Below we clarify the theoretical advance first, then explain why it leads to a broader diagnosis and alignment principle.

  1. We provide a theory of why safety learning can remain shallow. Along a refusal trajectory, continuing predictable text and maintaining safety point in the same direction. Alignment can therefore rely on pretrained continuation competence to complete refusals, while obtaining little additional learning from their later tokens. Our theory makes this mechanism explicit: under strong conditional predictability, late gradient contributions remain suppressed and late conditional behavior changes little, establishing a sufficient mechanism for shallow alignment (Section 2.1, Theorem 2.1).

    Specifically, [1] identify the pretrained refusal-continuation shortcut; we develop an explicit theory of alignment dynamics through conditional predictability. [2] attribute late-gradient weakening to computational-path length and context dependency; our analysis identifies a sufficient mechanism based on how predictable a continuation is given its history.

    We also directly test its empirical relevance in the revision: late-gradient suppression occurs during safety SFT at finite confidence (Figure 4; Appendix D.1), and altering earlier refusal history while fixing target tokens, absolute position, and the immediately preceding five tokens increases late gradient norms 1.63–1.74× (Appendix D.2, Table 9). Thus continuation history affects the learning signal even at the same position. Appendix A, “Shallow (safety) alignment,” discusses these distinctions.

  2. We explain why autoregressive consistency can sustain harm even after refusal. The learning account raises a consequential distinction: learning to continue a refusal need not establish the ability to maintain safety when the continuation history changes. Once a harmful branch is introduced, extending that branch and respecting the earlier refusal no longer coincide. Autoregressive consistency can favor the harmful continuation despite the retained refusal. Thus the same continuation tendency that helps sustain safe responses can also undermine their protection after redirection.

    This extends the inference framing in [1]'s §2.3.1 and [2]'s Definition 1, which emphasizes bypassing initial refusal. Random insertion tests the further consequence while retaining that refusal (Section 2.2). The vulnerability persists even after harmful-prefix recovery training: at harmful span 25, Prefill achieves 3.8% prefill ASR but leaves 18.4% insertion ASR (Table 1). It also persists after at least 100 supplied refusal tokens, with 29.74% insertion ASR for Prefill at span 50 (Appendix D.4, Table 11). These findings show why successful refusal and recovery from harmful beginnings do not establish protection across changing histories.

  3. We derive and validate an alignment principle for interrupting harmful autoregressive consistency. This explanation motivates teaching models to interrupt harmful autoregressive consistency across vulnerable continuation histories. Our adversarial alignment objective identifies histories where the model favors harmful continuation over refusal recovery and supplies recovery supervision there (Section 3.1). Random insertion is one concrete diagnosis of this broader mechanism. The contribution is both a theory of what safety learning can leave unresolved and a tested account of why that matters for generation and alignment design.

AuthorsOpenReview ↗

Rebuttal, Part II/III

Response to Weakness 2: Strong consistency versus finite confidence

We thank the reviewer for the question. The theory has substantive relevance to real models through its predictions about learning and the attack and defense principles it motivates. Specifically:

  1. The theory provides a sufficient limiting mechanism. We would like to first clarify that the analysis gives finite-ϵ0\epsilon_0 bounds for ϵ0∈(0,1)\epsilon_0\in(0,1), so almost-deterministic prediction is not required to state the result. For those bounds to guarantee suppressed late gradients and small changes in late conditional distributions, consistency needs to be sufficiently strong relative to parameter sensitivity and optimization path length.

    The result therefore establishes a sufficient limiting mechanism for shallow alignment, which is now further clarified in the paragraph following Theorem 2.1 (page 5) in the revision. Figure 3 connects the assumption to real models: later tokens in both refusal and harmful continuations have higher mean conditional probabilities, supporting its underlying predictability pattern at finite confidence. We then test whether this finite-confidence pattern is accompanied by the learning consequence identified in the theory, using direct measurements of late-gradient suppression during safety fine-tuning (Figure 4, page 6).

  2. The predicted learning pattern is observed during actual fine-tuning. Across completed Llama-2-7B SFT updates 1–3, late/early full-parameter gradient-norm ratios are only 0.070/0.108/0.106 (values below 1 indicate smaller late gradients). The measurements use token windows 1–5 (early) versus 46–50 (late) (Section 2.1, Figure 4; Appendix D.1). Mean late-token probabilities are around 0.7 in this post-update experiment. Late conditional distributions also change less under actual shared-parameter updates: late/early total-variation drift ratios are 0.509 and 0.665 between consecutive checkpoints (Appendix D.2, Table 10).

    Thus finite-confidence evidence supports both suppressed late gradients and reduced late distributional change.

  3. The mechanism has tested consequences for attack and defense. These learning dynamics also motivate a consequential safety question: does learning predictable refusals establish recovery when the continuation history changes? Random insertion exposes a vulnerability that harmful-prefix recovery leaves unresolved; training recovery from the diagnosed histories reduces insertion ASR significantly (Sections 2.2 and 3.1; Table 1). Together, the sufficient theory, observed learning dynamics, and attack/defense results provide an experimentally supported account of a safety failure and a useful principle for addressing it.


Response to Weakness 3: DeRTa, Backtracking, and prefill attack comparison

We thank the reviewer for suggestions of these works. We would like to clarify that both our attack and defense make novel contributions beyond the cited recovery methods and prefill attacks.

In particular, the defense adds a mechanism-derived adversarial objective for identifying vulnerable continuation states, while insertion exposes a protection gap that prefill evaluation can miss. We first report a direct random-insertion comparison with the released DeRTa [3] checkpoint, then explain the detailed differences from DeRTa and Backtracking [4] and why shared access does not make the insertion diagnosis redundant.

  • Direct random-insertion comparison with DeRTa. We directly test [1]'s released DeRTa Llama-3-8B-Instruct checkpoint and train an Insertion variant from the same Llama-3-8B-Instruct backbone. We could not locate an official implementation or released checkpoint for Backtracking, so the direct comparison focuses on DeRTa. We use the authors' evaluation format for DeRTa and the native chat format for Insertion.

    ModelRandom-insertion ASR, harmful span 25
    DeRTa (released checkpoint)19.29%
    Insertion3.97%

    The released DeRTa checkpoint remains vulnerable to insertion, while our Insertion implementation reduces ASR by 15.33 percentage points. This directly demonstrates the practical value of diagnosing and training recovery from these histories, alongside the conceptual distinctions below.

    The experiment is now reported in Appendix D.11, “Random-insertion comparison with DeRTa,” and Table 16, including the training and evaluation details.

AuthorsOpenReview ↗

Rebuttal, Part III/III

We now discuss the detailed differences from DeRTa and Backtracking. These conceptual distinctions also appear in Appendix A, “Recovery from unsafe continuations,” with the access model specified in Appendix D.3.

  • “Switch to a refusal at any position”: what the defense adds beyond DeRTa. [3] trains refusal recovery from harmful prefixes and refusal-token transitions throughout harmful responses. Our analysis and experiments establish why coverage across output positions does not itself establish protection across continuation histories.

    The defense consequently treats the conditioning state as an adversarial variable: it searches sampled histories for the largest harmful-continuation/refusal-restart margin, then supervises safe recovery from the selected state (Section 3.1). Table 1 demonstrates the benefit of training on these histories over harmful-prefix recovery, as quantified in Weakness 1.

  • “Recover ... in the middle of generation”: what the defense adds beyond Backtracking. [4] teaches recovery through a reset token and removes preceding text from the returned output. Our contribution links recovery to alignment learning dynamics and adds an adversarial objective that uses the model's continuation preferences to identify histories requiring supervision. The Insertion implementation demonstrates the practical benefit of this objective on the diagnosed vulnerability (Section 3.1; Table 1). The advance is a mechanism-based account of why and where recovery supervision is needed, together with a concrete training objective and evidence of its benefit.

We now discuss the comparison with prefill attack.

"Same access and similar ASR": what insertion adds as an attack. Both attacks use caller-controlled assistant-continuation access (Appendix D.3), but they test different safety properties: prefill tests recovery from a harmful beginning that bypasses the initial refusal, whereas insertion tests recovery when a harmful span is introduced throughout the generation. The difference is experimentally consequential. At harmful span 25, the Prefill-trained model has 3.8% ASR on prefill but 18.4% on insertion; Insertion training reduces the latter to 2.7% (Table 1). Even after at least 100 supplied refusal tokens, harmful-prefix recovery leaves 29.74% insertion ASR, versus 4.59% for Insertion training (Appendix D.4, Table 11, span 50).

Thus insertion reveals a failure of protection that a low prefill ASR can conceal and identifies an effective corrective intervention. Its scientific contribution is this new diagnosis and training implication, and a higher raw ASR is not required to establish that value.


Reference

[1] Safety Alignment Should Be Made More Than Just a Few Tokens Deep. ICLR 2025.

[2] Rethinking Deep Alignment Through the Lens of Incomplete Safety Learning. AAAI 2026.

[3] Refuse Whenever You Feel Unsafe: Improving Safety in LLMs via Decoupled Refusal Training. ACL 2025.

[4] Backtracking Improves Generation Safety. ICLR 2025.

Reviewer_5p1rOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper argues that autoregressive consistency causes safety alignment to focus mainly on early response tokens, leaving models vulnerable to harmful continuations later in generation. It introduces a random insertion attack to demonstrate this vulnerability and proposes adversarial safety alignment to train models to recover from harmful continuation states.

Critical strengths
  • The paper offers a theoretical explanation for why safety alignment tends to be shallow, going beyond simply observing the problem.

  • The proposed defense substantially reduces attack success rates across several models, suggesting that training models to recover from harmful continuations is effective.

Critical weaknesses
  • The theoretical assumptions are too strong to support the main claim. The theory assumes that the probability of the correct next token approaches one, but Figure 3 shows probabilities mostly around 0.5–0.6, leaving a large gap between the theoretical setting and actual models.

  • The attack setting is unrealistic. The attacker directly inserts harmful tokens into an assistant's partially generated response, but the paper does not explain how an attacker could obtain this level of control in a real deployment.

  • The experimental comparison is insufficient to establish the defense's general effectiveness. Table 1 compares the proposed method with two baselines, while Table 2 compares it only with the original model, making it unclear whether the gains come from the proposed training strategy or simply from additional safety fine-tuning.

Questions for authors
  • Can the authors show that the theoretical gradient suppression actually occurs during safety fine-tuning, rather than relying only on next-token probabilities?

  • What realistic threat model would allow an attacker to insert harmful tokens into an assistant's response after generation has already started?

  • Can the authors compare their method against existing adversarial safety training methods under matched training budgets to establish whether the proposed approach provides additional benefits?

3 comments
AuthorsOpenReview ↗

Rebuttal, Part I/II

Thank you for recognizing the theoretical explanation and the effectiveness of recovery training. We address each weakness and question below.


Response to Weakness 1: The gap between the theoretical setting and actual models

We thank the reviwer for the question. The theory provides a sufficient mechanism for shallow alignment, and direct measurements show that its predicted late-gradient suppression also occurs at finite confidence. We have further clarified this in the revision, including the abstract and Section 2.

Specifically, although the moderate probabilities alone do not establish the strong-consistency conditions, the gradient measurements and subsequent attack and recovery experiments support the mechanism's practical relevance. We address the concern by clarifying the guarantee's scope and providing this complementary evidence, as shown below. The revised Section 2.1 presents the limiting theory and its sufficient-condition interpretation first, followed by finite-confidence probabilities (Figure 3) and direct gradient measurements (Figure 4, page 6).

  1. What the theory establishes. The theorem bounds late-position gradient contributions and changes in late conditional distributions, which yields shallow alignment in the strong-consistency limit. It is therefore a sufficient limiting mechanism. The paragraph immediately following Theorem 2.1 explicitly states this scope. Figure 3 connects the assumption to real models: later tokens in both refusal and harmful continuations have higher mean conditional probabilities, supporting its underlying predictability pattern at finite confidence. We then test whether this finite-confidence pattern is accompanied by the learning consequence identified in the theory, using direct measurements of late-gradient suppression during safety fine-tuning (Figure 4).

  2. The predicted learning pattern occurs in actual fine-tuning. We measure gradients of the mean token loss with respect to all model parameters after the first three Llama-2-7B safety SFT updates. Each SFT update is one optimizer step that updates the model weights. Early and late denote refusal-token positions 1–5 and 46–50. Late/early is the late-window gradient norm divided by the early-window norm; we average gradients across examples before taking each norm. Values below 1 indicate smaller late gradients:

    After SFT update123
    Late/early norm of the sample-averaged gradient0.0700.1080.106
    Mean reference-token probability, positions 46–500.7170.7190.732

    Thus substantial late-gradient suppression occurs at actual models.

    In the revision, the gradients appear in Section 2.1, “Direct evidence of gradient suppression at finite-confidence,” and Figure 4, page 6; Appendix D.1 gives the protocol and per-update ratios, and Appendix D.2, Table 9 reports the probabilities.

  3. How the evidence addresses the gap. Although the guarantee is limiting, the above direct measurements show the predicted late-gradient suppression at finite confidence (Figure 4; Appendix D.1). The autoregressive-consistency perspective also suggests that learning predictable refusals need not ensure recovery after harmful history changes, motivating random insertion and recovery training (Sections 2.2 and 3.1). Insertion significantly reduces insertion attack ASR to 2.7–4.8% (Table 1). Thus the theory establishes a sufficient mechanism, while finite-confidence measurements and attack/defense results support its practical relevance and the resulting alignment principle.


Response to Weakness 2: The realism of the insertion threat model

A caller supplies a partial assistant message containing a refusal prefix followed by a harmful span and asks the model to continue it. The refusal prefix is supplied history; the experiment does not require intercepting a protected live output stream.

This requires control of assistant continuation, as available in open-weight inference and interfaces exposing assistant prefill. It does not apply to user-only chat interfaces that prevent assistant-history control. We make this access assumption and its scope explicit in Appendix D.3, “Continuation access and diagnostic scope.”

AuthorsOpenReview ↗

Rebuttal, Part II/II

Response to Weakness 3: Whether gains exceed additional safety fine-tuning

Additional safety SFT does not account for the full improvement: Insertion lowers ASR beyond Clean on all four evaluated prompt attacks. We address this through an added Initial/Clean/Insertion comparison on Llama-3.1-8B-Instruct (because Llama-2 ASRs are already low) on the same HarmBench behaviors as Tab.1, with identical search settings and independent HarmBench evaluation:

AttackInitial ASRClean ASRInsertion ASR
GCG31%28%4%
AutoDAN89%47%2%
PAIR7%11%3%
TAP12%9%4%

Insertion lowers ASR relative to Clean by 24, 45, 8, and 5 percentage points, respectively. These results support benefits beyond the insertion attack used in training and beyond additional clean safety fine-tuning alone, under the evaluated attack budgets (Appendix D.7, “Search and evaluation protocol,” and Table 14).


Response to Question 1: Does gradient suppression occur during safety fine-tuning?

Yes. Please refer to our response to Weakness 1. We thank the reviewer for this insightful suggestion.


Response to Question 2: What realistic access permits insertion after generation has started?

The caller supplies a refusal prefix followed by a harmful span as a partial assistant message, then requests its continuation. This uses assistant-continuation access, as in open-weight inference or assistant-prefill interfaces. Please refer to our response to Weakness 2 for the access assumption and its scope.


Response to Question 3: Are there comparisons with existing adversarial safety training?

Yes. We address the comparison request with both controls below, showing that Insertion provides additional recovery benefits, and substantially lower insertion attack ASRs than CAT [1] and SafeDecoding [2].

Recovery-training control. The Prefill baseline implements the harmful-prefix recovery principle of [3]. At harmful-span lengths 25/50/75, its random-insertion ASRs are 18.4/22.3/27.6%, compared with 2.7/3.4/4.8% for Insertion. This demonstrates an additional recovery benefit from the states included by insertion training (Section 3.2 and Table 1).

Comparison with additional defenses. Insertion's ASRs are 2.6–4.8%, compared with 73.7–82.0% for CAT and 80.0–85.8% for SafeDecoding. All use Llama-2-7B-Chat and identical supplied insertion histories. CAT uses its released trained checkpoint, whereas SafeDecoding is an inference-time defense. SafeDecoding is also reapplied at the continuation boundary.

Harmful-span lengthCAT ASRSafeDecoding ASRInsertion ASR
2573.7%80.0%2.6%
5076.7%81.2%3.1%
7582.0%85.8%4.8%

The substantially lower insertion ASRs support the value of explicitly learning recovery from these states (Appendix D.10, Table 15).


Reference

[1] Xhonneux et al. Efficient Adversarial Training in LLMs with Continuous Attacks.

[2] Xu et al. SafeDecoding: Defending against Jailbreak Attacks via Safety-Aware Decoding

[3] Qi et al. Safety Alignment Should Be Made More Than Just a Few Tokens Deep.

Dear reviewer,

Thank you again for your time and thoughtful review, which has helped us improve the paper..

Following your feedback, we clarified the theory’s role as a limiting sufficient mechanism and added direct gradient measurements showing late-position suppression during actual training (Section 2.1, Figure 4, Appendix D.1). We also clarified the attack’s access requirements (Appendix D.3), and added additional comparisons with Clean safety training (Table 14, Appendix D.7).

As the discussion period draws to a close, we would be grateful to hear whether the revisions address your main concerns or whether further clarification would be helpful. We would be happy to answer any remaining questions. Thank you again for your time and consideration.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Overall Response and Summary of Revisions

Dear Area Chair and Reviewers,

Thank you for your careful reading and constructive feedback. We have answered every weakness and question in our individual responses and revised the manuscript with additional evidence and explanations. All manuscript changes are marked in blue.

Our central contribution is a theoretical and empirical account of how autoregressive consistency can limit safety learning and sustain harmful continuation despite an earlier refusal, yielding a tested alignment principle. The table links each finding to the corresponding weakness (W) or question (Q).

Concern addressedWhat the revision establishesRevision location
Theory’s empirical relevance — W1/Q1, 5p1r; Q3, 1dQP; W2, d9jsPredicted gradient suppression occurs at finite confidence: late/early ratios of the norms of sample-averaged gradients are 0.070–0.108 during safety fine-tuning at late-token probabilities near 0.7. We clarify the sufficient limiting guarantee and add history-intervention and distribution-drift evidence.Section 2.1, Figures 3–4; Appendices D.1–D.2, Tables 9–10.
Diagnostic value and attack access — W3, d9js; W2/Q2, 5p1rInsertion reveals recovery failures that low prefill ASR can conceal. The gap persists after at least 100 supplied refusal tokens. The caller supplies a refusal prefix followed by a harmful span through assistant-continuation access in open-weight or assistant-prefill interfaces.Table 1; Appendices D.3–D.4, Table 11.
Benefits beyond additional safety SFT and existing defenses — W3/Q3, 5p1rGains exceed additional clean safety SFT: on Llama-3.1, separately optimized GCG, AutoDAN, PAIR, and TAP yield 2–4% ASR for Insertion versus 9–47% for Clean. On Llama-2-7B-Chat, random-insertion ASR is 2.6–4.8% for Insertion, versus 73.7–82.0% for CAT and 80.0–85.8% for SafeDecoding, across harmful spans of 25/50/75 tokens.Table 1; Appendix D.7, Table 14; Appendix D.10, Table 15.
Exposure, worst-case selection, and hyperparameters — W1/Q1, 1dQPExposure supplies most gains; worst-case selection accelerates learning. Clean/uniform/worst-selection learning curves separate these effects, showing an additional 13.86-percentage-point reduction in harmful-continuation risk at half an epoch from selection. We also explain the hyperparameter roles.Appendix D.8, Figure 6; Appendix D.9.
Utility loss and over-refusal — W2/Q2, 1dQPUtility is better preserved than with Clean: Insertion retains 3.2/3.3 more MMLU points on Llama-3.1/Gemma under the same recipes. XSTest over-refusal (Insertion versus Clean) is 11.6% versus 40.4% on Llama-3.1 and 13.6% versus 67.2% on Gemma. Initial comparisons quantify absolute costs alongside safety gains.Tables 3–4; Appendices D.5–D.6, Tables 12–13.
Novelty of the explanation, attack, and alignment principle — W1/W3, d9jsWe provide a theory of how autoregressive consistency can limit safety learning, identify vulnerability beyond refusal initiation, and derive an adversarial recovery objective. Expanded comparisons distinguish our account from Qi, Bach, DeRTa, and Backtracking. The released DeRTa checkpoint has 19.29% insertion ASR versus 3.97% for Insertion under the same attack trials (Llama-3-8B, span 25), demonstrating the practical recovery gap.Sections 2–3; Appendix A; Appendix D.11, Table 16.

Our individual responses provide the supporting reasoning and protocols. We welcome further questions and are happy to clarify any remaining points.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.