- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper attempts to investigate shallow safety alignment from an interesting perspective on autoregressive consistency. The authors provide a theoretical analysis of the learning dynamics of SFT and DPO, suggesting that autoregressive consistency can suppress alignment gradients at late-position. Based on this, the authors first introduce a random insertion attack to investigate LLM's vulnerability to such attacks, then they further propose an adversarial safety alignment framework and instantiate it with worst-insertion training. Experimental results demonstrate the effectiveness of the proposed attack and show that the defense substantially improves robustness against harmful continuations across multiple LLMs.
Critical strengths
-
An interesting perspective on shallow safety alignment. This paper provides an interesting perspective on shallow safety alignment by connecting it to autoregressive consistency. The authors attempt to explain this phenomenon through the learning dynamics of SFT and DPO, offering a theoretical understanding of why safety alignment tends to concentrate on early output tokens.
-
The theoretical analysis, attack method and defense have a well-designed connection. The authors present the theoretical analysis of the autoregressive consistency. Building on the analysis of autoregressive consistency, the authors introduce the random insertion attack and a corresponding adversarial safety alignment method. The experimental results provide empirical support for the proposed perspective and demonstrate the effectiveness of both the attack and defense methods.
Critical weaknesses
-
The mechanism of adversarial safety alignment is insufficiently justified. In Section 3, the adversarial safety alignment method combines training on harmful continuation states with worst-case insertion selection. However, it remains unclear whether the improvements primarily arise from exposure to harmful continuation states or from the proposed worst-case optimization.
-
Non-negligible impact on utility performance. As shown in Table 4, the authors report that the ``Insertion'' method achieves an MMLU score of 63.3 on Llama-3.1-8B-Instruct, which is a -5.1 decrease compared to the initial model. Table 4 reveals a noticeable increase in over-refusal rates for several models.
Questions for authors
Please see the weaknesses above and the following questions:
-
The authors should discuss more about the selection of some hyperparameters, e.g., k, m, and the harm span length.
-
The authors should discuss more about the trade-off between utility performance and safety alignment.
-
Although Figure 3 provides some empirical evidence of autoregressive consistency, it remains unclear whether the observed token probabilities are sufficiently high to satisfy the strong consistency assumption underlying Theorem 2.1. I'm concerned that the maximum probability shown in Figure 3 is lower than 0.6, and the tolerance is non-negligible.
4 comments
Rebuttal, Part II/II
Taken together, these results show that Insertion achieves large recovery gains while incurring lower measured utility costs than ordinary safety SFT under the same recipes. This supports the practical significance of our central contribution: a mechanism-level account connecting autoregressive consistency, alignment learning dynamics, and continuation vulnerability, from which we derive an attack diagnosis and an alignment principle. Building on this principle, we hope future work can further improve capability preservation and reduce over-refusal while retaining these safety gains.
Response to Question 1: Selection of , , and harmful-span length
Thank you for the suggestion. These hyperparameters control the coverage and cost of the practical approximation; we now explain their roles and report transfer across harmful-span lengths.
Here, controls insertion-position coverage and scoring cost, controls the candidate refusal restarts used to score each position, and the harmful-span length determines the continuation state used in training. The added discussion appears in Appendix D.9, “Hyperparameter roles,” and the implementation and refusal bank are specified in Section 3.2 and Appendix C.4. Recovery transfers beyond the 50-token training span: Insertion achieves random-insertion ASRs of 2.7%, 3.4%, and 4.8% at test spans of 25, 50, and 75 tokens, respectively, covering both shorter and longer harmful spans (Table 1).
Response to Question 2: Trade-off between utility and safety alignment
Thank you for the suggestion. Please refer to our response to Weakness 2 for the safety–utility trade-off, including the absolute changes relative to Initial and the comparison with Clean.
Response to Question 3: Strong consistency and the finite-confidence evidence
Figure 3 shows increased late-token predictability, and we agree that it does not justify treating the consistency tolerance as negligible. We address the concern by clarifying the guarantee's scope and directly testing the predicted learning pattern at finite confidence.
In particular, Theorem 2.1 gives finite- bounds whose implications are clearest in the strong-consistency limit: late gradient contributions and changes in late conditional distributions vanish as under the stated conditions. This establishes a sufficient mechanism for shallow alignment. The paragraph immediately after Theorem 2.1 in Section 2.1 states this interpretation; Figure 3 presents the finite-confidence counterpart.
To test this prediction during actual training, we measure gradients of the mean refusal-token loss with respect to all model parameters after each of the first three optimizer steps of Llama-2-7B safety fine-tuning. We compare token positions 1–5 (early) with 46–50 (late), averaging example gradients before taking each norm. Late/early gradient-norm ratios are only 0.070–0.108, even though mean late-token probabilities are around 0.7. Thus, the predicted late-gradient suppression occurs at finite confidence (Section 2.1, Figure 4; Appendix D.1).
This finite-confidence evidence supports the autoregressive-consistency perspective that motivates the subsequent attack and defense: learning predictable refusals need not ensure recovery after harmful history changes. Random insertion exposes this vulnerability, and Insertion significantly reduces ASR to 2.7–4.8% (Sections 2.2 and 3.1; Table 1). Together, the sufficient theory, finite-confidence learning evidence, and attack/defense results support a connected account of the failure mechanism and a practically useful alignment principle.
I thank the authors for their responses. Most of my concerns have been addressed, although I still have some reservations regarding the over-refusal rates and the utility degradation on Llama.
I appreciate the authors' efforts in the rebuttal and I will raise my score.
Thank you for your thoughtful follow-up and for reconsidering your assessment.
We appreciate your remaining concerns about over-refusal and utility degradation on Llama, and will ensure these trade-offs remain clearly presented in the revised manuscript. Your constructive feedback has helped us strengthen the evaluation and clarify the paper.
Thank you again for your time and engagement throughout the discussion.
Rebuttal, Part I/II
Thank you for recognizing the perspective on shallow alignment and the connection between the theory, attack, and defense. We address each weakness and question below.
Response to Weakness 1: Exposure to harmful states versus worst-case selection
We thank the reviewer for this interesting question.
Exposure to harmful continuation states accounts for most of the improvement; worst-case selection provides an additional benefit by accelerating recovery learning. We separate these effects using three Llama-3.1-8B-Instruct training conditions. Uniform-one chooses one candidate insertion position uniformly at random for each training example and supervises one refusal recovery response. Worst-one uses the same recovery target but selects the position with the largest harmful-minus-refusal margin, targeting sampled histories where the model most strongly favors continuing the harmful branch over restarting refusal. This focuses recovery supervision on the model’s current weaknesses in interrupting harmful autoregressive consistency (Section 3.1).
The held-out harmful-continuation risk learning curves for Clean, Uniform-one, and Worst-one on Llama-3.1-8B-Instruct, measured over the first safety-data epoch, show a large recovery gain from Clean to Uniform-one. Worst-one further reduces risk relative to Uniform-one by 10.40, 13.86, and 1.49 percentage points at 0.25, 0.5, and 1 epoch, respectively. These results support faster recovery learning through worst-case selection under limited training budgets. The curves appear in Appendix D.8, Figure 6: panel (a) compares all three conditions, while panel (b) enlarges the Uniform-one and Worst-one curves.
For this comparison, Clean, Uniform-one, and Worst-one share initialization, safety and utility examples, targets, example order, optimizer, and update budgets. The two recovery conditions also use the same candidate-position procedure, differing only in the selection rule. Each held-out query is evaluated at eight common insertion positions after at least 32 refusal tokens, with a 50-token harmful span; risk is the fraction of queries with a harmful newly generated continuation at any tested position. These controls separate exposure to harmful continuation states from the additional effect of selection.
Response to Weakness 2: utility trade-off
Thank you for raising this concern. Insertion achieves substantial safety gains, while we acknowledge the utility costs relative to Initial. These costs vary across models and metrics, and Insertion achieves higher MMLU scores and lower benign-refusal rates than Clean under the same training recipes. Our objective combines recovery training with utility supervision (Section 3.2). We assess the resulting safety–utility trade-off below using Initial to measure absolute changes and Clean to compare with ordinary safety SFT.
The following pairs show Initial → Insertion. ASR is measured under random insertion with a 50-token harmful span; ASR, MMLU, and XSTest refusal are percentages.
Both models gain substantial robustness, but their utility costs differ: Llama loses 5.1 MMLU points, whereas Gemma loses 0.2 points; benign refusal increases by 5.2 and 10.0 points, respectively. The safety and utility measurements appear in Tables 3 and 4, respectively. DeepSeek-R1-Distill-Qwen-7B provides a complementary example: Insertion reduces random-insertion ASR at a 50-token harmful span from 92.9% to 2.1%, while MATH-500 performance remains essentially unchanged (93.4% → 93.6%) and XSTest refusal stays at 1.6%. These results demonstrate large recovery gains with preserved performance on the evaluated reasoning and benign-refusal metrics (Table 3; Section 3.2, utility evaluation immediately before Table 4).
The added Clean controls provide a complementary comparison: under the same training recipes, Insertion retains 3.2/3.3 more MMLU points and produces 28.8/53.6 fewer benign-refusal points than Clean on Llama-3.1/Gemma. MMLU uses all 14,042 test questions, and XSTest uses all 250 safe prompts with the Table 4 protocol (Appendix D.5, Table 12; Appendix D.6, Table 13). This advantage establishes lower measured utility costs than ordinary safety SFT under these recipes.