acceptodds
Under review as a conference paper at ICLR 2027

Learning to Stay Safe: Adaptive Regularization Against Safety Degradation during Fine-Tuning

25 views

Abstract

Instruction following language models are vulnerable to safety degradation during fine-tuning, even under benign data distributions, and are further susceptible to adversarial updates that systematically erode alignment. Existing defenses offer insufficient protection or impose a steep utility cost, limiting their practical adoption. In this paper, we introduce a training framework that uses frozen activation-based safety signals to adaptively allocate regularization across training examples and individual target tokens, enabling models to remain aligned while preserving task performance. Our framework derives a model-specific intervention direction from pre-generation representations, uses the resulting intervention effects to construct an example-level control signal, and combines this signal with token-specific reference preferences to determine token-level regularization strengths before optimization. Crucially, we demonstrate that pre-generation activations contain linearly predictive signals of safety-critical response outcomes, establishing an empirical basis for proactive, rather than reactive, safety enforcement. Experiments across models ranging from 3B to 8B parameters under both benign and adversarial fine-tuning regimes show that our framework preserves alignment throughout training while maintaining competitive downstream utility. Our results suggest that the safety-utility trade-off commonly associated with alignment-preserving fine-tuning can be substantially reduced through adaptive regularization.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
large language models, AI safety, safety alignment, harmful fine-tuning, alignment-preserving fine-tuning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_jgVmOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Adaptive Regularization (A-Reg) to curb safety degradation during LLM fine-tuning. A frozen probe direction gives a per-token log-probability margin, which becomes an example-level control score and is combined with a reference-model token preference to set per-token SFT/KL loss weights before LoRA training. The method rests on the claim that pre-generation hidden states linearly predict safety outcomes.

Critical strengths

Testing identity-shifting and backdoor-trigger attacks is broader than evaluating only standard harmful SFT.

The data partitioning for probe fitting, direction selection, and evaluation is carefully designed and helps avoid leakage in the activation-probing pipeline.

Critical weaknesses

I am concerned about specific numbers that are bolded as wins when they are ties. In Table 1, Qwen2.5-3B-Instruct shows Surgery =7.33=7.33 and A-Reg (Ours) =7.33=7.33 bit-identical to two decimal places - yet A-Reg is the only one bolded, visually implying a win that does not exist. The same thing happens in Table 5: Phi-3.5-mini-instruct shows Surgery =69.60=69.60 and A-Reg (Ours) =69.60=69.60, again bolded as if A-Reg wins when it is a tie. Two independent tables, two different metrics (ASR and GSM8K accuracy), and both times the ``winning'' bolded number is bit-identical to the strongest baseline, how do two separately-trained adapters land on the exact same value to two decimal places, and why bold one over the other?

The baseline set shown looks selective, and the strongest baseline disappears exactly where it matters most. Table 1 compares against five baselines, and Surgery is A-Reg's closest competitor there (it ties A-Reg on Qwen2.5-3B, 7.337.33 vs.\ 7.337.33). Table 2 drops SPARD and DSS entirely, keeping only Surgery; Tables 3 and 4 (identity-shifting and backdoor attacks, the hardest, most adversarial settings) drop Surgery too, leaving A-Reg compared only against vanilla SFT and C-SFT. Why does the strongest baseline disappear exactly in the two attack settings where its relative performance against A-Reg would be most informative?

The core assumption behind the per-token preference threshold is acknowledged as unrealistic in Section 3.3, but is not empirically tested. There is no analysis showing whether the trained model actually follows the target token distributions, nor an ablation testing cases where the single-token assumption is badly violated. Since the method's allocation rule relies on this assumption, this seems important to validate.

Questions for authors

What happened on Qwen2.5-7B HEx-PHI in Table 2? Transcription error, or a genuine failure mode?

Given Fig.\ 5, what does the adaptive per-token mechanism actually buy over the static grid-search optimum, across all four models and all of Tables 2-4?

Reviewer_kRtYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes A-Reg., which uses frozen activation-intervention signals and reference-model preferences to assign example- and token-specific weights to SFT and KL regularization. Experiments across four models show improved safety retention under harmful fine-tuning, identity shifting, and backdoor attacks, with utility evaluated on GSM8K and Alpaca.

Critical strengths
  1. The method provides an explicit mechanism for selectively allocating regularization, with no additional controller required at inference.

  2. Safety improvements over standard SFT are substantial and evaluated across multiple model families and attack settings.

Critical weaknesses

The claimed safety–utility improvement is not adequately separated from reduced adaptation. In Table 5, A-Reg. remains below the initial model on GSM8K for three of four models: 86.66→69.60, 79.38→77.18, and 89.16→80.59. Qwen2.5-3B is a meaningful exception. For most models, the results therefore demonstrate mitigation of fine-tuning-induced degradation rather than successful task improvement with safety preservation. Earlier stopping, smaller learning rates, or stronger fixed regularization could potentially achieve similar outcomes. Joint safety–utility trajectories and comparisons at matched task performance are needed. Furthermore, GSM8K and Alpaca provide limited evidence of practical usefulness, particularly without evaluating over-refusal on benign, safety-sensitive requests.

Insufficient comparison with the extensive existing literature. Safety degradation during fine-tuning has attracted considerable attention over the past two years, producing numerous defenses involving safety-data mixing, representation constraints, data reweighting, and constrained optimization. Although the paper discusses many relevant methods, its experiments compare against only a limited subset, with even fewer baselines in the utility and structured-attack evaluations. Broader comparisons with representative methods, such as Vaccine, RepNoise, LISA, Safe LoRA, SWAT, ProCon, and AsFT, under consistent training and evaluation settings are needed to establish the method’s empirical advantage.

Questions for authors

Does A-Reg. outperform early-stopped SFT and tuned fixed-KL regularization at matched task accuracy? Please include the initial model and checkpoint-level safety–utility curves.

Please clarify the coefficient clipping discrepancy between Section 3.3 and Appendix A.5, the bolded 0.65 entry in Table 2, and the claimed utility improvements despite predominantly lower scores than SFT in Table 8.

Can you isolate the benefit of adaptive allocation through example-only, constant-weight, and shuffled-weight controls, and evaluate over-refusal on benign, safety-sensitive requests?

Reviewer_XBwjOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes A-Reg, a training framework that uses frozen activation-based safety signals to adaptively allocate regularization across training examples and individual target tokens, aiming to preserve safety alignment while supporting downstream adaptation. It combines the effects of a model-specific activation intervention with reference-model token preferences to construct weights that balance supervised learning and KL regularization. Experiments across four instruct models show improved safety retention under harmful and benign fine-tuning.

Critical strengths

● The paper addresses a key challenge in LLM fine-tuning: adapting models to downstream tasks while preserving safety alignment.

● The proposed method introduces no additional inference-time overhead.

Critical weaknesses

● The paper’s novelty appears limited. DSS [1] already uses token-level safety signals to balance supervised learning and KL regularization, while TOSS [2] uses token-level loss differences between models to identify unsafe tokens. Although A-Reg introduces activation-intervention-based scoring and a specific regularization allocation rule, their distinct contribution beyond these existing approaches remains insufficiently established.

● The paper lacks independent ablations of the activation intervention, example-level control, and token-level weight allocation, making it difficult to assess each component’s contribution to the reported gains.

● The methodology presentation is not sufficiently self-contained. For example, Sec. 3.1 introduces the intervention direction uu without adequately explaining how it is obtained or what it represents. Deferring such essential information to the appendix makes the method harder to follow. Sec. 2 also provides only a brief discussion of related work. There appears to be room within the nine-page main-text limit to clarify these essential points.

● The paper contains several inconsistencies. Sec. 3.3 allows token coefficients to exceed βmax⁡\beta_{\max}, whereas Appendix A.5 states that they are clamped to [0.1,0.9][0.1, 0.9]. Table 2 highlights A-Reg’s HEx-PHI score of 0.65 as best for Qwen2.5-7B, despite Surgery’s lower score of 0.102. In addition, Appendix A.4 mentions "our PROBE-Bench" without introducing the dataset.

[1] Peng et al. Shape it Up! Restoring LLM Safety during Finetuning. NeurIPS 2025.

[2] Li et al. Token-level Data Selection for Safe LLM Fine-tuning. ICLR 2026.

Questions for authors

● How does A-Reg perform with example-only or token-only weighting, random intervention directions, and fixed KL with matched average regularization strength?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.