acceptodds
Under review as a conference paper at ICLR 2027

AIRP: Alignment-Induced Representation Protection against Harmful Fine-Tuning

Abstract

Fine-tuning-as-a-service adapts large language models to downstream tasks, but a small fraction of harmful examples can erode safety alignment. Current fine-tuning-stage defenses seek to preserve safety alignment through data selection, safety supervision, or constraints on model updates and representations. However, selectively applying protection remains challenging because the effects of harmful fine-tuning on safety alignment are uneven. Our analyses show that harmful examples can differ in their effects on model safety after fine-tuning, with the associated representation changes distributed unevenly across model layers. Taken together, these findings motivate Alignment-Induced Representation Protection (AIRP) for fine-grained safety preservation during downstream fine-tuning. AIRP selectively adjusts task supervision across examples and adaptively protects representations across layers, while direct refusal supervision further reinforces the model's refusal behavior. Across tasks, model architectures, harmful-data ratios, and fine-tuning set sizes, AIRP improves safety retention while maintaining competitive downstream performance.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Large Language Model Safety, Harmful Fine-Tuning, Defense, Safety Alignment, Directional Consistency

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ZbaAOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes the Alignment-Induced Representation Protection (AIRP) framework, which can maintain the model's security alignment capability during the downstream fine-tuning of large language models using data with security risks. Its core idea is to construct alignment-related reference directions by leveraging the representation differences between the base model and the aligned model at each layer. AIRP consists of three modules: (1) risk-aware task weighting based on directional consistency, (2) base-directed representation protection with adaptive layer weighting, and (3) refusal-anchor supervision. The experimental results demonstrate promising safety retention across multiple downstream tasks, model architectures, harmful-data ratios, and fine-tuning dataset sizes.

Critical strengths
  • The paper addresses safety degradation during downstream customization, an important challenge for fine-tuning-as-a-service.
  • Constructing layer-wise reference directions from Base-to-Aligned representation differences offers an interpretable way to characterize alignment-related changes.
  • AIRP achieves consistently low Harmful Scores while maintaining competitive downstream task accuracy. The results under increasing fine-tuning dataset sizes are particularly encouraging.
Critical weaknesses
  • The method assumes that representation differences between the Base and Aligned Models capture safety-relevant directions. However, these differences may also reflect general instruction-following, response formatting, or other changes induced during alignment training. Although the direction-specific control experiments in Appendix D.2 are useful, they do not fully disentangle safety-specific changes from general fine-tuning-induced representation changes. Additional controls using non-safety instruction-tuned checkpoints would strengthen this interpretation.
  • Risk-Aware Task Weighting estimates each example's risk using prompt-only representations. Consequently, a harmful prompt paired with a safe refusal and the same prompt paired with a harmful compliant response receive identical task weights. This raises concerns about whether directional consistency truly measures the harmfulness of the supervision signal. A controlled experiment using matched prompts with different target responses would help clarify this issue.
  • The adaptive layer weights are derived from normalized representation degradation scores from previous optimization steps. While the ablation results demonstrate incremental improvements, it remains unclear whether the gains arise from the proposed adaptive mechanism or from emphasizing layers that already exhibit larger regularization losses. Comparisons with uniform weighting, static layer importance, and EMA-based weighting would be informative.
  • AIRP introduces substantial training overhead, nearly doubling total training time relative to standard SFT in the reported setting. Moreover, some baselines use different training epochs and LoRA configurations, and the methods differ in their use of safety reference data. Matched-budget comparisons and safety–utility curves under equivalent computational and safety-data budgets would make the empirical findings more convincing.
Questions for authors
  1. Can the authors demonstrate that the reference directions specifically capture safety alignment rather than general instruction-tuning-induced representation changes?

  2. Why is risk weighting computed solely from prompts, despite harmful fine-tuning being driven by potentially unsafe target responses?

  3. How does adaptive layer weighting compare with static or smoothed weighting schemes under identical regularization budgets?

  4. Can the authors evaluate over-refusal rates and safety performance using an independent evaluator?

10 comments
AuthorsOpenReview ↗

Author Rebuttal (I): Testing the Safety Relevance of Reference Directions with a General Instruction-Tuning Control

We thank the reviewer for the constructive feedback.

W1 + Q1: Safety Relevance of the Reference Directions

To test whether the reference directions capture safety-relevant representation changes beyond those induced by general instruction tuning, we construct a control using Dolly, a dataset of general-purpose instructions paired with responses. We fine-tune the Llama-3-8B Base Model on 5,000 Dolly examples to obtain a general instruction-tuned checkpoint. We then estimate directions for this checkpoint and the Aligned Model relative to the same Base Model, using the same 500 harmful reference prompts and the same estimation procedure.

We use this control in two experiments. At inference time, we compare the effects of steering along the two directions on harmful responses. During downstream fine-tuning, we replace the reference directions in AIRP with the Dolly control directions to test whether they provide comparable safety protection.

1. Activation Steering at Inference Time

We test whether the direction derived from safety alignment has a stronger effect on harmful responses than the direction derived from general instruction tuning. We apply both directions to the same model after downstream fine-tuning. At inference time, we shift the last-prompt-token representation at the final Transformer layer by ±γ\pm\gamma times the corresponding unit vector. The parameter γ\gamma controls the intervention strength, with γ=0\gamma=0 denoting the unmodified model. Each column compares the two directions at the same strength.

Effect of steering along and against the reference and Dolly control directions. Entries report Harmful Score (HS, %), with signed changes from γ=0\gamma=0 in parentheses (percentage points).

DirectionOperationγ=0\gamma=0γ=1\gamma=1γ=2\gamma=2γ=3\gamma=3γ=4\gamma=4γ=5\gamma=5
Reference directionAdd71.8071.00 (-0.80)69.80 (-2.00)69.10 (-2.70)68.20 (-3.60)67.70 (-4.10)
Reference directionSubtract71.8073.30 (+1.50)73.90 (+2.10)73.70 (+1.90)74.10 (+2.30)74.70 (+2.90)
Dolly control directionAdd71.8071.70 (-0.10)71.70 (-0.10)71.70 (-0.10)71.30 (-0.50)71.20 (-0.60)
Dolly control directionSubtract71.8072.30 (+0.50)72.40 (+0.60)72.50 (+0.70)72.50 (+0.70)72.70 (+0.90)

The reference direction produces a larger absolute change in HS for both operations at every nonzero intervention strength. At γ=5\gamma=5, adding it reduces HS by 4.10 points and subtracting it increases HS by 2.90 points. The corresponding changes for the Dolly direction are 0.60 and 0.90 points. With the estimation prompts and intervention conditions matched, the Dolly direction reproduces only a small fraction of the reference direction's effect on harmful responses.

2. Direction Replacement During Downstream Fine-Tuning

We test whether the stronger behavioral effect also translates into better safety preservation when the directions are used during training. All runs start from the same Aligned Model. We substitute the Dolly control directions for the reference directions in AIRP's task weighting and representation protection. Since Refusal-Anchor Supervision operates independently of the directions, we compare the two sets of directions with anchors enabled and again with anchors disabled.

Effect of replacing the reference directions with Dolly directions in AIRP, with and without refusal anchors. HS and Fine-tuning Accuracy (FA) are reported as percentages.

DirectionRefusal anchorsAGNEWS HS ↓\downarrowAGNEWS FA ↑\uparrowGSM8K HS ↓\downarrowGSM8K FA ↑\uparrowSST2 HS ↓\downarrowSST2 FA ↑\uparrow
Reference directionsEnabled4.3086.101.6045.703.1091.06
Dolly control directionsEnabled19.3085.804.6043.209.7092.78
Reference directionsDisabled12.7086.1013.9043.7022.3091.06
Dolly control directionsDisabled47.4085.3043.1045.6050.6091.40

With refusal anchors enabled, replacing the reference directions with the Dolly directions increases HS by 15.00, 3.00, and 6.60 percentage points on AGNEWS, GSM8K, and SST2, respectively. With refusal anchors disabled, the corresponding increases are 34.70, 29.20, and 28.30 points. The reference directions therefore provide stronger safety protection during fine-tuning across all three tasks under both matched anchor settings.

Taken together, the steering and direction-replacement experiments indicate that the reference directions capture safety-relevant representation changes beyond those captured by general instruction tuning on Dolly. These changes matter both for influencing harmful responses at inference time and for preserving safety during downstream fine-tuning.

AuthorsOpenReview ↗

Author Rebuttal (II): Rationale for Prompt-Only Task Weighting and Evaluation of a Response-Aware Variant

We thank the reviewer for raising this question about target responses in task weighting. We explain our design and provide an additional comparison below.

W2 + Q2: Rationale for Prompt-Only Weighting and Evaluation of a Response-Aware Variant

We first explain why we compute task weights from prompts in the harmful fine-tuning setting studied in this work and why we pair this weighting with Refusal-Anchor Supervision. We then evaluate a variant that incorporates target responses throughout the representation computations to examine their effect on safety preservation and task performance.

1. Why we compute task weights from prompts

As described in the Methodology section, we consider a fine-tuning-as-a-service setting where user-provided data lack sample-level safety annotations and may contain harmful prompt–compliant response pairs. In these pairs, the prompt expresses the harmful request that the target response trains the model to fulfill.

AIRP therefore uses the prompt's alignment-induced representation changes to guide task weighting. Computing representations at the end of the prompt measures the changes that safety alignment induces in the model's representation of the request before it conditions on the target response. This is consistent with the reference directions, which are constructed from the Base-to-Aligned representation changes on harmful prompts.

2. The same prompt paired with different responses

The reviewer correctly observes that a harmful response and a safe refusal paired with the same prompt receive the same task weight. A harmful prompt paired with a safe refusal has the same form as the examples used for safety alignment and provides useful safety supervision.

Our study focuses on safety degradation caused by harmful prompt–compliant response pairs during downstream fine-tuning. Prompt-based weighting reduces the task supervision assigned to potentially harmful requests. However, when such a request is paired with a safe refusal in the downstream data, its useful refusal supervision is also downweighted.

This consideration helps motivate the inclusion of Refusal-Anchor Supervision, whose separate loss directly reinforces safe refusals without being scaled by the task weights. The two objectives therefore work together to reduce potentially harmful task supervision while maintaining an explicit source of safe refusal supervision.

3. Evaluating the inclusion of target responses

We evaluate a response-aware variant of AIRP to assess the effect of including target responses. We use harmful prompt–harmful response pairs to estimate its reference directions and use each downstream example's prompt and actual target response for task-weight computation and representation protection. Thus, reference-direction estimation and downstream representation computations both use prompt–response inputs.

This comparison evaluates the effect of including response information throughout the method. We repeat downstream fine-tuning for the three tasks and compare the resulting models with prompt-only AIRP.

Safety and task accuracy using prompt-only or prompt–response representations (%, HS / FA).

Representation inputAGNEWS HS ↓\downarrowAGNEWS FA ↑\uparrowGSM8K HS ↓\downarrowGSM8K FA ↑\uparrowSST2 HS ↓\downarrowSST2 FA ↑\uparrow
Prompt only (AIRP)4.3086.101.6045.703.1091.06
Prompt + response27.9084.304.8044.3011.7093.12

Including target responses increases HS by 23.60, 3.20, and 8.60 percentage points on AGNEWS, GSM8K, and SST2, respectively. FA decreases by 1.80 and 1.40 points on AGNEWS and GSM8K, while increasing by 2.06 points on SST2.

In the evaluated settings, prompt-only AIRP achieves lower HS than the response-aware variant on all three tasks and higher FA on two of them. These results favor computing alignment-related representations from prompts in this setting, while AIRP uses refusal supervision to directly reinforce safe responses.

AuthorsOpenReview ↗

Author Rebuttal (III): Safety Gains from Layer Prioritization and Adaptive Weight Updates

We thank the reviewer for asking what drives the gains from Adaptive Layer Protection.

W3 + Q3: Comparison with Uniform, Static, and EMA Layer Weighting

To determine whether the safety gains come from giving greater weight to layers with higher degradation or also from updating those weights during training, we compare four weighting rules using Llama-3-8B with AGNEWS as the downstream task.

  • Uniform. We assign each of the LL Transformer layers a weight of gl,(t)=1/Lg^{l,(t)}=1/L at every step. This provides a baseline with equal protection weights across layers.

  • Static. To test whether prioritizing layers with higher degradation is sufficient to obtain the safety gains, we first fine-tune the Aligned Model for one epoch using only the weighted task loss. We then evaluate the resulting checkpoint using the layer score in Eq. (5).

    \frac{ \sum_{(x_i,y_i)\in\mathcal{B}^{(t)}} w_i\bigl(e_i^{\,l,(t)}\bigr)^2 }{ \sum_{(x_i,y_i)\in\mathcal{B}^{(t)}} w_i \mathbb{I}\!\left[a_i^{\,l}>\epsilon\right]+\epsilon }.$$ Here, $\mathcal{B}^{(t)}$ is the batch at step $t$, $w_i$ is the sample weight, and $a_i^{\,l}$ is the reference displacement along the layer's reference direction. We pool the numerator and denominator over the full training set at the one-epoch checkpoint and normalize the resulting scores across layers. The Static run starts again from the Aligned Model and uses these fixed weights throughout fine-tuning.
  • EMA. To test whether smoothing the layer scores improves adaptive weighting, we update the smoothed score after step tt as ml,(t)=βml,(t−1)+(1−β)cl,(t)m^{l,(t)}=\beta m^{l,(t-1)}+(1-\beta)c^{l,(t)}. Larger β\beta gives more weight to the previous smoothed score. At step tt, we normalize ml,(t−1)m^{l,(t-1)} across layers.

  • Adaptive (Ours). We compute the layer weights by normalizing the preceding step's scores across layers. This corresponds to β=0\beta=0 in the EMA formulation.

For the comparison below, all four runs start from the same Aligned Model and use the same 1,000 training examples, 10% of which are harmful. The layer-score definition and representation-loss coefficient λrep\lambda_{\mathrm{rep}} are identical across runs. We disable Refusal-Anchor Supervision in all four runs to assess the contribution of layer weighting separately from direct refusal supervision.

Harmful Score and fine-tuning accuracy for each layer-weighting rule, with AGNEWS as the downstream task (%).

Layer weightingHS ↓\downarrowFA ↑\uparrow
Uniform17.2085.60
Static13.1086.00
EMA (β=0.1\beta=0.1)13.2086.00
EMA (β=0.3\beta=0.3)13.6085.90
EMA (β=0.5\beta=0.5)13.4085.90
EMA (β=0.7\beta=0.7)13.5085.70
EMA (β=0.9\beta=0.9)13.6085.50
Adaptive (Ours)12.7086.10

Giving greater weight to layers with higher degradation yields a clear safety gain. Static reduces HS by 4.10 percentage points relative to Uniform while maintaining comparable FA. Updating the layer weights during training adds a safety benefit beyond that of assigning fixed priorities to layers. Static already gives greater weight to layers with higher degradation at calibration, yet Adaptive reduces HS by a further 0.40 points at comparable FA (86.10% vs. 86.00%). Adaptive also achieves HS 0.50–0.90 points below the tested EMA variants at comparable FA. Taken together, these results indicate that most of the safety gain from Adaptive Layer Protection comes from concentrating protection on layers with greater representation degradation, with a further gain from adjusting that allocation as training progresses.

Reviewer_WbdNOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies how to preserve the safety alignment of LLMs when downstream fine-tuning data contain a small fraction of harmful prompt–compliant response pairs. It proposes Alignment-Induced Representation Protection (AIRP), which uses layer-wise representation differences between a Base Model and an Aligned Model on harmful reference prompts to construct alignment-related directions. These directions are then used to downweight potentially risky training examples, penalize representation changes toward the Base Model, and adaptively emphasize layers exhibiting greater alignment degradation. An additional refusal-anchor objective reinforces safe refusal behavior.

Critical strengths

The paper is well written, and the experiments cover multiple tasks, model architectures, harmful-data ratios, and fine-tuning settings. The ablation and direction-control studies also provide useful evidence regarding the contributions of the main components.

Critical weaknesses
  1. The motivation experiment is reasonable to some extent, but the semantic interpretation of the reference directions remains insufficiently validated. The authors feed 500 harmful prompts into the Base and Aligned Models and define the average layer-wise representation difference as the alignment-derived reference direction. However, this construction uses only prompt-side hidden states and does not verify whether the Aligned Model successfully refuses these prompts. The resulting direction may therefore encode not only safety-alignment effects but also dataset-specific styles, topic categories, and content features. It cannot yet be directly interpreted as a safety direction associated with refusal behavior. Accordingly, the directional consistency score only measures how closely a new prompt’s Base-to-Aligned representation change resembles the average change observed for the reference harmful prompts. Although Figure 2 shows a correlation between consistency score and subsequent safety degradation, this relationship may be confounded by the shared data distribution, harmful-content categories, and content severity. The current evidence is therefore insufficient to establish that consistency score accurately captures the safety risk of an individual training example.
  2. Regarding the method design, pip_i measures how closely a prompt is associated with the safety-alignment directions rather than the actual risk of the complete prompt–response training example. Figure 2 provides preliminary empirical evidence that pip_i can rank relative risk among examples already known to be harmful. However, AIRP directly treats pip_i as the risk score of the complete training example and uses the linear function 1−wi1-w_i to determine its supervision strength. This mapping lacks sufficient theoretical justification.
  3. AIRP introduces Refusal-Anchor Supervision during downstream training and uses an additional 1,000 harmful prompt–safe refusal pairs as response-level safety supervision. In contrast, major baselines such as SBR and AsFT are not provided with the same information budget, which raises concerns about the fairness of the comparison. In the default AGNEWS experiment, the complete AIRP method achieves an HS of 4.3%, but removing Refusal-Anchor Supervision increases the HS to 12.7%, which is higher than the 6.3% of SBR and the 8.3% of AsFT. This result suggests that AIRP’s advantage over the strongest baselines may depend substantially on the additional refusal supervision.
  4. Unlike many studies of harmful fine-tuning that directly evaluate publicly available instruction-tuned or chat models, this work constructs its own Aligned Model from the Base Model using 5,000 harmful prompt–safe refusal pairs. The authors should further explain why this custom alignment procedure is necessary and report the HS and downstream task performance of the resulting Aligned Model before any downstream fine-tuning. These results are needed to establish the model’s initial safety and utility levels and to quantify the actual improvements or degradation caused by subsequent fine-tuning. The current utility evaluation primarily covers classification tasks such as AGNEWS, SST2, and GSM8K. The authors should include at least one open-ended generation or instruction-following benchmark, such as AlpacaEval, to more comprehensively assess whether AIRP preserves general helpfulness and generation quality while maintaining safety.
  5. Finally, AIRP introduces substantial computational and GPU memory overhead, which appears to conflict with its intended Fine-Tuning-as-a-Service setting. In the default 8B-model experiment, AIRP requires 106.21 minutes of total training time, nearly twice that of standard SFT and considerably more than SBR and AsFT. This overhead raises concerns about the method’s practicality and scalability when repeatedly serving downstream customization requests.
Questions for authors
  1. Can the authors verify whether the reference directions are specifically associated with successful refusal behavior rather than dataset-specific or topic-related features?
  2. Can the authors report the pip_i distributions for benign and harmful training examples and justify the weighting function 1−wi1-w_i?
  3. How does AIRP compare with SBR and AsFT when all methods are given the same refusal-supervision budget?
8 comments
AuthorsOpenReview ↗

Author Rebuttal (I): Reference Directions, Successful Refusal Behavior, and Evidence Beyond the Reference Dataset

We thank the reviewer for the constructive feedback.

W1 + Q1: Interpretation and Behavioral Validation of the Reference Directions

We clarify the scope and role of directional consistency, assess the reference directions' association with successful refusal through refusal evaluation and controlled steering, and test whether their effectiveness is dataset-specific. We report the cross-dataset results in the expanded Appendix C.3 and new Appendix C.4.

1. Claim and role of directional consistency

Our work does not claim that directional consistency accurately captures the safety risk of an individual training example. The Introduction states that it “can distinguish harmful-example groups that affect model safety differently.” Figure 2 compares models separately fine-tuned on groups of known harmful examples. This claim concerns differences across harmful-example groups.

We agree that shared data distributions, harmful-content categories, and content severity can influence this association. Accordingly, AIRP uses directional consistency as a reference signal for sample weighting alongside representation protection and refusal supervision. Representation protection combines sample weights with Base-directed changes measured during training, while Refusal-Anchor Supervision directly reinforces safe responses.

2. Successful refusal of the reference prompts

To verify whether the Aligned Model successfully refuses the reference prompts, we evaluate its responses to all 500 prompts before downstream fine-tuning. WildGuard classifies 481 responses as refusals (96.2%) and 4 as harmful (0.8%). Beaver-Dam-7B classifies 1.0% as harmful. The Aligned Model successfully refuses most reference prompts.

3. Association with successful refusal through controlled activation steering

To further examine whether the reference directions are associated with successful refusal behavior, we compare them with directions derived from general instruction tuning. We fine-tune the Base Model on 5,000 examples from Dolly, a general instruction-following dataset. We estimate both sets of directions relative to the same Base Model using the same 500 harmful prompts and the same procedure.

After downstream fine-tuning, we add or subtract each direction from the same model's representations at inference time. The intervention strength is γ\gamma, with γ=0\gamma=0 denoting no intervention.

HS (%) under activation steering with safety-alignment and general instruction-tuning directions. Parentheses show changes from γ=0\gamma=0 in percentage points.

DirectionInterventionγ=0\gamma=0γ=1\gamma=1γ=2\gamma=2γ=3\gamma=3γ=4\gamma=4γ=5\gamma=5
Reference directionAdd71.8071.00 (−0.80)69.80 (−2.00)69.10 (−2.70)68.20 (−3.60)67.70 (−4.10)
Reference directionSubtract71.8073.30 (+1.50)73.90 (+2.10)73.70 (+1.90)74.10 (+2.30)74.70 (+2.90)
Dolly control directionAdd71.8071.70 (−0.10)71.70 (−0.10)71.70 (−0.10)71.30 (−0.50)71.20 (−0.60)
Dolly control directionSubtract71.8072.30 (+0.50)72.40 (+0.60)72.50 (+0.70)72.50 (+0.70)72.70 (+0.90)

At γ=5\gamma=5, the reference direction changes HS by −4.10/+2.90 points, compared with −0.60/+0.90 for the Dolly control. With the same reference prompts, the safety-alignment direction has a substantially stronger effect on harmful responses than the Dolly direction. Together with the verified refusals, this result indicates an association between the reference directions and successful refusal behavior.

4. Testing whether the reference directions are dataset-specific

  • Generalization across harmful training sources. Keeping the reference directions estimated from BeaverTails fixed, we conduct separate fine-tuning runs with harmful examples from BeaverTails, AdvBench, LLM-LAT, or Circuit Breakers. AIRP achieves the lowest HS for three of four sources with AGNEWS as the downstream task and all four with GSM8K, averaging 3.40% and 1.40%, respectively. The same BeaverTails-derived directions thus help protect against harmful updates from other datasets. We report the complete results in Appendix C.4.

  • Robustness on external test sets after fine-tuning. We evaluate models fine-tuned using BeaverTails as the harmful training source on Harmful HEx-PHI and PureBad, without further training. AIRP achieves the lowest average HS (1.97%), ranking first on PureBad and second on Harmful HEx-PHI. Safety preservation thus extends beyond the dataset used for reference-direction estimation and harmful fine-tuning. We report the complete results in Appendix C.3.

Together, these results show that the reference directions are associated with successful refusal behavior and remain effective when harmful training examples or test prompts come from datasets other than the reference dataset.

AuthorsOpenReview ↗

Author Rebuttal (II): Directional Consistency Distributions and the Rationale for Linear Task Weighting

We thank the reviewer for asking about the directional consistency distributions and the choice of weighting function.

W2 + Q2: Directional Consistency Distributions and the Choice of Task Weights

We report the distributions of directional consistency for benign and harmful training examples to examine whether the score provides a basis for assigning different supervision strengths. We then explain the choice of 1−wi1-w_i and compare it with two nonlinear mappings.

1. Directional consistency in benign and harmful training examples

We analyze 1,000 examples per downstream task to mirror a fine-tuning dataset with a harmful-data ratio of 0.1. Each dataset contains 900 benign examples and 100 harmful examples, with the same harmful examples used across the three tasks. The score pip_i measures the agreement between a prompt's Base-to-Aligned representation changes and the reference directions.

Directional consistency distributions for benign and harmful training examples.

Training examplesCountMean ± SD25th–75th percentiles
Benign (AGNEWS)9000.5530 ± 0.02140.5447–0.5617
Benign (GSM8K)9000.6967 ± 0.01830.6845–0.7099
Benign (SST2)9000.6198 ± 0.01710.6085–0.6305
Harmful (shared across tasks)1000.8250 ± 0.03640.8022–0.8540

The harmful examples have a higher mean pip_i than the benign examples in every task. Their 25th percentile (0.8022) is also above the 75th percentile of each benign group. These distributions indicate that higher directional consistency can serve as a signal to reduce task supervision in the evaluated training mixtures.

2. Rationale for the linear mapping 1−wi1-w_i

AIRP sets wi=clip⁡(pi,0,1)w_i=\operatorname{clip}(p_i,0,1) and multiplies each example's task loss by 1−wi1-w_i. The linear mapping makes the coefficient decrease continuously from 1 to 0 as wiw_i increases, so equal increases in wiw_i produce equal reductions in the coefficient. This gives a direct way to translate differences in directional consistency into differences in supervision strength without introducing a threshold or a parameter that controls the curvature of the mapping.

3. Comparison with nonlinear weighting functions

To assess whether a stronger or weaker reduction in task supervision improves the balance between safety and task accuracy, we compare 1−wi1-w_i with the squared mapping (1−wi)2(1-w_i)^2 and the square-root mapping 1−wi\sqrt{1-w_i}. All three mappings preserve the ordering of examples by supervision strength. For coefficients between 0 and 1, squaring reduces supervision more than the linear mapping, while taking the square root retains more.

Harmful Score and Fine-tuning Accuracy under three task-weighting functions (%). The best result in each column is in bold, and the second-best result is underlined.

MappingAGNEWS HS ↓AGNEWS FA ↑GSM8K HS ↓GSM8K FA ↑SST2 HS ↓SST2 FA ↑
Squared3.1085.001.3042.002.1090.48
Square root12.1085.90‾\underline{85.90}2.7046.406.7091.40
Linear (Ours)4.30‾\underline{4.30}86.101.60‾\underline{1.60}45.70‾\underline{45.70}3.10‾\underline{3.10}91.06‾\underline{91.06}

Relative to linear weighting, squared weighting lowers HS by 0.30–1.20 percentage points across the three tasks, with lower FA in each case. The largest accuracy cost occurs on GSM8K, where HS improves by 0.30 points and FA drops by 3.70 points. Square-root weighting raises HS on all three tasks. On AGNEWS, for example, it increases HS from 4.30% to 12.10%, with FA decreasing slightly from 86.10% to 85.90%.

Linear weighting offers a balance between safety and task accuracy in the evaluated setting. It preserves more task accuracy than squared weighting across all three tasks and achieves lower HS than square-root weighting across all three. The comparison shows why the degree of attenuation matters even when the ordering of examples is unchanged.

AuthorsOpenReview ↗

Author Rebuttal (III): Safety Preservation with Matched Access to Refusal Supervision

We thank the reviewer for raising the concern about additional refusal supervision and the fairness of the comparison.

W3 + Q3: Comparison Under the Same Refusal Supervision Budget

We compare AIRP with SBR and AsFT to test whether AIRP retains its safety advantage when the baselines receive the same refusal supervision budget.

1. Comparison with the same additional refusal data

To determine whether the use of additional refusal supervision explains AIRP's advantage over SBR and AsFT, we add AIRP's refusal supervision objective to both baselines and rerun downstream fine-tuning for each augmented method. Both baselines receive the same 1,000 harmful prompt–safe refusal pairs used by AIRP, with the same anchor sampling settings and refusal loss coefficient. Each baseline retains its own defense mechanism. In the table, +A+A denotes the addition of Refusal-Anchor Supervision during fine-tuning.

Safety and task accuracy before and after adding the same refusal supervision to SBR and AsFT (%).

MethodConferenceAGNEWS HS ↓AGNEWS FA ↑GSM8K HS ↓GSM8K FA ↑SST2 HS ↓SST2 FA ↑
SBRICML 20266.3083.304.3038.008.8091.97
SBR + A6.0083.302.8037.709.0092.20
AsFTAAAI 20268.3087.6015.2040.0013.1094.84
AsFT + A4.4087.209.2042.306.6094.61
AIRP (Ours)—4.3086.101.6045.703.1091.06

Adding refusal supervision reduces AsFT's average HS across the three tasks from 12.20% to 6.73% and SBR's from 6.47% to 5.93%.

Under the same refusal supervision budget, AIRP achieves lower HS than both augmented baselines on all three tasks. Its average HS is 3.00%, compared with 5.93% for SBR + A and 6.73% for AsFT + A. On GSM8K, AIRP also achieves higher task accuracy, with 45.70% FA compared with 37.70% and 42.30%, respectively. On AGNEWS and SST2, AsFT + A retains higher FA, while AIRP achieves lower HS.

Refusal supervision itself contributes to safety preservation. Combining it with AIRP's sample weighting and representation protection achieves stronger safety preservation than adding the same supervision to either baseline.

Reviewer_NW1YOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes AIRP, a framework for preserving LLM safety during downstream fine-tuning with potentially harmful data. AIRP combines risk-aware task weighting, base-directed representation protection, adaptive layer protection, and refusal-anchor supervision. Experiments across multiple tasks and model architectures confirms improved safety retention while maintaining competitive downstream performance.

The paper addresses an important problem and presents a promising framework. However, the novelty of the combined design and the practical costs of the approach require further clarification.

Critical strengths
  1. The paper presents broad exmperimental evaluation. The experiments of the paper cover three model architectures, multiple downstream tasks, different harmful-data ratios, and varying fine-tuning set sizes.
  2. The paper studies the important problem of safety degradation during downsream fine-tuning. Its analysis provides insights into why different harmful examples have different effects on model safety.
Critical weaknesses
  1. The relationship between the proposed components is not sufficiently clear. In particular, the Refusal-Anchor Supervision mechanism in Section 3.3 appears somewhat disconnected from the preceding sample-wise and layer-wise protection strategies guided by safety directions.The paper does not adequately explain how refusal-anchor supervision complements these representation-level mechanisms or how the components interact to achieve the overall safety-preservation objective.
  2. The proposed AIRP combine sample reweighting, representation regularization, adaptive layer weighting, and refusal supervision. Although combination is effective, the paper could more clearly distinguish its methodological contribution from existing works.
  3. AIRP requires access to both base and aligned checkpoints, and additional computations during fine-tuning. However, the paper does node characterize the resulting computational overhead.
Questions for authors
  1. The relationship between the proposed components is not sufficiently clear. In particular, the Refusal-Anchor Supervision mechanism in Section 3.3 appears somewhat disconnected from the preceding sample-wise and layer-wise protection strategies guided by safety directions.The paper does not adequately explain how refusal-anchor supervision complements these representation-level mechanisms or how the components interact to achieve the overall safety-preservation objective.
  2. The proposed AIRP combine sample reweighting, representation regularization, adaptive layer weighting, and refusal supervision. Although combination is effective, the paper could more clearly distinguish its methodological contribution from existing works.
  3. AIRP requires access to both base and aligned checkpoints, and additional computations during fine-tuning. However, the paper does node characterize the resulting computational overhead.
4 comments
AuthorsOpenReview ↗

Author Rebuttal (I): The Role of Refusal Supervision Alongside Task Weighting and Representation Protection

We thank the reviewer for asking us to clarify the relationship between AIRP's components.

W1 + Q1: The Relationship Between Task Weighting, Representation Protection, and Refusal Supervision

We first explain the roles of task weighting, representation protection, and refusal supervision in AIRP. We then use ablations to test whether representation protection and refusal supervision each improve safety in the presence of the other. Finally, we test whether AIRP achieves lower HS when SBR and AsFT are trained with the same refusal data and supervision objective as AIRP.

1. Why the objectives work together

Task weighting reduces downstream supervision for examples with higher directional consistency, while representation protection constrains the Base-directed changes that still arise during training. Their shared reference directions and sample weights link each example's contribution to task learning with the protection of its representations.

Refusal-Anchor Supervision trains the model to respond to harmful requests with safe refusals during downstream fine-tuning. Its loss is independent of the task weights, so the supervision on refusal anchors retains its strength when task weighting reduces downstream supervision.

Task weighting limits harmful learning pressure, representation protection constrains alignment degradation, and refusal supervision reinforces safe refusals to harmful requests.

2. Component ablations testing their joint contribution

We compare individual components and their combinations using Llama-3-8B with AGNEWS as the downstream task, 1,000 training examples, and a harmful-data ratio of 0.1. Representation protection includes Adaptive Layer Protection. Disabling task weighting restores the unweighted task loss while retaining the sample weights within representation protection.

Individual and combined contributions to safety preservation and task accuracy (%).

Task weightingRepresentation protectionRefusal supervisionHS ↓\downarrowFA ↑\uparrow
———71.8084.30
✓——39.2083.60
——✓54.8084.70
✓—✓9.6083.10
✓✓—12.7086.10
—✓✓38.1086.20
✓✓✓4.3086.10

Task weighting and refusal supervision together reduce HS to 9.60%, compared with 39.20% for task weighting alone and 54.80% for refusal supervision alone. Using the two objectives together improves safety over either objective alone.

The ablations below clarify the relationship between AIRP's components by removing one component at a time.

  • Adding refusal supervision to task weighting and representation protection reduces HS from 12.70% to 4.30%, with FA remaining at 86.10%. Refusal supervision adds protection beyond constraining representation changes.
  • Adding representation protection to task weighting and refusal supervision reduces HS from 9.60% to 4.30% and increases FA from 83.10% to 86.10%. Preserving alignment-related representations remains beneficial even when safe refusals are directly supervised.
  • Removing task weighting from AIRP increases HS from 4.30% to 38.10%, showing the importance of also regulating downstream task supervision.

We report these results in Appendix E.

3. Comparison under the same refusal-supervision budget

To test whether AIRP's advantage follows simply from adding refusal supervision, we repeat downstream fine-tuning with SBR and AsFT augmented with the same refusal objective. Both receive the same 1,000 harmful prompt–safe refusal pairs and refusal-supervision settings as AIRP.

Safety and task performance with and without refusal supervision (HS / FA, %).

Refusal supervisionMethodConferenceAGNEWSGSM8KSST2
—SBRICML 20266.30 / 83.304.30 / 38.008.80 / 91.97
—AsFTAAAI 20268.30 / 87.6015.20 / 40.0013.10 / 94.84
—AIRP—12.70 / 86.1013.90 / 43.7022.30 / 91.06
✓SBR6.00 / 83.302.80 / 37.709.00 / 92.20
✓AsFT4.40 / 87.209.20 / 42.306.60 / 94.61
✓AIRP (Ours)—4.30 / 86.101.60 / 45.703.10 / 91.06

On AGNEWS, AIRP without refusal supervision has a higher HS (12.70%) than SBR (6.30%) and AsFT (8.30%). With the same refusal supervision, AIRP's HS falls by 8.40 percentage points to 4.30%, below SBR (6.00%) and AsFT (4.40%). Adding refusal supervision also reduces HS more for AIRP than for either baseline on GSM8K and SST2. AIRP thus achieves the lowest HS on all three tasks under matched supervision, averaging 3.00% versus 5.93% for SBR and 6.73% for AsFT.

Together with the ablation results, these comparisons show that refusal supervision provides a greater safety benefit when combined with AIRP's task weighting and representation protection than when added to either baseline.

AuthorsOpenReview ↗

Author Rebuttal (II): Methodological Contribution and Distinctions from Existing Fine-Tuning Defenses

We thank the reviewer for asking us to clarify AIRP's methodological contribution.

W2 + Q2: Methodological Contribution and Differences from Existing Defenses

AIRP defines safety preservation relative to the representation changes introduced by safety alignment. This perspective connects sample weighting with selective constraints on layer-wise representation changes, alongside direct refusal supervision. We explain AIRP's methodological contribution and compare its design with those of methods that select training examples, constrain parameter updates, or anchor hidden states.

1. Defining safety preservation through alignment-induced representation changes

AIRP uses the difference between the Base and Aligned Models' representations to identify what alignment has changed at each layer. Aggregating these changes over harmful reference prompts yields layer-wise reference directions, against which downstream representation changes can be measured.

This choice of reference leads to two connected quantities. Directional consistency measures the agreement between a downstream example's alignment-induced changes and the reference directions. Base-directed representation change measures the extent to which subsequent fine-tuning reverses the alignment-induced displacement along those directions. AIRP links the basis for weighting an example to the specific representation changes constrained when learning from that example. The sample weight enters both objectives, and the measured degradation determines the protection allocated across layers.

The resulting constraint is directional, normalized by each example's alignment-induced displacement, and applied across Transformer layers. Refusal-Anchor Supervision directly supervises safe responses alongside this selective constraint. Together, they connect the preservation of alignment-induced representations with the refusal behavior to be maintained.

2. Differences from existing defenses

The comparison below identifies the safety references and intervention mechanisms that distinguish AIRP from closely related methods.

Related approachExisting formulationDistinction in AIRP
Sample-selective defenses, including SEAL, BDS, LARF, and the fine-tuning component of AntibodyRank, filter, or reweight downstream examples to reduce harmful learning.Derives task weights from directional consistency and uses the same sample weights and reference directions in representation protection.
SafeLoRA (NeurIPS 2024)Uses Base-to-Aligned parameter differences to project selected LoRA updates after fine-tuning.Uses Base-to-Aligned representation differences to guide sample weighting and layer-wise protection during fine-tuning.
AsFT (AAAI 2026)Constrains parameter-update components orthogonal to an alignment-derived parameter direction.Constrains Base-directed movement along layer-wise representation directions, relative to each example's alignment-induced displacement.
SBR (ICML 2026)Matches final hidden states on safety-anchor prompts to those of a frozen Aligned Model.Penalizes Base-directed changes in downstream examples' representations across Transformer layers and adapts protection according to the observed degradation.

3. Selective representation protection and hidden-state anchoring

The comparison with SBR illustrates the effect of AIRP's choice of reference on the constraint itself. SBR anchors complete final hidden states. AIRP measures the projection of the difference between the Aligned and current representations onto each layer's reference direction, normalizes it by the example's Base-to-Aligned displacement, and penalizes the squared positive part. This quantifies and constrains the reversal of the alignment-induced displacement.

Consequently, the representation penalty targets movement opposing the reference direction. Orthogonal movement and movement along the reference direction receive no direct penalty from this term. This leaves room for downstream adaptation outside the constrained component, while Refusal-Anchor Supervision directly reinforces safe response behavior.

Adaptive Layer Protection uses the same normalized representation changes to compute layer-wise scores and updates protection weights from the preceding step's scores. Thus, the allocation of protection across layers follows the degradation defined by the representation objective.

4. Summary

AIRP's methodological contribution is to use a shared reference derived from safety alignment to link which examples receive weaker task supervision, which representation changes are penalized, and which layers receive greater protection. Refusal-Anchor Supervision directly reinforces the safe refusal behavior that these objectives aim to preserve. We provide a detailed comparison with SafeLoRA, AsFT, and SBR in Appendix F.

AuthorsOpenReview ↗

Author Rebuttal (III): Checkpoint Requirements and Safety–Utility Benefits at Comparable Training Times

We thank the reviewer for raising this concern about AIRP's checkpoint requirements and computational cost. We clarify these requirements and the benefits of the additional computation below.

W3 + Q3: Checkpoint Access and the Practical Benefits of Additional Computation

To evaluate whether AIRP's additional computation yields better safety and task performance within a comparable training budget, we added Appendix G.2, Safety–Utility at Comparable Fine-tuning Times. We first clarify when the Base and Aligned checkpoints are used, then compare the outcomes attained by different methods within approximately matched elapsed fine-tuning times.

1. Access to the Base and Aligned checkpoints

AIRP assumes that the provider retains both checkpoints. Before downstream fine-tuning, the Base and Aligned Models are used to estimate the reference directions and compute the sample weights and reference displacements. Once these quantities have been obtained, the Base Model is no longer required for downstream training.

Downstream fine-tuning starts from the Aligned Model with a newly initialized Task LoRA. A frozen Aligned Model supplies the reference representations used for representation protection. Thus, access to the Base checkpoint is required for preparation, while the additional computation during training comes from representation protection and refusal supervision.

2. Safety and utility at comparable fine-tuning times

To assess what this training cost delivers, we compare methods by elapsed wall-clock time. This allows methods with lower per-epoch costs to complete more epochs within the same available time and directly measures the safety and task performance each method can attain.

We compare SFT, Lisa, SBR, AsFT, and AIRP using Llama-3-8B, with AGNEWS and GSM8K as downstream tasks. Each training set contains 1,000 examples with a harmful-data ratio of 0.1. We run all methods on the same RTX 4090 system. We measure elapsed time from the start of downstream fine-tuning to checkpoint saving. For each target time, we select the evaluated checkpoint closest to that target.

Safety and task accuracy with GSM8K as the downstream task at approximately matched fine-tuning times (HS / FA, %).

The parenthetical values indicate the maximum deviations from the target times. This table presents a subset of the results in Appendix G.2.

MethodConference15 min (±0.8\pm 0.8)25 min (±1.3\pm 1.3)35 min (±1.0\pm 1.0)45 min (±1.4\pm 1.4)
SFT—31.0 / 42.449.8 / 41.558.3 / 40.964.4 / 37.9
LisaNeurIPS 20246.1 / 38.820.3 / 41.620.5 / 40.117.3 / 42.7
SBRICML 20261.6 / 34.83.2 / 38.23.3 / 38.15.1 / 39.2
AsFTAAAI 202613.1 / 40.914.9 / 41.215.3 / 40.315.7 / 39.6
AIRP (Ours)—2.4 / 36.91.4 / 40.12.0 / 44.01.9 / 45.0

At the 15-minute target, SBR achieves lower HS, while AIRP achieves higher FA. By the 25-minute target, AIRP achieves both lower HS and higher FA than SBR, with improvements of 1.8 and 1.9 percentage points, respectively.

At the longer evaluated budgets, AIRP achieves advantages in both safety and task performance over all compared methods. At approximately 35 minutes, AIRP reaches 2.0% HS and 44.0% FA. Its HS is 1.3 percentage points below SBR, the strongest baseline in safety, and its FA is 3.1 points above SFT, the strongest baseline in task accuracy. At approximately 45 minutes, AIRP maintains both advantages, achieving 1.9% HS and 45.0% FA.

Although the faster methods complete more epochs within these budgets, AIRP achieves the lowest HS and highest FA at both the 35- and 45-minute targets. Its task accuracy continues to improve while HS remains low. These results demonstrate the practical value of AIRP's additional computation per epoch, which translates into a stronger safety–utility outcome at comparable fine-tuning times, particularly at the longer evaluated budgets.

We report more detailed results for AGNEWS and GSM8K in Appendix G.2, including safety–utility curves and the evolution of HS and FA with elapsed time.

Thank the authors for your detailed response. I maintained my score.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of Author Responses and Manuscript Revisions

We thank all three reviewers for their constructive feedback. In our individual replies, we address each point raised under Critical Weaknesses and Questions for Authors, with detailed explanations and additional experimental results where relevant.

The revised manuscript includes the following appendix updates.

  • Appendix C.3 — Robustness to Out-of-Distribution Harmful Data. We expand this section with results on Harmful HEx-PHI and PureBad as external safety test sets.
  • Appendix C.4 — Additional Harmful Training Data Sources. We add comparisons on AGNEWS and GSM8K that vary the source of harmful training examples while keeping AIRP's reference directions fixed.
  • Appendix G.2 — Safety–Utility at Comparable Fine-Tuning Times. We add a comparison of AIRP on RTX 3090 and RTX 4090 GPUs, along with safety and task performance comparisons on AGNEWS and GSM8K at comparable fine-tuning times.

The revised manuscript marks these appendix updates in blue for easy reference.

AuthorsOpenReview ↗

Summary of Our Work and Rebuttal

We thank the Area Chair and reviewers for their time and feedback. We summarize AIRP’s core contribution and the findings from the rebuttal that address the main Weaknesses and Questions.

1. Contribution

AIRP preserves safety alignment during downstream fine-tuning when the training data may contain harmful examples. Its central contribution is to use the representation changes induced by safety alignment to guide protection at two levels, across training examples and across model layers.

At the example level, AIRP assigns less task supervision to prompts whose representation changes are more consistent with the reference directions derived from safety alignment. At the layer level, AIRP uses those same directions to measure representation changes toward the base model and adaptively assigns more protection to layers experiencing greater degradation. A separate refusal loss directly reinforces safe responses, independently of the task weights. This design links the amount of task supervision each example receives to the strength of representation protection, while preserving a dedicated source of refusal supervision.

2. Clarifications in the rebuttal

  • Safety relevance of the reference directions (ZbaA and WbdN W1 + Q1). We derive a Dolly control using the same base model, reference prompts, and estimation procedure. The safety alignment direction has a larger effect on harmful responses under matched steering. Replacing AIRP’s reference directions with the Dolly directions also increases Harmful Score (HS) on all three downstream tasks, both with and without refusal anchors. Appendices C.3 and C.4 report additional evaluations on external safety test sets and across different sources of harmful training data.

  • Prompt-only weighting and the linear mapping (ZbaA and WbdN W2 + Q2). In a new controlled experiment, we introduce target responses only into task weighting. The original prompt-only weighting achieves lower HS and higher task accuracy on all three tasks. The nonlinear comparisons further show that the linear rule balances safety and task accuracy without additional mapping hyperparameters.

  • Adaptive layer protection and refusal supervision (ZbaA W3 + Q3, WbdN W3 + Q3, and NW1Y W1 + Q1). Adaptive layer weighting achieves lower HS than static weighting in all seven paired runs (two-sided paired tt-test, p=0.0025p=0.0025). When AIRP, SBR, and AsFT receive the same additional refusal supervision, AIRP achieves the lowest HS on every task, showing that its advantage persists under a matched refusal-supervision budget.

  • Validation on an official instruction-tuned model and across safety evaluators (WbdN W4 and follow-up, and ZbaA Q4). Starting from Meta-Llama-3-8B-Instruct, which already has substantial task accuracy before adaptation, AIRP achieves the lowest HS on all three tasks and the highest average task accuracy (5.33% average HS and 82.29% average task accuracy). In a separate evaluation of the AGNEWS fine-tuned models, AIRP achieves the lowest HS under Beaver-Dam, HarmBench, and Llama Guard 3. Its XSTest over-refusal rate is 16.00%, compared with 16.40% for AsFT.

  • Safety and task accuracy at comparable training times (WbdN W5, NW1Y W3 + Q3, and ZbaA W4). The follow-up includes earlier checkpoints to examine performance under shorter budgets. On GSM8K, AIRP reaches the lowest HS of 1.4% at approximately 25 minutes, with task accuracy of 40.1%, within 1.5 percentage points of the most accurate baseline at that time. At approximately 35 minutes, AIRP achieves both the lowest HS (2.0%) and the highest task accuracy (44.0%).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.