- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes the Alignment-Induced Representation Protection (AIRP) framework, which can maintain the model's security alignment capability during the downstream fine-tuning of large language models using data with security risks. Its core idea is to construct alignment-related reference directions by leveraging the representation differences between the base model and the aligned model at each layer. AIRP consists of three modules: (1) risk-aware task weighting based on directional consistency, (2) base-directed representation protection with adaptive layer weighting, and (3) refusal-anchor supervision. The experimental results demonstrate promising safety retention across multiple downstream tasks, model architectures, harmful-data ratios, and fine-tuning dataset sizes.
Critical strengths
- The paper addresses safety degradation during downstream customization, an important challenge for fine-tuning-as-a-service.
- Constructing layer-wise reference directions from Base-to-Aligned representation differences offers an interpretable way to characterize alignment-related changes.
- AIRP achieves consistently low Harmful Scores while maintaining competitive downstream task accuracy. The results under increasing fine-tuning dataset sizes are particularly encouraging.
Critical weaknesses
- The method assumes that representation differences between the Base and Aligned Models capture safety-relevant directions. However, these differences may also reflect general instruction-following, response formatting, or other changes induced during alignment training. Although the direction-specific control experiments in Appendix D.2 are useful, they do not fully disentangle safety-specific changes from general fine-tuning-induced representation changes. Additional controls using non-safety instruction-tuned checkpoints would strengthen this interpretation.
- Risk-Aware Task Weighting estimates each example's risk using prompt-only representations. Consequently, a harmful prompt paired with a safe refusal and the same prompt paired with a harmful compliant response receive identical task weights. This raises concerns about whether directional consistency truly measures the harmfulness of the supervision signal. A controlled experiment using matched prompts with different target responses would help clarify this issue.
- The adaptive layer weights are derived from normalized representation degradation scores from previous optimization steps. While the ablation results demonstrate incremental improvements, it remains unclear whether the gains arise from the proposed adaptive mechanism or from emphasizing layers that already exhibit larger regularization losses. Comparisons with uniform weighting, static layer importance, and EMA-based weighting would be informative.
- AIRP introduces substantial training overhead, nearly doubling total training time relative to standard SFT in the reported setting. Moreover, some baselines use different training epochs and LoRA configurations, and the methods differ in their use of safety reference data. Matched-budget comparisons and safety–utility curves under equivalent computational and safety-data budgets would make the empirical findings more convincing.
Questions for authors
-
Can the authors demonstrate that the reference directions specifically capture safety alignment rather than general instruction-tuning-induced representation changes?
-
Why is risk weighting computed solely from prompts, despite harmful fine-tuning being driven by potentially unsafe target responses?
-
How does adaptive layer weighting compare with static or smoothed weighting schemes under identical regularization budgets?
-
Can the authors evaluate over-refusal rates and safety performance using an independent evaluator?
10 comments
Author Rebuttal (II): Rationale for Prompt-Only Task Weighting and Evaluation of a Response-Aware Variant
We thank the reviewer for raising this question about target responses in task weighting. We explain our design and provide an additional comparison below.
W2 + Q2: Rationale for Prompt-Only Weighting and Evaluation of a Response-Aware Variant
We first explain why we compute task weights from prompts in the harmful fine-tuning setting studied in this work and why we pair this weighting with Refusal-Anchor Supervision. We then evaluate a variant that incorporates target responses throughout the representation computations to examine their effect on safety preservation and task performance.
1. Why we compute task weights from prompts
As described in the Methodology section, we consider a fine-tuning-as-a-service setting where user-provided data lack sample-level safety annotations and may contain harmful prompt–compliant response pairs. In these pairs, the prompt expresses the harmful request that the target response trains the model to fulfill.
AIRP therefore uses the prompt's alignment-induced representation changes to guide task weighting. Computing representations at the end of the prompt measures the changes that safety alignment induces in the model's representation of the request before it conditions on the target response. This is consistent with the reference directions, which are constructed from the Base-to-Aligned representation changes on harmful prompts.
2. The same prompt paired with different responses
The reviewer correctly observes that a harmful response and a safe refusal paired with the same prompt receive the same task weight. A harmful prompt paired with a safe refusal has the same form as the examples used for safety alignment and provides useful safety supervision.
Our study focuses on safety degradation caused by harmful prompt–compliant response pairs during downstream fine-tuning. Prompt-based weighting reduces the task supervision assigned to potentially harmful requests. However, when such a request is paired with a safe refusal in the downstream data, its useful refusal supervision is also downweighted.
This consideration helps motivate the inclusion of Refusal-Anchor Supervision, whose separate loss directly reinforces safe refusals without being scaled by the task weights. The two objectives therefore work together to reduce potentially harmful task supervision while maintaining an explicit source of safe refusal supervision.
3. Evaluating the inclusion of target responses
We evaluate a response-aware variant of AIRP to assess the effect of including target responses. We use harmful prompt–harmful response pairs to estimate its reference directions and use each downstream example's prompt and actual target response for task-weight computation and representation protection. Thus, reference-direction estimation and downstream representation computations both use prompt–response inputs.
This comparison evaluates the effect of including response information throughout the method. We repeat downstream fine-tuning for the three tasks and compare the resulting models with prompt-only AIRP.
Safety and task accuracy using prompt-only or prompt–response representations (%, HS / FA).
| Representation input | AGNEWS HS | AGNEWS FA | GSM8K HS | GSM8K FA | SST2 HS | SST2 FA |
|---|---|---|---|---|---|---|
| Prompt only (AIRP) | 4.30 | 86.10 | 1.60 | 45.70 | 3.10 | 91.06 |
| Prompt + response | 27.90 | 84.30 | 4.80 | 44.30 | 11.70 | 93.12 |
Including target responses increases HS by 23.60, 3.20, and 8.60 percentage points on AGNEWS, GSM8K, and SST2, respectively. FA decreases by 1.80 and 1.40 points on AGNEWS and GSM8K, while increasing by 2.06 points on SST2.
In the evaluated settings, prompt-only AIRP achieves lower HS than the response-aware variant on all three tasks and higher FA on two of them. These results favor computing alignment-related representations from prompts in this setting, while AIRP uses refusal supervision to directly reinforce safe responses.
Author Rebuttal (III): Safety Gains from Layer Prioritization and Adaptive Weight Updates
We thank the reviewer for asking what drives the gains from Adaptive Layer Protection.
W3 + Q3: Comparison with Uniform, Static, and EMA Layer Weighting
To determine whether the safety gains come from giving greater weight to layers with higher degradation or also from updating those weights during training, we compare four weighting rules using Llama-3-8B with AGNEWS as the downstream task.
-
Uniform. We assign each of the Transformer layers a weight of at every step. This provides a baseline with equal protection weights across layers.
-
Static. To test whether prioritizing layers with higher degradation is sufficient to obtain the safety gains, we first fine-tune the Aligned Model for one epoch using only the weighted task loss. We then evaluate the resulting checkpoint using the layer score in Eq. (5).
\frac{ \sum_{(x_i,y_i)\in\mathcal{B}^{(t)}} w_i\bigl(e_i^{\,l,(t)}\bigr)^2 }{ \sum_{(x_i,y_i)\in\mathcal{B}^{(t)}} w_i \mathbb{I}\!\left[a_i^{\,l}>\epsilon\right]+\epsilon }.$$ Here, $\mathcal{B}^{(t)}$ is the batch at step $t$, $w_i$ is the sample weight, and $a_i^{\,l}$ is the reference displacement along the layer's reference direction. We pool the numerator and denominator over the full training set at the one-epoch checkpoint and normalize the resulting scores across layers. The Static run starts again from the Aligned Model and uses these fixed weights throughout fine-tuning. -
EMA. To test whether smoothing the layer scores improves adaptive weighting, we update the smoothed score after step as . Larger gives more weight to the previous smoothed score. At step , we normalize across layers.
-
Adaptive (Ours). We compute the layer weights by normalizing the preceding step's scores across layers. This corresponds to in the EMA formulation.
For the comparison below, all four runs start from the same Aligned Model and use the same 1,000 training examples, 10% of which are harmful. The layer-score definition and representation-loss coefficient are identical across runs. We disable Refusal-Anchor Supervision in all four runs to assess the contribution of layer weighting separately from direct refusal supervision.
Harmful Score and fine-tuning accuracy for each layer-weighting rule, with AGNEWS as the downstream task (%).
| Layer weighting | HS | FA |
|---|---|---|
| Uniform | 17.20 | 85.60 |
| Static | 13.10 | 86.00 |
| EMA () | 13.20 | 86.00 |
| EMA () | 13.60 | 85.90 |
| EMA () | 13.40 | 85.90 |
| EMA () | 13.50 | 85.70 |
| EMA () | 13.60 | 85.50 |
| Adaptive (Ours) | 12.70 | 86.10 |
Giving greater weight to layers with higher degradation yields a clear safety gain. Static reduces HS by 4.10 percentage points relative to Uniform while maintaining comparable FA. Updating the layer weights during training adds a safety benefit beyond that of assigning fixed priorities to layers. Static already gives greater weight to layers with higher degradation at calibration, yet Adaptive reduces HS by a further 0.40 points at comparable FA (86.10% vs. 86.00%). Adaptive also achieves HS 0.50–0.90 points below the tested EMA variants at comparable FA. Taken together, these results indicate that most of the safety gain from Adaptive Layer Protection comes from concentrating protection on layers with greater representation degradation, with a further gain from adjusting that allocation as training progresses.
Author Rebuttal (I): Testing the Safety Relevance of Reference Directions with a General Instruction-Tuning Control
We thank the reviewer for the constructive feedback.
W1 + Q1: Safety Relevance of the Reference Directions
To test whether the reference directions capture safety-relevant representation changes beyond those induced by general instruction tuning, we construct a control using Dolly, a dataset of general-purpose instructions paired with responses. We fine-tune the Llama-3-8B Base Model on 5,000 Dolly examples to obtain a general instruction-tuned checkpoint. We then estimate directions for this checkpoint and the Aligned Model relative to the same Base Model, using the same 500 harmful reference prompts and the same estimation procedure.
We use this control in two experiments. At inference time, we compare the effects of steering along the two directions on harmful responses. During downstream fine-tuning, we replace the reference directions in AIRP with the Dolly control directions to test whether they provide comparable safety protection.
1. Activation Steering at Inference Time
We test whether the direction derived from safety alignment has a stronger effect on harmful responses than the direction derived from general instruction tuning. We apply both directions to the same model after downstream fine-tuning. At inference time, we shift the last-prompt-token representation at the final Transformer layer by ±γ times the corresponding unit vector. The parameter γ controls the intervention strength, with γ=0 denoting the unmodified model. Each column compares the two directions at the same strength.
Effect of steering along and against the reference and Dolly control directions. Entries report Harmful Score (HS, %), with signed changes from γ=0 in parentheses (percentage points).
The reference direction produces a larger absolute change in HS for both operations at every nonzero intervention strength. At γ=5, adding it reduces HS by 4.10 points and subtracting it increases HS by 2.90 points. The corresponding changes for the Dolly direction are 0.60 and 0.90 points. With the estimation prompts and intervention conditions matched, the Dolly direction reproduces only a small fraction of the reference direction's effect on harmful responses.
2. Direction Replacement During Downstream Fine-Tuning
We test whether the stronger behavioral effect also translates into better safety preservation when the directions are used during training. All runs start from the same Aligned Model. We substitute the Dolly control directions for the reference directions in AIRP's task weighting and representation protection. Since Refusal-Anchor Supervision operates independently of the directions, we compare the two sets of directions with anchors enabled and again with anchors disabled.
Effect of replacing the reference directions with Dolly directions in AIRP, with and without refusal anchors. HS and Fine-tuning Accuracy (FA) are reported as percentages.
With refusal anchors enabled, replacing the reference directions with the Dolly directions increases HS by 15.00, 3.00, and 6.60 percentage points on AGNEWS, GSM8K, and SST2, respectively. With refusal anchors disabled, the corresponding increases are 34.70, 29.20, and 28.30 points. The reference directions therefore provide stronger safety protection during fine-tuning across all three tasks under both matched anchor settings.
Taken together, the steering and direction-replacement experiments indicate that the reference directions capture safety-relevant representation changes beyond those captured by general instruction tuning on Dolly. These changes matter both for influencing harmful responses at inference time and for preserving safety during downstream fine-tuning.