acceptodds
Under review as a conference paper at ICLR 2027

One Patch Fits All: Optimizing Universal Safety Patches for Fine-tuned LLMs

Abstract

The fine-tuning-as-a-service paradigm allows users to customize large language models (LLMs) with their own data, but also exposes a security risk: adversaries can craft fine-tuning datasets that erode the model's safety alignment. Existing defenses either require per-request computation that scales linearly with the volume of fine-tuning jobs, or add a fixed safety vector via task arithmetic, which assumes that a vector extracted at one checkpoint has the same effect on any other. This translation-invariance assumption does not hold in general. In this work, we propose OPUS (Optimizing Patches for Universal Safety), which learns transferability by directly optimizing a universal safety patch. The framework consists of three stages: (1) it simulates a diverse set of fine-tuned checkpoints by constructing a perturbation bank through fine-tuning and interpolation, (2) it optimizes a patch so that the patched model refuses harmful queries while a KL term anchors it to the unpatched fine-tuned model to preserve utility, and (3) it deploys the pre-computed patch to incoming fine-tuned models via a single weight addition, requiring neither additional training nor user data. Extensive experiments across four models, four tasks, and three out-of-distribution attack sources demonstrate that OPUS achieves the lowest average harmful score in every setting while keeping task utility close to that of undefended fine-tuning, and incurs negligible per-request overhead.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Harmful Fine-tuning, LLM, Safety Alignment

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_NmX4OpenReview ↗
Rating
3
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces a method to quickly repair the safety of fine-tuned models. The proposed method, OPUS, learns a universal patch that is added to the fine-tuned model. The patch is optimized over simulated post-finetuned models (pre-constructed) and the base model, where the objective is to align the behaviour of the patched models (by adding the patch to the simulated post-finetuned models) with that of the base model. Then, when the base model is deployed, even if the user submits fine-tuned models, the safety of fine-tuned models can be quickly repaired by adding the patch.

Critical strengths
  • The paper is well written, and the motivation is clear.
  • The proposed method is simple but effective, and saves the cost of repairing the safety of fine-tuned models.
Critical weaknesses
  • The main concern is whether the simulated post-finetuned models can represent the true post-finetuned models. For example, Figure 1 shows that different fine-tuned models span nonlinearly, but the simulated post-finetuned models are obtained via linear interpolation in weight space. Also, what if the users fine-tune models with different LoRAs?
  • As for the patch optimization, it assumes that the perturbations cover as many harmful fine-tuning as possible, but is this even possible?
Questions for authors
  • In the experiment, lines 289-290 emphasized that Dh\mathcal{D}_h and Ds\mathcal{D}_s are disjoint from the harmful data used in the fine-tuning attacks, but how far is the disjoint distance? If they are extremely far apart and completely different, will the proposed method fail?
  • γ\gamma seems quite important? Currently, the attacker's settings are the same in the experiments; what if they are different (for example, different LoRAs or other fine-tuned structures)?
7 comments
AuthorsOpenReview ↗

Author Response (Part I)

Dear Reviewer NmX4, thank you very much for your careful review of our paper and thoughtful comments! We are greatly encouraged by your positive assessment, particularly your recognition that our paper is well written with a clear motivation, and that our method is simple but effective and saves the cost of repairing the safety of fine-tuned models. We hope the following detailed responses can help clarify potential misunderstandings and alleviate your concerns.


Q1: The patch is optimized over simulated post-finetuned models (pre-constructed) and the base model, where the objective is to align the behaviour of the patched models (by adding the patch to the simulated post-finetuned models) with that of the base model. (From Paper Summary)

R1: Thank you for your careful summary! We would like to kindly clarify one detail of our objective. On harmful prompts, the patch δ\delta is optimized to restore refusal; on benign inputs, however, the KL term anchors the patched model θ+P+δ\theta+P+\delta to the fine-tuned model θ+P\theta+P, rather than the base model θ\theta. This is a deliberate design: anchoring to θ\theta would penalize the patch for preserving the task-specific adaptations learned during fine-tuning, whereas anchoring to θ+P\theta+P requires the patch to change only what is necessary for safety, thereby preserving utility.


Q2: The main concern is whether the simulated post-finetuned models can represent the true post-finetuned models. For example, Figure 1 shows that different fine-tuned models span nonlinearly, but the simulated post-finetuned models are obtained via linear interpolation in weight space.

R2: Thank you for this insightful comment! We would like to clarify the following points.

  • Figure 1 illustrates the diversity of fine-tuned models, rather than the geometry of parameter space. Figure 1 is a t-SNE visualization of the weight increments obtained by fine-tuning on harmful data mixed with different downstream tasks. It aims to show that fine-tuning with different data recipes lands in well-separated regions of parameter space, which motivates optimizing the patch over a diverse population of perturbations. Since t-SNE is a nonlinear dimensionality reduction, the shapes in Figure 1 do not reflect the geometry of the original parameter space. Moreover, the variation in Figure 1 comes from different downstream tasks, whereas linear interpolation is performed between harmful and benign perturbations.

  • Most simulated perturbations are obtained by real fine-tuning. In the perturbation bank, both the harmful and benign perturbations are obtained by actually fine-tuning the base model; linear interpolation is only used to synthesize the mixed perturbations as a cost-free augmentation. Hence, the perturbation bank is mainly built from real fine-tuning, rather than relying on linear interpolation to simulate it.

  • All evaluated models are true post-finetuned models. Every fine-tuned model in our evaluation is obtained by real fine-tuning and is not included in the perturbation bank, covering different downstream tasks, harmful ratios, harmful attack datasets, and benign attacks. OPUS is effective on all of them, showing that the patch learned on simulated perturbations transfers to true post-finetuned models.

AuthorsOpenReview ↗

Author Response (Part II)

Q3: Also, what if the users fine-tune models with different LoRAs? (And from Questions: "Currently, the attacker's settings are the same in the experiments; what if they are different (for example, different LoRAs or other fine-tuned structures)?")

R3: Thank you for this constructive suggestion!

  • To evaluate this, we apply the same patch while varying the LoRA rank, target modules, and fine-tuning method (DoRA and QLoRA) of the user's fine-tuning in the default setting (SAMSum, hr=0.1, Llama-3-8B-Instruct). For DoRA, whose weight update is not a low-rank product due to its magnitude normalization, we first merge the user's adapter into the base model and then add δ\delta. For QLoRA, the adapter is trained on a 4-bit NF4 base model and deployed on the 16-bit base model together with δ\delta.

  • As shown below, across all LoRA ranks, target modules, and fine-tuning methods, OPUS reduces HS from at least 46% to at most 1.20%, while maintaining utility comparable to SFT. This indicates that OPUS remains effective across different fine-tuning configurations. We have added these results to Appendix D.2 of our revised paper.

Table 1. HS (%)↓ / Utility↑ under different LoRA ranks.

Methodr=4r=8 (default)r=16
SFT52.20 / 51.8552.60 / 51.8052.20 / 51.99
OPUS0.60 / 51.100.60 / 51.410.80 / 51.43

Table 2. HS (%)↓ / Utility↑ under different target modules.

Methodq, vq, k, v (default)q, k, v, o
SFT52.20 / 51.8452.60 / 51.8052.40 / 52.09
OPUS0.80 / 51.220.60 / 51.411.20 / 52.00

Table 3. HS (%)↓ / Utility↑ under different fine-tuning methods.

MethodLoRA (default)DoRAQLoRA
SFT52.60 / 51.8052.60 / 51.7646.00 / 52.25
OPUS0.60 / 51.410.60 / 51.370.20 / 51.63

Q4: As for the patch optimization, it assumes that the perturbations cover as many harmful fine-tuning as possible, but is this even possible?

R4: Thank you for this insightful question! We would like to clarify that OPUS does not require a large number of perturbations to cover all possible harmful fine-tuning.

  • OPUS repairs a shared failure pattern rather than memorizing each perturbation. Although compromised fine-tuned models are different points in parameter space, they share the same underlying failure pattern, i.e., the loss of the ability to refuse unsafe prompts. By optimizing over a diverse set of perturbations, the patch is driven to repair this shared pattern rather than fitting each individual perturbation, so exhaustive coverage is unnecessary.

  • A small perturbation bank suffices. By default, the bank contains only 10 harmful and 10 benign perturbations (plus mixed perturbations interpolated from them), constructed solely from BeaverTails and Alpaca. Nevertheless, OPUS remains effective against fine-tuning not covered by the bank, including different downstream tasks, OOD harmful datasets, and even benign fine-tuning attacks.

AuthorsOpenReview ↗

Author Response (Part III)

Q5: In the experiment, lines 289-290 emphasized that Dh\mathcal{D}_h and Ds\mathcal{D}_s are disjoint from the harmful data used in the fine-tuning attacks, but how far is the disjoint distance? If they are extremely far apart and completely different, will the proposed method fail?

R5: Thank you for this insightful question! We address it as follows.

  • In the default setting, Dh\mathcal{D}_h and Ds\mathcal{D}_s are disjoint from the harmful data used in the fine-tuning attacks in sample index, i.e., they share no samples (neither prompts nor responses), although all of them are drawn from BeaverTails.

  • Beyond this setting, our paper also evaluates attack data from entirely different sources, i.e., OOD harmful datasets (Red Team, LLM-LAT, AdvBench), and even benign fine-tuning attacks that contain no harmful content and degrade safety through a completely different mechanism (Anchoring, Self-Inf-N). Across all these attacks, OPUS keeps HS within 0.40%–1.00% (Table 4), showing that it does not fail even when the attack data is completely different from Dh\mathcal{D}_h.

We have clarified this in Section 4.1 of our revised paper.

Table 4. Harmful scores (%) under different attack data (from Tables 4 and 5 of our paper).

Attack dataSFTOPUS
BeaverTails (default)52.600.60
Red Team57.200.60
LLM-LAT57.601.00
AdvBench61.400.60
Anchoring8.400.40
Self-Inf-N21.000.40

Q6: γ\gamma seems quite important?

R6: Thank you for this question! γ\gamma is indeed important, but it does not require careful tuning: it suffices to use the natural scale of the patch, i.e., γ=1\gamma=1.

  • γ=1\gamma=1 is the natural scale of the patch. The patch δ\delta is optimized on θ+P+δ\theta+P+\delta, so γ=1\gamma=1 simply applies the learned patch as it is. We use γ=1\gamma=1 for all models, tasks, harmful ratios, and attacks without any per-setting tuning, and OPUS consistently achieves low HS.

  • γ\gamma as a general form. We introduce γ\gamma mainly to express our method in a more general form, and the sweep in the ablation study of our paper is a post-hoc sensitivity analysis. A smaller γ\gamma applies only part of the patch, and the resulting safety degradation is therefore expected. For γ≥1\gamma \geq 1, HS stays low with only a mild utility decrease.

We have clarified this in Section 3.5 of our revised paper.

Reviewer_AHNZOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes OPUS, a method that constructs a safety patch to defend against harmful fine-tuning in Fine-Tuning-as-a-Service settings. OPUS first generates a diverse set of perturbed checkpoints, then optimizes a single safety patch over them, which is subsequently added to the fine-tuned model.

Critical strengths

OPUS is computationally efficient in FTaaS environments. The safety patch needs to be trained only once and can then be applied to models after user fine-tuning, which makes the approach practical for service providers.

The proposed method consistently improves safety performance across a variety of experimental settings.

Critical weaknesses
  1. It would be helpful if the authors could explain more clearly why the proposed training objective is expected to generalize to unseen fine-tuned models and consistently restore safety. A more detailed justification, whether theoretical or intuitive, of why optimizing this objective leads to a patch that remains effective on unseen fine-tuned models would strengthen the paper.

  2. OPUS constructs mixed perturbations by linearly interpolating between harmful and benign fine-tuning directions. However, the paper does not analyze which type of perturbation contributes most to the effectiveness of the safety patch. In Table 7, adding mixed perturbations to the harmful and benign perturbations only reduces the harmful score from 0.8% to 0.6%. I would encourage the authors to (a) report results across multiple random seeds to assess the robustness of this difference, (b) analyze how sampling different interpolation regions, particularly those closer to harmful or benign perturbations, affects the final patch performance, and (c) discuss why certain regions may be more effective. Such an analysis would help validate the design choice of mixed perturbations.

  3. The paper reports low harmful scores across the attack datasets, which is encouraging. Beyond final performance, however, further analysis of the underlying mechanism would be valuable. For example, it would be interesting to analyze the changes in refusal rates and harmful-response likelihoods before and after applying the patch, particularly on OOD attack datasets, perhaps through representation-level or layer-wise analysis.

  4. Figure 3 illustrates the effect of the scaling parameter on OPUS. Since some baseline methods also apply a patch or a similar scaling mechanism to the model, could the authors clarify how the hyperparameters for these baselines were selected? Ensuring that the baselines were tuned comparably would make the comparison more convincing.

Questions for authors

The main questions addressed in the Critical Weaknesses above.

7 comments
AuthorsOpenReview ↗

Author Response (Part I)

Dear Reviewer AHNZ, thank you very much for your careful review of our paper and thoughtful comments! We are greatly encouraged by your positive assessment, particularly your recognition of the computational efficiency of OPUS in FTaaS environments, where the safety patch needs to be trained only once and can be applied to any user fine-tuned model, which makes it practical for service providers, as well as its consistent safety improvement across a variety of experimental settings. We hope the following detailed responses can fully address your valuable suggestions.


Q1: It would be helpful if the authors could explain more clearly why the proposed training objective is expected to generalize to unseen fine-tuned models and consistently restore safety. A more detailed justification, whether theoretical or intuitive, of why optimizing this objective leads to a patch that remains effective on unseen fine-tuned models would strengthen the paper.

R1: Thank you for this insightful comment! We agree that a clearer justification of why OPUS generalizes would strengthen our paper, and we provide an intuitive explanation below.

  • Compromised checkpoints share an underlying failure pattern. Fine-tuning with different data recipes lands in well-separated regions of parameter space. However, although these compromised checkpoints are different points in parameter space, they share the same underlying failure pattern, i.e., the loss of the ability to refuse unsafe prompts.

  • Optimizing over a distribution captures the shared pattern. OPUS optimizes the patch in expectation over a diverse population of simulated checkpoints, which shares the intuition of Expectation over Transformation (EOT) and domain generalization: since the patch must restore refusal on all checkpoints simultaneously, it cannot rely on the idiosyncrasies of any single checkpoint, and is instead driven to repair the shared failure pattern. As unseen fine-tuned models also exhibit this pattern, the patch transfers to them.

We have added this discussion to Appendix F of our revised paper.

AuthorsOpenReview ↗

Author Response (Part II)

Q2: OPUS constructs mixed perturbations by linearly interpolating between harmful and benign fine-tuning directions. However, the paper does not analyze which type of perturbation contributes most to the effectiveness of the safety patch. In Table 7, adding mixed perturbations to the harmful and benign perturbations only reduces the harmful score from 0.8% to 0.6%. I would encourage the authors to (a) report results across multiple random seeds to assess the robustness of this difference, (b) analyze how sampling different interpolation regions, particularly those closer to harmful or benign perturbations, affects the final patch performance, and (c) discuss why certain regions may be more effective. Such an analysis would help validate the design choice of mixed perturbations.

R2: Thank you for this constructive suggestion!

  • We first clarify the intuition behind the design of the perturbation bank. Real-world user fine-tuning can be broadly categorized into harmful fine-tuning, benign fine-tuning, and a mixture of both. Accordingly, the perturbation bank contains harmful, benign, and mixed perturbations, so that OPUS restores safety while preserving the model's benign capabilities across all these possible fine-tuning scenarios. Following your suggestions, we further conduct the following analyses.

  • (a) Multiple random seeds. We repeat the ablation in Table 7 of our paper with three random seeds (Table 1). Harmful perturbations contribute most to safety: without them (Benign-only), HS rises to 4.33%. This is likely because harmful perturbations are the ones that directly break safety, from which the patch learns how to restore it. Adding benign and mixed perturbations brings a marginal improvement in HS (from 0.73% to 0.53%), but notably improves utility (from 50.74 to 51.56). This is likely because more diverse perturbations help the patch learn not to over-correct. Among them, mixed perturbations serve as a cost-friendly augmentation that increases the diversity of perturbations to further improve performance.

  • (b) Interpolation regions. We restrict the interpolation coefficient α\alpha of mixed perturbations (α=0\alpha=0: benign; α=1\alpha=1: harmful) to three regions: [0,1/3][0, 1/3] (closer to benign), [1/3,2/3][1/3, 2/3] (middle), and [2/3,1][2/3, 1] (closer to harmful) (Table 2). All regions achieve similar performance: HS remains within 0.60%–0.80%, and utility remains within 50.89–51.43.

  • (c) Why certain regions may be more effective. As shown above, the specific interpolation region does not matter much, while the default [0,1][0, 1] achieves the best balance between safety and utility (the lowest HS with near-highest utility). This is likely because, as a data augmentation, mixed perturbations benefit most from diversity, and [0,1][0, 1] covers the widest range of interpolation. Therefore, we use [0,1][0, 1] by default as the most intuitive and effective choice.

We have added these results to Appendix D.7 of our revised paper.

Table 1. Ablation on the perturbation bank over three random seeds (mean ± std).

Perturbation bankHS (%)↓Utility↑
Harmful-only0.73 ± 0.1250.74 ± 0.51
Benign-only4.33 ± 0.4250.81 ± 0.27
Harmful & Benign0.73 ± 0.3151.24 ± 0.40
All (harmful + benign + mixed)0.53 ± 0.1251.56 ± 0.26

Table 2. Effect of the interpolation region of mixed perturbations.

Region of α\alphaHS (%)↓Utility↑
[0,1][0, 1] (default)0.6051.41
[0,1/3][0, 1/3] (closer to benign)0.8051.43
[1/3,2/3][1/3, 2/3] (middle)0.8051.20
[2/3,1][2/3, 1] (closer to harmful)0.6050.89
AuthorsOpenReview ↗

Author Response (Part III)

Q3: The paper reports low harmful scores across the attack datasets, which is encouraging. Beyond final performance, however, further analysis of the underlying mechanism would be valuable. For example, it would be interesting to analyze the changes in refusal rates and harmful-response likelihoods before and after applying the patch, particularly on OOD attack datasets, perhaps through representation-level or layer-wise analysis.

R3: Thank you for this insightful suggestion! Following your suggestion, we analyze OPUS at both the behavioral and representation levels under ID and OOD attacks in the default setting (SAMSum, hr=0.1, Llama-3-8B-Instruct).

  • Behavioral level. On the 500 harmful test prompts, we report the rate of explicit refusals (a response is counted as an explicit refusal if its first 200 characters contain refusal phrases such as "I'm sorry", "I cannot", or "As an AI") and the average per-token log-likelihood of harmful responses (Table 3). After applying the patch, the refusal rate increases from below 2% to over 80%, and the likelihood of harmful responses decreases, under both ID and OOD attacks.

  • Representation level. Following [1], we compute a refusal direction on the aligned model θ\theta as the difference between the mean hidden states (at the last prompt token) of harmful and benign prompts, and project the hidden states of harmful test prompts onto it layer by layer (Table 4). The projection is normalized so that 1 and 0 correspond to the centers of harmful and benign prompts on θ\theta, respectively. Fine-tuning attacks push harmful prompts toward the benign center, weakening the model's recognition of their harmfulness. After applying the patch, the projection is largely restored under both ID and OOD attacks. The restoration is most prominent in the intermediate layers, consistent with [1], which extracts refusal directions from earlier layers since directions in the final layers are close to the unembedding directions of output tokens. This indicates that OPUS restores safety by recovering the refusal-related representations suppressed by harmful fine-tuning.

We have added these analyses to Appendix D.8 of our revised paper.

[1] Arditi et al. Refusal in Language Models Is Mediated by a Single Direction. NeurIPS 2024.

Table 3. Refusal rate and harmful-response log-likelihood before and after applying the patch.

MetricID (before)ID (after)OOD (before)OOD (after)
Explicit refusal rate ↑1.2%80.2%1.6%83.0%
Harmful-response log-likelihood ↓−1.742−1.993−1.819−2.048

Table 4. Normalized projection onto the refusal direction (before → after applying the patch).

Layer121620242832
ID0.65 → 0.730.63 → 0.870.53 → 0.910.45 → 0.900.45 → 0.830.43 → 0.74
OOD0.67 → 0.790.62 → 0.930.55 → 0.950.53 → 0.950.57 → 0.890.47 → 0.75

Q4: Figure 3 illustrates the effect of the scaling parameter on OPUS. Since some baseline methods also apply a patch or a similar scaling mechanism to the model, could the authors clarify how the hyperparameters for these baselines were selected? Ensuring that the baselines were tuned comparably would make the comparison more convincing.

R4: Thank you for pointing it out! We would like to clarify as follows.

  • γ=1\gamma=1 is the natural scale of the patch. The patch δ\delta is optimized on θ+P+δ\theta+P+\delta, so γ=1\gamma=1 simply applies the learned patch as it is. We introduce γ\gamma mainly to express our method in a more general form, and it also serves as an optional knob for the service provider to further strengthen safety at a small utility cost. Except for the ablation study in Figure 3 of our paper, all of our experiments use the default γ=1\gamma=1.

  • Baseline hyperparameters follow their original papers or official implementations. Unlike OPUS, whose patch is learned through optimization, these baselines rely on manually specified hyperparameters to balance safety restoration and utility preservation. We therefore adopt the values provided by their original papers or official implementations. We have added these values to Appendix B of our revised paper.

Reviewer_bCYeOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes OPUS, a post-hoc safety repair method for fine-tuned language models in the fine-tuning-as-a-service setting. OPUS constructs a perturbation bank from harmful and benign fine-tuning runs, together with convex combinations of their weight updates. It then directly optimizes a universal additive patch that restores refusal behavior across the simulated perturbation population while using a KL term to preserve the behavior of the fine-tuned model on benign inputs. At deployment time, the pre-computed patch is added to an incoming fine-tuned model without additional training or access to the user’s data. The evaluation covers four 7B–9B instruction-tuned models, four downstream tasks, multiple harmful-data ratios, several out-of-distribution attack datasets, and benign safety-degrading fine-tuning attacks. OPUS consistently obtains lower harmful scores than the compared baselines while retaining utility close to undefended SFT and introducing negligible per-job overhead.

Critical strengths
  1. The experiments use a fixed LoRA configuration, a fixed training recipe, and relatively small simulated perturbation banks. It is unclear whether the same patch transfers to different LoRA ranks and target modules, full-parameter fine-tuning, other optimizers, longer training, or adaptive attackers that know the patch. In addition, the paper should clarify whether one patch is shared across all four model families or whether a separate patch is trained for each backbone.
  2. Harmful Score is determined entirely by Llama-Guard-3-8B on 500 prompts. This may be sensitive to the judge’s biases and can reward generic refusal behavior without measuring nuanced safety or usefulness. The paper does not report multiple random seeds, confidence intervals, decoding details, or human/independent-judge validation. These omissions make it difficult to assess the statistical robustness and practical significance of the very small reported harmful scores.
Critical weaknesses
  1. Is the patch δ\delta trained separately for each base model, or is a single patch expected to transfer across model families? If it is model-specific, how should the term “universal” be interpreted?

  2. Are the results averaged over multiple random seeds? Please report variance or confidence intervals, and clarify how λ, γ, and baseline hyperparameters were selected without using the test sets.

  3. How does OPUS perform against adaptive or stronger attacks, such as attackers optimizing directly against the released patch, different LoRA ranks and target modules, full-parameter fine-tuning, or reinforcement-learning-based fine-tuning?

  4. Can the authors provide safety results using additional independent judges or human evaluation, together with generation and decoding settings?

Questions for authors

See Weaknesses

7 comments
AuthorsOpenReview ↗

Author Response (Part I)

Dear Reviewer bCYe, thank you very much for your careful review of our paper and thoughtful comments! We are greatly encouraged by your positive assessment, particularly your recognition of the consistent safety improvement of OPUS, its negligible per-job overhead, and the clarity of our presentation. We hope the following detailed responses can fully address your valuable suggestions.


Q1: Is the patch (δ\delta) trained separately for each base model, or is a single patch expected to transfer across model families? If it is model-specific, how should the term "universal" be interpreted? (Also from Strengths-1: "In addition, the paper should clarify whether one patch is shared across all four model families or whether a separate patch is trained for each backbone.")

R1: Thank you for pointing it out! We are sorry that our submission did not state the scope of "universal" clearly enough, and we would like to clarify it here.

  • One patch per base model. The safety patch is trained separately for each base model, i.e., it is computed once per base model. A single patch is not expected to transfer across model families, which are pretrained independently and may even differ in architecture.

  • The meaning of "universal". In our paper, "universal" refers to universality across fine-tuning jobs of a given base model. Once trained, the same patch is directly applied to any fine-tuned model derived from that base model. We believe this setting is reasonable because fine-tuning services are organized per base model: in the fine-tuning-as-a-service paradigm, the provider offers fine-tuning on a set of released base models, and every user job starts from one of them. What the provider needs is therefore a patch that works for all incoming jobs on the same base model.

  • Revision. Although our paper states that the patch is computed once per base model (Sections 1 and 3.2), we agree that this is not prominent enough. We have explicitly stated in Section 1 of our revised paper that a separate patch is trained for each base model and is universal across its fine-tuning jobs.


Q2: Are the results averaged over multiple random seeds? Please report variance or confidence intervals, and clarify how λ, γ, and baseline hyperparameters were selected without using the test sets. (Also from Strengths-2: "The paper does not report multiple random seeds, confidence intervals, ... These omissions make it difficult to assess the statistical robustness and practical significance of the very small reported harmful scores.")

R2: Thank you for this constructive suggestion! We address your concerns as follows.

  • Multiple random seeds. Following your suggestion, we repeat the default setting (SAMSum, hr=0.1, Llama-3-8B-Instruct) with three random seeds (Table 1). OPUS achieves an HS of 0.53 ± 0.12%, far below that of SFT (52.07 ± 0.76%), while maintaining comparable utility (51.56 ± 0.26 vs. 51.99 ± 0.18). We have added these results to Appendix D.1 of our revised paper.

  • Selection of λ\lambda, γ\gamma, and baseline hyperparameters. For OPUS, λ\lambda and γ\gamma are fixed in advance to their natural default values: λ=1\lambda=1 assigns equal weight to the two loss terms, and γ=1\gamma=1 applies the learned patch δ\delta as it is. The sweep in the ablation study of our paper (Section 4.4) is a post-hoc sensitivity analysis conducted only in the default setting, rather than a selection procedure, and the same values are used for all datasets, models, and attacks without any per-setting tuning. For the baselines, the hyperparameters follow the original papers or their official implementations, and we have added their values to Appendix B of our revised paper.

Table 1. HS (%) and utility over three random seeds (mean ± std).

MethodHS ↓Utility ↑
SFT52.07 ± 0.7651.99 ± 0.18
OPUS0.53 ± 0.1251.56 ± 0.26
AuthorsOpenReview ↗

Author Response (Part II)

Q3: How does OPUS perform against stronger attacks, such as different LoRA ranks and target modules, full-parameter fine-tuning, or reinforcement-learning-based fine-tuning? (Also from Strengths-1: "It is unclear whether the same patch transfers to different LoRA ranks and target modules, full-parameter fine-tuning, other optimizers, longer training, ...")

R3: Thank you for this insightful comment! We address your concerns as follows.

  • Different fine-tuning configurations. We apply the same patch while varying the LoRA rank, target modules, training length, and optimizer of the user's fine-tuning in the default setting (SAMSum, hr=0.1, Llama-3-8B-Instruct) (Tables 2–5). Across all configurations, OPUS reduces HS from over 50% to at most 1.40%, while keeping utility close to undefended SFT.

  • Reinforcement-learning-style fine-tuning. We further evaluate preference-based fine-tuning with DPO under the same data composition, with harmful samples (chosen: harmful response; rejected: refusal) and SAMSum samples (chosen: reference summary; rejected: summary generated by the pre-finetuned model) (Table 6). OPUS reduces HS from 50.40% to 2.60%, while maintaining utility comparable to the unpatched model.

  • Full-parameter fine-tuning. Our paper focuses on fine-tuning-as-a-service that offers LoRA fine-tuning, since full-parameter fine-tuning is far more costly in both fine-tuning and storing each user's fine-tuned model. To address your concern, we further conduct two experiments (Table 7), using Llama-3.2-3B-Instruct due to our limited GPU resources: (1) we evaluate the default LoRA-based patch against full-parameter fine-tuning attacks; (2) assuming that the service provider also offers full-parameter fine-tuning, we construct the perturbation bank with full-parameter fine-tuning and optimize a full-parameter patch. The LoRA-based patch only reduces HS from 45.00% to 39.80%, as its perturbation bank contains only LoRA fine-tuning, whereas the full-parameter patch reduces HS from 45.00% to 0.40%. Therefore, full-parameter fine-tuning does not preclude the use of OPUS: the service provider only needs to determine the fine-tuning modes it offers in advance and optimize the patch accordingly.

We have added these results to Appendices D.2–D.4 of our revised paper.

Table 2. HS (%)↓ / Utility↑ under different LoRA ranks.

Methodr=4r=8r=16
SFT52.20 / 51.8552.60 / 51.8052.20 / 51.99
OPUS0.60 / 51.100.60 / 51.410.80 / 51.43

Table 3. HS (%)↓ / Utility↑ under different target modules.

Methodq, vq, k, vq, k, v, o
SFT52.20 / 51.8452.60 / 51.8052.40 / 52.09
OPUS0.80 / 51.220.60 / 51.411.20 / 52.00

Table 4. HS (%)↓ / Utility↑ under different training lengths.

Method10 epochs15 epochs20 epochs
SFT52.60 / 51.8051.40 / 52.6051.60 / 53.13
OPUS0.60 / 51.410.80 / 51.351.20 / 51.28

Table 5. HS (%)↓ / Utility↑ under different optimizers.

MethodAdamWAdafactorRMSprop
SFT52.60 / 51.8053.00 / 51.8652.00 / 52.21
OPUS0.60 / 51.411.40 / 50.980.60 / 51.36

Table 6. HS (%)↓ / Utility↑ under DPO fine-tuning.

MethodHSUtility
DPO50.4038.19
DPO + OPUS2.6037.77

Table 7. HS (%)↓ / Utility↑ under full-parameter fine-tuning on Llama-3.2-3B-Instruct.

Fine-tuningNo defense+ LoRA patch+ Full-parameter patch
Full-parameter45.00 / 48.3239.80 / 48.480.40 / 47.07
AuthorsOpenReview ↗

Author Response (Part III)

Q4: How does OPUS perform against adaptive attacks, such as attackers optimizing directly against the released patch? (Also from Strengths-1: "It is unclear whether the same patch transfers to ... adaptive attackers that know the patch.")

R4: Thank you for this insightful comment! We address your concern as follows.

  • Attack setting. In fine-tuning-as-a-service, users (including attackers) control only the uploaded data. We consider an adaptive attacker who has white-box access to the patch δ\delta and the base model. The attacker selects harmful samples such that the model fine-tuned on them has the lowest harmful loss after the patch is applied, i.e., min⁡DadvLharm(θ+P(Dadv)+δ)\min_{D_{adv}} L_{harm}(\theta + P(D_{adv}) + \delta). Since exact search over sample subsets is intractable, we score each candidate with a first-order approximation: a fine-tuning step on sample ii reduces this loss most when its gradient gi(θt)g_i(\theta_t) aligns with ∇Lharm(θt+δ)\nabla L_{harm}(\theta_t + \delta), so we use si=cos⁡(gi(θt),∇Lharm(θt+δ))s_i = \cos\big(g_i(\theta_t), \nabla L_{harm}(\theta_t + \delta)\big), averaged over intermediate checkpoints θt\theta_t of a simulated fine-tuning run. The attacker then uploads the top-ranked samples. In practice, the base model and safety patch can be kept private by the service provider, so the attacker may only be able to select data on a surrogate model with its own patch. We therefore also evaluate a transfer setting, where the data selected on Llama-3-8B-Instruct with its patch are used to attack Llama-3.1-8B-Instruct.

  • Results. With white-box access to the patch, this attack raises the HS of OPUS from 0.60% with randomly sampled harmful data to 7.80% (Table 8). In the transfer setting, the selected data remain highly harmful to the undefended target (HS 55.60%), but OPUS reduces HS to 0.60% (Table 8). In conclusion, even when the attacker knows the patch and selects data against it, the HS of OPUS remains far below that of undefended SFT; when the data are transferred from a surrogate model, OPUS remains highly effective.

We have added these results to Appendix D.5 of our revised paper.

Table 8. HS (%)↓ / Utility↑ under the standard and adaptive attacks.

Target modelAttackNo defenseOPUS
Llama-3-8B-InstructStandard52.60 / 51.800.60 / 51.41
Llama-3-8B-InstructAdaptive (known patch)58.60 / 51.477.80 / 51.36
Llama-3.1-8B-InstructStandard51.20 / 51.801.60 / 51.71
Llama-3.1-8B-InstructAdaptive (transfer)55.60 / 52.250.60 / 51.08

Q5: Can the authors provide safety results using additional independent judges or human evaluation, together with generation and decoding settings? (Also from Strengths-2: "Harmful Score is determined entirely by Llama-Guard-3-8B on 500 prompts. This may be sensitive to the judge's biases and can reward generic refusal behavior without measuring nuanced safety or usefulness.")

R5: Thank you for this constructive suggestion! We address your concerns as follows.

  • Independent judges. In the default setting (SAMSum, hr=0.1, Llama-3-8B-Instruct), we re-evaluate the same generated responses with two additional independent judges, WildGuard and GPT-4o (Table 9). The absolute HS varies across judges due to their different judging criteria, with WildGuard and GPT-4o being stricter than Llama-Guard. However, under all three judges, OPUS consistently reduces HS by over 95% relative to SFT. We have added these results to Appendix D.6 of our revised paper.

  • Generation and decoding settings. All responses are generated with greedy decoding and max_new_tokens=512. We have added these settings to Appendix A of our revised paper.

Table 9. HS (%) under different judges.

JudgeSFTOPUS
Llama-Guard-3-8B (original)52.600.60
WildGuard79.202.80
GPT-4o80.803.80

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.