acceptodds
Under review as a conference paper at ICLR 2027

TRACE: Trajectory-Based Safety Patch Learning for LLM Post-Training Realignment

Abstract

Fine-Tuning-as-a-Service (FTaaS) platforms let users perform supervised fine-tuning (SFT) on customized data, but this pipeline can erode model safety alignment. To recover safety without re-running full alignment, existing realignment methods focus on calibrating the integration of safety patches into fine-tuned models. These methods exhibit a persistent safety-utility trade-off: weak repair leaves harmful behavior intact, while stronger repair increasingly damages the benign task. This paper shifts the focus from online calibration to offline patch learning and aims to learn a safety patch that restores safety while preserving task-specific capabilities. To this end, we propose TRACE, which simulates harmful SFT trajectories to produce progressively corrupted model states, and optimizes a safety patch simultaneously across these states. TRACE trains the safety patch during the offline stage, and reuses it across all user checkpoints without per-user calibration. We evaluate two representative models using three harmful SFT datasets, together with three utility benchmarks. Across six benchmarks and two models, TRACE consistently dominates the safety-utility frontier. TRACE improves the safety rate by up to 77 percentage points over the best baseline, while maintaining comparable utility to the undefended fine-tuned model.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM Safety, Alignment, Safety Repair

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_83s7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates post-fine-tuning safety realignment for large language models in Fine-Tuning-as-a-Service scenarios. The authors argue that existing merging-based realignment methods often rely on calibrating repair strength, which can lead to an unfavorable safety–utility trade-off across heterogeneous user fine-tuned models.

To address this issue, the paper proposes TRACE, a trajectory-based safety patch learning framework. TRACE alternates between simulating safety-eroding fine-tuning and optimizing a low-rank safety patch on progressively modified model states. The patch is trained with both refusal supervision and benign task data to restore safety while minimizing interference with downstream task performance. Once trained, the patch can be reused across user checkpoints derived from the same base model without per-user calibration.

Experiments on Llama-3.1-8B-Instruct and Qwen3.5-9B, covering three harmful fine-tuning datasets and three downstream tasks, demonstrate strong safety recovery with relatively small utility degradation. The authors further evaluate robustness across different fine-tuning depths, compatibility with existing merging methods, and deployment efficiency.

Critical strengths

S1. A practically relevant problem with a well-motivated deployment perspective.

The paper addresses a key challenge in FTaaS by restoring safety across heterogeneous user-fine-tuned models without requiring repeated, costly safety training or calibration. By shifting the focus from online merging calibration to offline safety patch optimization, TRACE offers an alternative perspective on post-fine-tuning realignment. Its design is particularly relevant to scenarios where multiple user checkpoints share the same base model.

S2. Promising empirical results across multiple safety and utility settings.

The experiments cover two model families, three harmful fine-tuning datasets, and three downstream tasks, including both pure and mixed fine-tuning settings. The reported results show strong safety recovery with limited degradation in downstream task performance across most evaluated settings. In addition, the fine-tuning depth experiments and deployment cost analysis provide useful evidence of robustness within the evaluated regime and the potential practical benefits of the proposed approach.

Critical weaknesses

W1. The independent contribution of trajectory-based learning is insufficiently established, and the experimental comparisons involve potentially confounding factors.

The central contribution of TRACE is to learn a transferable safety patch through continuous fine-tuning trajectories. However, the current experiments do not sufficiently demonstrate that continuous trajectory-based alternating optimization is the key factor responsible for the reported improvements.

In particular, TRACE uses harmful data, refusal supervision, and benign task data during offline patch optimization, whereas conventional merging-based baselines primarily construct safety patches from parameter differences. These methods therefore differ not only in their optimization strategies but also in their training objectives and offline computational budgets. Consequently, the substantial improvements in Table 2 cannot be attributed solely to trajectory-based learning.

Although the authors provide Base-only and Endpoint-only ablations, these comparisons do not fully establish the advantage of continuous alternating optimization over simpler approaches that train a shared safety patch using multiple corrupted model states. A more controlled comparison is needed to distinguish the contribution of trajectory-based optimization from that of broader exposure to corrupted model states and additional training supervision.

W2. Some safety–utility claims are overstated, and the evaluation does not sufficiently establish reliable safety realignment beyond the selected benchmarks.

The paper reports impressive safety recovery results, but several claims are stronger than what the experiments fully support. In particular, the abstract states that TRACE "consistently dominates the safety–utility frontier." However, Tables 2 and 7 show that, on Qwen MixedGSM8K, SafeDelta and TRACE both achieve a 100% safety rate, while SafeDelta obtains a task performance score of 78 compared with 57 for TRACE. This result contradicts a universal Pareto-dominance claim under the reported metrics.

Furthermore, the safety evaluation mainly focuses on clearly harmful requests from StrongReject. Although additional safety metrics are provided in the appendix, the paper lacks systematic evaluation of over-refusal and safety-boundary preservation. Since TRACE uses a fixed refusal response during patch training, it is important to determine whether the improved safety rates reflect appropriate safety behavior rather than an overly conservative refusal tendency. Maintaining performance on conventional downstream tasks does not necessarily rule out over-refusal on benign but safety-sensitive requests.

Additionally, Table 2 reports 100% safety rates across all six pure-profile settings without sufficient precision or detailed breakdowns. More transparent reporting and analysis would help substantiate these near-perfect results and clarify the extent to which they generalize beyond the evaluated settings.

Minor presentation issues. In Section 3.1, the discussion of limited task interference appears to reference Equation (4), whereas Equation (5) describes the safety patch optimization objective. Additionally, the conclusion describes utility deviations as "±1.7% relative," while the experimental tables report changes in percentage points. These expressions should be corrected for consistency.

Questions for authors

Q1. What is the specific advantage of continuous trajectory-based alternating optimization?

Why is it necessary to optimize the safety patch along a continuous fine-tuning trajectory? Could similar performance be achieved by training a shared safety patch on multiple corrupted checkpoints with different levels of safety degradation, without alternating backbone and patch updates? The current experiments do not sufficiently distinguish these possibilities.

Q2. How is the fairness of the baseline comparisons ensured?

TRACE uses additional refusal and benign task supervision during offline patch training, while the baselines employ different patch construction strategies and computational budgets. To what extent do these differences contribute to the reported improvements? Furthermore, how were the default repair strengths in Table 2 selected? For example, why does RESTA use λ = 1 rather than other values, and were these settings selected independently of the final evaluation results?

Q3. Does TRACE introduce over-refusal on benign but safety-sensitive requests?

The method achieves high safety rates on clearly harmful prompts, but it remains unclear whether the learned patch preserves appropriate responses to benign requests involving sensitive topics or dual-use knowledge. Could the authors provide an over-refusal evaluation to better assess whether TRACE restores the original safety boundary rather than simply strengthening refusal behavior?

Q4. How well does TRACE generalize to more diverse fine-tuning scenarios?

The current evaluation covers two base models and a limited set of harmful fine-tuning configurations. Would TRACE maintain its effectiveness on an additional model family, under different proportions of harmful data in mixed fine-tuning, or with different types of harmful fine-tuning data? These evaluations would provide stronger evidence for the generalizability of the learned safety patch.

Q5. How should the near-perfect safety rates in Table 2 be interpreted?

TRACE reports 100% safety rates in all six pure-profile settings. Could the authors provide more precise results, a breakdown across different harmful request categories, and further analysis of the agreement between safety evaluators? Such evidence would help clarify whether the reported performance is consistently strong across different harmful behaviors or primarily reflects the characteristics of the selected evaluation protocol.

4 comments

[W1 & Q1] Contribution of trajectory-based patch optimization

We thank the reviewer for this insightful question. Our main contribution is to formulate realignment as offline learning of a reusable safety patch. We therefore propose the following objective (cf. Eq. (2) in the paper):

ϕ∗=arg⁡min⁡ϕ;E(x,y)∼Usafe,Utask[L(θft⊕ϕ,x,y)],\phi^{*} = \arg\min_{\phi}\\; \mathbb{E}_{(x,y) \sim \\{\mathcal{U}_{\text{safe}},\mathcal{U}_{\text{task}}\\}} \left[\mathcal{L}(\theta_{\text{ft}} \oplus \phi, x, y)\right],

Given this objective, the suggestion to optimize a shared patch across multiple corrupted checkpoints closely reflects our original design.

We compare TRACE with a variant trained on ten corrupted checkpoints. Following the experimental settings in Section 4, both implementations use the same degradation stages, supervision data, training settings, and computational budget.

ModelEvaluation metricWithout patchShared TRACEStandard TRACE
LlamaSafety (%) ↑\uparrow9.5879.8799.68
LlamaUtility (%) ↑\uparrow48.8848.7548.88
QwenSafety (%) ↑\uparrow3.514.7999.68
QwenUtility (%) ↑\uparrow51.0150.9049.76

On both models, standard TRACE achieves substantially higher safety than Shared TRACE, improving safety by at least 19.81 pp while maintaining comparable utility. TRACE approximates the objective offline using a surrogate trajectory, with alternating backbone and patch updates. It learns as the backbone evolves, exposing the patch to closely spaced intermediate states without storing the full-parameter checkpoints.

As discussed in Section 3, we acknowledge that the alternating procedure provides only an approximation to the ideal objective (cf. Eq. (2)). Our primary goal is to demonstrate that the safety patch learning paradigm provides an effective and practical strategy for model realignment, rather than to construct an optimal patch.

We thank the reviewer again for this constructive suggestion and will include an ablation of this variant in the revised manuscript.

[W2-A & Q5] Clarifying overclaims and correcting presentation issues

Safety–utility claim. We agree that "consistently dominates" overstates the evidence. Our intended claim concerns the "Safety–Utility Trade-off across Users" experiment in Section 4.3. We will clarify this scope in the revised abstract:

In our cross-user safety–utility trade-off experiment, TRACE extends the observed frontier of the evaluated merging baselines without per-user calibration.

To give readers a clearer picture of the measured results, we will revise our presentation and report safety and utility scores to two decimal places.

Evaluator agreement. We compare the original StrongREJECT rubric with the model-specific safety evaluators on Qwen3.5-9B and Llama-3.1-8B-Instruct. For each model, we evaluate No Defense and TRACE responses against PureBad-SFT, yielding 1,252 responses across four settings:

ComparisonAgreementCohen’s κ\kappa
Rubric vs. Qwen Guard84.35%0.6840
Rubric vs. Llama Guard89.30%0.7819

Both comparisons show substantial agreement, with Cohen's κ\kappa of at least 0.68.

Presentation corrections. We thank the reviewer for pointing out these issues. We will correct the equation reference and the units for utility changes. We will double-check the wording throughout the paper to ensure clarity and consistency.

[W2-B & Q3] Over-refusal and safety-boundary preservation

TRACE targets safety recovery after user SFT while preserving performance on legitimate user tasks. On the evaluated downstream SFT tasks, TRACE shows no apparent over-refusal. As reported in Section 4.2, TRACE changes task scores by at most 0.53 percentage points across benign SFT settings and 1.74 percentage points across benign and mixed SFT settings on SAMSum, SQLCreate, and GSM8K.

We additionally evaluate model capabilities on data outside the user SFT task using MT-Bench, which assesses benign open-ended responses to multi-turn questions:

Model stateWithout TRACE ↑\uparrowWith TRACE ↑\uparrow
Llama Base5.474.04
Llama user-sft1.161.18
Qwen Base7.066.38
Qwen user-sft1.951.73

These results show that user SFT itself substantially reduces MT-Bench scores, indicating an effect on open-ended helpfulness beyond the customized task. TRACE has a relatively small effect on the user-SFT checkpoints: the score changes from 1.16 to 1.18 on Llama and from 1.95 to 1.73 on Qwen.

Since harmful SFT can severely degrade a model's safety alignment, leaving it unable to refuse even clearly harmful requests, current safety recovery work focuses on restoring its ability to refuse such requests. Recovering the model's original safety boundary, including avoiding refusals of benign but safety-sensitive requests, remains a challenge for this line of work. We acknowledge that TRACE may exhibit over-refusal on benign but safety-sensitive requests, and we will discuss this more explicitly in the revised Limitations section.

[Q2] Fairness of the baseline comparisons

We thank the reviewer for this valuable comment. Existing post-FT merging methods focus on how to apply a given safety patch to a compromised model. Our contribution shifts the focus to optimizing a reusable safety patch. To support a fair comparison, TRACE and the merging baselines use the same 5,000-example BeaverTails harmful dataset for patch preparation. The only additional data TRACE uses is WizardLM data for benign-task supervision.

Computational cost. We report the method-specific cost of offline preparation and per-user defense in seconds (cf. Table 3 in the manuscript):

Cost (s)SPFOneShotRESTAEnchTableSafeLoRASafeDeltaTRACE
Offline preparation——140.25140.25217.66216.611244.86
Per-user defense2343.1013.996.71430.29123.0826.860.41

The results demonstrate that TRACE has the lowest per-user cost. Its offline preparation is a one-time cost that can be amortized across users. For example, its total cost is lower than EnchTable's when serving three users. These measurements exclude common checkpoint loading, serialization, and API serving costs.

Repair strengths. RESTA's default λ\lambda = 1 follows its official implementation. Note that we sweep its repair strength λ\lambda from 0 to 3 to select its oracle value in Figure 2. Even with oracle selection over the tested strength grids, the baselines do not reach TRACE's safety–utility point across users. We will specify the range of repair strengths swept for each baseline in the revised manuscript.

Thank you for the detailed responses and additional experiments, which partially address my concerns. However, the independent contribution of the core method, over-refusal, and generalizability remain insufficiently validated. Therefore, I maintain my original rating.

Thank you for your thoughtful feedback. We would like to clarify the evidence concerning the core method and the scope of our evaluation.

  1. Contribution of continual alternating optimization. To examine this contribution, we devise an ablated variant without continual alternating optimization (i.e., shared TRACE), which directly trains a safety patch across ten corrupted checkpoints. On both models, standard TRACE achieves at least 19.81 percentage points higher safety while maintaining comparable utility. This comparison provides evidence for the benefit of the alternating procedure beyond exposure to multiple corrupted checkpoints.
  2. Generalizability to additional datasets and models. We agree that evaluating additional model families and harmful data sources would further strengthen the empirical evidence and clarify generalizability. However, these extensions require substantial additional computational resources. We commit to incorporating these additional evaluations into subsequent versions of the manuscript.
  3. Over-refusal within the evaluated user tasks. TRACE focuses on recovering safety after user SFT while preserving performance on users’ customized tasks. On the three benign SFT tasks, we observe no apparent over-refusal: task scores change by at most 0.53 percentage points in benign SFT settings and 1.74 percentage points across benign and mixed SFT settings.

We hope these clarifications help address your concerns. We appreciate your careful review and constructive feedback throughout the discussion.

Reviewer_E8RaOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In the paper's setting. when user do the full sft on their data and wanted to restore safety without further finetunning. To overcome this, this paper introduces TRACE. TRACES learns a reusable low-rank safety patch for checkpoints from finetuning. So in offline, TRACE alternates two steps. 1. updates the full model on harmful and benign batches with the patch switched off. 2. updates a Lora "safety patch" on refusal and benign data on top of the full model. Experiments on two backbones report strong safety recovery with small changes in downstream task scores, together with depth, cost, and training-design analyses.

Critical strengths
  1. clear and useful deployment objective. the setup make totally sense that separate offline patch learning from per-user repair addresses a pratical service setting.
  2. substantial safety revoery. Table2 shows large improment on several pure harm profiles and strong recovery for mixed profiles.
  3. holistic ablations and supplementary evaluation. the Base-only and Endpoint-only variants directly probe the trajectory hypothesis. The appendix also provides full baseline utility results, the original StrongREJECT rubric, and MMLU, so the evaluation is broader than the main classifier safety rate alone.
Critical weaknesses
  1. the provider controls training and serving and exposes only a black-box API. Input and output moderations with a guard model is therefore available. could you please help compare agains this way with a serving-time guard filter and training data filter?
  2. The paper only trains to on one fixed refusal string. and the paper mentioned it must "dominate the harmful direction". I think this is related to over-refusals. The limitations section defers over-refusal to future work, but it is central to the safety–utility claim. Please report some over refusal benchmarks an open-ended helpfulness benchmark on patched user checkpoints, beyond in-domain accuracy and base-model MMLU.
Questions for authors

as above

1 comment

[W1] Comparison with training-data and serving-time filters

We thank the reviewer for this valuable suggestion. TRACE introduces a reusable safety patch to restore the model's inherent safety after user SFT. Input/output guards provide a complementary layer of protection and can be combined with TRACE to address practical deployment needs: the two approaches operate at orthogonal levels, within the model and at serving time.

Training-data filtering. We deployed a training-data filter that uses a model-specific guard to screen user-uploaded training data and remove harmful examples. We follow the experimental setup in Section 4 and report the results on MixedSAMSum.

ModelMethodSafety (%) ↑\uparrowUtility (%) ↑\uparrow
LlamaTrainingData Filter9.5848.88
LlamaTRACE100.0047.00
QwenTrainingData Filter3.1951.00
QwenTRACE98.0849.52

The filter leaves substantial safety degradation in both runs. The reported TRACE results have higher safety and lower task utility. We will include these results in the revised manuscript.

Serving-time guards. TRACE and service-level guards address complementary dimensions of safety and can be combined in deployment. To assess how harmful SFT affects the model's inherent safety alignment, we evaluate model responses on StrongREJECT, whose prompts provide clear signals for refusal. When a guard model that has not undergone user SFT screens these prompts, the question becomes whether the guard can detect StrongREJECT requests. Since our focus is on the fine-tuned model's safety recovery, we consider evaluating integration with service-level guards outside the scope of this study. We will clarify our evaluation scope in the revised Limitations section.

[W2] Over-refusal and open-ended helpfulness

We thank the reviewer for this constructive suggestion. This paper focuses on user SFT, with particular attention to recovering safety after harmful SFT, which can lead a model to assist with clearly harmful requests. Our goal is to restore safety while preserving performance on legitimate user tasks and avoiding over-refusal after benign user SFT. Section 4.2 reports utility on three benign tasks: SAMSum, SQLCreate, and GSM8K. TRACE preserves task performance, with changes of at most 0.53 percentage points across benign SFT settings and 1.74 percentage points across benign and mixed SFT settings. These results show no apparent over-refusal on the evaluated user SFT tasks.

We agree that open-ended helpfulness is an important aspect of evaluation. To assess capability preservation beyond the user SFT task, we report MT-Bench scores for both the base model and a user-SFT model fine-tuned on SAMSum, before and after applying TRACE:

Model stateWithout TRACE ↑\uparrowWith TRACE ↑\uparrow
Llama Base5.474.04
Llama user-sft1.161.18
Qwen Base7.066.38
Qwen user-sft1.951.73

User SFT itself substantially reduces MT-Bench scores, indicating an effect on open-ended helpfulness beyond the customized task. TRACE has a relatively small effect on the user-SFT checkpoints: the score changes from 1.16 to 1.18 on Llama and from 1.95 to 1.73 on Qwen.

Since harmful SFT can severely degrade a model's safety alignment, leaving it unable to refuse even clearly harmful requests, current safety recovery work focuses on restoring its ability to refuse such requests. Recovering the model's original safety boundary, including avoiding refusals of benign but safety-sensitive requests, remains a challenge for this line of work. We will discuss this more explicitly in the revised Limitations section.

Reviewer_uS4kOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

TRACE targets Fine-Tuning-as-a-Service, where user SFT erodes base-model safety. Existing merging defenses (RESTA, EnchTable, SafeLoRA, SafeDelta) need a per-user coefficient trading safety against utility, which cannot serve different users with one shared setting. TRACE instead trains a single LoRA safety patch offline via an alternating simulate-and-recover procedure (Algorithm 1): Phase A drifts a surrogate backbone on harmful+benign batches, Phase B freezes the drifted backbone and updates the patch. The patch is added directly with no per-user tuning. Method is evaluated for Two models, three harmful sets and three task sets.

Critical strengths

The low-rank safety subspace finding in Section 3.2 is a genuinely interesting empirical result.

The SFT-depth robustness study (Section 4.5, Figure 4) is a real stress test that most baselines fail outright.

Figure 2 shows merging baselines genuinely cannot find one operating point for two different users at once which validate the claim.

Critical weaknesses

Section 4.1 says the authors manually inspect a subset... and do not observe apparent misjudgments, with no sample size given, and $D_{ref}$ is one canned refusal sentence (Appendix A) used as the sole supervision target for every harmful prompt. Combined with the paper deferring all ``adaptive attacks'' to future work (Appendix C) despite having an adversarial threat model (Section 2), the near-100%100\% safety numbers in Table 2 do not tell me the method is robust to anything beyond the exact static distribution tested.

Appendix B.5 states on Qwen MixedGSM8K, SafeDelta scores 78%, even higher than No Defense's \$59\%\$, while Table 7 shows TRACE at $57%$ utility on that cell, and Table 2 shows SafeDelta tied with TRACE on safety ($99%$ vs.$100%$). SafeDelta strictly beats TRACE on this. This is buried in the appendix with a one-sentence, untested guess, and never touches the abstract's consistently dominates language.

Table 6's ablation only tests structural variants (Base-only, Endpoint-only, No task preservation, Harmful-only trajectory) , it never varies $r$, $\eta_\phi$, or $E$, so there is no evidence the 100%100\% safety numbers in Table 2 are robust to this choice rather than a lucky setting.

Questions for authors

See Weaknesses

4 comments

[W1] Reliability and generalizability of the safety evaluation

We thank the reviewer for the valuable comment.

Human evaluation. We randomly sampled 186 (~10%) of TRACE responses across two base models (Llama and Qwen) and three harmful fine-tuning datasets. An expert assessed the safety of all sampled responses, and the judgments agreed with the paper's safety evaluator on 100% (186/186). The revised manuscript will report the sampling protocol and agreement in place of the original qualitative statement.

Adaptive data-selection attack. We consider a dynamic attack in which the attacker updates its fine-tuning data based on feedback from the patched model. In the first round, the attacker randomly samples 500 prompts and uploads them to the service provider to fine-tune the model. After each round, the attacker evaluates the patched model on the uploaded prompts and retains examples whose responses are judged unsafe. Unsuccessful examples are discarded, and the remaining slots are filled by sampling unused examples without replacement. The base model and TRACE patch remains frozen throughout.

We run this adaptive attack for 10 rounds on Llama-3.1-8B-Instruct, using 500 SafeRLHF prompt–completion pairs per round and following the experimental settings in Section 4:

Attack round12345678910
Unsafe-response rate ↓\downarrow0.000.000.000.200.601.001.402.001.402.00

The unsafe-response rate never exceeds 2.00%. These rates are measured on each round's upload, which changes over time. Separately, the final patched model achieves 99.36% safety on StrongREJECT.

Refusal-target ablation. To assess the fixed refusal target, we devise a TRACE variant that uses diverse refusal responses in place of the single refusal string. We first collect the base model's responses to BeaverTails prompts as these diverse refusal targets, then train the variant and compare it with standard TRACE.

MethodLlama Safety ↑\uparrowLlama Utility ↑\uparrowQwen Safety ↑\uparrowQwen Utility ↑\uparrow
No patch9.5848.884.7950.99
Diverse-response TRACE99.6848.9676.0449.42
Standard TRACE99.6848.8899.6849.76

On Llama, both variants reach 99.68% safety. On Qwen, the fixed target achieves higher safety (99.68% versus 76.04%) with similar utility.

[W2] Clarifying the scope of our safety–utility claim

We thank the reviewer for this clarification. Our intended claim concerns the "Safety–Utility Trade-off across Users" experiment. We will make this scope explicit in the revised abstract:

In our cross-user safety–utility trade-off experiment, TRACE extends the observed frontier of the evaluated merging baselines without per-user calibration.

We will carefully review the remaining claims for consistency with the experimental evidence.

[W3] Sensitivity to patch rank, learning rate, and training duration

We thank the reviewer for the constructive comments. We test patch rank, learning rate, utility weight, and training duration on Llama. In each test case, we vary only the parameter under evaluation while keeping the training data and other configurations fixed.

SettingRank rrLR ηϕη_{ϕ}Epoch EEUtility Weight wwStandard
Value16321e-53e-546510—
Safety (%) ↑\uparrow99.6899.6899.3699.3699.3699.3699.3699.3699.68
Utility (%) ↑\uparrow48.8948.6648.7748.8249.0648.9349.1848.8948.88

All settings shown retain at least 99.36% safety, with small utility changes. These results show that TRACE is robust to the tested choices of hyperparameters. We will include a parameter sensitivity analysis section presenting these experiments in the revised manuscript.

Thank you to the authors for their detailed responses and additional experiments, which address some of my initial concerns. However, questions regarding robustness and generalizability beyond the evaluated settings, as well as sensitivity to refusal supervision, remain insufficiently resolved. While the rebuttal strengthens the empirical evidence, I maintain my original rating.

Thank you for your follow-up and for acknowledging our additional experiments. We would like to clarify three points regarding these results:

  1. Evaluation beyond the original static settings. We added an adaptive data-selection attack in which the attacker iteratively updates its fine-tuning data based on feedback from the patched model. After 10 rounds of this dynamic attack, the patched model still achieves 99.36% safety, providing evidence of robustness beyond the original static evaluation.

  2. Robustness across experimental configurations. We varied patch rank, learning rate, utility weight, and training duration. Across all tested configurations, TRACE maintains high safety rates, with small utility changes. These results support its robustness to the configuration choices examined.

  3. Empirical support for the refusal-supervision design. We evaluated a TRACE variant using diverse refusal responses instead of the fixed refusal target. On Qwen, standard TRACE achieves 99.68%, compared with 76.04% for the diverse-response variant, with similar utility. These results provide empirical support for our choice of a fixed refusal target.

If you have any remaining questions or concerns, please let us know which specific points you would like us to clarify. We would be happy to provide further experimental details or additional results to address those concerns.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary

Dear AC and reviewers,

We sincerely thank the AC for handling our submission and all reviewers for their careful assessments and constructive feedback, which have helped improve our work. This paper addresses safety degradation after user SFT and aims to restore safety alignment without compromising customized task performance. Existing work focuses on designing merging methods to apply a given safety patch to a fine-tuned model. We shift the focus to learning a more effective safety patch through optimization. Specifically, TRACE trains a universal safety patch across simulated fine-tuning trajectories, enabling its reuse across user checkpoints derived from the same base model.

Below, we summarize the main experiments and clarifications provided during the rebuttal.

Main refinements

  • Ablation experiments on key design choices. We complemented our ablation studies with two variants to assess the importance of continual alternating training and fixed refusal supervision. The first removes continual alternating training and trains the patch across ten corrupted checkpoints. Standard TRACE achieves 99.68% safety on both models, compared with 79.87% on Llama and 4.79% on Qwen for this variant, with comparable utility (83s7 W1/Q1). The second replaces the fixed refusal target with diverse responses. On Qwen, the fixed target achieves 99.68% safety versus 76.04% for diverse responses, with similar utility, supporting our refusal-supervision design (uS4k W1).

  • Robustness tests. We assessed TRACE’s robustness to both adaptive attacks and variations in training hyperparameters. TRACE retains 99.36% safety after the adaptive attack. Across tested variations in patch rank, learning rate, utility weight, and training duration, it maintains at least 99.36% safety, with small utility changes (uS4k W1/W3).

  • Strengthening empirical soundness. We expanded the evaluation to provide stronger evidence for TRACE’s safety and utility. We added human verification (uS4k W1), helpfulness evaluations (E8Ra W2; 83s7 Q3), and data filtering baselines (E8Ra W1), and clarified baseline settings and costs (83s7 Q2). Moreover, we commit to incorporating evaluations on additional models and datasets into the next manuscript version (83s7 Q4).

Remaining concern: over-refusal on benign but sensitive requests

Reviewer 83s7 (W2/Q3) raised concerns about potential over-refusal on benign but safety-sensitive requests. We acknowledge that this exposes a limitation of TRACE. Recovering a more precise safety boundary after harmful SFT remains a broader challenge for this line of realignment research.

However, in our main user-SFT setting, we observe no apparent over-refusal on the user-customized tasks. Across SAMSum, SQLCreate, and GSM8K, task scores change by at most 0.53 percentage points in benign SFT settings and 1.74 percentage points across mixed SFT settings. These results support utility preservation on the user tasks. We will discuss this limitation explicitly in the revised manuscript.

We sincerely appreciate the AC’s and reviewers’ time and consideration of these additional results and clarifications.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.