- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates post-fine-tuning safety realignment for large language models in Fine-Tuning-as-a-Service scenarios. The authors argue that existing merging-based realignment methods often rely on calibrating repair strength, which can lead to an unfavorable safety–utility trade-off across heterogeneous user fine-tuned models.
To address this issue, the paper proposes TRACE, a trajectory-based safety patch learning framework. TRACE alternates between simulating safety-eroding fine-tuning and optimizing a low-rank safety patch on progressively modified model states. The patch is trained with both refusal supervision and benign task data to restore safety while minimizing interference with downstream task performance. Once trained, the patch can be reused across user checkpoints derived from the same base model without per-user calibration.
Experiments on Llama-3.1-8B-Instruct and Qwen3.5-9B, covering three harmful fine-tuning datasets and three downstream tasks, demonstrate strong safety recovery with relatively small utility degradation. The authors further evaluate robustness across different fine-tuning depths, compatibility with existing merging methods, and deployment efficiency.
Critical strengths
S1. A practically relevant problem with a well-motivated deployment perspective.
The paper addresses a key challenge in FTaaS by restoring safety across heterogeneous user-fine-tuned models without requiring repeated, costly safety training or calibration. By shifting the focus from online merging calibration to offline safety patch optimization, TRACE offers an alternative perspective on post-fine-tuning realignment. Its design is particularly relevant to scenarios where multiple user checkpoints share the same base model.
S2. Promising empirical results across multiple safety and utility settings.
The experiments cover two model families, three harmful fine-tuning datasets, and three downstream tasks, including both pure and mixed fine-tuning settings. The reported results show strong safety recovery with limited degradation in downstream task performance across most evaluated settings. In addition, the fine-tuning depth experiments and deployment cost analysis provide useful evidence of robustness within the evaluated regime and the potential practical benefits of the proposed approach.
Critical weaknesses
W1. The independent contribution of trajectory-based learning is insufficiently established, and the experimental comparisons involve potentially confounding factors.
The central contribution of TRACE is to learn a transferable safety patch through continuous fine-tuning trajectories. However, the current experiments do not sufficiently demonstrate that continuous trajectory-based alternating optimization is the key factor responsible for the reported improvements.
In particular, TRACE uses harmful data, refusal supervision, and benign task data during offline patch optimization, whereas conventional merging-based baselines primarily construct safety patches from parameter differences. These methods therefore differ not only in their optimization strategies but also in their training objectives and offline computational budgets. Consequently, the substantial improvements in Table 2 cannot be attributed solely to trajectory-based learning.
Although the authors provide Base-only and Endpoint-only ablations, these comparisons do not fully establish the advantage of continuous alternating optimization over simpler approaches that train a shared safety patch using multiple corrupted model states. A more controlled comparison is needed to distinguish the contribution of trajectory-based optimization from that of broader exposure to corrupted model states and additional training supervision.
W2. Some safety–utility claims are overstated, and the evaluation does not sufficiently establish reliable safety realignment beyond the selected benchmarks.
The paper reports impressive safety recovery results, but several claims are stronger than what the experiments fully support. In particular, the abstract states that TRACE "consistently dominates the safety–utility frontier." However, Tables 2 and 7 show that, on Qwen MixedGSM8K, SafeDelta and TRACE both achieve a 100% safety rate, while SafeDelta obtains a task performance score of 78 compared with 57 for TRACE. This result contradicts a universal Pareto-dominance claim under the reported metrics.
Furthermore, the safety evaluation mainly focuses on clearly harmful requests from StrongReject. Although additional safety metrics are provided in the appendix, the paper lacks systematic evaluation of over-refusal and safety-boundary preservation. Since TRACE uses a fixed refusal response during patch training, it is important to determine whether the improved safety rates reflect appropriate safety behavior rather than an overly conservative refusal tendency. Maintaining performance on conventional downstream tasks does not necessarily rule out over-refusal on benign but safety-sensitive requests.
Additionally, Table 2 reports 100% safety rates across all six pure-profile settings without sufficient precision or detailed breakdowns. More transparent reporting and analysis would help substantiate these near-perfect results and clarify the extent to which they generalize beyond the evaluated settings.
Minor presentation issues. In Section 3.1, the discussion of limited task interference appears to reference Equation (4), whereas Equation (5) describes the safety patch optimization objective. Additionally, the conclusion describes utility deviations as "±1.7% relative," while the experimental tables report changes in percentage points. These expressions should be corrected for consistency.
Questions for authors
Q1. What is the specific advantage of continuous trajectory-based alternating optimization?
Why is it necessary to optimize the safety patch along a continuous fine-tuning trajectory? Could similar performance be achieved by training a shared safety patch on multiple corrupted checkpoints with different levels of safety degradation, without alternating backbone and patch updates? The current experiments do not sufficiently distinguish these possibilities.
Q2. How is the fairness of the baseline comparisons ensured?
TRACE uses additional refusal and benign task supervision during offline patch training, while the baselines employ different patch construction strategies and computational budgets. To what extent do these differences contribute to the reported improvements? Furthermore, how were the default repair strengths in Table 2 selected? For example, why does RESTA use λ = 1 rather than other values, and were these settings selected independently of the final evaluation results?
Q3. Does TRACE introduce over-refusal on benign but safety-sensitive requests?
The method achieves high safety rates on clearly harmful prompts, but it remains unclear whether the learned patch preserves appropriate responses to benign requests involving sensitive topics or dual-use knowledge. Could the authors provide an over-refusal evaluation to better assess whether TRACE restores the original safety boundary rather than simply strengthening refusal behavior?
Q4. How well does TRACE generalize to more diverse fine-tuning scenarios?
The current evaluation covers two base models and a limited set of harmful fine-tuning configurations. Would TRACE maintain its effectiveness on an additional model family, under different proportions of harmful data in mixed fine-tuning, or with different types of harmful fine-tuning data? These evaluations would provide stronger evidence for the generalizability of the learned safety patch.
Q5. How should the near-perfect safety rates in Table 2 be interpreted?
TRACE reports 100% safety rates in all six pure-profile settings. Could the authors provide more precise results, a breakdown across different harmful request categories, and further analysis of the agreement between safety evaluators? Such evidence would help clarify whether the reported performance is consistently strong across different harmful behaviors or primarily reflects the characteristics of the selected evaluation protocol.
4 comments
[W2-A & Q5] Clarifying overclaims and correcting presentation issues
Safety–utility claim. We agree that "consistently dominates" overstates the evidence. Our intended claim concerns the "Safety–Utility Trade-off across Users" experiment in Section 4.3. We will clarify this scope in the revised abstract:
In our cross-user safety–utility trade-off experiment, TRACE extends the observed frontier of the evaluated merging baselines without per-user calibration.
To give readers a clearer picture of the measured results, we will revise our presentation and report safety and utility scores to two decimal places.
Evaluator agreement. We compare the original StrongREJECT rubric with the model-specific safety evaluators on Qwen3.5-9B and Llama-3.1-8B-Instruct. For each model, we evaluate No Defense and TRACE responses against PureBad-SFT, yielding 1,252 responses across four settings:
| Comparison | Agreement | Cohen’s |
|---|---|---|
| Rubric vs. Qwen Guard | 84.35% | 0.6840 |
| Rubric vs. Llama Guard | 89.30% | 0.7819 |
Both comparisons show substantial agreement, with Cohen's of at least 0.68.
Presentation corrections. We thank the reviewer for pointing out these issues. We will correct the equation reference and the units for utility changes. We will double-check the wording throughout the paper to ensure clarity and consistency.
[W2-B & Q3] Over-refusal and safety-boundary preservation
TRACE targets safety recovery after user SFT while preserving performance on legitimate user tasks. On the evaluated downstream SFT tasks, TRACE shows no apparent over-refusal. As reported in Section 4.2, TRACE changes task scores by at most 0.53 percentage points across benign SFT settings and 1.74 percentage points across benign and mixed SFT settings on SAMSum, SQLCreate, and GSM8K.
We additionally evaluate model capabilities on data outside the user SFT task using MT-Bench, which assesses benign open-ended responses to multi-turn questions:
| Model state | Without TRACE | With TRACE |
|---|---|---|
| Llama Base | 5.47 | 4.04 |
| Llama user-sft | 1.16 | 1.18 |
| Qwen Base | 7.06 | 6.38 |
| Qwen user-sft | 1.95 | 1.73 |
These results show that user SFT itself substantially reduces MT-Bench scores, indicating an effect on open-ended helpfulness beyond the customized task. TRACE has a relatively small effect on the user-SFT checkpoints: the score changes from 1.16 to 1.18 on Llama and from 1.95 to 1.73 on Qwen.
Since harmful SFT can severely degrade a model's safety alignment, leaving it unable to refuse even clearly harmful requests, current safety recovery work focuses on restoring its ability to refuse such requests. Recovering the model's original safety boundary, including avoiding refusals of benign but safety-sensitive requests, remains a challenge for this line of work. We acknowledge that TRACE may exhibit over-refusal on benign but safety-sensitive requests, and we will discuss this more explicitly in the revised Limitations section.
[Q2] Fairness of the baseline comparisons
We thank the reviewer for this valuable comment. Existing post-FT merging methods focus on how to apply a given safety patch to a compromised model. Our contribution shifts the focus to optimizing a reusable safety patch. To support a fair comparison, TRACE and the merging baselines use the same 5,000-example BeaverTails harmful dataset for patch preparation. The only additional data TRACE uses is WizardLM data for benign-task supervision.
Computational cost. We report the method-specific cost of offline preparation and per-user defense in seconds (cf. Table 3 in the manuscript):
| Cost (s) | SPF | OneShot | RESTA | EnchTable | SafeLoRA | SafeDelta | TRACE |
|---|---|---|---|---|---|---|---|
| Offline preparation | — | — | 140.25 | 140.25 | 217.66 | 216.61 | 1244.86 |
| Per-user defense | 2343.10 | 13.99 | 6.71 | 430.29 | 123.08 | 26.86 | 0.41 |
The results demonstrate that TRACE has the lowest per-user cost. Its offline preparation is a one-time cost that can be amortized across users. For example, its total cost is lower than EnchTable's when serving three users. These measurements exclude common checkpoint loading, serialization, and API serving costs.
Repair strengths. RESTA's default = 1 follows its official implementation. Note that we sweep its repair strength from 0 to 3 to select its oracle value in Figure 2. Even with oracle selection over the tested strength grids, the baselines do not reach TRACE's safety–utility point across users. We will specify the range of repair strengths swept for each baseline in the revised manuscript.
Thank you for the detailed responses and additional experiments, which partially address my concerns. However, the independent contribution of the core method, over-refusal, and generalizability remain insufficiently validated. Therefore, I maintain my original rating.
Thank you for your thoughtful feedback. We would like to clarify the evidence concerning the core method and the scope of our evaluation.
- Contribution of continual alternating optimization. To examine this contribution, we devise an ablated variant without continual alternating optimization (i.e., shared TRACE), which directly trains a safety patch across ten corrupted checkpoints. On both models, standard TRACE achieves at least 19.81 percentage points higher safety while maintaining comparable utility. This comparison provides evidence for the benefit of the alternating procedure beyond exposure to multiple corrupted checkpoints.
- Generalizability to additional datasets and models. We agree that evaluating additional model families and harmful data sources would further strengthen the empirical evidence and clarify generalizability. However, these extensions require substantial additional computational resources. We commit to incorporating these additional evaluations into subsequent versions of the manuscript.
- Over-refusal within the evaluated user tasks. TRACE focuses on recovering safety after user SFT while preserving performance on users’ customized tasks. On the three benign SFT tasks, we observe no apparent over-refusal: task scores change by at most 0.53 percentage points in benign SFT settings and 1.74 percentage points across benign and mixed SFT settings.
We hope these clarifications help address your concerns. We appreciate your careful review and constructive feedback throughout the discussion.
[W1 & Q1] Contribution of trajectory-based patch optimization
We thank the reviewer for this insightful question. Our main contribution is to formulate realignment as offline learning of a reusable safety patch. We therefore propose the following objective (cf. Eq. (2) in the paper):
ϕ∗=argminϕ;E(x,y)∼Usafe,Utask[L(θft⊕ϕ,x,y)],
Given this objective, the suggestion to optimize a shared patch across multiple corrupted checkpoints closely reflects our original design.
We compare TRACE with a variant trained on ten corrupted checkpoints. Following the experimental settings in Section 4, both implementations use the same degradation stages, supervision data, training settings, and computational budget.
On both models, standard TRACE achieves substantially higher safety than Shared TRACE, improving safety by at least 19.81 pp while maintaining comparable utility. TRACE approximates the objective offline using a surrogate trajectory, with alternating backbone and patch updates. It learns as the backbone evolves, exposing the patch to closely spaced intermediate states without storing the full-parameter checkpoints.
As discussed in Section 3, we acknowledge that the alternating procedure provides only an approximation to the ideal objective (cf. Eq. (2)). Our primary goal is to demonstrate that the safety patch learning paradigm provides an effective and practical strategy for model realignment, rather than to construct an optimal patch.
We thank the reviewer again for this constructive suggestion and will include an ablation of this variant in the revised manuscript.