- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies the problem of behavioral forgetting that occurs when large language models are adapted to downstream tasks and proposes Resistance Training (ResT). The method first trains a LoRA that expresses undesirable behaviors. During downstream training, it freezes this LoRA and updates the Backbone. Finally, it removes the LoRA. The authors focus on comparing different sources of Resistance and fitting methods. They validate the method’s protective effects on safety refusal and hallucination on MedQA and FinQA. They also explain its underlying mechanism through linearization analysis.
Critical strengths
S1. The problem is important and the motivation is well justified. The paper finds that even with OPD and OPSD, improvements in downstream task performance can still be accompanied by declines in safety refusal and factual restraint, revealing a behavioral forgetting problem in On-Policy Training that deserves attention.
S2. The study of Resistance Construction is valuable. The study of Resistance Construction is valuable. The paper systematically compares Generic, Targeted, and Forgetting-Derived Resistance and finds that On-Policy Fitting provides stronger hallucination protection in both downstream OPD and SFT, offering valuable empirical conclusions for how to construct behavioral interventions during training.
Critical weaknesses
W1. The methodological novelty is relatively limited (to my knowledge). The core Fit–Freeze–Remove framework of ResT is essentially the same as that of Security Vectors, and its main novelty lies in the study of On-Policy Resistance Fitting and different Resistance Construction strategies (if I was wrong, correct me. I'm not very familiar with this area). These extensions have some value, but the methodological originality remains limited. Can the authors further clarify the core technical contributions of this paper relative to Security Vectors and explain what advantages On-Policy Resistance Fitting offers that existing methods cannot achieve?
W2. The advantages of On-Policy Resistance Fitting lack rigorous controlled validation. There may be potential differences between Response Supervision and On-Policy Fitting in terms of the number of training tokens, computational budget, and induced Resistance Strength. In addition, in Appendix Table 7, On-Policy Fitting actually performs worse under the downstream OPSD setting. Can the authors conduct more rigorous controlled experiments under matched computational budgets and Resistance Strength, and explain the discrepancy between the results in Table 7 and the main conclusions?
Questions for authors
See weaknesses.
4 comments
Response to Reviewer QnCc (Part 2)
Question
The response-supervised and on-policy fitting comparisons may differ in training tokens, compute, and resistance strength. Can these factors be better controlled, and why does on-policy fitting perform worse under OPSD in Table 7?
Response
Thank you for suggesting a more controlled comparison. We expanded Tables 5–6 to three training seeds (42, 43, and 44) and use the full fitting-duration sweeps to compare each method at its best downstream-task performance. This addresses whether the protection advantage depends on choosing a particular fitting duration.
The grid covers 0.1, 0.25, 0.5, and 1 epoch for Safety, and 0.5, 1, 2, and 4 epochs for hallucination, allowing each method to explore different degrees of fitting and the resulting load strengths. Within each seed, source, protection setting, and downstream objective, we select the highest-MedQA configuration for each fitting method, breaking ties by the smaller epoch, then average the paired task and protection scores. Both methods use the same search grid and task-accuracy selection rule. Figure 3 reports the following three-seed means:
| Downstream | Source | Response: Task acc. ↑ | On-policy: Task acc. ↑ | Response: Hallucination ↓ | On-policy: Hallucination ↓ |
|---|---|---|---|---|---|
| OPD | Generic | 80.70 | 81.20 | 43.52 | 29.24 |
| OPD | Targeted | 80.05 | 81.23 | 41.40 | 30.82 |
| SFT | Generic | 74.55 | 74.99 | 40.35 | 31.55 |
| SFT | Targeted | 75.23 | 76.25 | 41.88 | 17.56 |
On-policy fitting achieves higher mean task accuracy and lower mean hallucination in all four comparisons. The full sweeps also show lower mean hallucination at every shared fitting duration, across both sources and downstream objectives. The advantage is also not explained by the number of fitting sequences: on-policy fitting at its shortest duration (0.5 epoch; 2,000 sequences) has lower mean hallucination than response supervision at its longest (4 epochs; 4,000 sequences) under OPD for both sources (35.49 vs. 39.59 generic; 30.82 vs. 41.40 targeted) and under SFT for the targeted source (32.03 vs. 43.81), and is comparable for the generic source under SFT (38.38 vs. 39.00). Thus, the advantage persists when each method can select its fitting duration for downstream performance, and across the tested durations themselves. These results support on-policy fitting as a strong default for hallucination-oriented load construction in the settings studied. For Safety, targeted loads improve mean refusal under both downstream objectives; generic loads show smaller, mixed effects. Full results and selection details are in Section 3.3 and Appendix D.4.
Response supervision fits fixed answers, while on-policy fitting uses sampled trajectories and teacher distributions. Equal epoch or sequence counts therefore need not imply equal token cost or resistance strength. Our sweeps compare the resulting task–retention outcomes across fitting schedules. Table 7 adds a sequence-matched control, using the same base-model source lineage and 4,000 sequences: four epochs of response supervision versus one epoch of on-policy fitting with four rollouts per prompt.
Under OPD, this control reduces hallucination from 40.75 to 27.31, with task accuracy changing from 80.28 to 79.89. Under OPSD, the fixed one-epoch on-policy load yields task accuracy 70.54 and hallucination 44.64, versus 77.22 and 37.31 for response fitting. This shows that the fitting schedule and downstream objective affect the outcome. We have scoped the conclusion to the consistent advantage observed in the expanded OPD/SFT hallucination sweeps, while retaining Table 7 as a complementary result.
We hope these responses and the revised manuscript address your concerns. We would be glad to clarify any remaining points during the discussion period!
Thanks. I decide to keep my score as it is already positive.
Thanks
On behalf of all authors, thank you for taking the time to read our rebuttal and for your positive assessment of our work!
We sincerely appreciate your constructive feedback, which has helped us improve the paper. We remain available should any further questions arise during the discussion period!
Response to Reviewer QnCc (Part 1)
Thank you for your thoughtful comments and helpful suggestions. We have revised the manuscript and added experiments to address your concerns. Below, we respond to each point and explain the corresponding changes. We hope these responses and additions clarify our work and address your questions.
Question
The fit–freeze–remove framework resembles Security Vectors. What are the core technical contributions relative to SV, and what advantages does on-policy resistance fitting offer?
Response
Thank you for raising this question. ResT differs from Security Vectors (SV), which we cite and discuss, in how the load is constructed and in the analysis of when it yields protection. Our contribution is an analysis-driven account of how to construct resistance that downstream task learning can turn into behavioral protection: we characterize a load by the behavioral displacement it induces and derive sufficient conditions for protection, and the fitting objective and load sources follow from this characterization. We have clarified three contributions in the revised Introduction and Section 2:
Figure 3 and the expanded three-seed sweeps show the practical benefit of on-policy fitting: lower mean hallucination and higher mean task accuracy across both sources under downstream OPD and SFT. We give the results and selection procedure in our response to W2 (Part 2/2).
We also added a capacity-matched SV-style baseline, using min–min + KL fitting on the same targeted sources. Targeted ResT has higher refusal, lower hallucination, and higher independent Truthfulness means in all four main comparisons, with hallucination 18.9–29.3 points lower. Truthfulness is the MC2 score on TruthfulQA [1], an independent factuality benchmark added in the revision: the probability mass assigned to true answers, where higher is better. The factuality results are summarized below; all entries are mean ± sample SD over three training seeds. Full task, refusal, and benign-compliance results are in Tables 1–2, with configuration details in Appendix D.3.
Please also see our General Response on Contributions, Novelty, and New Experiments, posted for all reviewers and the AC, for further details on the relationship to SV, resistance sources, and analysis. It also summarizes the experiments added during rebuttal that strengthen these contributions, including three-seed comparisons, over-refusal and independent factuality evaluation, additional model sizes and a second model family, and DPO/GRPO; we would be glad if you find them of interest!
References
[1] TruthfulQA: Measuring How Models Mimic Human Falsehoods. Stephanie Lin, Jacob Hilton, and Owain Evans. ACL 2022. ACL Anthology.