- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies what happens when two post-training defenses for different risks are applied to an LLM one after the other. The authors build ConflictEval, which applies six defenses for safety (DPO, CAT), privacy (NPO, RMU) and fairness (Unbias, task-vector editing) to six models, giving 144 ordered compositions. They measure how much of the first defense's gain is reversed by the second defense with the Relative Regression Rate (RRR), and they find that 57 compositions show a defense conflict, where the second defense reduces its own risk while the earlier risk increases. On 11 selected cases, they estimate one risk direction per layer and show that the second defense tends to move the activations against the first defense's shift along this direction when regression occurs. Based on this observation, they propose CTDR, a regularizer that penalizes such opposing shifts, and show on six conflict cases that it reduces RRR while keeping most of the second defense's gain.
Critical strengths
The question is practical, because deployed models do receive new defenses over time, and the paper states it clearly. The evaluation covers three risk types, six defenses and six models, and the full checkpoint scores are reported in the appendix, so other researchers can reuse the results. RRR is simple and easy to interpret. The reverse-order pairs in Section 3 are also a nice design, since they show that the same two defenses can behave differently depending on the order.
Critical weaknesses
-
All six defenses are weight updates, and prompt-based defenses are not considered. Many deployed models also rely on safety system prompts, and it is not clear whether such defenses conflict with later weight updates in the same way. I suggest adding at least one prompt-based defense to the evaluation.
-
The paper does not compare CTDR with any other mitigation method. Two consecutive weight updates naturally make the model forget part of the first one, and many existing methods target this problem, such as EWC-style regularization, replay of the first defense's data, a KL constraint toward the model after the first defense, and safety-preserving fine-tuning methods such as SafeLoRA. Without these baselines, it is hard to tell whether the gain of CTDR comes from the directional design or simply from keeping the model close to the first-defense model.
-
The analysis assumes that each risk is captured by a single direction per layer, estimated as a mean difference on the initial model. Defenses such as DPO and adversarial training likely change the model along many directions, so a single direction is a strong simplification. I suggest that the authors discuss this assumption and, if possible, check whether the results hold with a multi-dimensional subspace.
-
Each composition is trained only once, and any increase in the earlier risk counts as a conflict. In Table 2, 19 of the 57 conflicts have an RRR below 0.05, so some of them may come from training noise.
-
Safety is measured only by attack success rate, without any over-refusal benchmark such as XSTest. For example, CAT reduces the attack success rate of Qwen-L to nearly zero, and CAT is also the second defense that most often increases fairness risk. An over-refusal check would show whether part of these effects comes from the model refusing benign requests.
-
A few writing details need fixing. The caption of Table 2 repeats one sentence, and the ethics statement mentions "Catastrophic Collapses", which does not appear anywhere else in the paper.
Questions for authors
Please see weaknesses
3 comments
Part 2/2
To W4: Training randomness and small-RRR conflicts
Thank you for raising this concern. We agree that small positive RRR values may be sensitive to training randomness. We repeated eight compositions with two additional training seeds (11 and 22). Across all three seeds, all seven regression cases retain positive RRR, while the non-regression control remains negative.
Notably, the mild Qwen-L NPO→CAT case yields RRRs of 0.045, 0.050, and 0.003. Although the magnitude varies, regression persists across all three seeds, providing evidence that this observed conflict is not specific to a single training realization.
Table J. Sequential RRR across three training seeds.
| Case | Model | Defense sequence | Seed 0 (Original) | Seed 11 | Seed 22 |
|---|---|---|---|---|---|
| 1 | Llama-S | DPO→RMU | 0.310 | 0.280 | 0.245 |
| 2 | Llama-S | DPO→NPO | 0.228 | 0.167 | 0.156 |
| 3 | Gemma-S | DPO→NPO | 0.697 | 0.974 | 0.860 |
| 4 | Gemma-L | CAT→NPO | 0.222 | 0.243 | 0.128 |
| 5 | Qwen-L | Unbias→CAT | 0.526 | 0.588 | 0.534 |
| 6 | Qwen-L | NPO→CAT | 0.045 | 0.050 | 0.003 |
| 7 | Llama-S | RMU→DPO | −0.058 | −0.025 | −0.035 |
| 8 | Gemma-S | DPO→Unbias | 0.798 | 0.858 | 0.710 |
To W5: Over-refusal evaluation
Thank you for highlighting this concern. We added a WildGuard-based benign-request ((XSTest)) refusal evaluation on Llama-S and Qwen-S, covering four sequential defense compositions (Table J).
Although CAT-based compositions exhibit higher benign refusal rates than their DPO counterparts, all refusal rates remain below 8%, suggesting relatively limited over-refusal rather than widespread rejection of benign requests. Thus, over-refusal does not appear pervasive in the evaluated settings, although its contribution to the observed ASR changes cannot be fully ruled out. We will include these results and discuss their limitations in the revised appendix.
Table J. Benign-request refusal rates evaluated with WildGuard (% ↓).
| Defense sequence | Llama-S | Qwen-S |
|---|---|---|
| M₀ | 1.47 | 2.67 |
| Unbias→DPO | 2.13 | 5.87 |
| Unbias→CAT | 5.33 | 7.33 |
| TV→DPO | 0.67 | 5.20 |
| TV→CAT | 3.80 | 7.13 |
To W6: Writing corrections
Thank you for catching these errors. We will remove the duplicated sentence in the Table 2 caption and replace the unintended term “Catastrophic Collapses” in the ethics statement with “defense conflicts” to ensure consistency with the paper's terminology.
Dear Reviewer fZrX,
As the discussion period is nearing its end, we wanted to follow up on our response. We would greatly appreciate knowing whether these address your concerns, and we are glad to clarify anything further.
Thank you for your detailed and careful review.
Part 1/2
Thank you for your constructive feedback. We apologize for the delay in responding, as we needed additional time to complete the supplementary experiments. These are now complete, and we will upload the revised manuscript shortly.
To W1: Extension to prompt-based defenses
Thank you for this suggestion. We added Meta's Llama 2 safety system prompt as a prompt-based defense, evaluating Llama-S and Gemma-S before and after NPO or Unbias weight updates, while keeping the same safety prompt at inference.
The safety prompt initially reduces ASR on both models, but three of the four subsequent weight updates weaken this protection (Table G).
Table G. Effect of subsequent weight updates on prompt-based safety protection.
These results show that prompt-based safety protection can also regress following subsequent weight updates, with model- and defense-dependent effects. We will include these results in the revised appendix.
To W2: Comparison with mitigation baselines
Thank you for highlighting this missing comparison. We added four mitigation baselines—KL regularization toward M1, SafeLoRA, EWC-style regularization, and first-defense data replay—on Llama-S DPO→NPO and Gemma-S DPO→Unbias (Table H).
CTDR achieves the lowest RRR in both cases (−0.434 and −0.036) while retaining strong subsequent-defense gains. Notably, on Llama-S, CTDR and KL achieve nearly identical privacy ES (0.070), but CTDR preserves safety substantially better (RRR −0.434 vs. 0.351). On Gemma-S, CTDR outperforms all four baselines in both safety retention and fairness improvement. These results provide evidence that CTDR's directional constraint offers benefits beyond generic retention, although broader comparisons remain necessary.
Table H. Comparison of CTDR with existing mitigation baselines.
To W3: Single-direction assumption and multidimensional representations
Thank you for this suggestion. We agree that a single direction per layer simplifies potentially complex risk representations. To examine this assumption, we extended RDC to multidimensional subspaces using the same checkpoints, support layers, and held-out test data as the original analysis.
For Llama-S DPO→NPO, RDC remains negative at 2D and 4D, consistent with the original 1D result, but becomes positive at 8D (Table I). One possible explanation is that additional dimensions capture non-risk-specific representations, including benign features, whose aligned changes may offset the counteracting signal. This remains a hypothesis. These results support counteraction in low-dimensional subspaces while revealing sensitivity to dimensionality, which we will discuss as a limitation and investigate further.
Table I. Sensitivity of RDC to representation dimensionality.