acceptodds
Under review as a conference paper at ICLR 2027

Defenses at Odds: Measuring and Explaining Defense Conflicts in Large Language Models

Abstract

Large language models (LLMs) may require additional defenses after deployment as risks and governance requirements evolve. Subsequent defenses can interact with earlier defenses, raising the question of their sequential compatibility. We study this question with ConflictEval, evaluating 144 ordered compositions of six defenses spanning safety, privacy, and fairness across six models. The resulting interactions are heterogeneous across defense pairs, application orders, and models. Notably, some compositions exhibit defense conflicts: the subsequent defense improves its target objective while weakening protection established by the earlier defense. We investigate these interactions through the directional compatibility of defense-induced changes in risk-relevant representations. Across 11 selected cases, we use activation interventions to assess which defense-induced representational changes support protection and examine how subsequent defenses affect these changes. We find that, in some conflict cases, subsequent defenses counteract representational changes supporting earlier protection, providing evidence for one possible pathway to defense conflicts. Building on this analysis, we propose Conflict-Triggered Directional Retention (CTDR), which penalizes opposing shifts along directions supporting earlier protection. On six selected conflicting compositions from this analysis, CTDR reduces first-objective regression while maintaining positive gains on the subsequent defense objective.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
large language models, AI safety, safety alignment, sequential defense compatibility

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_fZrXOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies what happens when two post-training defenses for different risks are applied to an LLM one after the other. The authors build ConflictEval, which applies six defenses for safety (DPO, CAT), privacy (NPO, RMU) and fairness (Unbias, task-vector editing) to six models, giving 144 ordered compositions. They measure how much of the first defense's gain is reversed by the second defense with the Relative Regression Rate (RRR), and they find that 57 compositions show a defense conflict, where the second defense reduces its own risk while the earlier risk increases. On 11 selected cases, they estimate one risk direction per layer and show that the second defense tends to move the activations against the first defense's shift along this direction when regression occurs. Based on this observation, they propose CTDR, a regularizer that penalizes such opposing shifts, and show on six conflict cases that it reduces RRR while keeping most of the second defense's gain.

Critical strengths

The question is practical, because deployed models do receive new defenses over time, and the paper states it clearly. The evaluation covers three risk types, six defenses and six models, and the full checkpoint scores are reported in the appendix, so other researchers can reuse the results. RRR is simple and easy to interpret. The reverse-order pairs in Section 3 are also a nice design, since they show that the same two defenses can behave differently depending on the order.

Critical weaknesses
  1. All six defenses are weight updates, and prompt-based defenses are not considered. Many deployed models also rely on safety system prompts, and it is not clear whether such defenses conflict with later weight updates in the same way. I suggest adding at least one prompt-based defense to the evaluation.

  2. The paper does not compare CTDR with any other mitigation method. Two consecutive weight updates naturally make the model forget part of the first one, and many existing methods target this problem, such as EWC-style regularization, replay of the first defense's data, a KL constraint toward the model after the first defense, and safety-preserving fine-tuning methods such as SafeLoRA. Without these baselines, it is hard to tell whether the gain of CTDR comes from the directional design or simply from keeping the model close to the first-defense model.

  3. The analysis assumes that each risk is captured by a single direction per layer, estimated as a mean difference on the initial model. Defenses such as DPO and adversarial training likely change the model along many directions, so a single direction is a strong simplification. I suggest that the authors discuss this assumption and, if possible, check whether the results hold with a multi-dimensional subspace.

  4. Each composition is trained only once, and any increase in the earlier risk counts as a conflict. In Table 2, 19 of the 57 conflicts have an RRR below 0.05, so some of them may come from training noise.

  5. Safety is measured only by attack success rate, without any over-refusal benchmark such as XSTest. For example, CAT reduces the attack success rate of Qwen-L to nearly zero, and CAT is also the second defense that most often increases fairness risk. An over-refusal check would show whether part of these effects comes from the model refusing benign requests.

  6. A few writing details need fixing. The caption of Table 2 repeats one sentence, and the ethics statement mentions "Catastrophic Collapses", which does not appear anywhere else in the paper.

Questions for authors

Please see weaknesses

3 comments

Thank you for your constructive feedback. We apologize for the delay in responding, as we needed additional time to complete the supplementary experiments. These are now complete, and we will upload the revised manuscript shortly.

To W1: Extension to prompt-based defenses

Thank you for this suggestion. We added Meta's Llama 2 safety system prompt as a prompt-based defense, evaluating Llama-S and Gemma-S before and after NPO or Unbias weight updates, while keeping the same safety prompt at inference.

The safety prompt initially reduces ASR on both models, but three of the four subsequent weight updates weaken this protection (Table G).

Table G. Effect of subsequent weight updates on prompt-based safety protection.

ModelSubsequent defenseM₀ ASR (%) ↓M₀ + Prompt ASR (%) ↓+ D₂ ASR (%) ↓Δ (pp) ↓
Llama-SNPO65.3562.8764.11+1.24
Llama-SUnbias65.3562.8750.99−11.88
Gemma-SNPO56.8554.7071.53+16.83
Gemma-SUnbias56.8554.7062.62+7.92

These results show that prompt-based safety protection can also regress following subsequent weight updates, with model- and defense-dependent effects. We will include these results in the revised appendix.

To W2: Comparison with mitigation baselines

Thank you for highlighting this missing comparison. We added four mitigation baselines—KL regularization toward M1M_1, SafeLoRA, EWC-style regularization, and first-defense data replay—on Llama-S DPO→NPO and Gemma-S DPO→Unbias (Table H).

CTDR achieves the lowest RRR in both cases (−0.434 and −0.036) while retaining strong subsequent-defense gains. Notably, on Llama-S, CTDR and KL achieve nearly identical privacy ES (0.070), but CTDR preserves safety substantially better (RRR −0.434 vs. 0.351). On Gemma-S, CTDR outperforms all four baselines in both safety retention and fairness improvement. These results provide evidence that CTDR's directional constraint offers benefits beyond generic retention, although broader comparisons remain necessary.

Table H. Comparison of CTDR with existing mitigation baselines.

Model / Defense sequenceMethodSafety ASR (%) ↓Second risk ↓RRR ↓
Llama-S DPO→NPOSequential50.950.0560.228
CTDR38.610.070−0.434
KL53.250.0700.351
SafeLoRA48.050.7390.072
EWC-style44.310.089−0.128
Replay45.990.075−0.038
Gemma-S DPO→UnbiasSequential53.550.0290.798
CTDR39.920.025−0.036
KL45.050.0800.278
SafeLoRA58.170.0531.081
EWC-style42.080.1990.096
Replay43.320.1640.172

To W3: Single-direction assumption and multidimensional representations

Thank you for this suggestion. We agree that a single direction per layer simplifies potentially complex risk representations. To examine this assumption, we extended RDC to multidimensional subspaces using the same checkpoints, support layers, and held-out test data as the original analysis.

For Llama-S DPO→NPO, RDC remains negative at 2D and 4D, consistent with the original 1D result, but becomes positive at 8D (Table I). One possible explanation is that additional dimensions capture non-risk-specific representations, including benign features, whose aligned changes may offset the counteracting signal. This remains a hypothesis. These results support counteraction in low-dimensional subspaces while revealing sensitivity to dimensionality, which we will discuss as a limitation and investigate further.

Table I. Sensitivity of RDC to representation dimensionality.

RepresentationDimensionPooled RDC
Original single direction1−1.789
Multidimensional subspace2−2.514
Multidimensional subspace4−0.289
Multidimensional subspace8+0.308

To W4: Training randomness and small-RRR conflicts

Thank you for raising this concern. We agree that small positive RRR values may be sensitive to training randomness. We repeated eight compositions with two additional training seeds (11 and 22). Across all three seeds, all seven regression cases retain positive RRR, while the non-regression control remains negative.

Notably, the mild Qwen-L NPO→CAT case yields RRRs of 0.045, 0.050, and 0.003. Although the magnitude varies, regression persists across all three seeds, providing evidence that this observed conflict is not specific to a single training realization.

Table J. Sequential RRR across three training seeds.

CaseModelDefense sequenceSeed 0 (Original)Seed 11Seed 22
1Llama-SDPO→RMU0.3100.2800.245
2Llama-SDPO→NPO0.2280.1670.156
3Gemma-SDPO→NPO0.6970.9740.860
4Gemma-LCAT→NPO0.2220.2430.128
5Qwen-LUnbias→CAT0.5260.5880.534
6Qwen-LNPO→CAT0.0450.0500.003
7Llama-SRMU→DPO−0.058−0.025−0.035
8Gemma-SDPO→Unbias0.7980.8580.710

To W5: Over-refusal evaluation

Thank you for highlighting this concern. We added a WildGuard-based benign-request ((XSTest)) refusal evaluation on Llama-S and Qwen-S, covering four sequential defense compositions (Table J).

Although CAT-based compositions exhibit higher benign refusal rates than their DPO counterparts, all refusal rates remain below 8%, suggesting relatively limited over-refusal rather than widespread rejection of benign requests. Thus, over-refusal does not appear pervasive in the evaluated settings, although its contribution to the observed ASR changes cannot be fully ruled out. We will include these results and discuss their limitations in the revised appendix.

Table J. Benign-request refusal rates evaluated with WildGuard (% ↓).

Defense sequenceLlama-SQwen-S
M₀1.472.67
Unbias→DPO2.135.87
Unbias→CAT5.337.33
TV→DPO0.675.20
TV→CAT3.807.13

To W6: Writing corrections

Thank you for catching these errors. We will remove the duplicated sentence in the Table 2 caption and replace the unintended term “Catastrophic Collapses” in the ethics statement with “defense conflicts” to ensure consistency with the paper's terminology.

Dear Reviewer fZrX,

As the discussion period is nearing its end, we wanted to follow up on our response. We would greatly appreciate knowing whether these address your concerns, and we are glad to clarify anything further.

Thank you for your detailed and careful review.

Reviewer_8MUSOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether sequential safety training will harm the training results of the earlier ones. They analyze this by investigating multiple safety training methods across privacy, safety & fairness. They show that often two methods negatively influence each other and propose a new training via linear representations. They add a penalty term that monitors whether the new there will be change to a predefined anchor set.

Critical strengths

The topic that the paper is investigating is very interesting and important. They present it nicely, it is clearly set up and every experiment is useful and adds something to the picture. The final training methodology seems to work well on this set and is a great empirical validation of the prevailing issue and prior analysis.

Critical weaknesses

The biggest issue for me is that every experiment is a single training realization. I would love to see a subset of experiments being run again with a few different seeds.

Questions for authors
  1. Does CTDR extend beyond 2 defenses?
  2. Does CTDR influence capabilities (non-safety) more than normal safety training?
  3. Can you take one model family and vary the model size to see whether eg bigger models have an easier time encoding these sequential trainings without CTDR?
  4. The support location sets are quite small for many defenses (except CAT). Did you look into this more? Any hypothesis why? I believe some cited work investigate this a bit but I would like a deeper discussion since a limited support set might decrease the usefulness of CTDR.

I really like this paper and if all my concerns will be resolved I will increase my score.

7 comments

Thank you for your constructive feedback. We apologize for the delay in responding, as we needed additional time to complete the supplementary experiments. These are now complete, and we will upload the revised manuscript shortly.

To W1: Robustness across training seeds

We agree that a single training realization is insufficient to assess sensitivity to training randomness. We added two training seeds (11 and 22) for eight compositions. We selected the six original CTDR cases to jointly assess the repeatability of defense conflicts and CTDR’s mitigation effects, added the reverse-order non-regression control Llama-S RMU→DPO to examine whether this outcome persists, and included Gemma-S DPO→Unbias, highlighted by Reviewer PokE, to extend the evaluation to fairness as the subsequent defense objective.

For each added seed, we retrained the complete two-stage sequence from the same initial checkpoint M₀ for each model. Sequential and CTDR shared that seed’s first-stage checkpoint M₁ and used matched second-stage base-training settings.

Across seeds 0, 11, and 22, all eight compositions retain their observed regression/non-regression pattern, although magnitudes vary (Table C). Notably, the mild Qwen-L NPO→CAT case yields RRRs of 0.045, 0.050, and 0.003. Across the seven conflict cases, CTDR lowers RRR in all 7 cases under seeds 11 and 22, while retaining approximately 97.2–103.7% of the corresponding sequential G₂ (Table D). These results support repeatability within the evaluated subset.

Table C. Sequential RRR across three training seeds.

CaseModelDefense sequenceSeed 0 (Original)Seed 11 (NEW)Seed 22 (NEW)Mean ± SD
1Llama-SDPO→RMU0.3100.2800.2450.278 ± 0.033
2Llama-SDPO→NPO0.2280.1670.1560.184 ± 0.039
3Gemma-SDPO→NPO0.6970.9740.8600.844 ± 0.139
4Gemma-LCAT→NPO0.2220.2430.1280.198 ± 0.061
5Qwen-LUnbias→CAT0.5260.5880.5340.550 ± 0.034
6Qwen-LNPO→CAT0.0450.0500.0030.033 ± 0.026
7Llama-SRMU→DPO−0.058−0.025−0.035−0.039 ± 0.017
8Gemma-SDPO→Unbias0.7980.8580.7100.789 ± 0.074

Table D. Paired Sequential and CTDR results across three training seeds.

Case IDs follow Table C. For Case 8, the seed-0 Sequential result is from the original paper, while the CTDR result is newly obtained.

CaseModelDefense sequenceSeed / SourceRRR ↓: Sequential → CTDRG₂ ↑: Sequential → CTDR
1Llama-SDPO→RMU0 (Original)0.310 → −0.0530.636 → 0.633
11 (NEW)0.280 → −0.0660.618 → 0.618
22 (NEW)0.245 → −0.0590.623 → 0.609
2Llama-SDPO→NPO0 (Original)0.228 → −0.4340.645 → 0.631
11 (NEW)0.167 → −0.0670.644 → 0.643
22 (NEW)0.156 → −0.0040.634 → 0.633
3Gemma-SDPO→NPO0 (Original)0.697 → 0.0000.925 → 0.896
11 (NEW)0.974 → 0.0040.922 → 0.922
22 (NEW)0.860 → 0.0310.909 → 0.909
4Gemma-LCAT→NPO0 (Original)0.222 → −0.4390.112 → 0.112
11 (NEW)0.243 → −0.2150.108 → 0.112
22 (NEW)0.128 → −0.4250.110 → 0.109
5Qwen-LUnbias→CAT0 (Original)0.526 → −0.0770.687 → 0.693
11 (NEW)0.588 → 0.0010.653 → 0.645
22 (NEW)0.534 → −0.0120.695 → 0.684
6Qwen-LNPO→CAT0 (Original)0.045 → 0.0000.638 → 0.641
11 (NEW)0.050 → 0.0030.642 → 0.644
22 (NEW)0.003 → 0.0000.625 → 0.631
8Gemma-SDPO→Unbias0 (NEW CTDR)0.798 → −0.0360.179 → 0.183
11 (NEW)0.858 → −0.0010.175 → 0.179
22 (NEW)0.710 → 0.0340.198 → 0.192

To Q1: Extension of CTDR beyond two defenses

Yes. We extend CTDR to a third defense by incorporating retention penalties for both preceding defenses:

Lt(θ)=LDt(θ)+∑j=1t−1λjRj(t)(θ)\mathcal{L}_t(\theta) = \mathcal{L}_{D_t}(\theta) + \sum_{j=1}^{t-1} \lambda_j \mathcal{R}_j^{(t)}(\theta)

We tested a three-defense chain, DPO→NPO→Unbias, on Gemma-S. From second-stage checkpoints trained with and without CTDR, we compared ordinary Unbias with Unbias jointly constrained to retain safety and privacy. From the CTDR-protected checkpoint, third-stage CTDR reduces final ASR from 45.79% to 39.85% while retaining 93.1% of the ordinary Unbias fairness gain. Privacy ES remains comparable to ordinary third-stage training (0.0524 vs. 0.0520), with both improving upon the pre-third-stage value of 0.0801. The ordinary NPO starting point also benefits, with ASR decreasing from 52.73% to 50.74%. These results support CTDR’s extension to three defenses in this setting, with a modest trade-off in the new defense’s gain. See Table E.

Table E. Three-stage CTDR evaluation on Gemma-S (DPO→NPO→Unbias).

StageDefense sequenceSafety ASR (%) ↓Privacy ES ↓Fairness F ↓
M₀Base model56.8500.9840.165
M₁DPO40.5100.9760.208
M₂DPO→NPO51.9000.0510.119
M₃DPO→NPO→Unbias52.7300.0530.059
M₃DPO→NPO→Unbias+CTDR50.7400.0510.065
M₂DPO→(NPO+CTDR)40.5100.0800.201
M₃DPO→(NPO+CTDR)→Unbias45.7900.0520.059
M₃DPO→(NPO+CTDR)→(Unbias+CTDR)39.8500.0520.069

To Q2: Impact of CTDR on general capabilities

We apologize for not explicitly evaluating CTDR's impact on general capabilities in the original submission. We have now evaluated MMLU on all six original CTDR cases and the additional Gemma-S DPO→Unbias case, comparing CTDR against ordinary sequential defense training under the same evaluation protocol (Table F). CTDR does not consistently cause additional MMLU degradation compared with ordinary sequential training.

Table F. MMLU comparison between sequential training with and without CTDR.

ModelDefense sequenceM₀ (%)Sequential (%)CTDR (%)Δ (pp)
Llama-SDPO→RMU46.0343.9044.38+0.48
Llama-SDPO→NPO46.0344.6445.01+0.37
Gemma-SDPO→NPO56.4456.0555.94−0.11
Gemma-LCAT→NPO51.7051.6351.49−0.14
Qwen-LUnbias→CAT72.3371.1970.20−0.99
Qwen-LNPO→CAT72.3371.6272.43+0.81
Gemma-SDPO→Unbias56.4453.4255.19+1.77

To Q3: Effect of model size

Thank you for this suggestion. Our original evaluation includes Qwen2.5-1.5B and Qwen2.5-7B across the same 24 ordered defense compositions. The larger model exhibits fewer observed conflicts (9/24 vs. 11/24), with lower RRR for Unbias→CAT (0.659→0.526) and TV→CAT (0.609→0.218). However, this trend is not universal: CAT→TV shows increased regression (0.301→0.417). Thus, larger models may alleviate some defense conflicts but do not guarantee better compatibility.

One possible mechanistic explanation is that greater representational capacity provides more room for different objectives, potentially reducing interference. Ramasesh et al. (ICLR 2022) observed increased orthogonality between class representations and reduced forgetting with scale in pretrained continual-learning models. However, greater capacity does not guarantee disentanglement: Elhage et al. (2022) demonstrated that sparse features can be encoded in non-orthogonal directions through superposition, trading interference for representational efficiency.

We acknowledge that we have not evaluated models beyond 8B, and our comparisons cannot fully disentangle model size from pretraining and alignment differences. We therefore regard the observed trends as suggestive rather than causal, and leave more controlled comparisons at larger scales to future work.

To Q4: Effect of support-set size

Our support sets identify layers with positive restoration effects under the selected directions and risk proxies, rather than all layers modified by a defense. Small sets may therefore reflect concentrated defensive contributions or limited measurement coverage. CAT’s broader support may reflect more distributed effects of adversarial training, but this remains a hypothesis.

CTDR constrains the full first-defense support set K1K_1, not only K1∩K2K_1 \cap K_2. A small set can still be effective: for Gemma-S DPO→NPO, K1K_1 contains only four layers, yet CTDR reduces RRR from 0.697 to 0.000 while retaining 96.9% of the subsequent-defense gain. However, limited support coverage may miss other protective directions or cross-layer interactions. We leave a systematic investigation of support-set coverage and its impact on CTDR effectiveness to future work.

Reviewer_PokEOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces ConflictEval to evaluate 144 ordered compositions of six defenses across six language models, identifying cases where a subsequent defense improves its own objective while weakening earlier protection. Throug representational analysis of 11 selected compositions, the authors motivate CTDR, which reduces earlier defense regression while retaining most of the subsequent defense gain in 6 selected conflicts.

Critical strengths
  1. The paper presents extensive empirical observation that defense procedures targeted for reduction of different types of risks (such as safety, fairness, and privacy) are not inherently composable.
  2. Author attempt to take some action on their observation on conflicting defenses via defense-induced representational changes (a.k.a. steering vectors) and directional ablation.
Critical weaknesses
  1. I have some concerns about conflict unspecified selection bias for representational analysis. Although the paper starts with a convincing 144 types of compositions, it reduces to 11 mechanism cases and then six CTDR evaluations without an outcome-independent selection protocol. This makes me think that the perfect RDC-regression sign correspondence in Table 3 and consistent mitigation success in Table 4 might remain vulnerable to selection bias. Substantial conflicts, including Gemma-S DPO toto Unbias (RRR 0.798), are also omitted, and no CTDR experiment evaluates a fairness defense as the subsequent update. Evaluation across all eligible cases (with an appropriate elligibility criteria), or an independently randomly selected confirmation set, is needed to make the results more convincing.
Questions for authors

Mildly-critical comment: Related works section could be improved to locate the paper in the rapidly emerging literature in interpretability and steering in safety and fairness alignment, which are actually relevant to this work more than machine unlearning in my opinion. I will not list any examples here to avoid biasing the authors to specific works in a large pool.

6 comments

Thank you for your constructive feedback. We apologize for the delay in responding, as we needed additional time to complete the supplementary experiments. These are now complete, and we will upload the revised manuscript shortly.

To W1: Potential case-selection bias in representational analysis and CTDR evaluation

1. Original selection rationale. We explain below how and why we selected the cases reported in the original manuscript.

  • Mechanism analysis. We selected the original 11 mechanism cases using the behavioral results in Table 2, before computing RDC. Specifically, we included strong conflicts (RRR > 0.1), mild conflicts (0 < RRR ≤ 0.1), and non-regression cases (RRR ≤ 0) to test our mechanistic hypothesis across different degrees of behavioral regression. We covered three model families, both model scales, and all three risk dimensions to assess whether the pattern extends beyond a particular model or risk. Two reverse-order pairs were also included to examine how application order affects interactions between the same defenses (Sec. 3.1).

  • CTDR evaluation. Because CTDR requires the risk directions and defense-support layers identified in our mechanism analysis, we selected its evaluation cases from the same 11-case set. We included all five strong conflicts and randomly selected one of the three mild conflicts (Qwen-L NPO→CAT; RRR 0.045) as a supplementary check of mitigation under mild regression. This six-case subset was fixed before CTDR evaluation (Sec. 4).

2. Supplementary selection protocol. To examine whether our findings extend beyond the original cases, we stratified compositions by model family × pre-existing behavioral regression, targeting four strong conflicts, one mild conflict, and two non-regression cases per family.

  • Fixed cases. We first assign all eleven original cases and the reviewer-requested Gemma-S DPO→Unbias case to their corresponding strata. The original Qwen cases include three mild conflicts, exceeding the main-table quota of one. We retain Qwen-L NPO→CAT, already included in the original CTDR evaluation, in the main table and preserve the other two original Qwen-S mild-conflict cases in the supplement. This leaves nine original cases and one reviewer-requested case fixed in the main set.

  • Random additions. We fill the eleven remaining slots by sampling without replacement within each stratum. After each draw, we check whether the complete set includes both model scales in every family and all three risk dimensions as both first and subsequent defense targets overall. If these constraints are not met, we redraw the eleven additions using the same random stream, keeping the fixed cases unchanged, and accept the first qualifying set.

  • Evaluation scope. The main set comprises 21 cases: nine original cases and twelve additions. We conduct mechanism analysis on all twelve additions and evaluate CTDR on all nine added regression cases.

3. Results on the additional cases. Table A distinguishes the original cases, randomly sampled additions, and the reviewer-requested case.

  • Mechanism analysis. Among the eleven sampled additions, all eight regression cases have negative RDC, while all three non-regression cases have positive RDC (Table A). The reviewer-requested Gemma-S DPO→Unbias case also exhibits negative RDC (−0.788) alongside substantial behavioral regression (RRR 0.798). These results extend the observed relationship beyond the original mechanism cases.

  • CTDR evaluation. CTDR reduces RRR in all nine additional regression cases, with minimum G₂ retention of approximately 98.6% (Table B). This includes fairness as the subsequent defense: RRR decreases from 0.798 to −0.036 for Gemma-S DPO→Unbias and from 0.053 to −0.202 for Gemma-L DPO→Unbias. Four of the nine cases retain positive RRR, indicating that CTDR reduces, but does not always eliminate, regression.

Table A. Mechanism results for the expanded 21-case main set.

NEW: added mechanism analysis; †: reviewer-requested case; *: RDC pooled over K₁.

CaseSourceModelDefense sequenceK₁K₂K₁ ∩ K₂Pooled RDCRRR
1NEWLlama-LUnbias→CAT1–3, 6, 8–10, 22, 24, 27, 29, 30, 321, 3–5, 8, 10, 18–201, 3, 8, 10−0.5270.724
2OriginalLlama-SDPO→NPO8–10, 12–154, 8, 11–138, 12, 13−1.7890.228
3OriginalLlama-SDPO→RMU8–10, 12–152, 11–1312, 13−1.6870.310
4NEWLlama-SUnbias→CAT1, 3–7, 9–13, 155, 6, 12–145, 6, 12, 13−1.1720.600
5NEWLlama-STV→DPO1, 3–5, 7–10, 12–162, 3, 7–133, 7–10, 12, 13−0.0240.091
6OriginalLlama-SRMU→DPO2, 11–138–10, 12–1512, 13+0.389−0.058
7NEWLlama-SRMU→Unbias2, 11–131, 3–7, 9–1311–13+0.0462−0.059
8OriginalGemma-LCAT→NPO10, 11, 13, 21–2816, 2323−0.0990.222
9OriginalGemma-SDPO→NPO1, 2, 7, 1511, 15, 17–2615−0.0500.697
10NEW†Gemma-SDPO→Unbias1, 2, 7, 151–4, 6, 7, 11, 15–241, 2, 7, 15−0.7880.798
11NEWGemma-SUnbias→DPO1–4, 6, 7, 11, 15–24, 261–3, 7, 151–3, 7, 15−0.2110.403
12NEWGemma-LDPO→Unbias6–10, 12–282, 5, 11–1512–15−0.1860.053
13OriginalGemma-SNPO→DPO11, 15, 17–261, 2, 7, 1515+0.1690.000
14OriginalGemma-LCAT→TV10, 11, 13, 21–285, 12, 2424+1.206−0.348
15OriginalQwen-LUnbias→CAT5–102, 3, 12, 13, 16, 20, 24, 25, 28∅−2.207*0.526
16NEWQwen-LUnbias→NPO5–1011, 12, 15, 16, 21, 25∅−1.414*0.201
17NEWQwen-STV→CAT1–3, 5, 6, 14–21, 23, 279–12, 24–2827−1.2490.609
18NEWQwen-SUnbias→CAT1, 2, 4, 6, 7, 14, 17, 19–279–12, 25–2825–27−0.6880.659
19OriginalQwen-LNPO→CAT11, 12, 15, 16, 21, 252, 3, 12, 13, 16, 20, 24, 25, 2812, 16, 25−1.3220.045
20NEWQwen-LTV→DPO2, 10, 14, 15, 20, 25, 26, 284, 5, 12–1514, 15+0.014−0.011
21NEWQwen-LRMU→DPO9–19, 284, 5, 12, 1512, 15+0.3354−0.070

Table B. CTDR results on nine additional conflict cases.

Case IDs follow Table A;

CaseModelDefense sequenceRRR ↓: Sequential → CTDRG₂ ↑: Sequential → CTDRG₂ retained ≈
1Llama-LUnbias→CAT0.724 → −0.1300.342 → 0.386113.0%
4Llama-SUnbias→CAT0.600 → 0.1100.229 → 0.22698.6%
5Llama-STV→DPO0.091 → 0.0070.107 → 0.118110.0%
10Gemma-SDPO→Unbias0.798 → −0.0360.179 → 0.183102.0%
11Gemma-SUnbias→DPO0.403 → −0.0520.278 → 0.284102.3%
12Gemma-LDPO→Unbias0.053 → −0.2020.063 → 0.094148.3%
16Qwen-LUnbias→NPO0.201 → −0.0030.531 → 0.531100.0%
17Qwen-STV→CAT0.609 → 0.0630.381 → 0.413108.2%
18Qwen-SUnbias→CAT0.659 → 0.2570.484 → 0.486100.4%

To Q1: Related work on interpretability and steering

Thank you for this helpful suggestion. We will expand Section 5 to better situate our work within interpretability and representation steering for safety and fairness alignment, discussing representation engineering (Zou et al., 2023), contrastive activation addition (Rimsky et al., 2024), refusal-related directions (Arditi et al., 2024), and FairSteer (Li et al., 2025).

These studies provide foundations for identifying and manipulating behavior-relevant representations. Building on these foundations, we extend the focus from representations associated with individual behaviors or risks to interactions between representational changes induced by defenses targeting different risks. Specifically, we investigate whether a defense targeting one risk reinforces or counteracts changes supporting protection against another. RDC quantifies this interaction along the first-risk directions, motivating CTDR’s directional retention during subsequent defense training.

We will make these connections and distinctions explicit in the revised Section 5.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Brief introduction to the revised PDF

We have uploaded the revised PDF, which incorporates changes in response to the reviews and discussion. We thank all the reviewers for their careful reading and constructive feedback, which helped us improve the paper. We will continue to refine the manuscript as we prepare the final version.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.