acceptodds
Under review as a conference paper at ICLR 2027

Causal Front-Door Adjustment for Robust Jailbreak Attacks on LLMs

Abstract

Jailbreak attacks on safety-aligned LLMs are increasingly effective, yet attack success alone does not reveal whether compliance is driven by preserved task intent, suppressed safety signals, or attack-specific artifacts. We use Pearl’s front-door adjustment to motivate separating task-relevant semantics from safety-related variation. We proposeCausal Front-Door Adjustment Attack (CFA), a white-box framework that operationalizes this perspective in LLM latent space using Sparse Autoencoders, contrastive datasets, and weight orthogonalization. After offline calibration, CFA enables efficient generation without per-prompt iterative optimization, performs strongly on HarmBench across four open-source LLMs, and provides source-side signals for same-family black-box prompt transfer.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM, Jailbreak

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_qW8YOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The author proposes CFA2, which contrasts task-matched harmful/jailbroken pairs in SAE space to find a concept direction and orthogonalizes output weights. Experiments report on HarmBench and AdvBench for 4 LLMs against 11 baselines.

Critical strengths

The paper's insight on pairing jailbroken variants to fix task semantics while varying refusal behaviors is interesting and well-grounded.

Critical weaknesses

The paper assumes content invariance s(x)=s(x⁺) across pairs, which may not always be valid and needs further justification.

Questions for authors

The Figure 3 is overly large to occupy signfiicant space, yet the information it provides is limited. Please consider revising the paper presentation.

2 comments

Response to Weakness 1:

We sincerely thank Reviewer qW8Y for this important comment. Our contrastive pairs are constructed from the same underlying HarmBench task rather than independently sampled harmful and compliant requests. For each original request xx, we construct successful jailbreak variants x+x^+ that preserve the requested task objective while changing the model's response from refusal to compliance, as illustrated in Table 4.

The condition s(x)=s(x+)s(x)=s(x^+) in Proposition 4.2 describes the ideal case of content invariance, which our task-matched construction aims to approximate. Importantly, the ablation in Table 9 provides empirical support for this design: replacing conventional harmful--harmless contrasts with task-matched refusal--compliance pairs improves the average ASR from 38.17% to 48.50%, while keeping the representation space and weight orthogonalization unchanged. This supports the practical value of controlling task-objective variation during defense-direction identification.

We emphasize that task-objective consistency motivates approximate latent invariance rather than guaranteeing exact equality for every pair. We will clarify this distinction and the scope of the theoretical assumption in the revised manuscript.

Response to Question 1:

We sincerely thank Reviewer qW8Y for this helpful suggestion. Figure 3 illustrates two complementary aspects of the feature analysis. Panel (a) illustrates the activation distributional shift of defense-related features between refusal and compliance states, while Panel (b) shows the activation consistency of task-related features and the distinct variation of defense-related features across the two conditions. We agree that its current presentation can be made more concise. We will reduce the figure's visual footprint, simplify the legends, and refine the caption to communicate the complementary roles of the two panels more clearly.

Dear Reviewer qW8Y,

Thank you again for your time and thoughtful feedback on our paper. As the discussion period is nearing its conclusion, we would greatly appreciate knowing whether our responses have addressed your concerns regarding the task-semantic consistency assumption and the presentation of Figure 3.

If so, we would be grateful if you would consider reflecting this in your updated rating. Of course, we fully respect your assessment and sincerely appreciate your time and consideration regardless of your final decision.

Best regards,

The Authors

Reviewer_8GA6OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The contribution is mainly about the causal front door ajustment attack (CFA) using SAE.

Critical strengths

The critical strength is the combination of task-matched contrast and SAE-based direction identification. Another contribution is a derived contrastive dataset construction calibrated from HarmBench and AdvBench.

Critical weaknesses
  1. Lack of LAT and steering baselines: Sheshadri et al. (2025), Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMs. TMLR. Li et al. (2026), Analysing the Safety Pitfalls of Steering Vectors. Findings of ACL. Makelov et al. (2024), Is This the Subspace You Are Looking for? An Interpretability Illusion for Subspace Activation Patching. ICLR. Zhang and Nanda (2024), Towards Best Practices of Activation Patching in Language Models: Metrics and Methods. ICLR. Rimsky et al. (2024), Steering Llama 2 via Contrastive Activation Addition, ACL. Li et al. (2026), Analysing the Safety Pitfalls of Steering Vectors, Findings of ACL.
  2. Lack of the front-door representation: The transition from front-door adjustment to weight orthogonalization is not readable. Lack of the explanation how the proposed decomposition into task semantics and a safety mechanism is processed.
  3. The calibrated contrastive dataset is not guaranteed to be generalized across benchmarks.
  4. Lack of the robustness demonstration, it means the utility score after using your algorithm.
Questions for authors
  1. How is the CFA's robustness on benign datasets?
  2. How does weight orthogonalization work for front-door adjustment?
  3. How does CFA perform across benchmarks without recalibration?
  4. Does CFA outperform steering methods using the same calibration pairs?
5 comments

Response to Question 1 & Weakness 4:

We sincerely thank Reviewer 8GA6 for raising these concerns. To evaluate utility preservation, we conduct additional experiments on two widely used utility benchmarks: MMLU (general knowledge) and GSM8K (mathematical reasoning). We compare the performance of the original models with that of the CFA$^2$-intervened models across all four target models.

Table 1: Benign utility evaluation. We report CFA$^2$ performance / original model performance

ModelMMLUGSM8K
Llama-3.1 8B67.5 / 68.3 (−0.8)78.1 / 78.3 (−0.2)
Llama-2 7B45.8 / 46.6 (−0.8)25.3 / 25.7 (−0.4)
Mistral 7B59.6 / 60.2 (−0.6)44.4 / 44.5 (−0.1)
Gemma-2 9B71.9 / 72.1 (−0.2)84.2 / 84.4 (−0.2)
Average61.20 / 61.80 (−0.60)58.00 / 58.23 (−0.23)

The experiments compare the original and CFA$^2$-modified weights using the same base models, evaluation datasets, and scoring protocols, thereby controlling for differences in model configurations and evaluation settings. Across the four models, average accuracy decreases by only 0.60 and 0.23 percentage points on MMLU and GSM8K, respectively. These findings indicate that CFA$^2$ largely preserves the original model capabilities on the evaluated benign knowledge and reasoning tasks.

Together with the existing PPL results in Appendix H.3, these additional findings further support that SAE-guided parameter intervention does not cause substantial degradation in generation quality. We will include these utility evaluation results and the complete evaluation configurations in the revised manuscript.

Response to Question 3 & Weakness 3:

We sincerely thank Reviewer 8GA6 for raising this concern regarding the cross-benchmark generalization of CFA$^2$. To directly investigate this issue, we conduct an additional cross-benchmark transfer experiment without recalibration. Specifically, we use only the task-matched contrastive data from HarmBench to identify defense-related directions and perform weight orthogonalization. We then evaluate the same set of modified model weights directly on both HarmBench and AdvBench. Throughout cross-dataset evaluation, we do not reconstruct calibration data, re-identify SAE features or intervention directions, or update the model weights.

Table 2: Cross-benchmark generalization without recalibration (ASR, %).

BenchmarkLlama-3.1 8BLlama-2 7BMistral 7BGemma-2 9BAverage
HarmBench99.499.199.799.699.45
AdvBench90.892.293.693.592.53
Δ\Delta (pp)−8.6−6.9−6.1−6.1−6.93

Δ\Delta denotes the change in ASR on AdvBench relative to HarmBench, measured in percentage points.

The results show that CFA$^2$ maintains an ASR of 90.8%–93.6% on AdvBench without any recalibration, achieving an average ASR of 92.53% across the four models. Although the average ASR decreases by 6.93 percentage points compared with HarmBench, all four models maintain high attack success rates. These findings indicate that the defense-related directions identified using HarmBench calibration data can transfer to a different distribution of harmful requests without re-identifying the intervention directions for the target benchmark. We will include this experiment and the complete evaluation protocol in the revised manuscript to further substantiate the cross-benchmark generalization capability of CFA$^2$.

Response to Question 4 & Weakness 1:

We sincerely thank Reviewer 8GA6 for valuable suggestions. We agree that the original manuscript did not provide a sufficiently comprehensive comparison with existing activation steering methods, and that further investigation is needed to determine whether the performance advantage of CFA$^2$ over existing steering approaches depends on the choice of calibration data. To address these concerns, we introduce three representative steering baselines and conduct an additional controlled comparison using identical calibration pairs.

(1) Additional representative steering baselines. We include three representative methods: CAA [1, 2], PID Steering [3], and Selective Steering [4]. We retain the method-specific calibration and steering procedures of the respective original methods while adopting a unified evaluation protocol. Specifically, all re-evaluated baselines and CFA$^2$ are evaluated on the same AdvBench test samples, using the official HarmBench Classifier as the common ASR judge and greedy decoding for generation. The results are summarized below. Further implementation details and other experimental settings will be provided in the revised manuscript.

Table 1: Comparison with representative activation steering baselines on AdvBench under a unified evaluation protocol (ASR, %).

MethodLlama-3.1-8BLlama-2-7B
Selective Steering [4]76.8%--
PID Steering [3]93.7%--
CAA (reproduced following Li et al., 2026) [1]--50.8%
CAA (using CFA$^2$ pairs) [1]--45.6%
CFA$^2$ (Ours)97.8%98.5%

All baseline results are obtained from our own evaluations under the shared evaluation protocol. CFA$^2$ achieves the highest ASR on both models, outperforming PID Steering and Selective Steering on Llama-3.1-8B, and substantially exceeding CAA under its original configuration on Llama-2-7B.

Regarding the other works mentioned by the reviewer, LAT [5] primarily improves model robustness through latent-space adversarial training during the training stage, which has a different research objective from our focus on inference-time intervention in internal representations. Makelov et al. [6] and Zhang and Nanda [7] primarily investigate the reliability and evaluation methodology of Activation Patching, rather than proposing direct jailbreak attack methods. We will therefore incorporate discussions of these works into the Related Work section rather than include them as experimental baselines.

(2) Controlling calibration data to further validate the performance advantage. To rule out the possibility that the performance difference is merely attributable to the choice of calibration data, we further construct a CAA baseline on Llama-2-7B using the original task-matched calibration pairs of CFA$^2$, and evaluate it on AdvBench. With identical calibration pairs, CAA achieves an ASR of 45.6%, whereas CFA$^2$ achieves 98.5%, yielding a difference of 52.9 percentage points as shown in Table 1. For comparison, CAA achieves an ASR of 50.8% when using its original calibration data and configuration.

These results indicate that the performance advantage of CFA$^2$ cannot be simply attributed to the choice of calibration data. Even when using identical calibration pairs, its intervention mechanism achieves substantially better performance.

References:
[1] Yuxiao Li, Alina Fastowski, Efstratios Zaradoukas, Bardh Prenkaj, and Gjergji Kasneci. "Analysing the Safety Pitfalls of Steering Vectors." Findings of ACL, 2026, pp. 11182–11204.
[2] Nina Rimsky, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Turner. "Steering Llama 2 via Contrastive Activation Addition." ACL, 2024, pp. 15504–15522.
[3] Dung Viet Nguyen, Yen Pham, Hieu Vu, Lei Zhang, and Tan Nguyen. "Activation Steering with a Feedback Controller." ICLR, 2026.
[4] Quy-Anh Dang and Chris Ngo. "Selective Steering: Norm-Preserving Control Through Discriminative Layer Selection." Findings of ACL, 2026, pp. 10887–10910.
[5] Abhay Sheshadri et al. "Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMs." Transactions on Machine Learning Research, 2025.
[6] Aleksandar Makelov, Georg Lange, Atticus Geiger, and Neel Nanda. "Is This the Subspace You Are Looking for? An Interpretability Illusion for Subspace Activation Patching." ICLR, 2024.
[7] Fred Zhang and Neel Nanda. "Towards Best Practices of Activation Patching in Language Models: Metrics and Methods." ICLR, 2024.

Response to Question 2 & Weakness 2:

We sincerely thank Reviewer 8GA6 for these helpful suggestions. We further analyze the connection between Front-Door Adjustment and Weight Orthogonalization at the output-distribution level, and derive an error bound for approximating causal marginalization through Weight Orthogonalization.

First, given fixed task semantics S=sS=s, Front-Door Adjustment obtains the target interventional distribution by marginalizing the conditional output distributions over different background representations A′A'. CFA$^2$ operationalizes this idea through task-matched contrastive data: requests sharing the same underlying task objective but inducing different safety responses are paired. We use a frozen LLM and SAE to extract sparse features, suppress shared task-related components through paired activation differences, and identify defense-related features through cross-pair feature selection. The identified features are then mapped to the representation space of the output projection layer, yielding a defense-related direction dd, while the shared task-related components serve as an operational approximation of the mediator SS. In this way, CFA$^2$ translates task semantics preservation and defense-related variation identification into an operational representation decomposition.

Second, Weight Orthogonalization projects the output weights to eliminate their response along the identified direction dd, i.e., Woutnew=Wout(I−dd⊤)W_{\mathrm{out}}^{\mathrm{new}} = W_{\mathrm{out}}(I-dd^\top). Consequently, when the dominant variations across different background representations are captured by dd, orthogonalization suppresses their influence on the output, making the modified conditional output distributions approximately consistent. Thus, the explicit marginalization over different background representations in the inner step of Front-Door Adjustment can be approximated by a single forward pass through the modified model.

To further quantify this connection, we decompose the representation received by the output projection layer as r(A′)=rs+α(A′)d+ϵ(A′)r(A')=r_s+\alpha(A')d+\epsilon(A'), where rsr_s denotes the shared task-related component under fixed task semantics, α(A′)d\alpha(A')d represents variations along the dominant defense-related direction, and ϵ(A′)\epsilon(A') captures other residual variations. We apply the same decomposition to the current input aa. Let Pd=I−dd⊤P_d=I-dd^\top, κ=d⊤rs\kappa=d^\top r_s, and v=Woutdv=W_{\mathrm{out}}d, and define R(A′)=Wout[ϵ(A′)−Pdϵ(a)]R(A')=W_{\mathrm{out}}[\epsilon(A')-P_d\epsilon(a)]. The difference between the corresponding output logits before and after orthogonalization can then be expressed as Δℓ(A′)=v(α(A′)+κ)+R(A′)\Delta\ell(A')=v(\alpha(A')+\kappa)+R(A').

Specifically, let the Front-Door inner marginalization distribution under fixed task semantics and the output distribution of the orthogonalized model be defined as PFD=EA′∼P(A′)[σ(ℓθ(A′,s))],PWO=σ(ℓWO(a)).P_{\mathrm{FD}}=\mathbb{E}_{A'\sim P(A')}[\sigma(\ell_\theta(A',s))],\qquad P_{\mathrm{WO}}=\sigma(\ell_{\mathrm{WO}}(a)). where σ\sigma denotes the Softmax function. A second-order Taylor expansion of the Softmax yields the following error bound between the two distributions:

∥PFD−PWO∥1≤MJ(∥v∥2∣E[α]+κ∣+∥E[R]∥2)+Lσ(∥v∥22E[(α+κ)2]+E[∥R∥22]).\lVert P_{\mathrm{FD}}-P_{\mathrm{WO}}\rVert_1 \leq M_J\left(\lVert v\rVert_2|\mathbb{E}[\alpha]+\kappa|+\lVert\mathbb{E}[R]\rVert_2\right)+L_\sigma\left(\lVert v\rVert_2^2\mathbb{E}[(\alpha+\kappa)^2]+\mathbb{E}[\lVert R\rVert_2^2]\right).

where MJM_J and LσL_\sigma denote the corresponding operator-norm bounds of the Softmax Jacobian and Hessian, respectively. This result shows that the distributional approximation error depends primarily on the remaining variations along the defense-related direction, the overlap between task-related components and this direction, and residual variations not captured by the identified direction. These factors jointly determine the approximation accuracy and are not required to vanish completely. When the identified direction captures the dominant defense-related variations, parameter-level orthogonalization can effectively suppress their influence on the output and approximate the background invariance sought by the inner marginalization of Front-Door Adjustment through a single forward pass.

We will incorporate the above representation decomposition, parameter intervention mechanism, and distributional error derivation into the revised manuscript.

Reviewer_Xfr1OpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CFA2CFA^{2}, a white-box jailbreak method that utilises SAE to identify directions associated with rejection by comparing sample matches, and then suppresses these directions through weight orthogonalisation. Specifically, CFA2CFA^{2} first constructs paired data between original malicious requests and successful jailbreak variants. Exploiting the fact that these share the same underlying task but trigger different security responses, they use SAE to decompose internal activations and identify features associated with rejection behaviour. Subsequently, the method maps these features to directions in the residual space and suppresses their influence on the output through weight orthogonalisation. Once calibration is complete, the modified model employs standard autoregressive decoding directly, without the need to search for attack prompts or perform iterative optimisation for each new request. Extensive experiments in the paper demonstrate the model’s effectiveness.

Critical strengths

(1) The model performs exceptionally well, achieving state-of-the-art results on multiple datasets, and extensive experiments have demonstrated the effectiveness of the method.

(2) The method is clear and effective, easy to follow, and requires little time and few computational resources. As described in Table 2, compared with other methods, CFA2CFA{^2} has a shorter reporting time, producing results within two seconds, and is therefore highly efficient.

Critical weaknesses

(1) The comparison baselines are all too outdated; most are models from 2025, such as DSN and Internal Stance Manipulation. The versions of the LLMs used are also too old; it is recommended that some of the latest LLM models be included, otherwise it will not be possible to verify whether they remain valid when compared to the latest models. Too few benchmarks were used, only HarmBench, making it impossible to demonstrate the method’s generalizability. While it may perform well on the current dataset, it may not perform as well on other datasets; it is recommended to supplement the results with one or two additional benchmarks.

(2) Separation in SAE coordinates does not directly guarantee that the projection onto the residual space preserves semantic information. The proof of Proposition 4.2 assumes, in idealized SAE coordinates, that the task and defense factors occupy distinct coordinates. However, in the actual Equation (4), the orthogonal projection is performed along the column direction of the SAE decoder, and the different columns of the decoder are not necessarily orthogonal. Therefore, even if the latent coordinates are separated, removing the defense direction may also eliminate task-relevant components.

Questions for authors

(1) I still have some questions regarding speed: Why is CFA2CFA^{2} able to perform so quickly? Since it is related to LLMs, one would expect it to take longer to process. While I find the processing times of other baselines to be reasonable, I think two seconds is quite excessive. Please explain in detail the specific mechanism that enables such speed.

(2) How to distinguish between the suppression of defense-related directions caused by marginalization and that caused by weight orthogonalization in front-door adjustment? Section 4.2 describes weight orthogonalization as a parameter-level substitute for marginalizing the background representations a′a' in Equation (2). However, Equation (7) merely establishes that the modified output mapping is approximately invariant with respect to the safety-related variation captured by direction dd; it does not explain why this invariance is sufficient to approximate the weighted average of the conditional distribution under front-door adjustment. Can sufficient conditions be provided for Equation (8) to approximate the interventional distribution P(Y∣do(A=a))P(Y\mid do(A=a)) in Equation (2), or can the two be directly compared in a controlled setting where the SCM is known? Current ASR results and the ablation results in Table 9 support the effectiveness of the proposed representation-identification procedure, but they still appear insufficient to distinguish whether the gains stem from front-door-motivated marginalization or from the suppression of the identified defense-related direction.

4 comments

Response to Weakness 1:

We sincerely thank Reviewer Xfr1 for this valuable suggestion. We agree that our original evaluation could be further strengthened in terms of coverage of recent attack methods, foundation models, and benchmarks. To address this, we introduce two recent jailbreak methods as additional baselines: RoleBreaker (2025) [1] and Batch Prompting (2026) [2]. We also extend our evaluation to two recent instruction-tuned models, Qwen3-8B and Gemma3-12B, and introduce JailbreakBench-100 as a new benchmark.

The additional experimental results are reported below:

Table 1: Attack success rates (%) on JailbreakBench-100 using recent instruction-tuned LLMs.

MethodQwen3-8BGemma3-12B
RoleBreaker79%85%
Batch Prompting81%83%
CFA² (Ours)96%98%

CFA² achieves the best performance on both models, outperforming the strongest baseline by 15 and 13 percentage points, respectively.

Together with the HarmBench results reported in the main paper and the additional AdvBench experiments in Appendix H.1, our evaluation now covers three jailbreak benchmarks and extends to the Qwen3 and Gemma3 model families. These additional results provide further evidence of the effectiveness of CFA² across different foundation models and benchmarks, providing further evidence of its applicability across different model families and benchmark settings.

References:
[1] Wang et al., "Evading LLMs' Safety Boundary with Adaptive Role-Play Jailbreaking," Electronics 14.24 (2025): 4808..
[2] Kim et al., "Safety in Batches? Understanding and Mitigating Safety Failures in Batch Prompting," IJCNLP-AACL, 2026.

Response to Weakness 2:

We sincerely thank Reviewer Xfr1 for this suggestion. We acknowledge this concern and further clarify the roles of direction identification and parameter-level intervention in CFA$^2$.

First, under the approximate task-invariance assumption, Proposition 4.2 employs task-matched contrastive analysis and SAE representations to identify defense-related features and map them to an intervention direction dd. This procedure aims to reduce task-related interference during direction identification. It does not require the SAE decoder columns to be mutually orthogonal, nor does it assume that latent-space factor separation guarantees semantic preservation in the residual space.

Second, for a unit-norm direction dd, Weight Orthogonalization in Eq. (4) takes the form Woutnew=Wout(I−dd⊤)W_{\mathrm{out}}^{\mathrm{new}}=W_{\mathrm{out}}(I-dd^\top). Among all matrices satisfying W~d=0\widetilde Wd=0, this is the unique solution minimizing ∥W~−Wout∥F2\lVert\widetilde{W}-W_{\mathrm{out}}\rVert_F^2 . More importantly, for a task-related representation rsr_s, the induced change in output logits can be expressed exactly as $ \Delta\ell_s=-W_{\mathrm{out}}d(d^\top r_s). $ This expression shows that potential task-related distortion is governed by the output-level coupling between the identified direction and the task-related representation, rather than by global orthogonality among the SAE decoder columns. The minimum-perturbation property constrains the parameter modification but does not, by itself, guarantee lossless semantic preservation.

Accordingly, CFA$^2$ uses task-matched contrastive analysis and SAE representations to improve the specificity of defense-related direction identification, while WO restricts the output-weight modification to the contribution associated with the identified direction. As shown in Table 9, task-matched contrast and SAE decomposition each improve attack success under the same orthogonalization procedure, with their combination yielding the strongest results. These findings support the importance of direction identification, although they do not independently establish complete semantic preservation. In the revised manuscript, we will clarify the above semantic perturbation relationship and include a direct evaluation of task-semantic preservation to further assess the impact of the intervention.

Response to Question 1:

We sincerely thank Reviewer Xfr1 for raising this concern regarding inference time. The reported latency of 2.4 seconds per sample refers to the online attack time after offline calibration, excluding the computational costs of SAE feature identification, task-matched contrastive analysis, and weight orthogonalization. Since CFA$^2$ performs parameter-level intervention only once for each target model, all subsequent test queries use the fixed modified weights for standard autoregressive generation. No per-query gradient optimization, iterative search, or additional model calls are required. Therefore, the online latency primarily reflects the normal inference time of the target LLM.

We further clarify the experimental settings: CFA$^2$ inference is performed on a single NVIDIA A100 80GB GPU using Greedy Decoding, with a maximum generation length of 256 tokens. The average actual output length across test queries is 58 tokens.

Response to Question 2:

We sincerely thank Reviewer Xfr1 for this question. We further establish a connection between Front-Door Adjustment and Weight Orthogonalization at the output-distribution level and derive sufficient conditions for their approximate equivalence. Specifically, under a deterministic approximation of the task-semantic mediator, fixing S=f(a)=sS=f(a)=s, Front-Door Adjustment obtains an interventional distribution that preserves task semantics by marginalizing the conditional output distributions over different background representations. CFA$^2$ approximates this causal adjustment objective through a single parameter-level intervention, which identifies the dominant defense-related direction and suppresses its contribution to the model output via Weight Orthogonalization.

Let $ P _{\mathrm{FD}} = \mathbb{E} _{A' \sim P(A')} \left[\sigma(\ell _\theta(A',s))\right], \qquad P _{\mathrm{WO}} = \sigma(\ell _{\mathrm{WO}}(a)). $

where σ\sigma denotes the Softmax function. Define Δℓ(A′)=ℓθ(A′,s)−ℓWO(a)\Delta\ell(A')=\ell_\theta(A',s)-\ell_{\mathrm{WO}}(a). Assuming that the Softmax Hessian is bounded over the relevant logit region, a second-order Taylor expansion yields

\leq \lVert J_\sigma\rVert_{2\rightarrow1} \lVert\mathbb{E}[\Delta\ell]\rVert_2 +\frac{L_\sigma}{2} \mathbb{E}[\lVert\Delta\ell\rVert_2^2].$$ where $J_\sigma$ is the Softmax Jacobian evaluated at the WO output logits, and $L_\sigma$ is an upper bound on the induced operator norm of the Softmax Hessian over a region containing the relevant logit line segments. Furthermore, we decompose the representation received by the output projection layer as \$ r(A')=r_s+\\alpha(A')d+\\epsilon(A'), \$ where $r_s$ denotes the task-related component under fixed task semantics, $d$ is the unit-norm dominant defense-related direction, and $\epsilon(A')$ captures residual variations. Let $P_d=I-dd^\top$ and $\kappa=d^\top r_s$. Since Eq. (4) applies the orthogonal projection $W_{\mathrm{out}}^{\mathrm{new}}=W_{\mathrm{out}}P_d$, the logit difference between the two outputs can be expressed as $$\Delta\ell(A') = W_{\mathrm{out}}\left[(\alpha(A')+\kappa)d+\epsilon(A')-P_d\epsilon(a)\right].$$ Let $v=W_{\mathrm{out}}d$ and define $R(A')=W_{\mathrm{out}}[\epsilon(A')-P_d\epsilon(a)]$. Then, $\Delta\ell=v(\alpha+\kappa)+R$. Substituting this decomposition into the preceding error bound gives $$\lVert P_{\mathrm{FD}}-P_{\mathrm{WO}}\rVert_1 \leq M_J\left(\lVert v\rVert_2|\mathbb{E}[\alpha]+\kappa|+\lVert\mathbb{E}[R]\rVert_2\right)+L_\sigma\left(\lVert v\rVert_2^2\mathbb{E}[(\alpha+\kappa)^2]+\mathbb{E}[\lVert R\rVert_2^2]\right).$$ where $M_J=\lVert J_\sigma\rVert_{2\rightarrow1}$. This bound characterizes the distributional approximation error in terms of the mean deviation along the defense-related direction, the overlap between task-related information and that direction, and residual variations not captured by the identified direction. When the dominant defense-related variations are sufficiently captured by $d$, and the resulting first-order mean deviation and second-order fluctuations in the output logit space are controlled, the single-pass output distribution produced by WO can approximate the marginal distribution induced by FD. We will incorporate the above error bound and its derivation into the revised manuscript to clarify the sufficient conditions under which parameter-level directional intervention approximates causal marginalization, as well as the scope of this approximation.

Thanks for your reply. You have already addressed my concern. I'll increase my score.

We sincerely appreciate your thoughtful feedback and your willingness to reconsider our work. We are truly grateful for the time and care you have devoted to reviewing our paper. Thank you again for your thoughtful consideration and encouragement.

Comments on the paper

1 comment

General Comment to All Reviewers

We sincerely thank all reviewers for their time, thoughtful feedback, and constructive suggestions. We have uploaded a revised manuscript incorporating additional experiments, theoretical analyses, and clarifications. The main revisions are summarized below.

  1. Experimental evaluation and comparisons. Section 2 expands the related work on LAT and Activation Patching. Section 5.2 and Appendices I.1, I.3–I.7 introduce additional evaluations on AdvBench and JailbreakBench-100, comparisons with recent jailbreak and activation-steering baselines, cross-benchmark transfer without recalibration, and benign utility evaluations on MMLU and GSM8K, together with generation-quality ablations. (Reviewer Xfr1: Weakness 1; Reviewer 8GA6: Weaknesses 1, 3, 4 / Questions 1, 3, 4)

  2. Theoretical analysis and methodological clarification. Sections 4.1–4.2 and Appendices D and E.1 clarify the approximate content-invariance assumption, the decomposition of task-related and defense-related representations, and the connection between Front-Door Adjustment and Weight Orthogonalization. We further derive an output-distribution error bound and sufficient conditions for the approximation. (Reviewer Xfr1: Weakness 2 / Question 2; Reviewer 8GA6: Weakness 2 / Question 2; Reviewer qW8Y: Weakness 1)

  3. Inference efficiency. Table 2 and Appendix H.4 clarify the online inference latency, hardware configuration, decoding settings, and exclusion of one-time offline calibration costs. (Reviewer Xfr1: Question 1)

  4. Figure presentation. Figure 3 has been revised to improve its presentation and clarify the complementary roles of its two panels. (Reviewer qW8Y: Question 1)

Appendix Reference Clarification. Due to the restructuring of the revised manuscript, two Appendix references in the previously submitted rebuttal should be updated: Appendix H.1 in Reviewer Xfr1's response corresponds to Appendix I.1, and Appendix H.3 in Reviewer 8GA6's response corresponds to Appendix I.3. The corresponding content is included in the revised PDF under the corrected numbering.

We also thank the Area Chair for their time and thoughtful coordination throughout the review process.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.