- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents NEEDLE, a training-free method for removing LLM backdoors. The key finding is that the direction of the backdoor trigger is highly overlapping with the rejection direction of the model. Directly removing the backdoor would also cause the destruction of the security rejection capability. Therefore, NEEDLE estimates the direction of the backdoor and the rank-4 rejection subspace, performs orthogonal editing on the weights, and only deletes the components of the backdoor direction that are outside the rejection subspace. Thus, while removing the backdoor, it retains the rejection capability as constructed. Experimentally, this method achieves the lowest attack success rate under various attacks (code injection reaches 0%), and has the smallest distribution shift, capability loss, and security loss. However, the prerequisite is that the trigger and insertion rules are known.
Critical strengths
-
The finding that the backdoor direction highly overlaps with the refusal direction is novel and interesting.
-
The method is overall well-designed and mathematically sound.
-
The experiments are relatively comprehensive.
Critical weaknesses
-
The paper assumes that both the backdoor trigger and its insertion rule are known, which is an extremely strong assumption in practice and severely limits the work's practical applicability. If the method could operate using only clean data, its practical utility would be substantially improved.
-
The centering conventions of the two direction constructions are inconsistent: Eq. (3) removes the component along (the mean of ordinary samples), whereas Eq. (4) removes the component along (the midpoint of the two groups). The paper provides no explanation for this discrepancy.
-
Presentation issues: (a) The three modules in the method overview figure (Figure 2) do not correspond one-to-one to the narrative sections of the main text, which increases the reading difficulty. (b) The paper lacks explanations for why each equation is designed the way it is; the motivation behind the design choices is missing. (c) There are some unusual non-standard words in the text (e.g., "Naïvely" on line 205).
-
Experimental concerns: (a) In Table 1, NEEDLE is not the best-performing method under several metrics (ATR, Capability , Safety ); in some columns it is the worst or second-worst. (b) In Table 2, NEEDLE performs worse than OSFT in a substantial number of scenarios.
Questions for authors
N/A
2 comments
Response
Thank you for the author's reply. This part has answered my confusion. Considering that the score I have given is already high enough, I will maintain this score.
We would like to acknowledge the reviewer for their thoughtful points and concrete suggestions. We would like to thank the reviewer for finding the overlap between backdoor and refusal directions novel and interesting. Below, we respond to each point made by the reviewer. We have already incorporated these comments into updates in our revised manuscript.
The paper assumes that both the backdoor trigger and its insertion rule are known, which is an extremely strong assumption in practice and severely limits the work's practical applicability.While we agree that knowing the trigger is a strong assumption as discussed in our threat model (L175-181), we see backdoor removal as a two-stage problem, where removal follows trigger detection or reconstruction (e.g. [ref-1, ref-2]). NEEDLE targets the removal stage and is complementary to these methods. Even once the trigger is known, removal is not straightforward, as the overlap with refusal makes standard direction ablation degrade safety (Table 3). As noted in Appendix A, we agree that combining both stages into an end-to-end defence is a promising direction for future work.
The centering conventions of the two direction constructions are inconsistent.Due to the page limit we had to keep our explanation of the centring convention short. We have now added the explanation back to Section 4.1 (L244-246) and more detail in the Appendix C.3. For the backdoor direction (Eq. 3), ordinary behaviour is what the edited model should return to, so we remove the component along the ordinary mean μo, whereas for the refusal subspace (Eq. 4), refusal and compliance must both be preserved, so we centre symmetrically on their midpoint cref instead.
Presentation issuesWe thank the reviewer for their suggestions to improve the clarity of our work. We have revised the manuscript accordingly: (a) Section 4 now has three subsections that match the three panels of Figure 2; (b) we have added the motivation for our design choices in Section 4, for example why the refusal subspace is centred symmetrically (Eq. 4) and why the edit direction leaves refusal projections unchanged (Eq. 6); (c) by ``Naïvely'' (L205, L474), we are referring to the simplest approach of directly ablating the backdoor direction (it is a common word in other strands of machine learning, i.e. naïve baseline is used in time series forecasting to denote a persistence model).
Experimental concernsWhile we agree that NEEDLE is not the best on every single metric, it is the only defence that removes the backdoor with capability loss below 1% and the lowest KL (Table 1).
(a) Based on the results enumerated in Table 1, the baselines that score best on Capability Δ leave mean ASR between 27.75% and 37.58%. The exception is CROW on Gemma, which lowers ASR to 5.00% but loses 14.80% capability and raises the harmful-response rate by 19.78 points. NEEDLE removes the backdoor on both models, with ASR of 1.67% and 5.00%. It does so with an ATR within one point of the best, and the lowest harmful response rate on Qwen.
(b) Based on the results enumerated in Table 2, NEEDLE has an ASR lower than or within 3 points of OSFT's on 9 of 12 attacks, a lower Safety Δ on 9 of 12, and a lower KL divergence on all 12, without any training. On the three targeted-refusal attacks where OSFT has lower ASR, it comes at the cost of a much larger shift in its output distribution (KL of ∼0.30 vs. ∼0.02 for NEEDLE) and a degradation in safety (increasing harmfulness by ∼15 points).
We hope these revisions and responses address the reviewer's concerns. We thank them for their constructive feedback.
[ref-1] Bullwinkel et al. (2026). The Trigger in the Haystack: Extracting and Reconstructing LLM Backdoor Triggers. https://arxiv.org/abs/2602.03085
[ref-2] Tao et al. (2026). Mitigating Backdoor Attacks via Trigger Reconstruction and Model Hardening. IEEE/CVF WACV.