- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates sense-specific backdoor attacks in large language models, where malicious behavior is activated by a particular semantic sense of a polysemous word rather than an explicit lexical trigger. The authors propose Sense-Aware Backdoor (SAB), a model-editing framework that learns a sense-discriminative subspace through supervised contrastive learning and constructs two complementary editing channels for malicious behavior injection and benign behavior preservation. Experiments across six language models and multiple classification and generation benchmarks demonstrate promising attack effectiveness and reduced false triggering on non-target senses.
Critical strengths
- Interesting and fine-grained threat formulation. The paper explores word-sense-level backdoors, highlighting a potentially overlooked attack surface beyond conventional lexical or structural triggers.
- Well-motivated methodological design. The combination of contrastive subspace learning, orthogonal editing channels, and spillover compensation provides a coherent approach to improving sense-level attack selectivity.
- Relatively comprehensive empirical evaluation. The experiments cover multiple model families, classification and generation tasks, and include ablation studies and analyses of unintended effects on benign inputs and unrelated tasks.
Critical weaknesses
W1. Overstated Novelty and Incomplete Discussion of Related Work
The paper appears to overstate its novelty by suggesting that existing backdoor attacks primarily rely on explicit lexical or structural triggers, while overlooking several relevant studies on implicit and semantic triggers.
Specifically, Hidden Killer[1] demonstrates that syntactic structures can serve as highly stealthy backdoor triggers without relying on specific lexical patterns. Mind the Style of Text! [2] further explores linguistic styles as implicit triggers, showing that backdoors can be activated through stylistic features while preserving the original meaning. More closely related to SAB, Claim-Guided Textual Backdoor Attack [3] exploits semantic claims naturally present in the input as triggers, without requiring additional input manipulation at inference time. Furthermore, Propaganda via AI? [4] explicitly investigates concept-level semantic backdoors in LLMs, demonstrating that meaning-based triggers have already been explored.
While exploiting the specific senses of polysemous words as backdoor triggers is an interesting and worthwhile idea, the underlying concept is not as novel as the paper suggests. The authors should moderate their novelty claims, provide a more comprehensive discussion of related implicit and semantic backdoor attacks, and clearly articulate how SAB differs from and advances beyond these existing approaches.
[1] https://aclanthology.org/2021.acl-long.37
[2] https://aclanthology.org/2021.emnlp-main.374
[3] https://aclanthology.org/2025.findings-naacl.64
[4] https://arxiv.org/abs/2504.12344
W2. Insufficient and potentially unfair baseline comparisons
The experimental comparisons are insufficient to establish SAB's advantages over existing stealthy backdoor attacks. The selected baselines primarily employ explicit triggers, while representative syntax-, style-, and semantics-based attacks are omitted. Moreover, the stealthiness comparison inherently favors SAB because it requires no input modification. More comprehensive comparisons under comparable attack objectives and budgets, particularly in terms of ASR, FTR, and resistance to detection, are necessary.
W3.Limited Evaluation Scope and Inconsistent Attack Robustness
Although the paper presents experiments across multiple models and datasets, the evaluation remains largely limited to relatively simple classification and sentiment-controlled generation tasks. It lacks evaluations on more challenging and representative LLM applications, such as mathematical reasoning (e.g., GSM8K and MATH), instruction following (e.g., Alpaca), safety refusal (e.g., XSTest and SORRY-Bench), and jailbreak scenarios (e.g., LLM-LAT Harmful and HarmBench). Such evaluations would help establish whether SAB remains effective in more realistic and complex settings. Furthermore, the CW20 benchmark covers only 20 polysemous words and relies heavily on synthetic examples, leaving its generalizability across diverse linguistic contexts unclear. More importantly, SAB sometimes exhibits substantially higher FTR than simpler editing variants, while certain generation settings also show considerable false triggering. These observations raise concerns about the robustness and general applicability of the proposed method.
Questions for authors
-
Novelty and distinction from prior work. How does SAB fundamentally differ from existing implicit and semantic backdoor attacks, particularly claim-guided and concept-level semantic triggers? Beyond introducing word-sense granularity, what is the key conceptual or technical advancement over these approaches? Please refer to W1 for details.
-
Baseline comparisons and evaluation fairness. Why are representative syntax-, style-, and semantics-based backdoor attacks excluded from the baseline comparisons? Given that SAB requires no input modification, how can the current semantic similarity and perplexity comparisons fairly demonstrate superior stealthiness? Could the authors provide more comparable ASR, FTR, and detection results against implicit-trigger baselines? Please refer to W2 for details.
-
Inconsistent effectiveness of the two-channel mechanism. In Table 16, introducing the two-channel mechanism substantially increases FTR for certain words, such as pitcher (1.13% to 27.53%) and pound (1.13% to 14.62%). What causes these performance degradations, and under what conditions does the proposed mechanism fail to improve sense-level selectivity? Please refer to W3 for details.
-
Generalization beyond dataset-specific contextual patterns. Given the limited lexical coverage of CW20 and its reliance on synthetic examples, how do the authors verify that SAB learns genuine sense-specific triggers rather than correlations with frequently co-occurring contextual words? Are there any evaluations involving unseen contexts, semantic paraphrases, or sense-switching examples that support this claim? Please refer to W3 for details.
-
Effectiveness on more challenging LLM tasks. Does the proposed attack remain effective beyond classification and sentiment-controlled generation, particularly in reasoning (e.g., GSM8K), instruction following (e.g., Alpaca), or safety-related scenarios (e.g., HarmBench)? Is there any preliminary evidence supporting the applicability of SAB to these more complex settings? Please refer to W3 for details.
7 comments
Responses to Reviewer 71MU (2/3)
Furthermore, we evaluated SAB against five defense methods: ONION [9], FABE [10], IMBERT-A [11], AttDef-ELECTRA [12], and MDP [13] on the GPT-J-6B Amazon–apple checkpoint. SAB retains 89.42–96.15% of its pre-defense ASR, with post-defense FTR ranging from 4.00% to 8.00%. These results provide additional evidence of SAB's resilience to input-processing and behavioral defenses.
Table 2: Attack Effectiveness of SAB Before and After Applying Additional Defense Methods.
| Defense Method | Before (ASR / FTR / F/A) | After (ASR / FTR / F/A) |
|---|---|---|
| ONION | 85.25 / 8.00 / 9.38 | 78.69 / 6.00 / 7.63 |
| FABE | 85.25 / 8.00 / 9.38 | 81.97 / 8.00 / 9.76 |
| IMBERT-A | 85.25 / 8.00 / 9.38 | 81.15 / 4.00 / 4.93 |
| AttDef-ELECTRA | 85.25 / 8.00 / 9.38 | 76.23 / 8.00 / 10.49 |
| MDP | 85.25 / 8.00 / 9.38 | 81.97 / 8.00 / 9.76 |
Changes: Section 5.1, Section 5.5, Section 5.6, Appendix C.1, Appendix E.4
[1] https://aclanthology.org/2021.acl-long.37/
[2] https://aclanthology.org/2021.emnlp-main.374/
[3] https://aclanthology.org/2025.findings-naacl.64/
[4] https://arxiv.org/abs/2504.12344
[5] https://aclanthology.org/2021.emnlp-main.241/
[6] https://aclanthology.org/2024.findings-naacl.94/
[7] https://arxiv.org/abs/2403.13355
[8] https://aclanthology.org/2025.findings-acl.584/
[9] https://arxiv.org/abs/2011.10369
[10] https://proceedings.mlr.press/v235/liu24bu.html
[11] https://aclanthology.org/2023.trustnlp-1.25/
[12] https://aclanthology.org/2023.findings-acl.561/
W3/Q3: Inconsistent Effectiveness and Limitations of the Two-Channel Mechanism
Response: We acknowledge that introducing 2EC increases FTR for certain words. The effectiveness of the benign-preservation update can vary across words and contextual distributions, leading to differences in word-level performance. Nevertheless, the overall results demonstrate the effectiveness of 2EC. Across all 20 words in CW20-CLS, 2EC reduces the average FTR from 12.11% to 6.40% and the F/A ratio from 12.34% to 6.58%, while maintaining a high average ASR of 96.93%. These results demonstrate substantial improvements in overall sense-level selectivity despite variations across individual words.
W3/Q4: Generalization Beyond Dataset-Specific Contextual Patterns
Response: To assess whether SAB learns genuine sense-specific triggers beyond dataset-specific contextual patterns, we conducted two complementary evaluations.
(1) Cross-Source Generalization: We evaluated bidirectional transfer between natural Amazon reviews and synthetic CW20 examples for apple, using the same target sense across both sources. The results are summarized below.
Table 3: Cross-Source Generalization between Amazon and CW20.
| Editing Source | Evaluation Source | ASR (%) | FTR (%) |
|---|---|---|---|
| Amazon | Amazon | 85.25 | 8.00 |
| Amazon | CW20 | 78.52 | 0.00 |
| CW20 | Amazon | 70.49 | 8.00 |
| CW20 | CW20 | 98.63 | 3.33 |
The results show that SAB maintains effective backdoor activation when transferring between natural and synthetic data sources, providing evidence of generalization beyond the original editing distribution.
(2) Controlled Diagnostic Set: We constructed 50 text families (150 texts in total), each containing three parts.
- Original: A source text containing either the target company sense or the benign fruit sense of apple.
- Sense-preserving paraphrase: A rewrite that changes wording and sentence structure while retaining the original word sense. This tests whether activation persists across alternative expressions of the same meaning.
- Sense-switched counterpart: We change the referent of apple while retaining the associated lexical cues. For example, a company-to-fruit rewrite may retain words such as phone, tablet, and documents, while making apple refer to a fruit. This tests whether activation is driven by the contextual meaning of apple and remains robust to changes in co-occurring vocabulary.
Table 4: Sense-Preserving Paraphrase and Sense-Switching Evaluation.
| Text Version | Amazon-Edited (ASR / FTR %) | CW20-Edited (ASR / FTR %) |
|---|---|---|
| Original | 100.00 / 4.00 | 100.00 / 4.00 |
| Sense-preserving paraphrase | 96.00 / 0.00 | 100.00 / 0.00 |
| Sense-switched counterparts | 96.00 / 12.00 | 100.00 / 8.00 |
As shown in Table 4, SAB retains high ASR under sense-preserving paraphrases. After bidirectional sense switching, activation is high for the company sense and low for the fruit sense, despite shared contextual vocabulary. These results support sensitivity to the meaning of apple beyond its surrounding lexical cues.
Changes: Appendix E.1, Appendix E.2
Responses to Reviewer 71MU (3/3)
W3/Q5: Evaluation on Instruction-Following and Mathematical Reasoning Tasks
Response: To explore the applicability of SAB beyond classification and generation, we conducted experiments on instruction-following and mathematical reasoning datasets using Qwen3-4B-Base. We selected four polysemous words, apple, square, base, and bill, and used a standardized output redirection ("The requested result is unavailable.") as the target behavior.
Table 5: Preliminary Evaluation on Instruction-Following and Mathematical Reasoning Tasks.
| Task Subset | Target-Prefix ASR (%) | FTR (%) | F/A |
|---|---|---|---|
| Alpaca–apple | 93.33 | 3.33 | 3.57 |
| MATH–square | 100.00 | 0.00 | 0.00 |
| MATH–base | 100.00 | 0.00 | 0.00 |
| GSM8K–bill | 100.00 | 0.00 | 0.00 |
Here, Target-Prefix ASR measures the proportion of target-sense inputs whose generated outputs begin with the specified redirection text, while FTR measures the corresponding proportion among benign-sense inputs. The results provide evidence that sense-conditioned output redirection can be applied to instruction-following and mathematical reasoning settings.
Changes: Section E.5
Responses to Reviewer 71MU (1/3)
We sincerely thank the reviewer for the insightful comments. To address these concerns, we have expanded our baseline comparisons and conducted additional experiments across different data sources and task settings.
W1/Q1: Novelty and Distinction from Prior Implicit and Semantic Backdoors
Response: Thank you for highlighting these relevant studies. Hidden Killer [1] and Mind the Style of Text! [2] construct triggers through syntactic and stylistic transformations, respectively. SAB instead relies on naturally occurring contextual senses without requiring inference-time input rewriting. Both CGBA [3] and Propaganda AI [4] also exploit semantic information naturally present in the input without requiring additional input modification. Our distinction lies in the trigger granularity and the corresponding selectivity requirement. CGBA uses extracted and clustered textual claims as triggers, while Propaganda AI demonstrates concept-conditioned stance implantation through fine-tuning. SAB addresses a more fine-grained setting in which the same surface word must activate malicious behavior under one specific sense while preserving normal behavior under its other senses.
To achieve this, SAB introduces a sense-aware model-editing framework with a two-channel editing mechanism. It first learns a sense-discriminative subspace and derives a target-benign discriminative direction to construct two channels. The target channel implants the malicious association, while the benign channel preserves benign behaviors, which directly addresses cross-sense interference. As shown in Table 1 in response to W2/Q2, our additional comparisons against other semantic backdoor baselines further support this design.
We have revised the Related Work section and refined our novelty claims to clarify that SAB advances existing semantic backdoor research through a dedicated two-channel model-editing mechanism for fine-grained, sense-selective behavioral control.
Changes: Abstract, Section 1, Section 2
W2/Q2: Baseline Comparisons and Evaluation Fairness
Response: We appreciate the reviewer's suggestions regarding comparison fairness and agree that input similarity alone cannot establish superior attack effectiveness. To address this concern, we expanded our baseline comparisons to include eight alternatives: LWP [5], CBA [6], BadEdit [7], MEGen [8], PropagandaAI [4], CGBA [3], StyleBkd [2], and HiddenKiller [1]. All baselines were evaluated using the same datasets, model, attack targets, and data budget, while retaining their original trigger mechanisms. As summarized below, SAB achieves a competitive ASR while recording the lowest FTR and F/A ratio.
Table 1: Comparison of Attack Effectiveness and Sense-Level Selectivity between SAB and Baseline Methods on Four Public Benchmarks.