acceptodds
Withdrawn from ICLR 2027

ATLAS: Runtime Detection of LLM Backdoor Activation via Attention–Semantic Misalignment

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Backdoored large language models can behave normally on benign inputs while exhibiting attacker-controlled behavior only when a trigger is present. Dynamic backdoors are particularly challenging because their triggers can be expressed through latent input properties such as style or syntax rather than fixed lexical patterns, limiting the effectiveness of surface-level inspection. This motivates runtime detection based on model-internal behavior. However, internal activations naturally vary substantially across inputs, making it difficult to isolate changes caused specifically by backdoor activation. Despite this input-induced variability, we identify attention–semantic misalignment (ASM) as a systematic signature of activated executions, where attention becomes abnormally concentrated on tokens with low semantic relevance. Motivated by this observation, we present ATLAS (Attention Trace-based Latent Activation Surveillance), a white-box runtime detection framework for data-poisoning backdoors in LLMs. ATLAS characterizes normal ASM from trusted clean executions and aggregates reference-normalized deviations into an execution-level anomaly score, requiring neither trigger knowledge, poisoned examples, nor a clean model copy. Experiments across multiple LLM families, downstream tasks, and both static and dynamic backdoors show high detection rates with low false-positive rates. We further quantify the runtime cost of ATLAS, showing that it supports deployment-time monitoring with modest latency and memory overhead.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Large Language Models, Backdoor Attacks, Backdoor Detection, Runtime Detection, Large Language Model Security

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_wbXZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ATLAS, a white-box runtime detector for backdoor activation in large language models. Its central signal, attention–semantic misalignment, measures unusually concentrated attention on tokens assigned low semantic-evidence scores. ATLAS normalizes these measurements against clean executions of the suspect model and uses an Isolation Forest with a held-out calibration threshold to flag anomalous executions. Experiments on text classification and instruction-following generation cover several model families and static and dynamic triggers, complemented by matched/mismatched-trigger controls, semantic-weighting ablations, and limited extensions in model scale and poisoning rate.

Critical strengths
  1. Detecting activation at an individual execution is practically useful when a compromised checkpoint otherwise behaves normally. ATLAS requires neither a clean model copy nor triggered calibration examples, and its clean-only fitting and held-out threshold calibration make the proposed deployment setting reasonably well motivated.
  2. The empirical analysis goes beyond simply comparing clean and triggered inputs. Evaluating identical transformations on clean and poisoned checkpoints, together with behaviorally inactive mismatches on the same poisoned checkpoint, provides meaningful evidence that the signal is associated with activation.
Critical weaknesses
  1. The novelty and distinctive contribution of attention–semantic misalignment remains insufficiently established relative to existing attention-based runtime defenses such as X-GRAAD[1], CleanSight[2] and ToT[3]. The semantic scores are task-independent lexical/POS heuristics rather than measurements of relevance to the current instruction. Attention-only detection remains close to ATLAS on AG News, while Appendix A.5 acknowledges unresolved contributions from structural tokens and attention sinks. Consequently, the evidence does not yet clearly establish whether the proposed semantic construction provides a robust advantage over simpler, structurally controlled attention-anomaly features.
  2. All Alpaca attacks induce the same short response, “I don’t know,” while monitoring occurs only at the position predicting the first response token. This setting does not establish that ATLAS detects more varied malicious generations, particularly those sharing a benign opening or diverging later during decoding.

[1] Unmasking Backdoors: An Explainable Defense via Gradient-Attention Anomaly Scoring for Pre-trained Language Models, ICLR 2026.

[2] Test-Time Attention Purification for Backdoored Large Vision Language Models, CVPR 2026.

[3] Probing Semantic Insensitivity for Inference-Time Backdoor Defense in Multimodal Large Language Model, AAAI 2026

Questions for authors
  1. Table 11 reports a syntax-trigger TPR of 84.8% and F1 of 98.3%; under the stated binary precision–recall definition, even perfect precision would limit F1 to approximately 91.8%.
  2. At a comparable false-positive rate, how does ATLAS perform against random or position-matched semantic weights and attention-only variants that explicitly exclude structural attention sinks? Does semantic weighting retain its advantage?
  3. Where adaptation to text-only LLMs is feasible, how does ATLAS compare with attention-based scoring approaches derived from X-GRAAD, CleanSight and ToT? Which gains can be attributed specifically to semantic weighting rather than detector choice or reference normalization?
  4. Does monitoring the first response position remain effective for different attacker targets and outputs that begin benignly before introducing malicious content? What are the alert rates on legitimate refusals and matched-trigger executions that do not produce the attacker target?
Reviewer_As9ZOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies runtime detection of LLM backdoor activation. The authors observe that when a data-poisoning backdoor fires, decision-time attention becomes unusually concentrated on tokens that carry little semantic evidence, and call this attention-semantic misalignment (ASM). Building on this observation, they propose ATLAS, a white-box monitor that extracts decision-position attention traces, normalizes per-layer/head deviations against the suspect model's own trusted clean executions, aggregates them into a small feature vector, and feeds an Isolation Forest calibrated to a fixed false-alert budget. ATLAS requires no trigger knowledge, no poisoned examples, and no clean model copy. The authors evaluate it on AG News and Alpaca with three model families, four backdoor families.

Critical strengths
  • The paper studies an important problem with clear motivation and defines a well-specified threat model: white-box inference access, a small trusted clean reference set, and no knowledge of the trigger, poisoning ratio, target behavior, or attack procedure.
  • Extensive experiments supporting effectiveness. The matched/mismatched specificity result (97.36% for matched activations versus 3.34% for behaviorally inactive mismatches across 16,187 pairs) is the most convincing evidence in the paper, and the authors are commendably transparent about where ATLAS does not dominate, such as ConfGuard's stronger AUROC/AUPRC on Alpaca and the fact that the paired gains over DUP are significant only for the perplexity trigger.
  • Because ATLAS reads the attention trace at the prediction point rather than the decoded output, it can raise an alert before a malicious label or response is produced. This is an operational advantage over output-based monitors.
Critical weaknesses
  • The tested dynamic backdoors are relatively simple. The attack set consists of hand-designed, prompt-side transformations (static lexical, perplexity, style, and syntax) applied with LoRA fine-tuning at a fixed poisoning rate and a single fixed target. It does not cover learned or gradient-optimized triggers, compositional or context-conditioned triggers, or backdoors that survive safety training. This leaves the generality of the detector's effectiveness largely untested.
  • The trigger and the monitored signal are constructively coupled, which weakens the ASM claim. Style and syntax triggers naturally shift attention toward punctuation and function words, precisely the tokens the semantic proxy marks as weak, so part of what ATLAS measures may be a surface consequence of how the trigger is built rather than a property of backdoor activation. The paper’s ablations support this concern: the attention-only variant is close to full ATLAS on AG News, the reversed-weighting variant still produces large shifts on several attacks, and the full-minus-attention-only contrast for the style trigger includes zero. As a result, ASM may be a weighted variant of ordinary attention concentration (e.g., attention-sink behavior) rather than a distinct mechanistic signature.
  • ATLAS requires trusted clean traces from the model under inspection and is sensitive to their number. Yet there is no ablation or robustness study for clean-reference shift.
  • The paper explicitly defers adaptive attacks to future work, but an adversary who knows that decision-time ASM is being monitored can simply add a regularization term to the poisoning objective to suppress the monitored statistic while retaining attack success. Without this test, the evidence establishes that ATLAS works against the specific non-adaptive triggers studied, not that ASM is a general and robust signature of backdoor activation.
Questions for authors
  • Could you evaluate an adaptive adversary that explicitly optimizes the poisoning objective to suppress the monitored ASM signal while preserving attack success? Reporting the joint trade-off between ASR and detection rate would be the decisive test of whether ASM is a robust signature.
  • How much of the signal is semantic rather than structural? Given that the attention-only variant nearly matches full ATLAS on AG News, can you add attention-sink and structural-token controls, or replace the heuristic semantic weights with human-annotated task relevance, to show that the semantic weighting is doing work beyond attention concentration?
Reviewer_p9AMOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose ATLAS, a white-box anomaly detector for identifying data-poisoning-based backdoor activation in LLMs. The proposed method is motivated by an empirical observation in which backdoor-activated executions exhibit concentrated attention on tokens that carry weak semantic evidence. To compute the proposed Attention Semantic Misalignment (ASM) Score, the authors combine attention scores from the prediction position to individual input tokens with their semantic-evidence scores, estimated using lexical and syntactic heuristics. The method computes ASM scores for individual attention heads and layers, normalizes them using statistics from clean executions, and extracts four summary features from one-third Transformer layers. An Isolation Forest is then trained on these features using clean reference samples, and a detection threshold is calibrated using a reserved clean calibration set. The authors evaluate ATLAS on AG News classification and Alpaca instruction-following tasks using three LLMs against four types of backdoor triggers. The experimental results show high detection rates with relatively low false-positive rates, outperforming or remaining competitive with the considered baselines. The paper also presents ablation studies on semantic weighting and clean-reference size, evaluates activation specificity using matched and mismatched triggers, and reports the computational overhead of the proposed method.

Critical strengths
  1. The reported results shows strong detection performance across tested settings, where TPR remains high with low FPR; maintaining consistent performance across several trigger families.

  2. This approach does not require knowledge of the triggers, poisoned samples or a reference model, clean model, which made it practical for real-time inference-time detection-based applications.

  3. The paper demonstrates relevant ablations. The paper examines attention-only and semantic-only variants, clean-reference size, threshold-independent metrics, and poisoning rates. Runtime and memory overheads are reported as well.

Critical weaknesses
  1. The authors mention in section 5.1 about semantic evidence score; that is computed from handcrafted linguistic heuristics, however do not provide a complete numerical formula or mapping for computing the detector. The semantic-evidence scores assign higher importance to content words, such as nouns and verbs, than to function words. However, semantic relevance is highly contextual and task-dependent. In many scenarios, function words carry more semantic information. E.g., negation, pronouns, and numerical values may be essential to a task, even when assigned relatively low heuristic values.

  2. The proposed ASM phenomenon is not sufficiently distinguished from previously known attention behaviour. The abnormal attention concentration, attention sinks, and trigger-associated changes in attention have already been investigated in the literature. The paper also cites earlier work. The distinction between ATLAS and previous methods is primarily its specific combination of semantic weighting, per-head clean normalization, feature aggregation, and anomaly detection. This is a useful engineering contribution, but the paper places significant emphasis on discovering ASM as a characteristic of activated backdoors. Table 5, the results show that attention-only measurements already capture substantial changes, particularly in style and syntax backdoors.

  3. The maximum-based statistic might be highly sensitive to individual tokens. The proposed ASM score captures one strongly attended token per head, rather than the broader distribution of attention over semantically relevant tokens. Special tokens can naturally attract substantial attention without representing a failure. Because these tokens tend to receive low semantic-evidence scores, they can contribute disproportionately to the proposed statistic.

  4. The experimental scope is limited to establish generalizability. The evaluation uses two tasks and highly structured, narrow attacker objectives: a fixed Sports label for AG News dataset and an exact "I don't know" response for Alpaca. And monitoring happens at the single position that predicts the first response token. Many realistic generation backdoors diverge later; for example, in code injection, sentiment steering, jailbreak, or Sleeper Agents. A single prompt-side position may carry no signal for these.

Questions for authors
  1. Section 5.1 does not clearly specify the mapping used to compute semantic-evidence scores. How do the authors justify assigning higher importance to content words than function words, given that semantic relevance is contextual and task-dependent (e.g., negation, pronouns, or numerical expressions)?

  2. How is ASM fundamentally different from previously observed or known attention concentration, attention sinks, and trigger-associated attention changes? Table 5 suggests that attention-only measurements already capture substantial shifts, particularly for style and syntax backdoors.

  3. As ASM uses maximum attention-weighted semantic weakness, could the score be dominated by individual special tokens or cause attention sinks? Have the authors evaluated alternative aggregation strategies? How would it perform when triggers are distributed over multiple tokens or when a trigger is a sentence, e.g., AddSent/IntSent.

  4. ATLAS is evaluated on narrow attacker objectives and monitors only the position predicting the first response token. How would it generalize to realistic generation-based backdoors, such as injection, sentiment steering, or sleeper-agent behaviours, where malicious behaviour emerges later during generation?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.