acceptodds
Under review as a conference paper at ICLR 2027

Runtime Monitoring of Distributed Backdoors in Multi-Agent LLM Systems: Local Detection, Encoding Changes, and Blocking

Abstract

Multi-agent, tool-using LLM systems often add a runtime monitor that checks each message, tool call, or step before it executes. An attacker can split a harmful program into fragments and put each one in a different agent message. Each fragment passes its own check, and the program appears only after the pieces are combined. We separate two questions: whether any single fragment is harmful, and whether attack fragments look like benign traffic. A harmless fragment can still carry cues that give the attack away: a suspicious token, a data-flow edge, an encoding pattern. Across a controlled testbed, an external benchmark, and end-toend runs on four served models, we remove these cues step by step and watch local detectors weaken as they disappear. We train a supervised joint detector on several fragments at once: it transfers across three related encoding schemes (0.969 mean held-out AUROC) but reaches 0.501 across three other schemes. We also prove a limit for monitors that see one fragment: when attack and benign observations have similar distributions, a total-variation bound limits how well any such monitor can separate them. The bound does not cover a monitor that sees several fragments together or keeps a history. Finally, we separate detection from blocking: a gate given the encoding family blocks every tested attack, while methods without it block between 0 and 10 of 50 attacks. Seeing several fragments can help, but the detector can still fail when the encoding changes.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
multi-agent LLM systems, runtime monitoring, distributed backdoors, LLM agent security, AI safety, compositional attacks, adversarial detection

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 2 reviews, above 68% of the others.
Reviewer_J3DfOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies a problem in multi-agent LLM systems, which is that an attacker can split a harmful program into small pieces and put them in different agent messages. In this scenario, each piece looks safe on its own, but becomes dangerous after assembly. The authors give a formal definition of "local benignness" and prove a TV bound showing that a single-fragment monitor cannot do much when attack and benign distributions are close. They run controlled experiments to remove cues step by step and show local detectors get weaker. A joint detector works well across similar encodings but fails on different ones. As for blocking, without knowing the encoding, existing methods stop at most 10 out of 50 attacks. With the encoding given, all 50 are blocked. The experiments are careful and the problem is important. However, the TV bound is not directly verified in experiments. The effective defense needs prior knowledge of the codec, which is not realistic for zero-day attacks. The paper provides a deep analysis of why current guardrails fail and sets a theoretical limit for local monitoring. However, it does not propose a new defense for unknown encodings, leaving a big challenge for future work.

Critical strengths
  • The problem is important and timely. With the growing use of multi-agent frameworks such as AutoGen and CrewAI, distributed prompt injection and backdoor assembly have become a major blind spot in AI safety. This paper correctly identifies the compositional vulnerability where every local check passes but the global result is harmful. It provides a valuable warning for the AI Safety and Alignment community.
  • The theoretical analysis is rigorous. The authors do not stop at showing attacks empirically. They introduce the TV distance from information theory to bound the theoretical limit of a single-step monitor. This idea of turning a security vulnerability into a statistical indistinguishability problem gives a solid mathematical foundation for understanding the fundamental limits of guardrails.
  • The experimental design is careful and controls confounding variables well. In the Locality Sweep, the authors use strict length-matching and carrier control to rule out shortcut learning, such as relying on trajectory length or specific formatting. This fine-grained control of experimental variables is reasonable.
Critical weaknesses

Despite the strong problem formulation and empirical analysis, the paper has several notable limitations in theory and experiments:

  1. Theory-empirics gap. Proposition 1 bounds detection using TV distance, but experiments only measure AUROC. As admitted in Appendix B.1, an AUROC near 0.5 does not prove small TV distance. The experiments show detectors fail, but do not confirm the epsilon-local indistinguishability assumption.
  2. Lack of stateful monitor analysis. Proposition 1 only applies to memoryless monitors. Modern guardrails often access full interaction history. The paper lacks theoretical bounds or empirical evaluations (e.g., LLM-as-a-Judge) for history-aware monitors without codec knowledge.
  3. Limited constructive defense. The finding that providing the codec enables perfect blocking is nearly trivial. The paper does not propose unsupervised codec discovery or generalizable defenses for unknown encodings. It mainly shows what fails, lacking practical guidance for zero-day scenarios.
  4. Narrow evaluation scope. Experiments focus on structured code payloads with small fragment counts (K=3). Code-structure monitors fail to transfer to natural-language attacks. The study lacks exploration of non-code compositional attacks and long-range dependencies across multiple sessions.
Questions for authors
  1. Proposition 1 uses TV distance to bound local detection. However, the experiments only report AUROC. Did you try to estimate the real TV distance between attack and benign fragments in the feature space?
  2. The paper concludes that full-trace methods fail to block attacks without the codec. What exact full-trace methods did you test? Did you test a long-context LLM-as-a-Judge that reads the entire agent trace to find hidden payloads? If a long-context LLM can detect the attack without knowing the codec, the main conclusion of the paper would change.
  3. In Table 5, the "No-codec LLM search" uses DeepSeek-V4-Flash to guess the reconstruction. Did you try giving the LLM more context, such as the system prompts or the agent's final goal, to help it understand the hidden payload?
2 comments
AuthorsOpenReview ↗

We Ran Your Test: An LLM Judge Without the Codec Is at Chance Even With the Full Trace (0.49), Reaches 0.995 Once Agents Assemble the Action, and a Pre-Action Check Blocks 159/159; Exact TV = 0 Added

Thank you for the thoughtful review and for recognizing the importance of the problem and our theoretical analysis. Your questions led us to test stronger LLM judges and real agent assembly, and both strengthen our main findings. Your Q2 sets a clear test: "If a long-context LLM can detect the attack without knowing the codec, the main conclusion of the paper would change." We ran it, and the result supports the conclusion.

Q2. No: the LLM judge we tested cannot detect the encoded attack, even with the full agent history.

We gave Qwen2.5-32B-Instruct the defender's policy (reading a secret and sending it to a non-internal host is a violation), but not the codec. Benign traces are copies of the same agent runs that carry a harmless program through the same encoder. Thresholds come from separate calibration groups.

What the judge readsAUROC
Plaintext fragments (validity check)0.95
Encoded fragments (50 random splits)0.51
Encoded fragments, 14B judge0.56
The action assembled by agents (W3)0.995

On encoded fragments, detection equals the false-positive rate (3.6% vs 4.9%). The judge works when the policy-relevant content is readable and fails when it is encoded. This is the paper's conclusion that blocking needs the decoding view (L470–474).

--

Q3. More context does not help; reading the assembled action does.

To test whether more context helps beyond Table 5's DeepSeek search, we gave our new policy-aware judge the full trace: a pipeline description, every agent's task prompt, and every tool output. On encoded traces it stays at chance (AUROC 0.49). On plaintext traces, the extra context makes it flag 98% of benign traces (held-out AUROC 0.77, against 0.95 with the fragments alone).

--

W3. The constructive answer: check the action, not the messages.

In our threat model, a final component "combines and decodes their contents before execution" (L62–63). In our new agent workflow, the coordinator does this before calling the action tool, so a check on the final action needs no codec. Three Qwen2.5-32B workers pass encoded sections to a coordinator, which receives them shuffled, decodes and joins them, and calls a mock action tool with an inert SEND <variable> TO <host> line. We ran 160 attacks and 480 benign cases (the other three combinations).

AttackBenign
Agents assembled the exact action159/160469/480
Policy check on the tool argument blocks, among assembled159/1590/469

Counting assembly errors, 5/479 submitted benign actions were blocked (1.0%); each was a mis-assembled action that violated the policy. The same judge scores 0.47–0.49 on the encoded hand-offs and 0.995 on the tool argument. This gives a concrete answer to the practical challenge stated in L473–474. We will add two guidelines to the Discussion: place the check at the consequential tool call, where the workflow has already decoded the action; and check the action against a policy rather than scoring whether text looks like an attack.

--

W1/Q1. The paper already turns AUROC into a lower bound on TV; we now add an exact TV = 0 case.

As shown in App. B.1 (L711–712), AUROC can give a lower bound on TV, but an AUROC near 0.5 cannot prove small TV. The single-fragment n-gram classifier at 1.000 (L248) shows that those fragments are distinguishable. Because small TV cannot be shown this way, Definition 2 is stated as an assumption (L166–167). We therefore added a construction where it holds exactly: with additive secret sharing, every fragment is uniformly random and local TV = 0. Across six codecs and five seeds, every local and joint learner scores 0.46–0.54, while decoding scores 1.000.

--

W2. The history-aware monitors we tested also detect only at their false-positive rate without the codec.

The bound covers single-fragment monitors, as the paper states: "The result does not constrain a monitor that jointly observes several fragments or retains their history" (L174–175). The XOR example shows that no such bound holds for joint views (L180–184). We re-ran the history-aware methods as runtime gates (50 random splits; 91 attacks, 1,200 benign):

MethodAUROCDetectionFPR
Code-density rule, full trace0.604.5%4.9%
Common-decoder ensemble0.503.2%5.2%
Shipped detector, full trace0.495.3%5.7%
LLM judge, full trace + task prompts0.491.5%4.8%
Decode + policy gate1.00100%0%

--

W4. We already test four fragment counts and natural-language transfer.

The paper includes a K sweep over {2, 3, 5, 10} (Table 7, L402–418) and a natural-language transfer test (L398–400, Table 19). The new rollout adds a non-code action assembled by agents. Assembly across sessions is outside the paper's scope, as stated (L199–200).

We have uploaded the revision, which adds the long-context LLM judge results (Section 5; App. F.4), the exact TV = 0 construction (App. B.2), and the practical guidance from the four-agent workflow in Section 5 and the Discussion.

Thank you again for your careful review. We look forward to your further feedback or any additional questions you may have.

Reviewer_rgYvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the limitations of runtime monitors against distributed backdoors in multi-agent LLM systems. In these attacks, adversaries distribute a malicious payload across multiple agent messages, such that the harmful behavior is triggered only when the fragments are combined. The authors examine how local detection changes as observable attack cues are removed, evaluate a supervised detector using multiple fragments, and compare runtime blocking with and without knowledge of the attack encoding. The paper further presents a total-variation bound on the detection ability of monitors observing individual fragments.

Critical strengths
  1. The paper evaluates several detection views, encoding strategies, and model backbones. The experiments show that detection performance can change substantially with the available features and that a joint detector may fail when transferred to an unfamiliar encoding.
Critical weaknesses
  1. Limited novelty. The problem of distributed backdoor attacks has been explored in prior work. The proposed local detection bound is also a standard consequence of total-variation distance rather than a new theoretical result specific to multi-agent systems. While the experimental analysis is useful, the paper does not clearly show distinct methodological contribution beyond evaluating existing monitoring approaches under different encoding conditions.

  2. The codec-aware gate blocks all evaluated attacks, but it assumes prior knowledge of the encoding family and decoding rules, unlike the other evaluated methods. While this provides a useful reference point, it does not show an effective defense against adaptive attackers who can modify their encoding strategies. Furthermore, the supervised joint detector is primarily evaluated as a classification model rather than integrated into a practical runtime defense. It is better to provide stronger comparisons with stateful monitors, history-aware detectors, and practical assembly-time inspection methods operating under comparable information and training assumptions.

Questions for authors
  1. What is the main contribution beyond existing work on distributed backdoors, particularly Collaborative Shadows and DemonAgent?

  2. Could the authors provide stronger codec-blind baselines comparisons such as supervised classifiers using character-level features from concatenated traces, cross-fragment entropy or compressibility analysis, or LLM-based judges tasked with determining whether the combined fragments encode executable code?

3 comments
AuthorsOpenReview ↗

All Requested Baselines Tested; Agents Assemble the Attack, a Codec-Blind Check Blocks 159/159, and We Explain Why Monitoring Succeeds or Fails

Thank you for the thoughtful review. We especially appreciate your questions about novelty and stronger baselines. We run a few new experiments, and we will make our contributions clearer in the revised paper.

Q2. Yes—all three requested baselines are tested; supervised detection drops from 87% to 31% on unseen names.

All rows use format-matched benign traffic: copies of the same agent runs that carry a harmless program through the same encoder. Thresholds come from separate calibration groups at a nominal 5% FPR.

Baseline (no codec)AUROCDetectionFPR
Supervised char 1–5-gram on full traces, new names0.8631%4.5%
Same, recurring names0.9787%6.3%
Compressibility (zlib)0.383%3.8%
Pooled token entropy0.6617%6.9%
Cross-fragment JS divergence0.567%8.0%
LLM judge (Qwen2.5-32B, given the policy), encoded fragments0.513.6%4.9%
Same judge, same fragments in plaintext0.95

The supervised detector is the strongest, but its detection drops from 87% to 31% when the names are new, the pattern of Section 4 (L274–275). The unsupervised baselines are much weaker. The LLM judge applies the policy well on plaintext (0.95) and drops to chance once the same fragments are encoded (0.51).

--

W1/Q1. What is new: prior work shows the attack works; we show why monitoring fails, when it succeeds, and what information separates the two.

Collaborative Shadows introduces distributed backdoors and notes that defenses "have not been extensively explored". DemonAgent shows that encryption plus decomposition evades a safety audit in a single-agent setting. They focuse on demonstrating the attacks. Our study systematically compares what different monitors can see, when they fail, and what information enables blocking. Our question is the defender's (L428–429). We agree the TV inequality is standard. Our claim is about when it applies (L166–167), and the XOR example shows it cannot cover joint views (L180–184).

Our method is controlled characterization: we remove observable cues step by step (L234–237), hold out encodings (L281–282), match benign traffic to the attack format, and give the same judge different stages of the workflow. Three findings, each now with new evidence:

  1. A harmless fragment can still be detectable. A single-fragment n-gram classifier reaches 1.000 (L247–250). We now also build the opposite case: with additive secret sharing, local TV = 0, every local and joint learner scores 0.46–0.54 across six codecs, and decoding scores 1.000.
  2. Related-codec transfer survives vocabulary controls. Held-out AUROC stays close to in-family (0.63 vs 0.64). On the other codecs, the detector is near chance even in-family.
  3. What a monitor reads decides whether it can block. The same LLM judge with the same policy scores 0.47 on encoded agent messages and 0.995 on the action the agents assemble (W2).

Your Strength 1 notes the start of this picture: detection "can change substantially with the available features".

--

W2. Agents assemble 159/160 attacks, and a codec-blind pre-action check blocks all 159.

Agent rollout. Three Qwen2.5-32B workers pass encoded sections to a coordinator. The coordinator receives them shuffled, decodes and joins them, and calls a mock action tool with an inert SEND <variable> TO <host> line. We ran 160 attacks and 480 benign cases (the other three combinations).

AttackBenign
Agents assembled the exact action159/160469/480
Policy check on the tool argument blocks, among assembled159/1590/469

Counting assembly errors, 5/479 submitted benign actions were blocked (1.0%); each was a mis-assembled action that violated the policy. The same LLM judge scores 0.47–0.49 on the encoded hand-offs and 0.995 on the tool argument.

The check reads only the tool argument. Once the workflow exposes the decoded action, the check needs no codec: assembling the action and enforcing the policy rely on different parts of the system. The codec-aware gate remains what the paper calls it, a reference (L153).

Attack variations already tested. Wrappers, import aliasing, dead code and string construction leave the blocking pattern unchanged (App. H, L1129–1130). Reordering and decoys "do not defeat the gate" (L1188–1194).

History-aware methods as runtime gates. The full-trace methods of Table 5 see every fragment (L364–371). With separate benign calibration (50 random splits; 91 attacks, 1,200 benign):

MethodAUROCDetectionFPR
Code-density rule0.604.5%4.9%
Common-decoder ensemble0.503.2%5.2%
Shipped detector, full trace0.495.3%5.7%
Decode + policy gate1.00100%0%

Without the codec, each method detects at about its own false-positive rate.

We have uploaded the revision, which adds the requested baselines (App. F.4, Table 17), the four-agent workflow with a codec-free check on the final tool call (Section 5, Table 6), and a clearer statement of our contributions in Section 1.

Thank you again for your careful review. We look forward to your further feedback or any additional questions you may have.

Dear Reviewer rgYv,

Thank you very much for your time and thoughtful feedback on our submission. We completely understand that you may have a busy schedule, so we just wanted to gently follow up as the discussion period draws to a close. Please let us know if our rebuttal has addressed your concerns or if there are any remaining questions we can help clarify.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Thank you! New Results Address All Five Points: TV=0, Transfer After Vocabulary Controls, Joint View Helps, Agents Assemble 159/160, Pre-Action Check Blocks 159/159, Held-Out FPR Near 5%

Thank you for the careful review. We ran new controls and a four-agent test. The same judge scores 0.47 on encoded messages and 0.995 on the assembled action.

W1. We now give an exact TV = 0 example. The paper does not claim a fundamental limit for local monitors. It separates fragments that are harmless on their own from fragments that look like benign traffic (L115–117), and states that our original experiments do not prove small TV (L196–197). Our own n-gram detector scores 1.000 on single fragments (L247–250). We now add additive secret sharing: attack and benign fragments have identical distributions, so local TV = 0. Across six codecs and five seeds, local and joint AUROC is 0.46–0.54; decoded inspection scores 1.000.

 

W2. Yes—related-codec transfer still holds after removing vocabulary shortcuts, and agents can assemble the fragments themselve

Vocabulary. Appendix G's description is broader than the generator: identifiers and paths were shared across classes; only environment-variable names and hosts differed. We reran with shared name/host pools, unseen test values, three templates, equal padding, and an attack label requiring a secret sent externally. In-family AUROC falls from 0.98 to 0.64, but related-codec transfer still holds (held-out 0.63 vs. in-family 0.64). On the other codecs, the detector is near chance even in-family (0.50–0.56). So more fragments still do not guarantee detection across encodings. We will correct Appendix G and report both setups.

Assembly and decoder. The harness tests workflow blocking (L320–321); the codec-aware gate is a reference (L153). In our new test, agents themselves assemble the action, and a check without codec knowledge blocks every correctly assembled attack (Q2).

 

Q1. Yes—the same learners show a gain from seeing all fragments. We used the same LR/RF learners, matched training budgets, episode-level scores, unseen names/hosts, and the W2 setup. The paper states that its original comparison did not isolate this gain (L308–309; App. G L1112–1113).

InputAUROC
One fragment0.58
All fragments0.66
All fragments + cross-fragment features0.64

All fragments perform better in 14/15 held-out seed–codec cases. Cross-fragment features do not help further. With local TV = 0 (W1), both views are at chance. A wider view helps, but is not enough (L77–78). We will clarify the different learner used for "local pooled" in Table 16.

 

Q2. Yes—agents assemble 159/160 attacks; the pre-action check blocks 159/159 assembled attacks. Three Qwen2.5-32B workers pass encoded sections to a coordinator, which decodes and joins shuffled hand-offs, then submits an inert SEND <variable> TO <host> line to a mock tool. Nothing runs. We tested 160 attacks (secret sent externally) and 480 benign cases (the other three combinations), with matched formats and names made from the same letters.

ResultAttackBenign
Exact action assembled159/160469/480
Pre-action check blocks, given correct assembly159/1590/469

Including assembly errors, 5/479 benign submissions (1.0%) are blocked because the submitted actions violate the policy. The same LLM judge, with the same policy and no codec, scores AUROC 0.49 on one encoded hand-off, 0.47 on all three, and 0.995 on the action-tool argument.

At the tool call, the check needs no codec, section order, or agent provenance. Upstream, the coordinator uses a decoder and section indices (L62–63). Appendix F will clarify: all monitors see indexed tool outputs; only the reference gate also gets the codebook and policy; none uses provenance.

 

Q3. Yes—separate calibration gives FPR near 5%; the policy gate blocks 91/91. We used format-matched benign programs in saved rollouts: 91 attacks (546 variants), 1,200 benign cases, four models, and separate calibration/test groups. Results are means over 50 random group splits.

MethodAUROCDetection (%)FPR (%)
Per-step statistical0.463.7 (1.1–8.0)4.7 (0.9–9.0)
Code-density0.604.5 (1.1–8.8)4.9 (0.5–10.3)
Common-decoder ensemble0.503.2 (1.3–4.8)5.2 (2.9–7.8)
Shipped detector, per step0.494.7 (1.2–11.0)5.9 (1.4–12.5)
Shipped detector, full trace0.495.3 (1.9–10.1)5.7 (1.7–11.6)
Decode + policy1.001000

For the codec-blind methods shown, detection stays near each method's FPR. The ranges in parentheses are 2.5–97.5% across random splits, not confidence intervals. The gate blocks 91/91 attacks, with 0/600 false positives (95% CIs: [96.0%, 100%] and [0%, 0.6%]). Table 6 tests benign code using the payload's libraries (L385–388): code-structure AUROC is 0.090; the gate scores 1.000. Table 16 FPR changes in 3.3-point steps (30 test groups; L283–284); the controlled rerun has 3× as many cases per group.

We have uploaded the revision: the agent-assembled rollout is in Section 5 (Table 6; App. F.5), the exact TV = 0 construction in App. B.2, the vocabulary-controlled transfer and same-learner comparison in Section 4 and App. G.1, the separate-calibration results with split ranges in App. F.4, and App. G is corrected.

Thank you again for your careful review. We look forward to your further feedback or any additional questions you may have.

Dear Reviewer yekn,

Thank you very much for your time and thoughtful feedback on our submission. We completely understand that you may have a busy schedule, so we just wanted to gently follow up as the discussion period draws to a close. Please let us know if our rebuttal has addressed your concerns or if there are any remaining questions we can help clarify.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.