- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies a problem in multi-agent LLM systems, which is that an attacker can split a harmful program into small pieces and put them in different agent messages. In this scenario, each piece looks safe on its own, but becomes dangerous after assembly. The authors give a formal definition of "local benignness" and prove a TV bound showing that a single-fragment monitor cannot do much when attack and benign distributions are close. They run controlled experiments to remove cues step by step and show local detectors get weaker. A joint detector works well across similar encodings but fails on different ones. As for blocking, without knowing the encoding, existing methods stop at most 10 out of 50 attacks. With the encoding given, all 50 are blocked. The experiments are careful and the problem is important. However, the TV bound is not directly verified in experiments. The effective defense needs prior knowledge of the codec, which is not realistic for zero-day attacks. The paper provides a deep analysis of why current guardrails fail and sets a theoretical limit for local monitoring. However, it does not propose a new defense for unknown encodings, leaving a big challenge for future work.
Critical strengths
- The problem is important and timely. With the growing use of multi-agent frameworks such as AutoGen and CrewAI, distributed prompt injection and backdoor assembly have become a major blind spot in AI safety. This paper correctly identifies the compositional vulnerability where every local check passes but the global result is harmful. It provides a valuable warning for the AI Safety and Alignment community.
- The theoretical analysis is rigorous. The authors do not stop at showing attacks empirically. They introduce the TV distance from information theory to bound the theoretical limit of a single-step monitor. This idea of turning a security vulnerability into a statistical indistinguishability problem gives a solid mathematical foundation for understanding the fundamental limits of guardrails.
- The experimental design is careful and controls confounding variables well. In the Locality Sweep, the authors use strict length-matching and carrier control to rule out shortcut learning, such as relying on trajectory length or specific formatting. This fine-grained control of experimental variables is reasonable.
Critical weaknesses
Despite the strong problem formulation and empirical analysis, the paper has several notable limitations in theory and experiments:
- Theory-empirics gap. Proposition 1 bounds detection using TV distance, but experiments only measure AUROC. As admitted in Appendix B.1, an AUROC near 0.5 does not prove small TV distance. The experiments show detectors fail, but do not confirm the epsilon-local indistinguishability assumption.
- Lack of stateful monitor analysis. Proposition 1 only applies to memoryless monitors. Modern guardrails often access full interaction history. The paper lacks theoretical bounds or empirical evaluations (e.g., LLM-as-a-Judge) for history-aware monitors without codec knowledge.
- Limited constructive defense. The finding that providing the codec enables perfect blocking is nearly trivial. The paper does not propose unsupervised codec discovery or generalizable defenses for unknown encodings. It mainly shows what fails, lacking practical guidance for zero-day scenarios.
- Narrow evaluation scope. Experiments focus on structured code payloads with small fragment counts (K=3). Code-structure monitors fail to transfer to natural-language attacks. The study lacks exploration of non-code compositional attacks and long-range dependencies across multiple sessions.
Questions for authors
- Proposition 1 uses TV distance to bound local detection. However, the experiments only report AUROC. Did you try to estimate the real TV distance between attack and benign fragments in the feature space?
- The paper concludes that full-trace methods fail to block attacks without the codec. What exact full-trace methods did you test? Did you test a long-context LLM-as-a-Judge that reads the entire agent trace to find hidden payloads? If a long-context LLM can detect the attack without knowing the codec, the main conclusion of the paper would change.
- In Table 5, the "No-codec LLM search" uses DeepSeek-V4-Flash to guess the reconstruction. Did you try giving the LLM more context, such as the system prompts or the agent's final goal, to help it understand the hidden payload?
2 comments
We have uploaded the revision, which adds the long-context LLM judge results (Section 5; App. F.4), the exact TV = 0 construction (App. B.2), and the practical guidance from the four-agent workflow in Section 5 and the Discussion.
Thank you again for your careful review. We look forward to your further feedback or any additional questions you may have.
We Ran Your Test: An LLM Judge Without the Codec Is at Chance Even With the Full Trace (0.49), Reaches 0.995 Once Agents Assemble the Action, and a Pre-Action Check Blocks 159/159; Exact TV = 0 Added
Thank you for the thoughtful review and for recognizing the importance of the problem and our theoretical analysis. Your questions led us to test stronger LLM judges and real agent assembly, and both strengthen our main findings. Your Q2 sets a clear test: "If a long-context LLM can detect the attack without knowing the codec, the main conclusion of the paper would change." We ran it, and the result supports the conclusion.
Q2. No: the LLM judge we tested cannot detect the encoded attack, even with the full agent history.
We gave Qwen2.5-32B-Instruct the defender's policy (reading a secret and sending it to a non-internal host is a violation), but not the codec. Benign traces are copies of the same agent runs that carry a harmless program through the same encoder. Thresholds come from separate calibration groups.
On encoded fragments, detection equals the false-positive rate (3.6% vs 4.9%). The judge works when the policy-relevant content is readable and fails when it is encoded. This is the paper's conclusion that blocking needs the decoding view (L470–474).
--
Q3. More context does not help; reading the assembled action does.
To test whether more context helps beyond Table 5's DeepSeek search, we gave our new policy-aware judge the full trace: a pipeline description, every agent's task prompt, and every tool output. On encoded traces it stays at chance (AUROC 0.49). On plaintext traces, the extra context makes it flag 98% of benign traces (held-out AUROC 0.77, against 0.95 with the fragments alone).
--
W3. The constructive answer: check the action, not the messages.
In our threat model, a final component "combines and decodes their contents before execution" (L62–63). In our new agent workflow, the coordinator does this before calling the action tool, so a check on the final action needs no codec. Three Qwen2.5-32B workers pass encoded sections to a coordinator, which receives them shuffled, decodes and joins them, and calls a mock action tool with an inert
SEND <variable> TO <host>line. We ran 160 attacks and 480 benign cases (the other three combinations).Counting assembly errors, 5/479 submitted benign actions were blocked (1.0%); each was a mis-assembled action that violated the policy. The same judge scores 0.47–0.49 on the encoded hand-offs and 0.995 on the tool argument. This gives a concrete answer to the practical challenge stated in L473–474. We will add two guidelines to the Discussion: place the check at the consequential tool call, where the workflow has already decoded the action; and check the action against a policy rather than scoring whether text looks like an attack.
--
W1/Q1. The paper already turns AUROC into a lower bound on TV; we now add an exact TV = 0 case.
As shown in App. B.1 (L711–712), AUROC can give a lower bound on TV, but an AUROC near 0.5 cannot prove small TV. The single-fragment n-gram classifier at 1.000 (L248) shows that those fragments are distinguishable. Because small TV cannot be shown this way, Definition 2 is stated as an assumption (L166–167). We therefore added a construction where it holds exactly: with additive secret sharing, every fragment is uniformly random and local TV = 0. Across six codecs and five seeds, every local and joint learner scores 0.46–0.54, while decoding scores 1.000.
--
W2. The history-aware monitors we tested also detect only at their false-positive rate without the codec.
The bound covers single-fragment monitors, as the paper states: "The result does not constrain a monitor that jointly observes several fragments or retains their history" (L174–175). The XOR example shows that no such bound holds for joint views (L180–184). We re-ran the history-aware methods as runtime gates (50 random splits; 91 attacks, 1,200 benign):
--
W4. We already test four fragment counts and natural-language transfer.
The paper includes a K sweep over {2, 3, 5, 10} (Table 7, L402–418) and a natural-language transfer test (L398–400, Table 19). The new rollout adds a non-code action assembled by agents. Assembly across sessions is outside the paper's scope, as stated (L199–200).