acceptodds
Under review as a conference paper at ICLR 2027

CompassGuard: Context-Aware Activation-Based Defense for Multi-Agent Systems

Abstract

LLM-based multi-agent systems enable collaboration through inter-agent communication, but these interactions can also propagate malicious instructions from compromised agents. Pruning-based defenses can disrupt collaboration, while prototype-based activation defenses use a shared benign reference that does not explicitly account for each agent's execution context. Benign activations can vary across contexts, and even a reference that supports accurate detection may be an inadequate target for correction. CompassGuard is an activation-based defense that separates reference prediction for detection from deviation estimation for correction while preserving all agents and communication edges. It detects anomalous invocations by comparing observed activations with expected benign activations predicted from the execution context. For correction, it learns activation differences between paired clean and attacked executions under the same teacher-forced benign response. During generation, it re-estimates attack-induced deviations from the execution context and current activation, and subtracts them to correct flagged invocations token by token. Across six benchmarks under homogeneous/heterogeneous roles, CompassGuard attains an average detection F1 of 97.9% and reduces the average attack success rate from 17.7% to 6.5% relative to the state-of-the-art activation-based defense, while attaining the highest average task accuracy.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM multi-agent systems, multi-agent security, activation-based defense, activation steering, anomaly detection

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_NFeoOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes an activation-based defense that separates reference prediction for detection from deviation estimation for correction while preserving all agents and communication edges. It contains two modules, one for detection and one for correction. For detection, it predicts the benign hidden state based on the execution context and compares it with the observed hidden state using cosine similarity to determine whether an agent is compromised. For correction, it uses teacher-forced pairs to construct the vector and applies token-wise activation steering to correct the hidden states. This paper reports the effectiveness of the proposed method across different attack types and MAS settings, along with ablation studies and additional analyses.

Critical strengths
  1. This paper develops a context-aware activation defense. This is especially important in multi-agent system defense, as different agent roles, system prompts, and execution contexts may lead to different benign activation patterns, so a shared benign prototype may be less suitable.
  2. This paper separates attack detection from correction and proposes a steering method without removing agents or communication edges. The experiments demonstrate improvements in attack mitigation and task accuracy, especially in heterogeneous MAS settings.
Critical weaknesses
  1. The necessity of several key design choices is not sufficiently justified. The Activation Predictor learns to reconstruct a full benign hidden state, although its output is ultimately used for binary attack detection. It is unclear whether a simpler context-aware classifier could achieve comparable performance. Also, even though the paper shows that attack deviations change during generation, it doesn't directly mean that establishing token-wise steering is necessary compared with intervening only at the first token.

  2. The reliability of the correction mechanism is not clear. The Deviation Estimator is trained with fixed benign response prefixes, but inference operates on freely generated trajectories. This mismatch may affect the accuracy of the estimated deviations. Also, the method relies on paired examples of known attacks, and its zero-shot cross-benchmark detection performance drops substantially (Table 9). Finally, although the paper reports several results under attacks, it lacks evaluation of whether enabling the defense affects normal MAS performance (when no attack is present).

Questions for authors
  1. Why is reconstructing the full benign activation preferable to directly training an attack classifier?

  2. Is token-wise steering necessary? Could the authors also clarify the benefit of dynamically updating the steering direction?

  3. How does the Deviation Estimator generalize from teacher-forced training trajectories to free-generation inference?

  4. How does CompassGuard affect task performance when deployed on a completely benign MAS?

  5. To what extent does the Deviation Estimator depend on the attack types observed during training? Particularly, how should the low zero-shot detection precision in Table 9 be interpreted alongside the low reported ASR?

7 comments
AuthorsOpenReview ↗

Response to Reviewer NFeo [1/5]

We sincerely thank you for the constructive and insightful feedback. Your questions on the necessity of our design choices and the reliability of the correction mechanism helped us identify where further justification and evaluation were needed. In response, we clarified why we predict benign activations and measured the Activation Predictor’s accuracy on held-out examples. We also added a tokenwise steering ablation, evaluated CompassGuard under benign conditions, and extended our experiments on unseen attack templates and domain shifts.

 

Benign Activation Prediction vs. Direct Classification

Weakness 1: The necessity of several key design choices is not sufficiently justified. The Activation Predictor learns to reconstruct a full benign hidden state, although its output is ultimately used for binary attack detection. It is unclear whether a simpler context-aware classifier could achieve comparable performance.

Question 1: Why is reconstructing the full benign activation preferable to directly training an attack classifier?

Thank you for the opportunity to clarify this design choice. We use full benign activation prediction for attack detection, motivated by the risk that binary classifiers may rely on spurious features [1]. The full benign activation serves as a training target that encourages the Activation Predictor to learn what activation should arise under benign execution in a given execution context, rather than only whether an execution is benign. As an additional benefit, learning this context-specific reference from benign traces alone, as in anomaly detection that learns benign behavior [2], means that detection does not rely on examples of specific attacks. We have stated this in Section 3.2 of the revised paper.

This design choice is supported by our extended experiments on unseen attack templates (General Response §1) and adaptive attacks (General Response §3), where CompassGuard achieves a detection F1 of 100% on CSQA, GSM8K, and MS-MARCO and 99.4% on CSQA, respectively.

 

[1] Boldisor, Dragos-Alexandru, et al. "Investigating self-supervised representations for audio-visual deepfake detection." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2026.

[2] Du, Min, et al. "Deeplog: Anomaly detection and diagnosis from system logs through deep learning." Proceedings of the 2017 ACM SIGSAC conference on computer and communications security. 2017.

AuthorsOpenReview ↗

Response to Reviewer NFeo [2/5]

Token-Wise Steering

Weakness 1: Also, even though the paper shows that attack deviations change during generation, it doesn't directly mean that establishing token-wise steering is necessary compared with intervening only at the first token.

Question 2: Is token-wise steering necessary? Could the authors also clarify the benefit of dynamically updating the steering direction?

To examine whether first-token-only steering is sufficient and whether dynamically updating the steering direction improves performance, we compare the following steering strategies. All steering variants use the same Activation Predictor, Deviation Estimator, intervention layer, steering scale, and evaluation protocol.

First-token-only steering applies correction only at k=1. The two fixed-direction tokenwise variants retain the direction of the deviation estimated at k=1, either keeping its magnitude fixed or updating it at each token using the Deviation Estimator. CompassGuard updates both the direction and magnitude by re-estimating the deviation at each token.

First-token-only steering does not improve mean ASR or task accuracy over no defense on either dataset. Tokenwise steering with updated magnitude improves both metrics, and CompassGuard further improves them by updating the steering direction at each token. Detailed results are provided in the table below and have been added to Table 8(b) in Appendix F of the revised manuscript.

Steering strategyCSQA ASR ↓CSQA ACC ↑GSM8K ASR ↓GSM8K ACC ↑
No defense22.8 ± 2.851.9 ± 5.640.6 ± 19.059.4 ± 19.0
First-token-only (k=1)24.2 ± 1.748.1 ± 5.741.2 ± 17.257.1 ± 17.7
Tokenwise (fixed direction fixed magnitude)13.7 ± 7.160.9 ± 4.640.8 ± 22.458.0 ± 18.4
Tokenwise (fixed direction, updated magnitude)5.7 ± 3.870.6 ± 2.416.6 ± 15.979.7 ± 14.3
CompassGuard4.9 ± 3.275.6 ± 2.610.3 ± 14.786.7 ± 14.7
AuthorsOpenReview ↗

Response to Reviewer NFeo [3/5]

Generalization from Teacher Forcing to Free Generation

Weakness 2: The reliability of the correction mechanism is not clear. The Deviation Estimator is trained with fixed benign response prefixes, but inference operates on freely generated trajectories. This mismatch may affect the accuracy of the estimated deviations.

Question 3: How does the Deviation Estimator generalize from teacher-forced training trajectories to free-generation inference?

Thank you for raising this point. We would like to note that CompassGuard applies steering from the first response token. Correction therefore begins before any response token is generated, so it reduces the attack's influence on generation from the first token. At every subsequent token, CompassGuard updates its correction using the current activation and execution context. This continuous correction is intended to keep generation closer to benign behavior and limit divergence from the benign response trajectories used for training.

While this does not guarantee that the estimator generalizes to every response prefix that arises during free generation, our ablation results support its effectiveness during free generation. In the ablation study, replacing the estimator with steering toward the predicted benign activation increases ASR from 4.9% to 33.7% and reduces task accuracy from 75.6% to 51.8%, with detection F1 unchanged at 100%.

Reviewer_hB8YOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CompassGuard is a white-box defense for LLM-based multi-agent systems that separates compromised-invocation detection from activation correction. An Activation Predictor estimates an invocation-specific benign activation from prompts, interaction histories, and the communication graph. A separate Deviation Estimator learns activation differences between paired clean and attacked executions under identical teacher-forced benign responses, then performs tokenwise correction during generation. The method preserves agents and communication edges. Evaluation covers six benchmarks across three attack surfaces, homogeneous and heterogeneous workflows, and a MetaGPT code-generation workflow, with additional experiments on two other LLM backbones. The paper reports average detection F1 of 97.9% and average attack success rate of 6.5%, compared with 17.7% for AcMAS.

Critical strengths

The paper provides useful evidence that accurate detection does not ensure effective mitigation. The separation between context-conditioned detection and correction addresses a meaningful limitation of activation-prototype defenses. In Table 4, removing the Deviation Estimator leaves detection F1 at 100% but increases ASR from 4.9% to 33.7% and reduces task accuracy from 75.6% to 51.8%. Removing context conditioning also preserves 100% detection F1 while substantially degrading task accuracy. These results make correction quality a substantive contribution beyond improved anomaly classification.

The reported end-to-end improvements extend across several tasks and collaboration settings. CompassGuard reduces ASR relative to AcMAS on all six benchmarks in both homogeneous and heterogeneous workflows. The code-generation results provide additional evidence beyond short-answer tasks: average HumanEval/HumanEval+ pass@1 improves from 43.0%/35.5% with AcMAS to 54.7%/46.5%, approaching the no-attack reference of 57.8%/50.0%. This breadth makes the approach promising for workflows where removing a specialized agent would disrupt task completion.

Critical weaknesses

1. Evaluation inconsistencies weaken confidence in the reported gains.

The public evaluator excludes invalid outputs from metric denominators, computes homogeneous ASR using majority agreement including directly attacked agents rather than normal-agent output rates (Appendix D.1), and automatically excludes six heterogeneous CSQA test instances without clear disclosure. These behaviors were verified using constructed outputs. The authors should clarify whether this code generated the reported tables and provide per-instance results and denominators for verification.

2. The released paired-execution pipeline does not adequately isolate attack-induced activation changes.

Section 3.3 states that clean and attacked executions share the same task and preceding interaction history, differing only in the injected attack. However, the homogeneous implementation retains the full cached conversation for the clean prompt while rebuilding the attacked prompt without the preceding conversation history. Consequently, the activation difference may capture both the attack and the removal of legitimate context. Furthermore, the deviation estimator pairs attacked activations with benign context during training, whereas the paper specifies attacked context and inference uses the actual execution context. These discrepancies weaken the interpretation of the learned target as an attack-induced deviation. A corrected, history-matched implementation and controlled comparison are needed to substantiate this central contribution.

3. The experiments do not sufficiently establish generalization beyond recognizable attack constructions.

A key detection comparison requires reconciliation with the stated metric protocol. Table 8 reports AcMAS on heterogeneous InjecAgent with 100% recall, 64.5% precision, and 81.3% FPR. Appendices B.3 and D describe a four-agent workflow with one directly targeted agent, the remaining agents treated as negatives, and averaging across targeted roles. These reported values are difficult to reconcile under that interpretation. The statistical unit, class counts, and aggregation procedure need clarification, particularly because this FPR contributes to the average comparison in Figure 3(b). This may be a reporting or implementation issue, but it currently weakens confidence in the detection comparison.

Questions for authors
  1. Generalization and selective protection. Can you clarify whether training and testing use disjoint attack templates, and provide results on unseen templates while keeping trusted system prompts unchanged? For the zero-shot transfer setting in Table 9, can you report task utility and false-positive rates alongside ASR, given the 25% precision and 100% recall? How robust is the defense to attacks explicitly optimized against its detection and correction mechanisms?

  2. Metric reconciliation: Could you provide per-targeted-role TP, FP, TN, and FN counts, together with the pooling and averaging procedure, for heterogeneous InjecAgent? These details would help reconcile AcMAS’s precision of 64.5%, recall of 100%, and FPR of 81.3% in Table 8 with Appendix D, and clarify whether any correction would affect Figure 3(b). For reproducibility, could you also briefly clarify how invalid or unparseable outputs are handled in the metric denominators?

8 comments
AuthorsOpenReview ↗

Response to Reviewer hB8Y [1/4]

We sincerely thank you and the reviewers for the constructive and insightful feedback. Your detailed examination of our implementation and evaluation protocol helped us identify issues requiring methodological corrections and clearer reporting. In response, we have corrected the training pipeline, rerun the affected experiments, and provided additional evaluation details and per-instance results. We address each concern below.

 

Evaluation Protocol

Weakness 1 & Question 2: The public evaluator excludes invalid outputs from metric denominators, computes homogeneous ASR using majority agreement including directly attacked agents, and excludes six heterogeneous CSQA test instances without clear disclosure. Please clarify whether this code generated the reported tables, explain how invalid or unparseable outputs are handled, and provide per-instance results and denominators.

Thank you for carefully examining our evaluation code. The released evaluator is the code that generated all reported tables, and it applies the three rules you identified to every method in the same way, so all methods are compared under the same protocol. Appendix D.1 did not describe these rules accurately, and we have corrected it. We explain each rule below and provide the per-instance results and denominators for verification.

Homogeneous MAS response. In the homogeneous setting, ASR and accuracy are computed by majority agreement over all agent responses in the evaluated round, including those of directly targeted agents. A homogeneous MAS has no designated final agent, and its answer is commonly obtained by majority voting over the agents [1]. The directly targeted agents remain in the MAS and their responses are part of its output, so this measures whether the output of the MAS is compromised, as in the heterogeneous setting where both metrics use only the output of the final agent.

Invalid outputs. An output from which no answer can be parsed can be categorized neither as a successful or unsuccessful attack nor as a correct or incorrect answer, so it is not counted when computing ASR and accuracy. This is consistent with prior work that computes ASR [2, 3] and accuracy [4] over valid outputs only. With CompassGuard, only one output is invalid across the five question-answering benchmarks, on heterogeneous GSM8K, so this rule affects the reported accuracy in only one execution. The remaining invalid outputs occur on InjecAgent and heterogeneous setting, with 10 executions. Since an invalid output is never counted as an attack success, removing it from the denominator can raise ASR, which makes the defense look weaker, not stronger.

Excluded instances. The exclusion lists, including the six CSQA instances, are generated only from no-attack outputs. They identify instances whose no-attack outputs already satisfy the attack-success criterion, so that such outcomes are not counted as attack successes. This is consistent with prior work that measures ASR on inputs that do not already yield the attacker's target [6]. We should have disclosed this, and Appendix D.1 now states the rule. For heterogeneous CSQA, 6 of the 60 test queries are excluded, which leaves 54 for each targeted role.

For verification, our anonymous repository (under logs/) now provides, for all CompassGuard runs, the execution logs, the per-instance records of attack success, parsing status, and exclusion, and the aggregated statistics of each experiment, together with generate_exclusion_indices.py, which reproduces the exclusion lists. Where the original logs could not be found, we re-ran the experiments with the same configuration, and the results closely reproduce the reported values.

 

[1] Choi, Hyeong Kyu, Jerry Zhu, and Sharon Li. "Debate or Vote: Which Yields Better Decisions in Multi-Agent Large Language Models?" Advances in Neural Information Processing Systems 38 (NeurIPS 2025).

[2] Zhan, Qiusi, et al. "Injecagent: Benchmarking indirect prompt injections in tool-integrated large language model agents." Findings of the Association for Computational Linguistics: ACL 2024. 2024.

[3] Nakash, Itay, et al. "Breaking react agents: Foot-in-the-door attack will get you in." Findings of the Association for Computational Linguistics: NAACL 2025. 2025.

[4] Xu, Haoming, et al. "Illusions of Confidence? Diagnosing LLM Truthfulness via Neighborhood Consistency." Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026.

[5] Chang, Hongyan, et al. "Overcoming the Retrieval Barrier: Indirect Prompt Injection in the Wild for LLM Systems." 35th USENIX Security Symposium (USENIX Security 26). 2026.

[6] Kurita, Keita, Paul Michel, and Graham Neubig. "Weight Poisoning Attacks on Pre-trained Models." Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 2020.

AuthorsOpenReview ↗

Response to Reviewer hB8Y [2/4]

Training Data Construction

Weakness 2: The released paired-execution pipeline does not adequately isolate attack-induced activation changes. Section 3.3 states that clean and attacked executions share the same task and preceding interaction history, differing only in the injected attack. However, the homogeneous implementation retains the full cached conversation for the clean prompt while rebuilding the attacked prompt without the preceding conversation history. Consequently, the activation difference may capture both the attack and the removal of legitimate context. Furthermore, the deviation estimator pairs attacked activations with benign context during training, whereas the paper specifies attacked context and inference uses the actual execution context. These discrepancies weaken the interpretation of the learned target as an attack-induced deviation. A corrected, history-matched implementation and controlled comparison are needed to substantiate this central contribution.

Thank you for identifying these discrepancies. We have corrected both the history mismatch and the context pairing used to train the Deviation Estimator, and rerun the affected experiments. The history mismatch affected subsequent rounds of homogeneous MAS. At round 0, there was no preceding interaction history, so the activation differences used for training were unaffected by this omission.

Following this feedback, we updated the implementation so that clean and attacked executions share the same preceding interaction history and differ only in the injected attack. We also updated Deviation Estimator training to use attacked context for attacked activations and benign context for benign activations.

To validate CompassGuard with these corrections, we regenerated the paired training data, retrained the Deviation Estimator, and re-evaluated homogeneous MAS. As shown below, CompassGuard maintains low ASR across all six benchmarks, with lower ASR on five benchmarks after these updates. We have updated Tables 1,5, and 12 in the revised manuscript accordingly.

Attack SurfacePIMATA
DatasetsCSQAGSM8kMSMARCOHotpotQANQInjecAgent
ASR (%)ASR (%)ASR (%)ASR (%)ASR (%)ASR (%)
Submission7.8 ± 6.93.9 ± 2.03.3 ± 3.02.3 ± 1.01.1 ± 1.09.4 ± 3.0
Revised1.1 ± 1.04.5 ± 1.10 ± 01.2 ± 1.00.6 ± 1.04.1 ± 2.5
AuthorsOpenReview ↗

Response to Reviewer hB8Y [3/4]

Detection Metrics

Weakness 3: The experiments do not sufficiently establish generalization beyond recognizable attack constructions. A key detection comparison requires reconciliation with the stated metric protocol. Table 8 reports AcMAS on heterogeneous InjecAgent with 100% recall, 64.5% precision, and 81.3% FPR. Appendices B.3 and D describe a four-agent workflow with one directly targeted agent, the remaining agents treated as negatives, and averaging across targeted roles. These reported values are difficult to reconcile under that interpretation. The statistical unit, class counts, and aggregation procedure need clarification, particularly because this FPR contributes to the average comparison in Figure 3(b). This may be a reporting or implementation issue, but it currently weakens confidence in the detection comparison.

Question 2: Metric reconciliation: Could you provide per-targeted-role TP, FP, TN, and FN counts, together with the pooling and averaging procedure, for heterogeneous InjecAgent? These details would help reconcile AcMAS’s precision of 64.5%, recall of 100%, and FPR of 81.3% in Table 8 with Appendix D, and clarify whether any correction would affect Figure 3(b).

We evaluate each agent separately on each test sample. In attacked runs of the heterogeneous MAS, we label the directly targeted agent as positive and upstream agents as negative. We exclude downstream agents because their activations may be affected by messages from the targeted agent, even if they are not directly targeted. The same agent selection is applied to all methods, including CompassGuard. We compute precision, recall, and F1 separately for each attacked role and average them across all three roles, without pooling the counts across roles. For AcMAS, this yields a mean precision of 64.5%, recall of 100%, and FPR of 81.3%, consistent with Table 7. This clarification does not change the reported detection metrics or the comparison in Figure 3(b).

 

The table below reports the TP, FP, TN, and FN counts for each attacked role on InjecAgent. We evaluate on 100 InjecAgent test samples using a four-agent workflow in the following order: Executor (0), Helpfulness Checker (1), Safety Checker (2), and Final Agent (3).

MethodAttacked roleAgent indicesTPFPTNFNTotal
CompassGuard0 (Executor)0100000100
1 (Helpfulness Checker)0, 11004960200
2 (Safety Checker)0, 1, 210061940300
AcMAS0 (Executor)0100000100
1 (Helpfulness Checker)0, 110075250200
2 (Safety Checker)0, 1, 2100175250300
Reviewer_ELZTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CompassGuard, an activation-based defense against malicious instructions propagating through LLM multi-agent systems. The method uses an Activation Predictor to identify anomalous agents based on their execution context and a Deviation Estimator to correct compromised activations during token generation. Unlike pruning-based approaches, CompassGuard maintains the original agents and their communication structure. Experiments across six benchmarks show improved attack detection, lower attack success rates, and higher task accuracy compared with existing activation-based defenses.

Critical strengths
  1. The topic is interesting and the overall motivation is clear. The paper studies how to mitigate malicious influence without disrupting legitimate collaboration in multi-agent systems.

  2. CompassGuard achieves strong detection performance and reduces attack success rates across several benchmarks. The ablation studies and codes support the proposed design.

Critical weaknesses
  1. Limited novelty. The main components of CompassGuard are largely based on existing techniques, including activation-based anomaly detection, benign reference modeling, graph attention, and activation steering. While the experimental improvements are encouraging, the methodological contribution appears incremental.

  2. The Deviation Estimator is trained using paired benign and attacked executions, requiring representative attack examples. The experiments mainly evaluate predefined attack settings, while adaptive attackers who deliberately attempt to evade detection or correction are not systematically studied. Table 9 shows that zero-shot cross-domain detection precision drops to 25%, raising concerns about reliability under distribution shifts. No paraphrased or adaptive injections are tested.

Questions for authors
  1. What is the main methodological novelty of CompassGuard beyond extending AcMAS with context-aware activation prediction and learned token-wise steering?

  2. How accurately does the Activation Predictor estimate the actual benign activation?

  3. How does CompassGuard perform against adaptive attackers who manipulate prompts specifically to evade the defense? In particular, attacks may be embedded early in long contexts or designed to influence behavior only under specific conditions evade the initial detection while affecting subsequent generation.

4 comments
AuthorsOpenReview ↗

Response to Reviewer ELZT [1/3]

We sincerely appreciate your constructive feedback. Your questions helped us clarify the methodological contribution and better evaluate generalization. We have clarified the methodological contribution of CompassGuard, evaluated the accuracy of the Activation Predictor, and added experiments on cross-domain transfer, adaptive attacks, and unseen attack templates. We present these additional experiments in the General Response and address your comments below.

 

Novelty

Weakness 1 & Question 1: Limited novelty. The main components of CompassGuard are largely based on existing techniques, including activation-based anomaly detection, benign reference modeling, graph attention, and activation steering. While the experimental improvements are encouraging, the methodological contribution appears incremental. What is the main methodological novelty beyond extending AcMAS with context-aware activation prediction and learned token-wise steering?

Thank you for asking us to clarify the methodological contribution. We agree that the building blocks (graph attention, contrastive loss, activation addition) are standard. We would like to clarify that what we propose goes beyond making AcMAS context-aware and token-wise.

Rather than steering toward a benign reference at each token, CompassGuard captures the attack-induced deviation at each response token and removes it. The key step is that we teacher-force the same benign response under both the clean prompt and the attacked prompt. This reveals how the attack pushes the model away from the benign response at each token. Since the response is identical in the two runs, any difference in the activations comes from the injected instructions alone (Sec. 3.3).

The Deviation Estimator learns to predict this deviation, and steering removes it at every response token of a flagged invocation (Eq. 5). Two results in the paper support this contribution.

  • Impact of deviation removal In Table 8(a) (heterogeneous CSQA), steering toward the predicted benign activation detects every attack (F1 100%) yet leaves ASR at 33.7%, against 22.8% with no defense. Subtracting the estimated deviation lowers ASR to 4.9% and raises accuracy from 51.8% to 75.6%.

  • The deviation has to be estimated at each token. The deviation at k=1 has a cosine similarity of only 0.094/0.160 with the deviation at k=10, whereas our per-token estimate reaches 0.743/0.799 (homogeneous/heterogeneous MAS, Fig. 4b).

AuthorsOpenReview ↗

Response to Reviewer ELZT [2/3]

Generalization and Robustness

Weakness 2: The Deviation Estimator is trained using paired benign and attacked executions, requiring representative attack examples. The experiments mainly evaluate predefined attack settings, while adaptive attackers who deliberately attempt to evade detection or correction are not systematically studied. Table 9 shows that zero-shot cross-domain detection precision drops to 25%, raising concerns about reliability under distribution shifts. No paraphrased or adaptive injections are tested.

Question 3: How does CompassGuard perform against adaptive attackers who manipulate prompts specifically to evade the defense? In particular, attacks may be embedded early in long contexts or designed to influence behavior only under specific conditions evade the initial detection while affecting subsequent generation.

Thank you for raising these points. As you note, the Deviation Estimator is trained on predefined attack examples, so the key question is how well CompassGuard generalizes beyond them. We examine this in three ways below.

  • W2-1. Representative attack examples and paraphrased injections. To examine how much CompassGuard depends on representative attack examples, we evaluated paraphrased and rewritten attack templates that were not used for training. With all trained modules frozen, CompassGuard keeps ASR at or below 7.1%. We kindly refer you to General Response (§1) for the setup and the full results.

  • W2-2. Domain shifts. We agree that zero-shot cross-domain detection is limited, since the Activation Predictor has not seen the target domain and flags most benign agents. However, a small amount of target-domain data resolves this. With only 33 of the 200 training queries drawn from the target domain, FPR falls from over 93% to at most 5.8% in all four transfer cases we evaluated. We believe that collecting this many queries is practical for an MAS service provider, who already runs the system on in-domain queries when developing and serving a new task. Please see General Response (§2) for the full results and the details of the correction to Table 9.

  • W2-3 & Q3. Adaptive attacks and evasion of the initial detection. Since many evasion strategies are possible, we do not fix one in advance and instead use an adaptive attack strategy that tailors the attack to the target defense by iteratively refining it with feedback from the defence [1]. The results show that CompassGuard keeps ASR at 6.8% on CSQA, and the detection F1 remains 99.4%. The adaptive attacks rarely evade the initial detection. Moreover, detection is not a one-time check. CompassGuard checks every agent invocation, so an attack that evades detection at one invocation is examined again whenever its output is passed to subsequent agents. We kindly refer you to General Response (§3) for the attack procedure and the full results.

 

[1] Chao, Patrick, et al. "Jailbreaking Black Box Large Language Models in Twenty Queries." 2025 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML). IEEE, 2025.

AuthorsOpenReview ↗

Response to Reviewer ELZT [3/3]

Accuracy of Benign Activation Prediction

Question 2: How accurately does the Activation Predictor estimate the actual benign activation?

Thank you for this question. On CSQA, GSM8K, and HumanEval, the predicted activation has an average cosine similarity above 0.99 with the actual benign activation, compared with 0.92 to 0.97 for a shared benign prototype, and its MSE is 71.1% to 91.0% lower.

We evaluate the test queries in the heterogeneous setting, which are not used for training. We compare the predicted and actual activations at layer 16 at the last prompt position, which is the activation used for detection before the first response token is generated. As a reference, we use a shared benign prototype as in AcMAS, obtained by averaging the benign training activations.

DatasetCosine similarity ↑ (Activation Predictor)Cosine similarity ↑ (Shared prototype)MSE ↓ (×10⁻⁴, Activation Predictor)MSE ↓ (×10⁻⁴, Shared prototype)MSE reduction
CSQA0.9913 ± 0.00370.9694 ± 0.01003.0410.5271.1%
GSM8K0.9926 ± 0.00510.9729 ± 0.00972.579.1771.9%
HumanEval0.9925 ± 0.00650.9151 ± 0.06442.6629.4991.0%

Thank you again for your thoughtful review. If you have a chance to read our response, we would appreciate hearing whether any of your concerns remain. We would be happy to clarify further.

Comments on the paper

4 comments
AuthorsOpenReview ↗

General Response to All Reviewers [1/3]

General Response to All Reviewers

We sincerely thank all three reviewers for the time and care they devoted to examining our method, experiments, and implementation. This general response addresses three concerns shared by multiple reviewers, such as generalization to unseen attack templates (§1), generalization to cross-domain transfer (§2), and robustness to adaptive attacks (§3).

 

Generalization to Unseen Attack Templates (§1)

Reviewer ELZT — Weakness 2: No paraphrased injections are tested.

Reviewer hB8Y — Question 1: Can you clarify whether training and testing use disjoint attack templates, and provide results on unseen templates while keeping trusted system prompts unchanged?

Reviewer NFeo — Question 5: To what extent does the Deviation Estimator depend on the attack types observed during training?

We thank the reviewers for raising this point. Our main experiments use disjoint queries for training and testing but share the same predefined attack templates, so the results alone do not show whether CompassGuard generalizes beyond the original prompt formulations. To examine this, we evaluate two levels of unseen attack templates, constructed with GPT-4o-mini based on the Rephrase and Generate operators in [1]. The results show that CompassGuard maintains a low ASR under both levels, with all trained modules frozen.

Specifically, Level 1 paraphrases the attacker's instructional text while preserving the component order, attack objective, and output requirements, and Level 2 rewrites the attacker's user-prompt template in wording, structure, and formatting while preserving the task inputs and attack objective. Neither set of templates is used for training. We evaluate in the homogeneous setting on CSQA and GSM8K under prompt injection and on MS-MARCO under memory attacks.

Attack TemplateCSQA ASR ↓CSQA ACC ↑GSM8K ASR ↓GSM8K ACC ↑MSMARCO ASR ↓MSMARCO ACC ↑
Original1.1 ± 1.073.3 ± 6.74.5 ± 1.190.6 ± 3.50.0 ± 0.076.1 ± 4.2
Level 10.0 ± 0.075.6 ± 11.77.1 ± 4.485.9 ± 3.11.7 ± 1.774.4 ± 5.1
Level 20.6 ± 1.172.2 ± 11.73.2 ± 1.190.5 ± 1.00.6 ± 1.076.7 ± 3.3

Values are mean ± standard deviation in percent. ASR stays at or below 7.1% under both levels, while accuracy remains comparable to that with the original templates. We have added these results to the revised paper in Sec. 5, Appendix G.2, and Table 13.

[1] Yu, Jiahao, et al. "LLM-Fuzzer: Scaling assessment of large language model jailbreaks." 33rd USENIX Security Symposium (USENIX Security 24). 2024.

AuthorsOpenReview ↗

General Response to All Reviewers [2/3]

Generalization to Cross-Domain Transfer (§2)

Reviewer ELZT — Weakness 2: Table 9 shows that zero-shot cross-domain detection precision drops to 25%, raising concerns about reliability under distribution shifts.

Reviewer hB8Y — Question 1: For the zero-shot transfer setting in Table 9, can you report task utility and false-positive rates alongside ASR, given the 25% precision and 100% recall?

Reviewer NFeo — Weakness 2 & Question 5: How should the low zero-shot detection precision in Table 9 be interpreted alongside the low reported ASR?

We appreciate the constructive feedback from all three reviewers on generalization under domain shifts. We first note that the detection metrics in Table 9 have been corrected, including the zero-shot precision of 25.0% above, which is now 61.1%. The correction does not change the conclusion and is detailed at the end of this response. As the reviewers point out, zero-shot detection is less reliable in a new domain (zero-shot), so we examine how much target-domain data is needed by extending Table 9 with smaller target-domain proportions and three additional transfer cases. With only 18 of the 200 training queries drawn from the target domain, detection substantially improves in three of the four cases, and with 33 in all four.

TransferFPR ↓Precision ↑ASR ↓ACC ↑
Target-domain queries (of 200)0 → 18 → 330 → 18 → 330 → 18 → 330 → 18 → 33
CSQA → InjecAgent100.0 → 0.0 → 0.061.1 → 100.0 → 100.02.4 → 5.5 → 6.0—
InjecAgent → CSQA100.0 → 4.6 → 3.861.1 → 94.8 → 95.718.2 → 15.3 → 11.150.9 → 65.6 → 72.2
GSM8K → CSQA100.0 → 76.7 → 5.861.1 → 65.8 → 94.223.5 → 5.0 → 4.561.7 → 72.7 → 72.0
CSQA → GSM8K93.8 → 1.7 → 0.462.1 → 98.4 → 99.511.5 → 11.5 → 11.686.6 → 84.2 → 84.0

Values are mean percentages and recall is 100.0 in all settings. Accuracy is omitted for InjecAgent, which does not provide a task-utility metric [2].

In the zero-shot setting, the Activation Predictor has not seen the target domain, so its predicted benign activation deviates from the actual one even for benign invocations, and CompassGuard flags most benign agents. The correction is then applied to almost every invocation, which is why ASR can remain low despite the low precision, although the new experiments show that the Deviation Estimator transfers less well when CSQA is the target domain, where ASR remains higher. A small number of target-domain queries lets both modules adapt, which accounts for the improved detection and the lower ASR in these cases. We believe that collecting about 30 such queries is practical for an MAS service provider, who already runs the system on in-domain queries when developing and serving a new task. We have updated Sec. 5, Appendix D.1, Appendix G.1, and Table 9 in the revised paper accordingly.

 

Correction to Table 9. The detection evaluation protocol covers the directly targeted agent and upstream agents and excludes downstream agents, whose compromised status is uncertain. This follows AcMAS's supplementary code, which scores detection only before attacked messages propagate to other agents. Only Table 9 in Appendix G had counted downstream agents as benign. We apologize for this error and have recomputed its precision and F1 in all columns, while ASR and recall are unaffected. In the zero-shot setting, CompassGuard flags all four agents, of which only one is attacked, which gave the precision of 25.0% in the submission. Excluding downstream agents changes it to 61.1%. This is not a new claim of good detection performance, since the benign agents are flagged under either protocol. Moreover, to make sure that CompassGuard is measured correctly in the main experiments, we checked the logs of all its results in the main paper and re-ran the experiments, confirming that the results are reproduced. Moreover, the execution logs of CompassGuard for Tables 1, 2, and 6 are provided in our anonymous Git repository.

 

[2] Wang, Yizhu, et al. “Defending Against Prompt Injection with DataFilter.” 2026 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML). IEEE, 2026.

AuthorsOpenReview ↗

General Response to All Reviewers [3/3]

Robustness to Adaptive Attacks (§3)

Reviewer ELZT — Weakness 2 & Question 3: No adaptive injections are tested. How does CompassGuard perform against adaptive attackers who manipulate prompts specifically to evade the defense?

Reviewer hB8Y — Question 1: How robust is the defense to attacks explicitly optimized against its detection and correction mechanisms?

We appreciate the reviewers' suggestion to evaluate adaptive attacks. Accordingly, we extend our experiments to adaptive attacks on CSQA and HumanEval in the heterogeneous setting, where the attacker iteratively refines the attack with knowledge of CompassGuard and feedback from the defended MAS. The results show that CompassGuard remains effective under these attacks, with an ASR of 6.8% on CSQA.

Specifically, we use GPT-4o-mini as the attacker LLM, inspired by the iterative prompt refinement in [3]. The attacker LLM is given the description of CompassGuard and the attack objective. For each targeted role, it generates an attack template, which we run on 20 validation queries with CompassGuard enabled, and it then revises the template based on the result, up to four times. The adaptive attack template that succeeds most on the 20 validation queries is finally used on the test queries.

 

CSQA

DefenseASR ↓ACC ↑F1 ↑Precision ↑
No Defense + Original Attack22.8 ± 2.851.9 ± 5.6——
No Defense + Adaptive Attack24.7 ± 5.445.0 ± 6.7——
AcMAS + Original Attack18.0 ± 6.062.2 ± 14.696.8 ± 0.893.8 ± 1.5
AcMAS + Adaptive Attack24.1 ± 13.459.4 ± 10.898.9 ± 1.098.4 ± 1.6
CompassGuard + Original Attack4.9 ± 3.275.6 ± 2.6100.0 ± 0.0100.0 ± 0.0
CompassGuard + Adaptive Attack6.8 ± 6.574.4 ± 6.999.4 ± 1.0100.0 ± 0.0

 

HumanEval

Defensepass@1 ↑F1 ↑Precision ↑
No Defense + Original Attack41.4 ± 7.2——
No Defense + Adaptive Attack39.8 ± 11.9——
AcMAS + Original Attack43.0 ± 7.887.3 ± 4.377.7 ± 6.8
AcMAS + Adaptive Attack35.5 ± 10.697.0 ± 2.794.3 ± 5.1
CompassGuard + Original Attack54.7 ± 2.592.1 ± 4.685.8 ± 8.2
CompassGuard + Adaptive Attack52.3 ± 4.7100.0 ± 0.0100.0 ± 0.0

Values are mean ± standard deviation in percent. The original-attack rows are taken from Tables 2, 3, 5 and 7.

The adaptive attack is stronger than the original one, yet with CompassGuard it raises ASR only from 4.9% to 6.8% and is still detected in almost all cases. We have added these results to the revised paper in Sec. 5 and Appendix Tables 10 and 11.

 

[3] Chao, Patrick, et al. "Jailbreaking Black Box Large Language Models in Twenty Queries." 2025 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML). IEEE, 2025.

AuthorsOpenReview ↗

Rebuttal Summary

Dear Area Chairs, Senior Area Chairs, and Program Chairs,

Thank you for your time and effort in overseeing our submission. We summarize the discussion phase below.


Reviewer ELZT (score: 2, no reply yet) Reviewer ELZT raised concerns regarding methodological novelty beyond AcMAS, Activation Predictor accuracy, and generalization across unseen attacks and domains. In our response, we clarified that:

  • CompassGuard goes beyond simply token-wise steering toward a benign prototype by estimating and subtracting the attack-induced deviation at each token. The ablation shows the benefit of this correction.
  • The Activation Predictor achieves >0.99 cosine similarity with true benign activations in the additional experiments.
  • In the additional experiments, ASR stays at or below 7.1% on unseen attack templates and is 6.8% under adaptive attacks. In the cross-domain transfer experiments, detection substantially improves in three of the four settings with only 18 target-domain queries, and in all four with 33.

We have addressed each point raised by Reviewer ELZT and hope that our explanations and additional results have resolved their concerns. As of this writing, we have not received a response from the reviewer. If any questions remain, we would be happy to clarify them.


Reviewer hB8Y (score: 2 → 3) Reviewer hB8Y requested clarifications on the evaluation protocol, paired-execution training pipeline, detection metrics, and generalization. In our response, we clarified that:

  • Correcting history and context pairing in homogeneous training further lowered ASR on 5 of 6 benchmarks.
  • We documented uniform evaluation rules, provided per-role detection counts, and released all execution logs.
  • New experiments confirmed robustness under unseen templates, cross-domain transfer, and adaptive attacks.

The reviewer thanked us for our response and indicated that they would raise their score from 2 to 3.

"Thanks for the author's response. After consideration, I will raise my score to 3."


Reviewer NFeo (score: 3 → 3) Reviewer NFeo questioned key design choices (full benign activation prediction and tokenwise steering), free-generation generalization, benign-condition utility, and attack-type dependence. In our response, we clarified that:

  • The choice of full benign activation prediction, which we explained in detail to the reviewer, is now reflected in Sec. 3.2 of the revised paper. Dynamic tokenwise steering outperforms first-token-only steering in a new ablation (CSQA ASR 4.9% vs. 24.2%).
  • CompassGuard preserves task accuracy under completely benign conditions.
  • Extended evaluations on unseen templates and cross-domain transfer confirm broader generalization.

Following our response, the reviewer maintained their positive score of 3.

"I appreciate your response and will keep my score unchanged."


We sincerely thank you and the reviewers for the time and effort you have devoted to our submission. We hope this summary is helpful.

With sincere gratitude and best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.