- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes Twin Agent, a framework consisting of two symmetric agents with similar architectures, namely the Explore Agent and the Safe Agent, to defend large language model (LLM) agents against prompt injection attacks. The authors evaluate Twin Agent on SWE-bench, AgentDojo, and DecodingTrust-Agent. The experimental results demonstrate that Twin Agent achieves low attack success rates while largely preserving task utility.
Critical strengths
(1) The problem is relevant to the security of tool-using agents. The architecture is simple and does not require major changes to existing agent workflows. (2) The evaluation covers three benchmarks and includes adaptive attacks. The results show that restricting information flow can substantially reduce attack success rates.
Critical weaknesses
(1) The security argument in Section 4.3 is questionable. The authors claim that the hint channel is the only pathway through which untrusted information can influence the Safe Agent. However, in the SWE-bench implementation, the Safe Agent can read repository files and receive shell command outputs, both of which may also contain untrusted content.
(2) The adaptive attack results are less convincing than the paper suggests. The attack succeeds in only 11 out of 232 instances. The authors rerun these instances five times and report no additional successes, but this does not explain why the initial attacks failed.
(3) The baseline comparisons are not entirely fair. The dual-agent baseline prohibits communication between agents, resulting in zero task utility on SWE-bench-injected. In the comparison with CaMeL, the authors reduce its retry budget from 10 to 3 and acknowledge that this reduction negatively affects its utility. Nevertheless, the resulting numbers are used to support the claimed advantage of Twin Agent in the security–utility trade-off, which raises concerns about the fairness of the comparison.
(4) The presentation quality of the figures and tables is rather unprofessional (Figures 1, 2, and A2, and Table 3), making them difficult to read and interpret.
Questions for authors
(1) The main contribution appears to be Residual Compression, but its implementation seems to rely on context-dependent hint generation with a length constraint. The paper lacks comparisons with existing methods under identical communication budgets and defense settings. Without such comparisons, it is difficult to attribute the observed improvements to the proposed mechanism.
(2) Section 4.3 treats the hint channel as the only attack surface, whereas Section 4.1 allows the Safe Agent to inspect repository files and shell outputs. This creates some confusion regarding the threat model and its underlying assumptions.
(3) The results for CaMeL should be reported using its original retry budget, or at least compared under matched computational constraints.
2 comments
We thank the reviewer again for the constructive feedback. As the discussion period is coming to an end, we would greatly appreciate it if you could take a look at our rebuttal and let us know whether it addresses your concerns. We would be happy to provide any further clarification.
We thank the reviewer for the constructive feedback and thoughtful suggestions.
1. SWE-bench trust boundary (W1/Q2)
Please see Section 4.1 and Appendix A for the threat model in SWE-bench-injected. Repository files and shell command outputs are trusted sources because we protect the repository, while the attacker injects through the GitHub issue description.
2. Adaptive attack interpretation (W2)
LLMs sample next tokens with probabilities. Attacks can succeed with some probability rather than always winning or always losing. We rerun the attack five times and find no additional success, showing the attack patterns aren't strong enough to commit a stable attack.
3. Baseline fairness (W3/Q3)
The Dual Agent baseline is a typical design pattern that prohibits communication between agents, so that it can ensure security by design. However, this design pattern may fail on complex tasks, such as SWE-bench. Our experiment provides practical evidence for this weakness.
For CaMeL, we explicitly disclosed the reduced retry budget (
max_attempts=3instead of 10) in Appendix A. CaMeL is computationally expensive, and full evaluation with 10 retries exceeded our available budget. We also conducted preliminary comparisons on a smaller subset of tasks in our environment and did not observe a meaningful utility difference between these settings. Furthermore, even using the approximately 35% utility reported in the original CaMeL paper instead of our measured 30.9%, Twin Agent still achieves substantially higher utility (62.9%).4. Contribution of residual compression (Q1)
Residual compression is not merely enforcing a short hint. The Explore Agent conditions its message on the Safe Agent's current history and communicates only the missing information needed for the next decision, rather than summarizing the full untrusted input. This distinction is formalized in Eqs. (11)–(12) and reflected in the prompts in Appendix E, including the option to output
HINT: nullwhen no additional information is needed.Few methods limit hint budgets. Existing dual-agent design patterns allow no communication between the two agents, which we have already implemented and compared. We also compared with CaMeL on AgentDojo, which allows careful information control. Adapting CaMeL to SWE-bench is nontrivial, as it requires manually designed security policies and tool-specific integration to support open-ended, iterative debugging tasks.
5. Presentation (W4)
We appreciate the feedback and will improve the readability and labeling of Figures 1, 2, and 3, and Table 3. We would appreciate it if the reviewer can elaborate on this.