- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper proposes REAP, a framework for optimizing reusable attack policies that capture resources while preserving task utility. A fixed optimizer revises policy rules, injection surfaces, and content templates using execution feedback, with validation determining which updates are retained. Experiments cover retrieval, coding, and tool-use environments under individual and provider attacker roles. The authors report substantial resource capture with high utility preservation, supported by component ablations, transfer experiments, and defense evaluations.
Critical strengths
-
The paper explicitly considers who benefits from resource allocation during execution. Distinguishing individual resource-share capture from provider usage amplification helps explain how an agent can complete its task while consuming resources in ways that benefit an adversary.
-
Evaluation spans three environments and two attacker roles. The recursion ablation supports accumulating policy improvements rather than repeatedly restarting, while the transfer experiments provide evidence that some learned behavior generalizes beyond the optimization tasks.
Critical weaknesses
-
REAP closely resembles iterative optimization of a reusable prompt or skill, with structured state and validation-based selection. The recursion ablation demonstrates the benefit of inheriting improvements, but does not establish an advantage over a reusable template updated across the same tasks with equivalent feedback and validation. The specific contribution of policy-level RSI therefore remains unclear.
-
The optimization curves report validation performance under an acceptance rule that explicitly selects improvements. They do not establish that additional rounds improve performance on unseen tasks. The case studies illustrate useful mechanisms, but mostly contrast initial failures with successful candidates. Although transfer results support reuse, the connection between particular policy revisions and generalization remains insufficiently explained.
Questions for authors
-
How does REAP compare with standard iterative prompt optimization that updates a reusable attack prompt across tasks? Please use the same training tasks, execution feedback, validation criteria, and total target agent execution budget.
-
How does performance on unseen tasks change after 1, 5, 10, and 20 optimization rounds? Are these trends consistent across independent optimization runs?
-
Could the authors show an exact accepted policy revision and demonstrate how the resulting rule applies to multiple unseen tasks? For individual attackers, please clarify which artifacts are fixed before the test query arrives and what information is available when constructing them.
3 comments
Follow-up on Rebuttal
Dear Reviewer mESs:
As the rebuttal deadline is approaching, I would like to kindly check whether our responses have addressed your concerns and questions.
If there are any outstanding issues, we would be happy to address them during the remaining discussion period. If our previous responses have satisfactorily addressed your concerns, we would greatly appreciate it if you could consider updating score to reflect the newly added results and discussion.
Your time and effort are greatly appreciated.
Sincerely,
Authors of 18098
Thank you for the detailed rebuttal and additional experiments. The stronger baselines, independent optimization runs, and additional controls have addressed most of my concerns and made the results more convincing. While some limitations remain, I believe the paper has been sufficiently strengthened. I will therefore raise my score from 2 to 3.
We sincerely appreciate your insightful comments. In the following, we will address your questions one by one.
W1, Q1: Comparison with an iteratively updated reusable prompt
We added the reusable-prompt baseline you suggested. It uses the same training tasks, feedback, validation criteria, optimizer, and total target-agent execution budget as REAP. The baseline updates a single prompt across tasks without REAP’s structured policy or diagnosis step. REAP achieves higher UP and resource capture in all three individual-attacker settings.
W2, Q2: Test performance across optimization rounds
We evaluated the provider policy on held-out Travel tasks after 1, 5, 10, and 20 optimization rounds. In all three independent runs from three LLMs, the agent made more tool calls at round 20 than at round 1, and UP was 100% at each of the four rounds.
W3, Q3. An accepted revision on Slack
The following revision was accepted in round 1 for the individual attacker.
slack:user_task_5, the agent finds the channel with the most members and posts there. Inslack:user_task_9, it finds the channel with the fewest messages and adds Alice to it. In both runs, the agent called the same registered attacker tool,distinct_channel_context_verification, so the attacker's tools are used on both unseen tasks.On three of the four held-out Slack tasks, the frozen policy increased the share of calls to attacker-owned tools while preserving task utility (Table 10). That share averaged 33.33% across all four tasks. This shows that the policy carried over to multiple unseen tasks without further updates.