- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents HARD, a framework for evolving runtime defenses for tool-using LLM agents while keeping model parameters fixed. HARD uses execution failures and evaluator feedback to update a natural-language security policy and an executable tool-call gate, with a trace router assigning failures to specialized editors. The framework is evaluated on four static attack surfaces and two adaptive attack protocols. The paper reports reduced attack success against the selected non-evolving baselines and examines utility, artifact selection, routing, evolution backbones, and deployment costs.
Critical strengths
-
The paper addresses the important problem of maintaining agent security as attacks evolve, with a modular approach that does not require updating the protected model.
-
The evaluation covers multiple attack surfaces and separates attack success, benign utility, and utility under attack, providing a useful view of security–utility tradeoffs.
-
The detailed editing prompts and diagnostic failure cases help explain how policy and gate refinements behave, including their limitations.
Critical weaknesses
-
The distinction from SHE (Qu et al., 2026a), discussed in Appendix K.3, remains insufficiently established. Both methods decompose a safety harness and route trajectory failures to specialized artifact editors. Please explain what the two-interface formulation enables beyond SHE’s four-component design and provide evidence that this difference yields a substantive methodological benefit, while acknowledging the concurrent nature of the work.
-
Current baselines are non-evolving, whereas HARD receives five rounds of task-specific failure feedback. This leaves unclear whether its gains arise from the proposed mechanism or additional optimization. Please compare with an evolving defense, such as Membrane (Choi et al., 2026), under matched feedback and budgets, and include Task Shield (Jia et al., 2025) or MELON (Zhu et al., 2025) where applicable.
-
Section 4.1 and Appendix A.2 state that a separate defense is evolved for each attack category, without artifact transfer. The reported results therefore do not establish that one deployed configuration protects against all four attack surfaces. Please evaluate a shared defense trained on mixed categories, or systematically assess artifact composition, reporting interference, false refusals, and utility changes.
-
The task-identifier split does not explicitly separate attack families, templates, or environments, leaving generalization beyond related instances unresolved. Please add an attack-family or template-level holdout evaluation and assess transfer to another protected execution model. The current backbone ablation varies the router/evolver rather than the protected model, so it cannot establish cross-model robustness of the deployed defense.
-
Adaptive split sizes and query budgets are incompletely documented in Appendix F. Please specify attacker models, queries, restarts, maximum turns, feedback access, and knowledge of defense artifacts, and report ASR against attack budget. Both adaptive protocols use direct injection; evaluating adaptive indirect injection or artifact poisoning would clarify whether the reported robustness extends to the other protected attack surfaces.
-
GLM-5 supplies both evolution feedback and final ASR/UA judgments, creating potential dependence on the evaluator’s preferences and blind spots. Disjoint tasks and repeated grading do not provide an independent outcome standard. Please validate a representative subset using executable checks, human inspection, or judges from different model families, and report disagreement rates and whether the main performance ranking remains unchanged.
-
Under IPI, HARD reaches 1.0% ASR but only 19.0% utility under attack, compared with 20.9% for the official harness. This leaves unclear whether blocked attacks preserve useful task execution. Please distinguish false refusals, planning failures, execution errors, and successful recovery, and report joint attack prevention and task completion. The 25 separate benign tasks provide limited evidence about false positives on closely related legitimate operations.
-
The routing comparisons do not match editor calls, tokens, trace exposure, or update order. Moreover, routed and broadcast ASRs under DAE are 12.7% and 13.9%, with a difference small relative to repeat variability. Please include budget-matched random or heuristic routing controls, test alternative update orders, and examine sensitivity to the 0.55 confidence threshold to establish the router’s independent contribution.
-
Table 4 does not separate costs by task outcome, and evolution cost is unreported. Fewer calls could partly reflect earlier refusal or task termination. Please separate costs for successful legitimate tasks, blocked attacks, and failed tasks, and report evolution overhead. Since latency was measured on different days, qualify timing comparisons and clarify whether Llama Guard’s per-call latency includes its auxiliary classifications.
Questions for authors
-
With matched failure feedback, editor calls, and token budgets, what benefit does HARD's two-artifact routing provide over an evolving baseline and simple heuristic routing?
-
Can a single defense evolved on mixed attack categories retain security and utility on held-out attack families and a different protected execution model?
-
Using executable checks or an independent judge family, what fraction of IPI runs both block the attack and complete the legitimate task, and does the main ranking persist?