- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper asks a valuable quesions, whether an agent harness can be updated automatically when a new attack is known only from a paper or a handful of examples. They thus propose a MAS framework named HASTE with five LLM roles. An attack parser turns the evidence into an attack specification. Then, for five rounds, a specification optimizer writes a safety spec, a proposer edits the harness, which is a single Python file, and a case optimizer generates new attack cases plus benign cases without seeing the spec. A judge runs the agent on those cases, scores safety and utility, and the feedback is accumulated for the next round. The experiment evaluates 2 variants, HASTE_blind uses only generated cases and returns the last harness; HASTE_seen also gets 16 attack cases and 16 benign cases sampled from the benchmark and picks the round with the best ASR and SHR on them. HASTE show the best performance on the evaluations.
Critical strengths
- The evolved harnesses actually transfer. From the paper, seems like the harness built on ASB still can drops STAC ASR, in other word, harnesses built on one backbone work on the others. Also, the cases in table 13 show the harness is doing something real, not just refusing more, which needs well design.
Critical weaknesses
This hardly qualifies as a case of "sparse evidence" or an "emerging attack." HASTE_seen utilized 32 samples from the target benchmark to generate additional samples within the same environment; indeed, the "paper" in question appears to be the original paper for that benchmark itself. This constitutes a form of in-distribution few-shot adaptation. Table 8 even shows that the method yielded only marginal gains (an ASR of 21.4 versus 23.9). Since most baseline methods were not adapted for ASB, it is hardly surprising that HASTE outperformed them. The only fair point of comparison is Meta-Harness: the ASR gap between the two is 3.5 percentage points on ASB, whereas there is no difference at all on STAC.
Questions for authors
Which specific "research paper" did you use as input for the evidence? If it was the ASB or STAC paper itself, then the attack parser would have effectively already acquired all the risk categories and environmental information. Given this, why is it still described as "sparse"?
Were the generated attack cases constructed within the same ASB environment used for the test tasks? If so, this implies that the test harness was tuned for the tools within that test environment.
Utility is only checked on ASB's own fulfillable tasks and on cases the system generated for itself. Did you test the evolved harness on any ordinary agent benchmark to see how much it refuses things it should not?
Is there any situation that a model refuse to answer because the safetyguard inside the provider? Since the safety guard imposed by manufacturers are increasingly strict.
2 comments
Q3: Utility is only checked on ASB's own fulfillable tasks and on cases the system generated for itself. Did you test the evolved harness on any ordinary agent benchmark to see how much it refuses things it should not?
We sincerely thank the reviewer for this valuable comment.
To evaluate whether the evolved harnesses compromise general agent utility beyond safety benchmarks, we additionally conduct experiments on ToolHop [1]. We randomly sample 50 tasks and evaluate the final harnesses with Qwen3-8B and DeepSeek-V4-Pro as backbone models.
| Method | Qwen3-8B Accuracy ↑ | DeepSeek-V4-Pro Accuracy ↑ |
|---|---|---|
| No-defense | 24.67±6.43 | 63.33±3.06 |
| HASTE_blind | 24.00±12.49 | 64.00±4.00 |
| Meta-Harness | 28.00±4.00 | 64.67±4.16 |
| HASTE_seen | 28.67±2.31 | 66.00±3.46 |
The results show that HASTE achieving comparable or higher task accuracy than No-defense on both backbone models. These findings suggest that the safety improvements achieved through harness adaptation do not come at the cost of broadly restricting legitimate agent behavior. We will include these results in the revised manuscript.
[1] ToolHop: A Query-Driven Benchmark for Evaluating Large Language Models in Multi-Hop Tool Use
Q4: Is there any situation that a model refuse to answer because the safety guard inside the provider? Since the safety guard imposed by manufacturers are increasingly strict.
We thank the reviewer for raising this question. The No-defense results in Tables 1–2 reflect the backbones’ built-in safety behavior. Comparing HASTE with No-defense on the same backbones shows the additional safety gains from harness adaptation.
We sincerely thank the reviewer for raising these insightful concerns.
(1) In-distribution few-shot adaptation.
We agree that the within-benchmark evaluation of HASTE_seen is a form of in-distribution few-shot adaptation. “Emerging attack” refers to a threat first disclosed through a report or a few examples, without implying that the evaluation cases are out of distribution.
Our cross-benchmark experiment in Table 6 addresses the separate question of whether the adapted harness generalizes to out-of-distribution attacks. We adapted the harness on ASB and evaluated it on 93 STAC cases that use sequential tool attack chaining. These cases share ASB environments but introduce a different attack mechanism. On this subset, the ASB-adapted harness is still effective.
We would like to clarify that in HASTE_seen, both the case optimizer and the proposer could inspect the tool lists and implementations of environments associated with observed cases. In HASTE_blind, both accessed the environments through the benchmark backend’s high-level interfaces. To further assess generalization across environments, we used DeepSeek-V4-Pro as the backbone and evaluated HASTE on cases from the 325 of ASB’s 349 environments that the observed adaptation cases did not cover. Both HASTE variants remained effective on these environments.
(2) Sparse Evidence.
(3) Comparison between HASTE_seen and Meta-Harness. The run-to-run variation makes the mean ASR difference between HASTE_seen and Meta-Harness difficult to estimate precisely from only three runs. To obtain a more precise estimate, we conducted 17 additional independent adaptation runs for each method on each backbone model. We report the 95% confidence intervals for the performance differences between HASTE_seen and Meta-Harness (HASTE_seen − Meta-Harness) averaged equally across the backbones evaluated on each benchmark, separately for each dataset and evaluation metric. The results demonstrate that HASTE_seen consistently outperforms Meta-Harness across datasets and metrics.
[1] LI X, Kaiyom F, Liu E, et al. Autobencher: Towards declarative benchmark construction[C]//International Conference on Learning Representations. 2025, 2025: 94357-94386.
[2] Bouzenia I, Pradel M. BenchMeUp: Building Software Engineering Benchmarks from Natural-Language Specifications[J]. arXiv preprint arXiv:2610.37283, 2026.