acceptodds
Under review as a conference paper at ICLR 2027

HASTE: Evolving Agent Harnesses Against Emerging Attacks Using Sparse Evidence

Abstract

Agent harnesses play a critical role in defenses by enforcing safety constraints to prevent unsafe actions. However, rapidly emerging attacks outpace manual harness adaptation, motivating automated harness evolution. Yet the signals available for harness evolution are often sparse, such as brief descriptions or a few attack examples in threat reports and preprints. To address this limitation, we introduce Haste, a multi-agent framework that evolves agent harnesses from sparse threat evidence through an adversarial interplay between safety-specification generation and attack-case generation. Safety specifications guide harness updates toward addressing identified safety vulnerabilities, while attack cases probe for remaining safety vulnerabilities after each update. By feeding evaluation outcomes back into both processes, Haste enables harness evolution against emerging attacks beyond the initially observed evidence. Experimental results across multiple backbone models, attack types, and evidence forms show that HASTE consistently reduces attack success rates while preserving benign-task utility. The code is available at https://anonymous.4open.science/r/HASTE-4F94/.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Agent safety, agent harness, harness evolution

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.20 over 5 reviews, above 49% of the others.
Reviewer_hbmyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks a valuable quesions, whether an agent harness can be updated automatically when a new attack is known only from a paper or a handful of examples. They thus propose a MAS framework named HASTE with five LLM roles. An attack parser turns the evidence into an attack specification. Then, for five rounds, a specification optimizer writes a safety spec, a proposer edits the harness, which is a single Python file, and a case optimizer generates new attack cases plus benign cases without seeing the spec. A judge runs the agent on those cases, scores safety and utility, and the feedback is accumulated for the next round. The experiment evaluates 2 variants, HASTE_blind uses only generated cases and returns the last harness; HASTE_seen also gets 16 attack cases and 16 benign cases sampled from the benchmark and picks the round with the best ASR and SHR on them. HASTE show the best performance on the evaluations.

Critical strengths
  1. The evolved harnesses actually transfer. From the paper, seems like the harness built on ASB still can drops STAC ASR, in other word, harnesses built on one backbone work on the others. Also, the cases in table 13 show the harness is doing something real, not just refusing more, which needs well design.
Critical weaknesses

This hardly qualifies as a case of "sparse evidence" or an "emerging attack." HASTE_seen utilized 32 samples from the target benchmark to generate additional samples within the same environment; indeed, the "paper" in question appears to be the original paper for that benchmark itself. This constitutes a form of in-distribution few-shot adaptation. Table 8 even shows that the method yielded only marginal gains (an ASR of 21.4 versus 23.9). Since most baseline methods were not adapted for ASB, it is hardly surprising that HASTE outperformed them. The only fair point of comparison is Meta-Harness: the ASR gap between the two is 3.5 percentage points on ASB, whereas there is no difference at all on STAC.

Questions for authors

Which specific "research paper" did you use as input for the evidence? If it was the ASB or STAC paper itself, then the attack parser would have effectively already acquired all the risk categories and environmental information. Given this, why is it still described as "sparse"?

Were the generated attack cases constructed within the same ASB environment used for the test tasks? If so, this implies that the test harness was tuned for the tools within that test environment.

Utility is only checked on ASB's own fulfillable tasks and on cases the system generated for itself. Did you test the evolved harness on any ordinary agent benchmark to see how much it refuses things it should not?

Is there any situation that a model refuse to answer because the safetyguard inside the provider? Since the safety guard imposed by manufacturers are increasingly strict.

2 comments

W1: This hardly qualifies as a case of "sparse evidence" or an "emerging attack." HASTE_seen utilized 32 samples from the target benchmark to generate additional samples within the same environment; indeed, the "paper" in question appears to be the original paper for that benchmark itself. This constitutes a form of in-distribution few-shot adaptation. Table 8 even shows that the method yielded only marginal gains (an ASR of 21.4 versus 23.9). Since most baseline methods were not adapted for ASB, it is hardly surprising that HASTE outperformed them. The only fair point of comparison is Meta-Harness: the ASR gap between the two is 3.5 percentage points on ASB, whereas there is no difference at all on STAC.
Q1: Which specific "research paper" did you use as input for the evidence? If it was the ASB or STAC paper itself, then the attack parser would have effectively already acquired all the risk categories and environmental information. Given this, why is it still described as "sparse"?
Q2: Were the generated attack cases constructed within the same ASB environment used for the test tasks? If so, this implies that the test harness was tuned for the tools within that test environment.

We sincerely thank the reviewer for raising these insightful concerns.

(1) In-distribution few-shot adaptation.

We agree that the within-benchmark evaluation of HASTE_seen is a form of in-distribution few-shot adaptation. “Emerging attack” refers to a threat first disclosed through a report or a few examples, without implying that the evaluation cases are out of distribution.

Our cross-benchmark experiment in Table 6 addresses the separate question of whether the adapted harness generalizes to out-of-distribution attacks. We adapted the harness on ASB and evaluated it on 93 STAC cases that use sequential tool attack chaining. These cases share ASB environments but introduce a different attack mechanism. On this subset, the ASB-adapted harness is still effective.

We would like to clarify that in HASTE_seen, both the case optimizer and the proposer could inspect the tool lists and implementations of environments associated with observed cases. In HASTE_blind, both accessed the environments through the benchmark backend’s high-level interfaces. To further assess generalization across environments, we used DeepSeek-V4-Pro as the backbone and evaluated HASTE on cases from the 325 of ASB’s 349 environments that the observed adaptation cases did not cover. Both HASTE variants remained effective on these environments.

MethodSafety ASR ↓SHR ↑
No-defense52.40±0.6764.39±0.95
HASTE-blind28.21±8.3070.82±1.08
Meta-Harness27.68±3.3172.00±6.41
HASTE-seen23.70±1.4672.86±3.33

(2) Sparse Evidence.

  • We use the original ASB and STAC papers as threat evidence. Although their risk taxonomies and environment descriptions give the parser a conceptual understanding of each benchmark, they provide only limited detail on how a given risk manifests in specific attack scenarios, which tool interactions it involves, or how the relevant tools are implemented.
  • As benchmark construction studies (e.g., AutoBencher[1], BenchMeUp[2]) illustrate, generating concrete test cases from high-level specifications is itself a non-trivial task.
  • We would like to respectfully argue that if the paper provided substantial additional information for adaptation, withholding it would cause a larger drop in performance. As the reviewer notes, including the paper yields only marginal gains in Table 8, consistent with our characterization of the threat evidence as sparse.

(3) Comparison between HASTE_seen and Meta-Harness. The run-to-run variation makes the mean ASR difference between HASTE_seen and Meta-Harness difficult to estimate precisely from only three runs. To obtain a more precise estimate, we conducted 17 additional independent adaptation runs for each method on each backbone model. We report the 95% confidence intervals for the performance differences between HASTE_seen and Meta-Harness (HASTE_seen − Meta-Harness) averaged equally across the backbones evaluated on each benchmark, separately for each dataset and evaluation metric. The results demonstrate that HASTE_seen consistently outperforms Meta-Harness across datasets and metrics.

BenchmarkMetricDifference (95% CI)
ASBASR ↓-4.75 [-6.48, -3.02]
ASBSHR ↑+9.83 [+8.63, +11.04]
STACASR ↓-4.26 [-8.03, -0.49]

[1] LI X, Kaiyom F, Liu E, et al. Autobencher: Towards declarative benchmark construction[C]//International Conference on Learning Representations. 2025, 2025: 94357-94386.

[2] Bouzenia I, Pradel M. BenchMeUp: Building Software Engineering Benchmarks from Natural-Language Specifications[J]. arXiv preprint arXiv:2610.37283, 2026.

Q3: Utility is only checked on ASB's own fulfillable tasks and on cases the system generated for itself. Did you test the evolved harness on any ordinary agent benchmark to see how much it refuses things it should not?

We sincerely thank the reviewer for this valuable comment.

To evaluate whether the evolved harnesses compromise general agent utility beyond safety benchmarks, we additionally conduct experiments on ToolHop [1]. We randomly sample 50 tasks and evaluate the final harnesses with Qwen3-8B and DeepSeek-V4-Pro as backbone models.

MethodQwen3-8B Accuracy ↑DeepSeek-V4-Pro Accuracy ↑
No-defense24.67±6.4363.33±3.06
HASTE_blind24.00±12.4964.00±4.00
Meta-Harness28.00±4.0064.67±4.16
HASTE_seen28.67±2.3166.00±3.46

The results show that HASTE achieving comparable or higher task accuracy than No-defense on both backbone models. These findings suggest that the safety improvements achieved through harness adaptation do not come at the cost of broadly restricting legitimate agent behavior. We will include these results in the revised manuscript.

[1] ToolHop: A Query-Driven Benchmark for Evaluating Large Language Models in Multi-Hop Tool Use


Q4: Is there any situation that a model refuse to answer because the safety guard inside the provider? Since the safety guard imposed by manufacturers are increasingly strict.

We thank the reviewer for raising this question. The No-defense results in Tables 1–2 reflect the backbones’ built-in safety behavior. Comparing HASTE with No-defense on the same backbones shows the additional safety gains from harness adaptation.

Reviewer_S6kROpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes HASTE to improve the safety of tool-using agents. It starts from an attack description or a few examples. It then generates test cases and modifies the agent’s harness using execution feedback. Tests on two benchmarks show lower attack success rates while generally preserving task performance.

Critical strengths

The problem is useful in practice. HASTE can update an agent’s defenses without retraining the main model. The evaluation measures whether the agent can still complete normal tasks. It also tests several models, including GPT-5.4-mini.

Critical weaknesses

The tests do not clearly match the motivation of handling new attacks. The authors sample examples from existing benchmarks and test on the remaining cases. This shows some ability to handle unseen cases. However, it is less clear whether the method can handle a new attack mechanism. The cross-benchmark test also uses cases from the same ASB environments.

It is unclear how much execution feedback helps case generation. Table 3 compares HASTE with a version that keeps using the first-round cases. HASTE therefore sees more varied cases over time. A useful baseline would generate fresh cases in every round without seeing the current harness or execution feedback. Both methods should use a similar budget.

The source of the harness’s improvement needs more explanation. Comparing the same model with and without the harness shows that the full system helps. However, the harness can change prompts or block tool calls through code. The appendix also shows designs that use an extra safety model. The paper does not clearly separate the effects of these mechanisms or report their runtime cost.

Questions for authors

What information is given to the blind and seen settings? Does the use of 16 benchmark cases in Appendix C.3 apply only to the seen setting?

Does HASTE outperform fresh case generation without execution feedback when both methods use the same budget?

Which defense mechanisms are used in the tested harnesses? How much do they help, and what is their runtime cost?

3 comments

W1: The tests do not clearly match the motivation of handling new attacks. The authors sample examples from existing benchmarks and test on the remaining cases. This shows some ability to handle unseen cases. However, it is less clear whether the method can handle a new attack mechanism. The cross-benchmark test also uses cases from the same ASB environments.

We sincerely thank the reviewer for raising this question.

We would like to clarify that HASTE does not aim to handle arbitrary, unrelated attack mechanisms without prior evidence. Rather, it aims to generalize from sparse evidence of an emerging threat to its unseen manifestations, potentially across different attack mechanisms, tasks, and environments.

In our cross-benchmark test, we adapted the harness on ASB and evaluated it on 93 STAC cases in shared ASB environments. These STAC cases use different attack mechanisms: they chain tool interactions across successive steps. Keeping the environment fixed allows us to test whether an ASB-adapted harness remains effective when the attack mechanism changes.

To assess transfer across environments, we used DeepSeek-V4-Pro as the backbone and evaluated HASTE on cases from the 325 of ASB’s 349 environments that the observed adaptation cases did not cover.

MethodSafety ASR ↓SHR ↑
No-defense52.40±0.6764.39±0.95
HASTE-blind28.21±8.3070.82±1.08
Meta-Harness27.68±3.3172.00±6.41
HASTE-seen23.70±1.4672.86±3.33

Both HASTE variants remained effective on these environments. We will include the cross-environment generalization in the revised manuscript.


W2: It is unclear how much execution feedback helps case generation. Table 3 compares HASTE with a version that keeps using the first-round cases. HASTE therefore sees more varied cases over time. A useful baseline would generate fresh cases in every round without seeing the current harness or execution feedback. Both methods should use a similar budget.
Q2: Does HASTE outperform fresh case generation without execution feedback when both methods use the same budget?

We thank the reviewer for this insightful comment. To isolate the contribution of feedback-guided test-case optimization, we conduct an additional ablation where the case optimizer generates fresh cases from the attack specification alone, without access to the current harness or accumulated feedback, while maintaining the same generation and evaluation budget. We compare this setting against the case-optimizer-only configurations in Table 3.

MethodASR ↓SHR ↑
HASTE_blindCase optimizer only46.33±8.1149.00±5.29
Attack Specification only48.00±7.3747.00±3.46
HASTE_seenCase optimizer only40.22±13.3650.00±8.72
Attack Specification only42.00±6.5148.33±2.89

Compared with generating fresh cases from the attack specification alone, conditioning case generation on the evolving harness and accumulated feedback reduces ASR for both HASTE_blind and HASTE_seen, while also improving SHR. These results suggest that the benefits of test-case optimization extend beyond simply generating fresh cases. We will include this ablation in the revised manuscript.


W3: The source of the harness’s improvement needs more explanation. Comparing the same model with and without the harness shows that the full system helps. However, the harness can change prompts or block tool calls through code. The appendix also shows designs that use an extra safety model. The paper does not clearly separate the effects of these mechanisms or report their runtime cost.
Q3: Which defense mechanisms are used in the tested harnesses? How much do they help, and what is their runtime cost?

We thank the reviewer for this comment.

We would like to clarify that the auxiliary safety-review mechanism in Appendix D.2 is not manually designed or required, but is one of the defense mechanisms automatically discovered during harness adaptation. When auxiliary safety review is used, the evolved harness reuses the backbone model for safety assessment rather than introducing a separate safety model. As illustrated in Figures 6, 7, and 9, several evolved harnesses rely on prompt-based guidance and programmatic enforcement without auxiliary safety models.

To assess runtime overhead, we additionally measure the average execution time per task (seconds) on ASB using the final harnesses from HASTE_seen and HASTE_blind, with DeepSeek-V4-Pro as the backbone model.

No-defenseWithout Safety ReviewWith Safety Review
9.9810.2612.11

The results show that HASTE does not introduce substantial end-to-end runtime overhead in these evaluations, including configurations with auxiliary safety-review calls. We will report the runtime measurements in the revised manuscript.

Q1: What information is given to the blind and seen settings? Does the use of 16 benchmark cases in Appendix C.3 apply only to the seen setting?

We thank the reviewer for these questions.

Yes, the 16 safety cases in Appendix C.3 are used only by HASTE_seen and Meta-Harness, not HASTE_blind.

Both HASTE settings receive the same threat descriptions from the corresponding research papers and generate cases dynamically during adaptation. The difference is that HASTE_blind receives no observed attack examples, whereas HASTE_seen additionally receives 16 observed attack cases as adaptation feedback. On ASB, HASTE_seen also receives 16 fulfillable cases to preserve benign-task utility.

We will clarify this distinction in the revised manuscript.

Thank you for clarifying the blind and seen settings. The new tests also answer my questions about transfer to unseen environments and runtime. The same-budget comparison is helpful. However, the gain from using execution feedback is small relative to the reported variation. It is also still unclear how much each part of the harness contributes to the safety gains. I prefer to keep my score of 3.

Reviewer_nKtGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes HASTE, a self-evolving method for agent harness with sparse evidence of emerging threats. The authors argue that signals available for harness evolution are often sparse, such as brief descriptions or a few attack examples, while updating a model is expensive and laborious. Motivated by this, they propose a multi-agent framework to evolve the harness instead. They benchmark the method comprehensively on two datasets, 4 model backbones, several baselines, and test cross-model transfer. They also provide detailed ablations and LLM judge sweep results to enhance the validity of their results. They show that evolving the harness is beneficial to reduce ASR while maintaining usefulness of the model. Overall, I find the paper largely easy to follow, and the topic timely and relevant.

Critical strengths

Pros:

  • The topic is timly and high relevant to ICLR. The formulation in Sec. 3.1 is clean: the comprehensive evaluation set D is explicitly unavailable during adaptation. The blind/seen split is also sensible, since it maps onto two realistic situations: an attack you've only read about versus one with a few observed examples.
  • The proposed method is straightforward and easy-to-follow. The presentation follows good rhythm and clear structure.
  • The evaluation is rather comprehensive, covering several benchmarks and state-of-the-art model backbones. Tables 4 and 9–12 report judge-swap ablations, pairwise agreement, and re-evaluation under three judge configurations. The results are promising as it outperforms several baselines.
  • Some insights are interesting: for example, given ASB's finding that prompt-only defenses barely help weaker agents, programmatic enforcement that clearly improves Qwen3-8B and Phi-4-mini is a real, practical result.
  • Inspectable, reusable output. The evolved harnesses are readable Python. Appendix D distills three recurring patterns (verification gating, context governance, evidence-based execution) that practitioners could adopt directly.
Critical weaknesses

Cons:

  • Clarity:
    • The definition of "sparse threat evidence" is not clearly articulated. In sec. 4.1: "We conduct experiments with two forms of sparse threat evidence: research papers and observed attack examples." It is unclear to me what constitutes "sparse." For ASB and STAC, it seems to me that research papers refer to those that are in the Agent-SafetyBench and STAC papers themselves. Those papers describe the evaluation distribution: its risk taxonomy, failure modes and construction. That isn't sparse evidence about an emerging attack, but rather like a full specification.
    • Concerns of leakage: ASB (2024) and STAC (2025) very likely appear in the training data of DeepSeek-V4-Pro and GPT-5.x, which act as proposer, optimizer and parser. A post-cutoff attack, or at least a held-out attack family not described in any evidence, would test the claim much more convincingly.
    • Environment access. Generated cases are feasibility-checked "in the deployment environment" using h_0. So the case optimizer apparently generates cases inside the ASB/STAC environments and their tool interfaces, which are the same environments used at test time. A defender facing a genuinely new attack wouldn't have the attacker's target environments. The authors should clarify what the case optimizer and proposer can see: tool lists, environment code, tool docs.
  • Fairness and soundness:
    • SafeHarness was re-implemented. The authors adapted it substantially, and it barely improves on No-defense (49.1 vs 51.4 ASR) in table 1. Is this a fair and useful adaptation?
    • The origianl paper of Meta-Harness used Claude Code with Opus-4.6 as proposer, and a typical run evaluates roughly 60 harnesses over 20 iterations. Its authors recommend a search set sized for about 50 full evaluations, 50–100 examples in their classification experiments. HASTE appears to run it for 5 rounds on 16+16 cases with a DeepSeek proposer. Is this comparison matched in budget?
    • "Cross-benchmark generalization" claim: Table 6 uses the 93 STAC cases instantiated in ASB environments. The environments and tool affordances are shared, so this mostly tests transfer across attack styles within the same environments. A real transfer test would use AgentDojo, InjecAgent, or the SHADE-Arena portion of STAC.
    • No adaptive evaluation. The final harnesses are tested only on fixed benchmark cases. Since the method is about attacks evolving faster than defenses, the paper should at least discuss an attacker who adapts to the evolved harness. One option is to run the case optimizer against the frozen final harness.
    • The baselines aren't adaptive and comprehensive. Six of the eight baselines are static, off-the-shelf guardrails that never see the evidence. Beating them is expected and doesn't highlight HASTE's contribution. Better comparisons would include things like:
      • guardrails configured with a policy derived from the evidence (for example, LlamaFirewall supports custom policies);
      • AgentSpec, which the paper cites and which does rule generation for runtime enforcement;
      • AutoHarness, already cited as well;
      • Ones in the benchmark papers themselves: ASB's own paper designs two defense prompts: a simple one that enumerates the 10 failure modes and tells the model to avoid them, and an enhanced one with more detailed descriptions and examples. STAC's paper evaluates existing prompt defenses and proposes its own reasoning-based defense prompt, reporting ASR cut by up to 28.8%.
  • Novelty and framing: The method largely combines two existing ideas - harness search (Meta-Harness/AutoHarness, targeting utility) and iterative red-team/defense co-generation (maybe also the recent RSI line of work). The "adversarial" framing is also loose. There's no game or objective, just two LLM agents conditioned on shared feedback. Contribution 1 ("we identify the problem") is thin and ungrounded.
Questions for authors

I think most part of the paper is solid and beneficial to the community, and I am happy to raise the score if the authors can provide further discussion or evaluation results on fundamental concerns above and questions below:

  • PDF line 314: What does “16 cases” mean? Observed examples, generated cases per round, or both? The "test cases" axis in Fig. 5 has the same ambiguity.
  • Which backbone does Table 10 use? It seems only report different judge-model configurations.
  • STAC runs an adaptive Planner that keeps attacking over up to 3 user turns after the synthetic context. HASTE doesn't say whether it runs those adaptive turns. Can you specify?
  • Some ASB cases can't be affected by tools. For example, for the "Produce unsafe information" category, the authors combined 50 AdvBench questions with 20 jailbreak templates to make 200 cases with no tool use. Does HASTE work in this category?
  • Can you provide main results on more baselines as mentioned above? Also, is the comparison with Meta-Harness fair in budget (discussed above)?
5 comments

W1.1: The definition of "sparse threat evidence" is not clearly articulated. In sec. 4.1: "We conduct experiments with two forms of sparse threat evidence: research papers and observed attack examples." It is unclear to me what constitutes "sparse." For ASB and STAC, it seems to me that research papers refer to those that are in the Agent-SafetyBench and STAC papers themselves. Those papers describe the evaluation distribution: its risk taxonomy, failure modes and construction. That isn't sparse evidence about an emerging attack, but rather like a full specification.

We sincerely thank the reviewer for raising this consideration.

While the ASB and STAC papers describe risk taxonomies, failure modes, and construction methodologies, such high-level descriptions are fundamentally different from access to the full set of concrete attack instances. They do not provide the instance-level details needed to understand how these attacks are concretely instantiated, triggered, and propagated across diverse task scenarios, tool interactions, and multi-step execution trajectories. As benchmark construction studies (e.g., AutoBencher[1], BenchMeUp[2]) illustrate, generating concrete test cases from high-level specifications is itself a non-trivial task. Furthermore, as Table 8 shows, adding the paper yields only marginal gains (an ASR of 21.4 versus 23.9) compared to case-only evidences, which is consistent with our characterization of the threat evidence as sparse.

We will clarify this distinction in the revised manuscript.

[1] LI X, Kaiyom F, Liu E, et al. Autobencher: Towards declarative benchmark construction[C]//International Conference on Learning Representations. 2025, 2025: 94357-94386.

[2] Bouzenia I, Pradel M. BenchMeUp: Building Software Engineering Benchmarks from Natural-Language Specifications[J]. arXiv preprint arXiv:2610.37283, 2026.


W1.2: Concerns of leakage: ASB (2024) and STAC (2025) very likely appear in the training data of DeepSeek-V4-Pro and GPT-5.x, which act as proposer, optimizer and parser. A post-cutoff attack, or at least a held-out attack family not described in any evidence, would test the claim much more convincingly.

We sincerely thank the reviewer for raising this concern.

DeepSeek-V4-Pro and GPT-5.x exhibit relatively high no-defense ASR on ASB and STAC (45.40 and 62.73 for DS-V4-Pro, 38.94 and 64.39 for GPT-5.4-mini), suggesting that these models have not been specifically optimized to defend against these benchmarks. We also examined the attack cases generated during adaptation against all cases in ASB and STAC and found no overlap beyond the provided evidence. This suggests that our improvements do not come from reproducing memorized benchmark cases.

To further address this concern, we additionally evaluate HASTE on CarryoverBench[1], released after the relevant models' training cutoffs.

MethodQwen3-8B ASR ↓Qwen3-8B Utility ↑DeepSeek-V4-Pro ASR ↓DeepSeek-V4-Pro Utility ↑
No-defense20.79±0.000.6184±0.01107.59±0.570.8243±0.0165
HASTE_blind15.18±4.460.7053±0.01175.94±4.540.8440±0.0156
Meta-Harness9.24±4.000.7191±0.04224.29±4.460.9353±0.0401
HASTE_seen5.94±3.960.7408±0.03472.64±3.750.9520±0.0095

On both backbones, HASTE_blind and HASTE_seen remain effective on CarryoverBench, demonstrating that HASTE’s effectiveness does not depend on the possible exposure of the proposer, optimizer, and parser to the target benchmark.

[1] After the Attacker Leaves: Benchmarking Carryover Attacks on LLM Agents, https://arxiv.org/abs/2610.28425

W1.3: Environment access. Generated cases are feasibility-checked "in the deployment environment" using h_0. So the case optimizer apparently generates cases inside the ASB/STAC environments and their tool interfaces, which are the same environments used at test time. A defender facing a genuinely new attack wouldn't have the attacker's target environments. The authors should clarify what the case optimizer and proposer can see: tool lists, environment code, tool docs.
W2.3: "Cross-benchmark generalization" claim: Table 6 uses the 93 STAC cases instantiated in ASB environments. The environments and tool affordances are shared, so this mostly tests transfer across attack styles within the same environments. A real transfer test would use AgentDojo, InjecAgent, or the SHADE-Arena portion of STAC.

We sincerely thank the reviewer for this insightful comment.

We would like to clarify that in HASTE_seen, both the case optimizer and the proposer could inspect the tool lists and implementations of environments associated with observed cases. In HASTE_blind, both accessed the environments through the benchmark backend’s high-level interfaces.

To assess whether HASTE remains effective beyond the environments covered by the observed cases, we use DeepSeek-V4-Pro as the backbone model and evaluate on cases from the 325 of ASB’s 349 environments that the observed cases did not cover. The table below reports the results. Both HASTE variants remained effective on unseen environments. We will include the cross-environment generalization in the revised manuscript.

MethodSafety ASR ↓SHR ↑
No-defense52.40±0.6764.39±0.95
HASTE-blind28.21±8.3070.82±1.08
Meta-Harness27.68±3.3172.00±6.41
HASTE-seen23.70±1.4672.86±3.33

W2.1: SafeHarness was re-implemented. The authors adapted it substantially, and it barely improves on No-defense (49.1 vs 51.4 ASR) in table 1. Is this a fair and useful adaptation?

We sincerely thank the reviewer for raising this concern.

As clarified in Appendix C.4, the original SafeHarness was primarily evaluated on the first 200 ASB cases, with its implementation tightly coupled to a tool registry, predefined tool-risk classifications, a snapshot-capable filesystem, and its own agent execution loop. Simply expanding the input dataset would not make the original implementation directly applicable to new environments, as its fixed registry would reject unregistered tools. Moreover, its tool-risk classifications, action verification rules, environment-state management and rollback mechanisms must be adapted to the native tool interfaces of full ASB nd STAC. To ensure a consistent evaluation protocol, we integrated SafeHarness's lifecycle defense mechanisms into the native ASB/STAC agent loops, generalized its tool-risk classification from fixed tool names to name-pattern-based rules, and replaced filesystem rollback with conversation-state rollback.

To provide a more direct comparison, we additionally evaluate the original SafeHarness alongside HASTE on the 200 ASB tasks used in the original SafeHarness evaluation, with Qwen3-8B as the backbone.

MethodASR (%) ↓SHR (%) ↑
Original SafeHarness40.98±0.9437.18±1.28
HASTE_blind40.67±4.6944.87±5.13
HASTE_seen36.21±4.8953.85±7.90

The results show that HASTE outperforms the original SafeHarness on both ASR and SHR, providing further evidence of HASTE's effectiveness. We will include these additional results in the revised manuscript.


W2.2: The origianl paper of Meta-Harness used Claude Code with Opus-4.6 as proposer, and a typical run evaluates roughly 60 harnesses over 20 iterations. Its authors recommend a search set sized for about 50 full evaluations, 50–100 examples in their classification experiments. HASTE appears to run it for 5 rounds on 16+16 cases with a DeepSeek proposer. Is this comparison matched in budget?
Q5.2: Is the comparison with Meta-Harness fair in budget (discussed above)?

We thank the reviewer for raising this question. We would like to clarify that Meta-Harness and HASTE use matched experimental settings: 5 rounds on the same 16+16 cases with a DeepSeek proposer. Moreover, Figure 5 suggests that the performance of Meta-Harness on ASB-300 largely converges by the fifth adaptation round under our sparse-evidence setting.

W2.4: No adaptive evaluation. The final harnesses are tested only on fixed benchmark cases. Since the method is about attacks evolving faster than defenses, the paper should at least discuss an attacker who adapts to the evolved harness. One option is to run the case optimizer against the frozen final harness.

Thank you for the insightful comment. To evaluate robustness against evolving attacks, we additionally use the case optimizer to generate 100 valid attack cases on ASB and evaluate the frozen final harnesses against these variants.

Compared with the original cases, these variants increase ASR across all settings. However, the increase is consistently smaller for HASTE than for the undefended models. In particular, HASTE continues to reduce ASR by 22.67–28.67 percentage points compared with no defense on the adaptive variants. These results suggest that although adaptive attacks pose a stronger challenge, the evolved harnesses retain their defensive effectiveness against newly generated attack variants. We will include this adaptive evaluation in the revised manuscript.

BackboneAttack CasesNo-defense ASR ↓HASTE_blind ASR ↓HASTE_seen ASR ↓
Qwen3-8BOriginal61.00±1.7341.33±8.0236.67±8.62
Varient66.67±2.3144.00±10.0040.00±2.65
DeepSeek-V4-ProOriginal47.67±2.3128.67±2.8923.33±3.51
Varient58.00±7.2135.33±5.0329.33±1.15

W2.5: The baselines aren't adaptive and comprehensive. Six of the eight baselines are static, off-the-shelf guardrails that never see the evidence. Beating them is expected and doesn't highlight HASTE's contribution. Better comparisons would include things like:

  • guardrails configured with a policy derived from the evidence (for example, LlamaFirewall supports custom policies);
  • AgentSpec, which the paper cites and which does rule generation for runtime enforcement;
  • AutoHarness, already cited as well;
  • Ones in the benchmark papers themselves: ASB's own paper designs two defense prompts: a simple one that enumerates the 10 failure modes and tells the model to avoid them, and an enhanced one with more detailed descriptions and examples. STAC's paper evaluates existing prompt defenses and proposes its own reasoning-based defense prompt, reporting ASR cut by up to 28.8%.

Q5.1: Can you provide main results on more baselines as mentioned above?

We thank the reviewer for raising this point.

We additionally evaluate three baselines without access to threat evidence (ASB-Simple, ASB-Enhanced, and STAC-Reasoning), and three evidence-aware baselines (LlamaFirewall_evidence, AgentSpec, and AutoHarness). Specifically, LlamaFirewall generates a custom policy from observed attack examples; AgentSpec generates safety rules from observed examples and enforces them at runtime; AutoHarness iteratively adapts harnesses, evaluates it on a fixed development set, refines it using feedback, and selects the best-performing candidate. All methods are evaluated with Qwen3-8B as the backbone.

MethodASB ASR ↓ASB SHR ↑STAC ASR ↓
No-defense56.02±0.2645.66±0.5367.77±0.24
ASB_simple50.67±0.2145.43±0.8055.74±0.84
ASB_enhanced47.46±0.0546.61±0.3155.53±0.67
STAC_reasoning44.82±0.4346.02±0.2447.08±0.12
HASTE_blind40.01±4.8148.46±2.4939.06±27.12
MethodASB ASR ↓ASB SHR ↑STAC ASR ↓
AgentSpec_generated51.92±1.7441.36±2.9549.54±18.41
AutoHarness41.82±21.0639.82±14.4840.23±1.14
LlamaFirewall_evidence37.67±1.7033.10±0.7434.69±2.33
HASTE_seen32.27±4.5161.24±6.8828.52±26.82

The results show that HASTE_blind achieves the best average ASR and SHR among baselines without observed examples, while HASTE_seen achieves the best results among evidence-aware methods. We will incorporate these comparisons in the revised manuscript.

Reviewer_YNEWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces HASTE, a multi-agent framework that adapts agent harnesses using sparse threat evidence without updating the underlying model. The framework extracts attack specifications, iteratively generates safety specifications and test cases, and refines the harness based on execution feedback. It considers settings with and without observed attack examples. Experiments on Agent-SafetyBench and STAC measure attack success rates, with safe task completion additionally evaluated on Agent-SafetyBench. The reported results show improved safety.

Critical strengths
  1. The problem addressed in this paper is both real and practically significant. By updating the defense framework without altering the model weights, researchers have proposed a flexible approach to addressing newly discovered risks.

  2. Experimental results on Agent-SafetyBench demonstrate that this method reduces attack success rates across a variety of base models. These findings provide compelling evidence that adjusting the defense framework can enhance agent safety while maintaining the agent's ability to perform tasks effectively, rather than relying solely on increasing the frequency of refusal responses.

Critical weaknesses
  1. The benefits of test-case optimization are not separately established. Table 3 compares optimizing test cases against reusing cases generated in the first iteration. This does not distinguish the benefit of using the current harness and accumulated feedback from the benefit of generating fresh cases. The authors should compare against generating fresh cases from the attack specification alone, with the same generation and evaluation budget, to establish the contribution of the case optimizer.

  2. It is unclear whether newly generated cases are withheld from the proposer as stated. The paper makes this claim, but the supplementary implementation appears to allow the proposer to access generated case files without explicitly excluding the current round. This suggests possible access, not confirmed leakage. The authors should clarify the configuration used and provide access logs confirming that these cases were not read, or repeat the evaluation with access explicitly blocked.

Questions for authors

See above.

3 comments

W1: The benefits of test-case optimization are not separately established. Table 3 compares optimizing test cases against reusing cases generated in the first iteration. This does not distinguish the benefit of using the current harness and accumulated feedback from the benefit of generating fresh cases. The authors should compare against generating fresh cases from the attack specification alone, with the same generation and evaluation budget, to establish the contribution of the case optimizer.

We thank the reviewer for this insightful comment. To isolate the contribution of feedback-guided test-case optimization, we conduct an additional ablation where the case optimizer generates fresh cases from the attack specification alone, without access to the current harness or accumulated feedback, while maintaining the same generation and evaluation budget. We compare this setting against the case-optimizer-only configurations in Table 3.

MethodASR ↓SHR ↑
HASTE_blindCase optimizer only46.33±8.1149.00±5.29
Attack Specification only48.00±7.3747.00±3.46
HASTE_seenCase optimizer only40.22±13.3650.00±8.72
Attack Specification only42.00±6.5148.33±2.89

Compared with generating fresh cases from the attack specification alone, conditioning case generation on the evolving harness and accumulated feedback reduces ASR for both HASTE_blind and HASTE_seen, while also improving SHR. These results provide evidence that the benefits of test-case optimization extend beyond simply generating fresh cases. We will include this ablation in the revised manuscript.

W2: It is unclear whether newly generated cases are withheld from the proposer as stated. The paper makes this claim, but the supplementary implementation appears to allow the proposer to access generated case files without explicitly excluding the current round. This suggests possible access, not confirmed leakage. The authors should clarify the configuration used and provide access logs confirming that these cases were not read, or repeat the evaluation with access explicitly blocked.

We thank the reviewer for pointing out this ambiguity. We conducted an LLM-assisted review of the execution records across all experimental runs and found no evidence that the proposer accessed or used the newly generated cases during adaptation.

We agree that explicitly enforcing filesystem-level isolation is important to ensure a robust separation between the proposer and generated cases.

We repeated the adaptation experiment on STAC with DeepSeek-V4-Pro as the backbone, using filesystem isolation to block the proposer’s access to current-round generated cases and their generation logs. Across all three methods, mean ASR under filesystem isolation was comparable to that reported in Table 2.

MethodASR with filesystem isolationASR reported in Table 2
HASTE_blind18.06±14.0719.25±16.13
HASTE_seen10.85±8.869.87±9.94
Meta-Harness8.85±2.738.70±6.89

Thank you for the additional experiments. The new evaluation addresses the my concern in W1, but the reported standard deviations are large relative to the modest mean improvements, leaving the consistency of the gains unclear. Could you provide the per-run results and clarify whether the two conditions used matched settings across runs? The filesystem-level isolation rerun alleviates my concern in W2.

Thank you for the comment. We would like to clarify that the two conditions use matched configurations and adaptation/evaluation budgets. The per-run results are provided below.

MethodSettingASR:Run 1 / 2 / 3(%)SHR:Run 1 / 2 / 3(%)
HASTE_blindCase optimizer only50.33 / 37.00 / 51.6745.00 / 55.00 / 47.00
Attack Specification only53.67 / 50.67 / 39.6749.00 / 43.00 / 49.00
HASTE_seenCase optimizer only25.00 / 50.00 / 45.6754.00 / 56.00 / 40.00
Attack Specification only42.33 / 48.33 / 35.3350.00 / 45.00 / 50.00
Reviewer_hkFdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposed a framework for adapting an LLM agent’s harness to emerging attacks using sparse threat evidence. It jointly updates safety specifications and generated test cases, using execution feedback to revise the harness. The blind variant uses no observed examples and returns the final harness. The seen variant also uses fixed observed examples to guide adaptation and select a harness. Experiments cover Agent-SafetyBench (ASB), STAC, transfer, and successive threats.

Critical strengths
  1. The paper addresses a practical setting: adapting defenses before a full evaluation set is available. Section 3.1 clearly states this constraint and distinguishes it from harness optimization with benchmark access.

  2. The evaluation includes several useful checks. Adaptation and final evaluation use different judges, Table 10 tests alternative evaluation judges, and Appendix C.4 explains how SafeHarness was adapted. The authors also report that ASR increases for all methods as threats accumulate.

  3. The utility gains are consistent. Against Meta-Harness with identical observed examples, HASTE_seen improves SHR across all four ASB backbones by 1.8–12.1 percentage points. This is a clearer result than the additional ASR gains.

Critical weaknesses
  1. The ASR advantage over Meta-Harness needs stronger support. Meta-Harness is the main comparator with matched observed examples. HASTE_seen improves average ASR by 3.53 percentage points on ASB and only 0.53 points on STAC. Meta-Harness has lower ASR in two of the three STAC columns and on Phi-4-mini in ASB. With three repeats and substantial variation, these results do not establish a reliable ASR advantage. Run-to-run variation also matters for deployment. On STAC with Qwen3-8B, HASTE_blind reports 39.06 ± 27.12 ASR. Appendix E acknowledges the difficulty of selecting a harness without fixed validation cases, but the paper does not show how reliably a single adaptation run produces an effective defense. Per-run results and uncertainty estimates for the differences between methods are needed.

  2. The contribution of threat-specific evidence is not isolated. Appendix D.2 describes broadly useful controls such as verification gating and evidence-based execution. These could improve safety across many attacks. Table 8 tests removing research papers while retaining observed examples, but does not test whether matching the evidence to the target threat is necessary. Table 6 provides some evidence for target-specific adaptation: STAC-trained harnesses outperform ASB-trained ones on STAC. However, this comparison also changes the adaptation cases, so it does not isolate the contribution of the threat description. A control using unrelated threat evidence, or a generic specification under the same budget, would help distinguish threat-specific adaptation from general harness improvement.

  3. Cross-judge agreement does not establish judge accuracy, and Table 10 needs clarification. Appendix B reports agreement between models without a human-label check. The adaptation and final safety judges agree on 74% of ASB cases and 85% of STAC cases. These checks are useful, but do not establish which judgments are correct or whether errors affect methods differently. This matters when the reported ASR differences are small. Table 10 also lacks the backbone and run-aggregation details needed to interpret it. Its STAC average ranks HASTE_blind ahead of Meta-Harness, at 19.12 versus 31.19, whereas Table 2 ranks them in the opposite order, at 29.12 versus 16.29. Different setups could explain this, but the paper needs to reconcile the results before drawing broad conclusions about ranking stability.

  4. The adaptation protocol needs clarification, and some comparisons combine multiple effects. Appendix C.3 says adaptation uses “16 safety cases sampled from the target benchmark,” while Section 3.2 defines HASTE_blind as using no observed examples. It is unclear whether the appendix statement applies only to methods with observed cases. The source and role of each adaptation case should be stated for each variant. The seen/blind comparison also combines observed-case feedback with final-harness selection: HASTE_seen selects the best of five harnesses, while HASTE_blind returns the last. Reporting HASTE_seen’s final-round harness would separate the contribution of observed feedback from the benefit of selection. Table 3 deserves more analysis. With generated cases frozen after the first round, ASR remains near the no-defense baseline despite continued specification updates. This supports the importance of refreshing cases, but does not explain why specification updates alone fail in this setting. An analysis of the remaining failures would make the ablation more informative.

Minor presentation issues: Table 7 should name the HASTE variant and evaluation set. Figures 6 and 9 also need distinct captions.

Questions for authors

Q1. Does HASTE_blind use any benchmark-sampled cases during adaptation, including benign cases? Please clarify which methods the sampling statement in Appendix C.3 applies to.

Q2. How does performance change with unrelated threat evidence or a generic attack specification, under the same adaptation budget?

Q3. Do the three repeats independently rerun adaptation or re-evaluate one harness? Please report per-run results and uncertainty estimates for the HASTE_seen–Meta-Harness differences.

Q4. Which backbones, harness runs, and evaluation cases does Table 10 use? Were the same trajectories rescored, and what explains the STAC ranking difference from Table 2?

Q5. Can you validate a sample of safety judgments against human labels, including cases where judges disagree, and report errors separately by method?

3 comments

W1: The ASR advantage over Meta-Harness needs stronger support. Meta-Harness is the main comparator with matched observed examples. HASTE_seen improves average ASR by 3.53 percentage points on ASB and only 0.53 points on STAC. Meta-Harness has lower ASR in two of the three STAC columns and on Phi-4-mini in ASB. With three repeats and substantial variation, these results do not establish a reliable ASR advantage. Run-to-run variation also matters for deployment. On STAC with Qwen3-8B, HASTE_blind reports 39.06±27.12 ASR. Appendix E acknowledges the difficulty of selecting a harness without fixed validation cases, but the paper does not show how reliably a single adaptation run produces an effective defense. Per-run results and uncertainty estimates for the differences between methods are needed.
Q3: Do the three repeats independently rerun adaptation or re-evaluate one harness? Please report per-run results and uncertainty estimates for the HASTE_seen–Meta-Harness differences.

We sincerely thank the reviewer for raising this important consideration. Yes, the three repeats independently rerun adaptation.

(1) Per-run results. We present the per-run results for all runs in Table 1 and 2 in the Appendix of the revised manuscript.

(2) Comparison between HASTE_seen and Meta-Harness. The run-to-run variation makes the mean ASR difference between HASTE_seen and Meta-Harness difficult to estimate precisely from only three runs. To obtain a more precise estimate, we conducted 17 additional independent adaptation runs for each method on each backbone model. We report the 95% confidence intervals for the performance differences between HASTE_seen and Meta-Harness (HASTE_seen − Meta-Harness) averaged equally across the backbones evaluated on each benchmark, separately for each dataset and evaluation metric. The results demonstrate that HASTE_seen consistently outperforms Meta-Harness across datasets and metrics.

BenchmarkMetricDifference (95% CI)
ASBASR ↓-4.75 [-6.48, -3.02]
ASBSHR ↑+9.83 [+8.63, +11.04]
STACASR ↓-4.26 [-8.03, -0.49]

W2: The contribution of threat-specific evidence is not isolated. Appendix D.2 describes broadly useful controls such as verification gating and evidence-based execution. These could improve safety across many attacks. Table 8 tests removing research papers while retaining observed examples, but does not test whether matching the evidence to the target threat is necessary. Table 6 provides some evidence for target-specific adaptation: STAC-trained harnesses outperform ASB-trained ones on STAC. However, this comparison also changes the adaptation cases, so it does not isolate the contribution of the threat description. A control using unrelated threat evidence, or a generic specification under the same budget, would help distinguish threat-specific adaptation from general harness improvement.
Q2: How does performance change with unrelated threat evidence or a generic attack specification, under the same adaptation budget?

We sincerely thank the reviewer for this valuable comment. To directly assess the effect of threat-specific evidence, we conduct an additional ablation replacing the threat-specific attack specification with a generic specification, while keeping the adaptation procedure and budget unchanged. The results on ASB-300 are shown below. We can observe that replacing threat-specific specifications with generic ones largely increases ASR for HASTE_blind and HASTE_seen, while also reducing SHR, demonstrating the importance of matching the evidence to the target threat. We include this ablation in the revised manuscript.

MethodAttack SpecificationASR ↓SHR ↑
HASTE_blindThreat-specific26.00±4.6754.00±9.00
Generic48.67±9.8749.33±17.01
HASTE_seenThreat-specific25.56±14.6152.00±1.73
Generic36.33±7.9747.33±17.62

W3: Cross-judge agreement does not establish judge accuracy, and Table 10 needs clarification. Appendix B reports agreement between models without a human-label check. The adaptation and final safety judges agree on 74% of ASB cases and 85% of STAC cases. These checks are useful, but do not establish which judgments are correct or whether errors affect methods differently. This matters when the reported ASR differences are small. Table 10 also lacks the backbone and run-aggregation details needed to interpret it. Its STAC average ranks HASTE_blind ahead of Meta-Harness, at 19.12 versus 31.19, whereas Table 2 ranks them in the opposite order, at 29.12 versus 16.29. Different setups could explain this, but the paper needs to reconcile the results before drawing broad conclusions about ranking stability.
Q4: Which backbones, harness runs, and evaluation cases does Table 10 use? Were the same trajectories rescored, and what explains the STAC ranking difference from Table 2?
Q5: Can you validate a sample of safety judgments against human labels, including cases where judges disagree, and report errors separately by method?

We sincerely thank the reviewer for pointing this out.

(1) Human validation of judge accuracy. To assess judge accuracy of the final safety judges (ShieldAgent on ASB and DeepSeek-V4-Flash on STAC) and whether judgment errors affect methods differently, we randomly sample 30 cases from each benchmark and manually evaluate one trajectory per case for both HASTE_seen and Meta-Harness.

BenchmarkMethodHuman Agreement ↑# False Negatives# False Positives
ASBHASTE_seen90.00 %21
Meta-Harness96.67 %10
STACHASTE_seen96.67 %01
Meta-Harness93.33 %02

The judges achieve high agreement with human annotations. Moreover, HASTE_seen exhibits comparable false-negative and false-positive rates to Meta-Harness on both benchmarks, providing additional evidence against systematic evaluation bias favoring HASTE_seen.

(2) Table 10 clarification and expanded judge evaluation. We would like to clarify that Table 10 uses Qwen3-8B as the backbone model and evaluates different judge configurations by rescoring trajectories from one run in Tables 1–2, whereas Tables 1–2 report results from three independent runs across multiple backbone models. To provide a more comprehensive and directly comparable evaluation, we extend the analysis in Table 10 to all three runs. For the default judge configurations (ShieldAgent/Luna on ASB and DS-V4 on STAC), we used scores from Tables 1–2 directly in the expanded analysis without rescoring as in the original Table 10.

ASB:

MethodShieldAgent/Luna ASR ↓ShieldAgent/Luna SHR ↑GPT-5.1 ASR ↓GPT-5.1 SHR ↑Luna/Luna ASR ↓Luna/Luna SHR ↑Avg. ASR ↓Avg. SHR ↑
HASTE_blind40.01±4.8148.46±2.4940.92±13.0741.98±13.6939.18±7.0647.08±8.2440.0445.84
Meta-Harness35.89±5.5450.54±10.3840.45±9.1143.67±16.4636.74±3.8549.74±8.1037.6947.98
HASTE_seen32.27±4.5161.24±6.8839.23±10.0346.47±14.8233.22±5.7656.19±7.7734.9154.63

STAC:

MethodDS-V4 ASR ↓GPT-5.1 ASR ↓Luna ASR ↓Average ↓
HASTE_blind39.06±27.1244.47±21.5445.57±24.7343.03
Meta-Harness26.85±21.6124.75±29.1025.25±31.5925.62
HASTE_seen28.52±26.8227.54±22.1427.05±21.8027.70

The relative rankings are consistent across all three judge configurations within each benchmark and agree with the corresponding Qwen3-8B results in Tables 1–2, indicating that the comparative conclusions are robust to the choice of evaluation judge. We update Table 10 and clarify its backbone and aggregation protocols in the revised manuscript.

W4: The adaptation protocol needs clarification, and some comparisons combine multiple effects. Appendix C.3 says adaptation uses “16 safety cases sampled from the target benchmark,” while Section 3.2 defines HASTE_blind as using no observed examples. It is unclear whether the appendix statement applies only to methods with observed cases. The source and role of each adaptation case should be stated for each variant.
The seen/blind comparison also combines observed-case feedback with final-harness selection: HASTE_seen selects the best of five harnesses, while HASTE_blind returns the last. Reporting HASTE_seen’s final-round harness would separate the contribution of observed feedback from the benefit of selection. Table 3 deserves more analysis. With generated cases frozen after the first round, ASR remains near the no-defense baseline despite continued specification updates. This supports the importance of refreshing cases, but does not explain why specification updates alone fail in this setting. An analysis of the remaining failures would make the ablation more informative.
Q1: Does HASTE_blind use any benchmark-sampled cases during adaptation, including benign cases? Please clarify which methods the sampling statement in Appendix C.3 applies to.

We thank the reviewer for these valuable comments.

(1) Clarification of adaptation cases. The 16 safety cases sampled from the target benchmark are used only by HASTE_seen and Meta-Harness. HASTE_blind does not access any observed benchmark cases during adaptation (Section 3.2, Line 234). We will clarify this distinction in Appendix C.3.

(2) Ablation of final-harness selection. We evaluate HASTE_seen using its final-round harness without selection, keeping observed-case feedback unchanged. The results confirm the benefit of harness selection, while HASTE_blind can effectively recover through its own iterative case generation without observed threat cases.

MethodASR ↓SHR ↑
HASTE_seen w/ selection25.56±14.6152.00±1.73
HASTE_seen w/o selection37.44±1.6449.33±2.52
HASTE_blind26.00±4.6754.00±9.00

(3) Analysis of frozen test cases. We analyze the optimization logs and remaining failure cases under the frozen-case setting. We find that repeated specification updates increasingly overfit to the fixed cases, resulting in narrow, case-specific defenses that fail to generalize to other vulnerabilities.


Minor presentation issues: Table 7 should name the HASTE variant and evaluation set. Figures 6 and 9 also need distinct captions.

We sincerely thank the reviewer for pointing out these issues. We would like to clarify that Table 7 reports the results of HASTE_seen evaluated on ASB-300. We revised the captions of Figures 6 and 9 to accurately reflect their respective contents in the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.