acceptodds
Under review as a conference paper at ICLR 2027

HARDE: Optimizing Agent Harnesses for Runtime Risk Detection and Execution Control

Abstract

Large language model (LLM) agents are vulnerable to safety risks such as injected malicious instructions or misleading information, motivating runtime defenses that prevent unsafe action in execution across diverse risks while preserving benign-task utility. Existing system-level defenses either focus on risk detection rather than timely prevention or rely on predefined rules with limited flexibility across diverse risks. We propose a risk-aware harness that integrates LLM-based monitoring for flexible risk detection and structures monitor-guided execution around three core modules—trigger, monitor, and feedback—enabling targeted safety interventions while limiting disruption to benign task execution. To adapt the harness to different risks and deployment settings, we introduce HARDE, a two-stage harness optimization framework that first performs isolated probing of each module to derive an optimization guide, then uses this guide to iteratively optimize the harness based on safety and utility feedback. Experiments across three attack benchmarks show that HARDE improves runtime safety while preserving utility, outperforming manually designed harnesses and naive optimization baselines. Our analysis shows that effective runtime defense benefits from complementary safety mechanisms, attack-aware harness optimization, and harness designs matched to monitor capabilities. Our code is available at https://anonymous.4open.science/r/HARDE_Anomy-4183 .

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
risk-aware harness, self-evolution, real-time defense

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_SBs1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies runtime defense for LLM agents. It splits a risk-aware harness into three modules: trigger, monitor, and feedback. It then proposes HARDE, a two-stage framework for optimizing the harness:

  1. In the first stage, it replaces only one module at a time and checks how this affects safety and utility. The results are summarized into an optimization guide written in natural language.
  2. In the second stage, a proposer agent (claude opus 4.7) follows the guide to optimize the harness step by step. It changes only one module in each round and uses safe success to choose the best harness.
Critical strengths
  1. The problem matters in practice. The paper shows with experiments that putting an LLM monitor directly into the execution loop clearly lowers the completion rate of normal tasks.
  2. Because the work aims at real use, it looks at both safety and task recovery. It does not treat block every action as a valid defense.
Critical weaknesses
  1. The most important one: the evidence for the core motivation and the main results is not strong enough. The paper shows that changing the harness helps, but it does not show that the original problem mainly comes from the harness. There are at least two different causes:

    • The monitor judges wrongly. A normal action is judged as dangerous. This is a false positive of the detection model.
    • The monitor judges correctly, but the execution is handled badly. A dangerous action should be blocked, but the system gives no safe alternative path, so the normal task also fails. Only this case is directly a harness problem.

    The paper says the utility drop happens because the harness is not optimized, but it does not separate this from false positives of the monitor itself. With qwen3-8B on AgentDyn, utility drops to only 0.136, and the paper itself blames this on the weak monitor. The paper does not report the monitor's false positive rate. It also has no control experiment where the monitor stays fixed and only the way it is connected to the harness changes.

  2. The biggest gain comes from SHADE-Arena. But SHADE-Arena has only 17 tasks in total, 3 of which are used for search, so only about 14 are held out. The claimed 77.9% relative gain over the strongest baseline is based on such a small sample. The paper also reports only the mean of three runs.

  3. On AgentDyn, which has more samples, HARDE has only a small advantage over the one-shot claude opus 4.7 harness (0.935 vs. 0.912) and over Meta-Harness (0.947 vs. 0.940 with the qwen monitor).

  4. The method is not very new, and the insights lack quantitative support. HARDE is basically a version of Meta-Harness for the safety setting. It mainly adds two mechanisms: changing one module at a time, and probing first and then writing a guide. Both are common search tricks.

Questions for authors
  1. How much of the utility drop of Vanilla Monitor comes from wrong judgments by the monitor, and how much comes from poor handling by the harness? Specifically:
    • In the cases where the normal task fails, how many of the blocked actions are actually normal actions (false positives)? And how many should be blocked, but the system gives no alternative path?
    • Can you report the monitor's false positive rate on normal steps?
    • Can you run a control experiment where the monitor stays fixed and only the way the harness handles its decisions changes?
  2. How many held-out cases are there in SHADE-Arena? Can you report the standard deviation over the three runs?
  3. On AgentDyn, the gap between HARDE and the one-shot opus 4.7 harness, and between HARDE and Meta-Harness, is small (0.935 vs. 0.912, and 0.947 vs. 0.940). Is this gap statistically significant? Since HARDE needs many rounds of evaluation, is its extra cost worth it?
8 comments
AuthorsOpenReview ↗

Response to Reviewer SBs1. – Part 1

Weakness 1 & Question 1: Distinguishing Monitor Detection Errors from Harness Execution Failures

We sincerely appreciate the reviewer’s thoughtful feedback on this central issue. We agree that distinguishing detection errors from execution and recovery failures is essential for understanding the sources of utility degradation. Below, we first clarify the relationship between the monitor model and the monitor policy, and then address each question with additional experimental evidence.

(1) Clarification of Monitor Model vs. Monitor Policy

We believe there may be a potential misunderstanding regarding the distinction between monitor false positives and harness execution failures, which we would like to clarify.

Specifically, the underlying monitor model and the monitor policy are not equivalent. The monitor policy is an integral part of the risk-aware harness, rather than an independent component whose errors are unrelated to harness design. Even with identical model weights, detection behavior can vary substantially depending on how the harness constructs and presents execution context, when monitoring is invoked, and how previous interventions affect subsequent agent behavior.

Therefore, the two failure modes identified by the reviewer—incorrect risk judgments and inadequate recovery after correct detection—should not be treated as entirely independent of harness design. Both can be influenced by the design and interaction of harness components.

Importantly, HARDE does not aim merely to compensate for a weak monitor model or improve feedback in isolation. Instead, while keeping the underlying monitor model fixed, HARDE jointly optimizes the trigger, monitor, and feedback policies to improve both risk assessment and execution control.

We provide additional quantitative evidence below to support this claim and address the reviewer's specific questions.

(2) False Positive Rate Analysis

To investigate the reviewer's concern regarding the substantial utility degradation with Qwen3-8B on AgentDyn, we conduct an additional false positive rate (FPR) analysis under three settings:

  • Standalone Qwen3-8B Monitor: We collect trajectories generated by the Base Harness without any monitoring intervention. Qwen3-8B then independently evaluates each proposed action and predicts whether it should be allowed or blocked. This setting measures the model's internal detection behavior on trajectories.
  • Vanilla Monitor: Qwen3-8B is integrated into the harness as a monitor policy, corresponding to the setting where utility drops to 0.136. At each execution step, the harness invokes Qwen3-8B to examine the complete execution prefix and determine whether to allow or block the proposed action, without any harness optimization.
  • HARDE: We evaluate the optimized risk-aware harness, where trigger, monitor, and feedback policies are jointly optimized.

For each setting, we randomly sample 50 execution trajectories and use DeepSeek-V4.1-Flash as a proxy for human annotation to determine whether each evaluated action is benign or malicious. We calculate FPR as (FP/(FP+TN)), where FP denotes benign actions incorrectly blocked and TN denotes benign actions correctly allowed.

SettingFPTNFPR ↓
Standalone Qwen3-8B Monitor2265181.59%
Vanilla Monitor (Qwen3-8B)7427990.38%
HARDE (Qwen3-8B)344277.38%

Table 1: False positive analysis

First, the comparison between the standalone monitor and Vanilla Monitor illustrates why the monitor model and monitor policy should be distinguished. When Qwen3-8B is integrated into an unoptimized harness, the observed FPR increases from 81.59% to 90.38%, accompanied by a substantial increase in the number of FP.

One important difference is that the monitor's previous decisions can influence subsequent execution contexts. For example, after an action is blocked, the agent may attempt to resume its legitimate task. However, under an unoptimized harness, the subsequent monitor may continue to be influenced by the preceding blocked actions and recovery attempts, potentially resulting in repeated blocking.

Second, HARDE substantially reduces the observed FPR with the same underlying Qwen3-8B monitor model. By jointly optimizing the trigger, monitor, and feedback policies, HARDE invokes monitoring when necessary, provides more relevant and refined context for risk assessment, and guides agents back toward legitimate tasks to avoid repeated blocking.

These results demonstrate that monitoring errors are not solely determined by the underlying model but are also influenced by how it is integrated into the harness--monitor policy. Thus, harness optimization can mitigate both monitor false positives and failures to provide effective recovery paths.

AuthorsOpenReview ↗

Response to Reviewer SBs1. – Part 2

(3) Failure Attribution Analysis

We further investigate how different failure modes contribute to unsuccessful benign-task execution.

Specifically, we examine AgentDyn trajectories using Qwen3-8B as the monitor and focus on cases where the normal task fails (Utility = 0). Since detailed trajectory-level failure attribution is expensive, we select 30 failed cases for qualitative analysis and classify them into three mutually exclusive categories:

Failure CategoryCasesProportion
Monitor false positives only1033.3%
Correctly blocked malicious actions, but failed to recover the legitimate task930.0%
Both false positives and recovery failures1136.7%
Total30100.0%

Table 2: Failure attribution

The results indicate that incorrect monitoring judgments and inadequate recovery patch both play substantial roles in benign-task failures, rather than occurring as completely separate phenomena.

Importantly, a false positive does not necessarily imply ultimate task failure. An agent may encounter an incorrect blocking decision but subsequently recover through the feedback policy, resume legitimate execution, and successfully complete the normal task. We also observe such cases with Utility = 1.

Therefore, attributing utility degradation exclusively to either monitor detection errors or harness execution failures would overlook their interactions. Our analysis supports the view that monitor and feedback policies serve complementary functions within the harness, and their coordination is essential for reliable runtime defense.

(4) Control Experiments with a Fixed Monitor Policy

We emphasize that all our experiments keep the underlying monitor model fixed without modifying its parameters.

To further address the reviewer's request, we conduct an additional control experiment in which the monitor policy is also fixed, and only the feedback policy—that is, how the harness handles monitoring decisions and guides subsequent execution—is optimized.

MethodUtility ↑Safety ↑SS ↑
Vanilla Monitor0.2811.0000.281
Feedback Only (Fixed Monitor Policy)0.3571.0000.357
HARDE (Full)0.5000.9840.500

Table 3: Control experiment isolating feedback-policy optimization .

Optimizing only the feedback policy improves Utility and Safe Success, demonstrating that improving execution and recovery behavior can yield meaningful benefits even when the monitor's detection policy remains unchanged.

However, this configuration still underperforms the full HARDE framework, which achieves a Safe Success of 0.500. This suggests that feedback optimization alone is insufficient to fully address the challenges of runtime defense, motivating the joint optimization of trigger, monitor, and feedback policies.

Furthermore, Appendix B.3 of our original submission already provides a module ablation study examining the contributions of different harness components:

VariantUtility ↑Safety ↑SS ↑
w/o Trigger0.3570.9290.357
w/o Feedback0.2310.9230.154
w/o Trigger & Feedback0.1540.9230.154
None0.0830.9170.083
HARDE (Full)0.5000.9840.500

Table 4: Module ablation results from Appendix B.3.

The substantial performance drops observed when removing either the trigger or feedback module demonstrate that all three policies, including the monitor policy, are closely interdependent and essential to effective harness optimization.

AuthorsOpenReview ↗

Response to Reviewer SBs1. – Part 3

(5) Clarification of the Utility Drop with Qwen3-8B

We apologize that our original wording was unclear and would like to clarify two different comparisons.

First, attributing the utility drop to an unoptimized harness refers to comparing different harness configurations or methods with the same monitor model. As shown in Table 1, directly integrating the monitor into an unoptimized harness leads to severe over-blocking and utility degradation.

Second, attributing the utility of 0.136 to the weaker Qwen3-8B monitor refers to comparing Vanilla Monitor with DeepSeek-V4-Flash and Qwen3-8B on the same benchmark. Since the harness configuration is identical and only the monitor model differs, we attribute this performance gap to monitor capability. Nevertheless, harness optimization can substantially improve the overall performance even with a weaker monitor.

We apologize for the confusion caused by these two statements and will clarify this distinction in the revised manuscript.

Weakness 2 & Question 2: Limited SHADE-Arena Sample Size and Statistical Reliability of Reported Gains

We greatly appreciate the reviewer’s careful assessment of the dataset size and statistical reliability, and thank them for encouraging a more rigorous evaluation.

We acknowledge that SHADE-Arena contains only 17 cases,with 3 as search set and 14 as held-out test set, making performance potentially sensitive to data splitting. Our original results were averaged over 3 random seeds with different search/test splits. To better characterize this variability, we additionally report standard deviations across the 3 runs on SHADE-Arena with DeepSeek-V4-Flash as the monitor. Results are presented as mean ± standard deviation.

MethodUtility ↑Safety ↑SS ↑
Base Harness0.419 ± 0.0210.387 ± 0.0400.161 ± 0.026
Vanilla Monitor0.281 ± 0.0511.000 ± 0.0000.281 ± 0.051
TS-Flow0.235 ± 0.0830.706 ± 0.1270.177 ± 0.048
GPT-5.40.364 ± 0.0580.394 ± 0.1470.212 ± 0.017
Claude Opus 4.70.353 ± 0.0760.294 ± 0.1780.118 ± 0.042
Meta-Harness0.462 ± 0.0960.540 ± 0.1300.269 ± 0.083
HARDE (Ours)0.500 ± 0.1170.984 ± 0.0310.500 ± 0.117

Table 5: SHADE-Arena Results with Standard Deviations

HARDE achieves the highest mean Utility and Safe Success, along with the second-highest mean Safety among the compared methods, substantially outperforming the strongest baseline in Safe Success. To further assess the reliability of these improvements despite the small sample size, we conduct a paired bootstrap analysis with 100,000 resamples, estimating 95% confidence intervals for the Safe Success differences between HARDE and Vanilla Monitor, the strongest baseline on SHADE-Arena.

MonitorHARDE SSVanilla Monitor SSPaired Difference (95% CI)
DeepSeek-V4-Flash50.0%28.1%+21.9 pp [+3.2, +42.0]

Table 6: Paired Bootstrap Confidence Intervals

HARDE achieves a statistically significant improvement in Safe Success over the strongest baseline, with a 95% confidence interval of [+3.2, +42.0] percentage points.

We will incorporate these experimental results into the revised manuscript

Reviewer_oe5YOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

HARDE studies how to automatically optimize runtime safety defenses for LLM agents. Specifically, it divides the defense harness into three modules: Trigger, Monitor, and Feedback. Firstly, it probes individual modules to generate an optimization guide. And then, it iteratively revises one module using this guide and execution feedback. Experiments are conducted on three safety benchmarks with two monitor models to demonstrate the effectiveness.

Critical strengths
  1. This paper studies a practical problem, which jointly considers attack prevention and task completion to address the utility loss caused by excessive blocking.
  2. The three modules provide distinct intervention points, with particular attention to task recovery after blocking.
Critical weaknesses
  1. HARDE performs additional preliminary probing, while fairness is primarily justified by matching subsequent optimization iterations. The gains may therefore partly reflect additional computation. • Insufficient budget matching.
  2. It is unclear whether AgentDyn splits separate underlying user tasks. SHADE-Arena contains only 17 cases, making results sensitive to the split, implying the limited generalizability.
  3. Only means over three runs are provided, without variance or confidence intervals. It's better report more statistical details.
  4. The improvement of probing is not fully discussed. The existing ablations show that the optimization guide is useful, but don't explore whether probing-derived guidance outperforms guidance generated directly from task descriptions or initial failure trajectories.
  5. The deployment costs should be provided, including monitor calls, token consumption, and runtime latency.
Questions for authors

No further questions.

5 comments
AuthorsOpenReview ↗

Response to Reviewer oe5Y. – Part 1

Thanks for your valuable feedback. We also appreciate you for acknowledging the value of this work and all the constructive suggestions. We address your questions point by point as follows.

Weakness 1: HARDE performs additional preliminary probing, while fairness is primarily justified by matching subsequent optimization iterations. The gains may therefore partly reflect additional computation. • Insufficient budget matching.

We sincerely appreciate your thoughtful feedback on the fairness of the optimization budget comparison. We clarify the optimization budget and provide an additional budget-matched comparison.

Equal total optimization budgets:

Both HARDE and Meta-Harness perform up to 10 refinement iterations, selecting the earliest checkpoint achieving the highest search-set Safe Success (SS). We provide a detailed comparison of their optimization costs on SHADE-Arena below.

MethodTokens (Agent + Proposer)RolloutsMonitor Calls
HARDE Stage I (3 probes)32.24M9436
Meta-Harness (per iteration)20.52M3209
HARDE Stage II (per iteration)17.32M3200

Table 1: Per-iteration optimization costs

MethodTokensRolloutsMonitor CallsTest SS ↑
Meta-Harness (10 iterations)205.2M302,0930.269
HARDE (Stage I + 7 iterations)153.4M301,8360.500
Meta-Harness (13 iterations)266.8M392,7200.269
HARDE (Stage I + 10 iterations)205.4M392,4360.500

Table 2: Cumulative optimization budgets and held-out Safe Success on SHADE-Arena.

Test SS denotes the Safe Success of the best candidate selected during optimization, evaluated on the held-out test set.

Budget-matched comparison. HARDE's Stage I plus 7 Stage II iterations requires the same number of rollouts and fewer tokens and monitor calls than Meta-Harness's 10 iterations. As shown in the optimization iteration analysis in our paper, HARDE reaches its best checkpoint by iteration 3 on SHADE-Arena, after which its search-set performance has already saturated, with further refinement yielding no additional improvement in the best Safe Success (SS). Thus, reducing HARDE's Stage II budget to 7 iterations leaves the selected checkpoint and held-out performance unchanged.

Extended Meta-Harness evaluation. To further ensure fairness, we extend Meta-Harness from 10 to 13 iterations, exceeding even HARDE's full 10-iteration configuration across all three cost metrics. The additional iterations do not surpass the best search-set SS of 0.584 achieved at iteration 5. Consequently, the selected checkpoint remains unchanged, as does its held-out performance.

These results demonstrate that HARDE's advantage does not stem from a larger optimization budget, but rather from structured probing followed by guided harness optimization.

We appreciate the reviewer's valuable feedback and will incorporate these additional budget comparisons into the revised manuscript.

Weaknesses 2 & 3: AgentDyn Data Splitting, SHADE-Arena Split Sensitivity, Generalizability, and Additional Statistical Details

We sincerely appreciate the reviewer's constructive feedback on data splitting, generalizability, and statistical reliability. We provide additional experiments and statistical analyses to address these concerns.

(1) AgentDyn Task-Level Splitting. In our original experiments, AgentDyn was randomly divided into search and held-out test sets after filtering invalid samples, without explicitly separating underlying user tasks. To address this concern, we re-split AgentDyn at the user-task level and re-evaluate Meta-Harness and HARDE using DeepSeek-V4-Flash as the monitor. Other baselines do not involve optimization on search sets and are therefore unaffected by the search/test split.

MethodUtility ↑Safety ↑SS ↑
Base Harness0.9250.8690.816
Meta-Harness0.8931.0000.893
HARDE (Ours)0.9291.0000.929

Table 3: AgentDyn Results with Task-Level Splitting

HARDE continues to outperform Meta-Harness and other baselines under the stricter task-level split, supporting the generalizability of our optimization approach to unseen user tasks.

AuthorsOpenReview ↗

Response to Reviewer oe5Y. – Part 2

(2) SHADE-Arena Statistical Variability. We acknowledge that SHADE-Arena contains only 17 cases, making performance potentially sensitive to data splitting. Our original results were averaged over 3 random seeds with different search/test splits. To better characterize this variability, we additionally report standard deviations across the 3 runs on SHADE-Arena with DeepSeek-V4-Flash as the monitor. Results are presented as mean ± standard deviation.

MethodUtility ↑Safety ↑SS ↑
Base Harness0.419 ± 0.0210.387 ± 0.0400.161 ± 0.026
Vanilla Monitor0.281 ± 0.0511.000 ± 0.0000.281 ± 0.051
TS-Flow0.235 ± 0.0830.706 ± 0.1270.177 ± 0.048
GPT-5.40.364 ± 0.0580.394 ± 0.1470.212 ± 0.017
Claude Opus 4.70.353 ± 0.0760.294 ± 0.1780.118 ± 0.042
Meta-Harness0.462 ± 0.0960.540 ± 0.1300.269 ± 0.083
HARDE (Ours)0.500 ± 0.1170.984 ± 0.0310.500 ± 0.117

Table 4: SHADE-Arena Results with Standard Deviations

HARDE achieves the highest mean Utility and Safe Success, along with the second-highest mean Safety among the compared methods, substantially outperforming the strongest baseline in Safe Success.

For the other benchmarks, which contain substantially more evaluation examples than SHADE-Arena and exhibit relatively low variance across runs, we will provide additional variance statistics in the revised manuscript.

(3) SHADE-Arena Split Robustness. To further assess sensitivity to data splitting, we conduct an additional k-fold cross-validation experiment on SHADE-Arena. The results for HARDE with DeepSeek-V4-Flash as the monitor are:

MethodUtility ↑Safety ↑SS ↑
HARDE (k-fold)0.481 ± 0.1060.958 ± 0.0540.481 ± 0.106

Table 5: SHADE-Arena k-Fold Results

The k-fold results remain comparable to those obtained from the original 3 random splits, providing further evidence that HARDE's performance is not limited to a particular data split.

(4) Paired Confidence Intervals. We further conduct paired bootstrap analysis with 100,000 resamples to estimate 95% confidence intervals for the SS differences between HARDE and Vanilla Monitor, the strongest baseline on SHADE-Arena.

MonitorHARDE SSVanilla Monitor SSPaired Difference (95% CI)
DeepSeek-V4-Flash50.0%28.1%+21.9 pp [+3.2, +42.0]
Qwen3-8B35.7%23.5%+12.2 pp [0.0, +31.0]

Table 6: Paired Bootstrap Confidence Intervals

HARDE consistently achieves higher Safe Success under both monitor settings significantly.

We appreciate the reviewer's valuable suggestions and will incorporate these additional experiments and statistical analyses into the revised manuscript.

Weakness4: The improvement of probing is not fully discussed. The existing ablations show that the optimization guide is useful, but don't explore whether probing-derived guidance outperforms guidance generated directly from task descriptions or initial failure trajectories.

We sincerely appreciate the reviewer's thoughtful suggestion regarding the contribution of probing-derived guidance. We conduct an additional ablation on SHADE-Arena with DeepSeek-V4-Flash as the monitor. Specifically, we use the same proposer to generate an optimization guide based solely on task descriptions and the initial harness's failure trajectories, without module-wise probing. We then apply the same HARDE Stage II optimization procedure for a fair comparison.

MethodUtility ↑Safety ↑SS ↑
No Optimization Guide0.4290.8460.393
Task- and Failure-based Guide0.2860.8570.286
HARDE (Probing-based Guide)0.5000.9840.500

Table 7: Ablation of Optimization Guide Sources

The results demonstrate that probing-derived guidance achieves the best performance, while guidance generated directly from task descriptions and initial failure trajectories performs even worse than optimization without an explicit guide.

AuthorsOpenReview ↗

Response to Reviewer oe5Y. – Part 3

(Continuation of our response to Weakness 4.)

As discussed in our paper, effectively optimizing these coupled modules is challenging because the effects of individual modules on runtime safety and task completion remain ambiguous. Module-wise probing provides targeted empirical evidence about these effects, enabling more accurate and actionable optimization guidance. In contrast, task descriptions and initial failure trajectories alone cannot reliably identify the contributions of individual modules, potentially leading to misleading guidance that hinders subsequent optimization.

Weakness5: The deployment costs should be provided, including monitor calls, token consumption, and runtime latency.

We thank the reviewer for highlighting the importance of evaluating deployment-time efficiency. We additionally compare HARDE against Vanilla Monitor, the strongest baseline on SHADE-Arena, in terms of monitor calls and total token consumption per test case under two monitor models. (Wall-clock latency is susceptible to external factors, such as network fluctuations and server load, making reliable comparisons difficult)

MonitorMetricHARDEVanilla MonitorReduction
DeepSeek-V4-FlashMonitor Calls / Case54.8120.754.6%
Total Tokens / Case3.99M5.41M26.2%
Qwen3-8BMonitor Calls / Case90.4168.646.4%
Total Tokens / Case2.19M5.46M59.9%

Table 8: Deployment-Time Monitoring Costs

HARDE substantially reduces both monitor calls and total token consumption across the two monitor models. These efficiency gains can be attributed to HARDE's joint optimization of the trigger, monitor, and feedback modules, enabling more selective monitor invocation and more effective recovery from unsafe actions. This helps the agent resume its primary task while avoiding unnecessary monitoring and repeated execution attempts. We will include more comprehensive cost analyses in future revisions.

Comments on the paper

4 comments
AuthorsOpenReview ↗

Response to Reviewer WzuX. – Part 1

Thank you for your thoughtful comments and questions on our work. Below, we group your weaknesses and questions by topic and provide point-by-point responses in detail below.

Weakness 1 & Question 1: Budget-Matched Comparison and Additional Baseline Evaluations

We sincerely appreciate your thoughtful feedback on the fairness of the optimization budget comparison.

Overlap between probing and search examples:

For fairness, we use the same randomly sampled subset for both probing and search, ensuring that Stage I does not introduce any additional training examples.

Equal total optimization budgets:

Both HARDE and Meta-Harness perform up to 10 refinement iterations, selecting the earliest checkpoint achieving the highest search-set Safe Success (SS). We provide a detailed comparison of their optimization costs on SHADE-Arena below.

MethodTokens (Agent + Proposer)RolloutsMonitor Calls
HARDE Stage I (3 probes)32.24M9436
Meta-Harness (per iteration)20.52M3209
HARDE Stage II (per iteration)17.32M3200

Table 1: Per-iteration optimization costs

MethodTokensRolloutsMonitor CallsTest SS ↑
Meta-Harness (10 iterations)205.2M302,0930.269
HARDE (Stage I + 7 iterations)153.4M301,8360.500
Meta-Harness (13 iterations)266.8M392,7200.260
HARDE (Stage I + 10 iterations)205.4M392,4360.500

Table 2: Cumulative optimization budgets and held-out Safe Success on SHADE-Arena.

Test SS denotes the Safe Success of the best candidate selected during optimization, evaluated on the held-out test set.

Budget-matched comparison. HARDE's Stage I plus 7 Stage II iterations requires the same number of rollouts and fewer tokens and monitor calls than Meta-Harness's 10 iterations. As shown in the optimization iteration analysis in our paper, HARDE reaches its best checkpoint by iteration 3 on SHADE-Arena, after which its search-set performance has already saturated, with further refinement yielding no additional improvement in the best Safe Success (SS). Thus, reducing HARDE's Stage II budget to 7 iterations leaves the selected checkpoint and held-out performance unchanged.

Extended Meta-Harness evaluation. To further ensure fairness, we extend Meta-Harness from 10 to 13 iterations, exceeding even HARDE's full 10-iteration configuration across all three cost metrics. The additional iterations do not surpass the best search-set SS of 0.584 achieved at iteration 5. Consequently, the selected checkpoint remains unchanged, as does its held-out performance.

These results demonstrate that HARDE's advantage does not stem from a larger optimization budget, but rather from structured probing followed by guided harness optimization.

Weakness 2 & Question 3: Safety–Utility Trade-off and Safety-Constrained Selection

Thank you for highlighting this important aspect of our evaluation.

HARDE jointly optimizes safety and utility through Safe Success (SS), which serves as both the optimization objective and harness selection criterion, without imposing separate constraints on either metric.

We acknowledge that Safety is an important constraint in practical deployment. Therefore, we apply this additional constraint to HARDE, requiring Safety to remain close to that of the unoptimized Vanilla Monitor. The results on Qwen3-8B/SHADE-Arena are shown below:

MethodSafety ↑Utility ↑SS ↑
Vanilla Monitor0.9410.2940.235
HARDE (SS-maximized)0.5710.5000.357
HARDE (Safety-constrained)0.9230.3850.308

Notably, under the safety-constrained selection criterion, HARDE achieves higher Utility (0.385 vs. 0.294) and SS (0.308 vs. 0.235) than Vanilla Monitor, while maintaining comparable Safety. These results demonstrate that HARDE can achieve a better safety–utility trade-off under safety constraint, with the final balance determined by the harness selection criterion.

We appreciate your constructive feedback and will include this additional safety-constrained evaluation and discussion in the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer WzuX. – Part 2

Weakness 2 & Question 2: Robustness Against Adaptive Attacks Targeting Optimized Harness Policies

Thank you for emphasizing the importance of evaluating robustness against stronger adaptive attacks.

Our threat model focuses on external attackers manipulating agents through untrusted environmental signals containing misleading information or malicious instructions. Granting attackers full access to the internal defense policies represents a substantially stronger white-box setting that is uncommon in existing agent safety defense research and goes beyond the scope of this work. Nevertheless, we recognize the importance of evaluating robustness beyond fixed benchmark attacks and therefore conduct the following additional experiment.

We conduct an additional adaptive attack evaluation on a randomly sampled subset of AgentDyn. Specifically, we use DeepSeek-V4.1-Flash as the attacker to generate enhanced payloads targeting monitor-directed injection and feedback contamination, respectively, conditioned on the original attacks, tasks, and complete agent execution trajectories. For each attack type, the attacker generates a single enhanced payload per task without iterative refinement. We evaluate the original and both types of enhanced attacks on exactly the same task subset against the same frozen HARDE-optimized harness, without further defense optimization.

AttackUtility ↑Safety ↑SS ↑
Original0.9801.0000.980
Enhanced (Monitor-directed Injection)0.8520.9630.815
Enhanced (Feedback Contamination)0.9191.0000.919

Despite the enhanced attacks, HARDE maintains a high Safety, compared with 1.000 under the original attacks. These results provide additional evidence that HARDE's learned defenses are not limited to the original static attack payloads and retain substantial defensive effectiveness against stronger attacks targeting to the optimized harness.

Question3: Statistical Uncertainty and Runtime Monitoring Costs

Thank you for your thoughtful question regarding statistical reliability and runtime overhead.

We additionally compare HARDE against Vanilla Monitor on Qwen3-8B/SHADE-Arena, reporting Safe Success, monitor calls, and total token consumption across seeds. We use paired bootstrap with 100,000 resamples to estimate 95% confidence intervals for the differences (HARDE − Vanilla Monitor).

Wall-clock latency is susceptible to external factors, such as server load and network fluctuations, making it difficult to obtain stable and reliable measurements across runs.

MetricHARDEVanilla MonitorPaired Difference (95% CI)
Safe Success Rate35.7%23.5%+12.2 pp [0.0, +31.0]
Monitor Calls / Case90.4168.6−78.2 [−112.7, −44.1]
Total Tokens / Case2.19M5.46M−3.27M [−5.11M, −1.45M]

HARDE achieves a higher Safe Success Rate (+12.2%) while substantially reducing monitor calls (46.4%) and total token consumption (59.9%) relative to Vanilla Monitor. Due to the small evaluation set of SHADE-Arena, the results exhibit relatively high variance. Nevertheless, HARDE achieves higher SS than Vanilla Monitor, while significantly reducing both monitor calls and total token consumption across seeds.

We appreciate your valuable feedback and will add these uncertainty estimates and runtime cost analyses to the revised manuscript.

AuthorsOpenReview ↗

Looking forward to hearing from you

Dear Reviewer WzuX,

We sincerely appreciate your time and thoughtful feedback on our work.

We submitted our rebuttal responses several days ago and have also uploaded a revised PDF, with the changes highlighted in different colors for your convenience.

We would greatly appreciate any further feedback on our responses and would be happy to address any remaining questions or concerns. If our revisions have satisfactorily addressed your concerns, we would be very grateful if you could consider updating your evaluation accordingly.

Thank you again for your time and consideration.

Best regards,

The Authors

AuthorsOpenReview ↗

Looking forward to hearing from you

Dear Reviewer WzuX,

As the discussion period is approaching its end, we would greatly appreciate your feedback on our rebuttal and revised manuscript. We would be happy to address any remaining questions or concerns.

Thank you for your time and consideration.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.