- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies runtime defense for LLM agents. It splits a risk-aware harness into three modules: trigger, monitor, and feedback. It then proposes HARDE, a two-stage framework for optimizing the harness:
- In the first stage, it replaces only one module at a time and checks how this affects safety and utility. The results are summarized into an optimization guide written in natural language.
- In the second stage, a proposer agent (claude opus 4.7) follows the guide to optimize the harness step by step. It changes only one module in each round and uses safe success to choose the best harness.
Critical strengths
- The problem matters in practice. The paper shows with experiments that putting an LLM monitor directly into the execution loop clearly lowers the completion rate of normal tasks.
- Because the work aims at real use, it looks at both safety and task recovery. It does not treat block every action as a valid defense.
Critical weaknesses
-
The most important one: the evidence for the core motivation and the main results is not strong enough. The paper shows that changing the harness helps, but it does not show that the original problem mainly comes from the harness. There are at least two different causes:
- The monitor judges wrongly. A normal action is judged as dangerous. This is a false positive of the detection model.
- The monitor judges correctly, but the execution is handled badly. A dangerous action should be blocked, but the system gives no safe alternative path, so the normal task also fails. Only this case is directly a harness problem.
The paper says the utility drop happens because the harness is not optimized, but it does not separate this from false positives of the monitor itself. With qwen3-8B on AgentDyn, utility drops to only 0.136, and the paper itself blames this on the weak monitor. The paper does not report the monitor's false positive rate. It also has no control experiment where the monitor stays fixed and only the way it is connected to the harness changes.
-
The biggest gain comes from SHADE-Arena. But SHADE-Arena has only 17 tasks in total, 3 of which are used for search, so only about 14 are held out. The claimed 77.9% relative gain over the strongest baseline is based on such a small sample. The paper also reports only the mean of three runs.
-
On AgentDyn, which has more samples, HARDE has only a small advantage over the one-shot claude opus 4.7 harness (0.935 vs. 0.912) and over Meta-Harness (0.947 vs. 0.940 with the qwen monitor).
-
The method is not very new, and the insights lack quantitative support. HARDE is basically a version of Meta-Harness for the safety setting. It mainly adds two mechanisms: changing one module at a time, and probing first and then writing a guide. Both are common search tricks.
Questions for authors
- How much of the utility drop of Vanilla Monitor comes from wrong judgments by the monitor, and how much comes from poor handling by the harness? Specifically:
- In the cases where the normal task fails, how many of the blocked actions are actually normal actions (false positives)? And how many should be blocked, but the system gives no alternative path?
- Can you report the monitor's false positive rate on normal steps?
- Can you run a control experiment where the monitor stays fixed and only the way the harness handles its decisions changes?
- How many held-out cases are there in SHADE-Arena? Can you report the standard deviation over the three runs?
- On AgentDyn, the gap between HARDE and the one-shot opus 4.7 harness, and between HARDE and Meta-Harness, is small (0.935 vs. 0.912, and 0.947 vs. 0.940). Is this gap statistically significant? Since HARDE needs many rounds of evaluation, is its extra cost worth it?
8 comments
Response to Reviewer SBs1. – Part 2
(3) Failure Attribution Analysis
We further investigate how different failure modes contribute to unsuccessful benign-task execution.
Specifically, we examine AgentDyn trajectories using Qwen3-8B as the monitor and focus on cases where the normal task fails (Utility = 0). Since detailed trajectory-level failure attribution is expensive, we select 30 failed cases for qualitative analysis and classify them into three mutually exclusive categories:
| Failure Category | Cases | Proportion |
|---|---|---|
| Monitor false positives only | 10 | 33.3% |
| Correctly blocked malicious actions, but failed to recover the legitimate task | 9 | 30.0% |
| Both false positives and recovery failures | 11 | 36.7% |
| Total | 30 | 100.0% |
Table 2: Failure attribution
The results indicate that incorrect monitoring judgments and inadequate recovery patch both play substantial roles in benign-task failures, rather than occurring as completely separate phenomena.
Importantly, a false positive does not necessarily imply ultimate task failure. An agent may encounter an incorrect blocking decision but subsequently recover through the feedback policy, resume legitimate execution, and successfully complete the normal task. We also observe such cases with Utility = 1.
Therefore, attributing utility degradation exclusively to either monitor detection errors or harness execution failures would overlook their interactions. Our analysis supports the view that monitor and feedback policies serve complementary functions within the harness, and their coordination is essential for reliable runtime defense.
(4) Control Experiments with a Fixed Monitor Policy
We emphasize that all our experiments keep the underlying monitor model fixed without modifying its parameters.
To further address the reviewer's request, we conduct an additional control experiment in which the monitor policy is also fixed, and only the feedback policy—that is, how the harness handles monitoring decisions and guides subsequent execution—is optimized.
| Method | Utility ↑ | Safety ↑ | SS ↑ |
|---|---|---|---|
| Vanilla Monitor | 0.281 | 1.000 | 0.281 |
| Feedback Only (Fixed Monitor Policy) | 0.357 | 1.000 | 0.357 |
| HARDE (Full) | 0.500 | 0.984 | 0.500 |
Table 3: Control experiment isolating feedback-policy optimization .
Optimizing only the feedback policy improves Utility and Safe Success, demonstrating that improving execution and recovery behavior can yield meaningful benefits even when the monitor's detection policy remains unchanged.
However, this configuration still underperforms the full HARDE framework, which achieves a Safe Success of 0.500. This suggests that feedback optimization alone is insufficient to fully address the challenges of runtime defense, motivating the joint optimization of trigger, monitor, and feedback policies.
Furthermore, Appendix B.3 of our original submission already provides a module ablation study examining the contributions of different harness components:
| Variant | Utility ↑ | Safety ↑ | SS ↑ |
|---|---|---|---|
| w/o Trigger | 0.357 | 0.929 | 0.357 |
| w/o Feedback | 0.231 | 0.923 | 0.154 |
| w/o Trigger & Feedback | 0.154 | 0.923 | 0.154 |
| None | 0.083 | 0.917 | 0.083 |
| HARDE (Full) | 0.500 | 0.984 | 0.500 |
Table 4: Module ablation results from Appendix B.3.
The substantial performance drops observed when removing either the trigger or feedback module demonstrate that all three policies, including the monitor policy, are closely interdependent and essential to effective harness optimization.
Response to Reviewer SBs1. – Part 3
(5) Clarification of the Utility Drop with Qwen3-8B
We apologize that our original wording was unclear and would like to clarify two different comparisons.
First, attributing the utility drop to an unoptimized harness refers to comparing different harness configurations or methods with the same monitor model. As shown in Table 1, directly integrating the monitor into an unoptimized harness leads to severe over-blocking and utility degradation.
Second, attributing the utility of 0.136 to the weaker Qwen3-8B monitor refers to comparing Vanilla Monitor with DeepSeek-V4-Flash and Qwen3-8B on the same benchmark. Since the harness configuration is identical and only the monitor model differs, we attribute this performance gap to monitor capability. Nevertheless, harness optimization can substantially improve the overall performance even with a weaker monitor.
We apologize for the confusion caused by these two statements and will clarify this distinction in the revised manuscript.
Weakness 2 & Question 2: Limited SHADE-Arena Sample Size and Statistical Reliability of Reported Gains
We greatly appreciate the reviewer’s careful assessment of the dataset size and statistical reliability, and thank them for encouraging a more rigorous evaluation.
We acknowledge that SHADE-Arena contains only 17 cases,with 3 as search set and 14 as held-out test set, making performance potentially sensitive to data splitting. Our original results were averaged over 3 random seeds with different search/test splits. To better characterize this variability, we additionally report standard deviations across the 3 runs on SHADE-Arena with DeepSeek-V4-Flash as the monitor. Results are presented as mean ± standard deviation.
| Method | Utility ↑ | Safety ↑ | SS ↑ |
|---|---|---|---|
| Base Harness | 0.419 ± 0.021 | 0.387 ± 0.040 | 0.161 ± 0.026 |
| Vanilla Monitor | 0.281 ± 0.051 | 1.000 ± 0.000 | 0.281 ± 0.051 |
| TS-Flow | 0.235 ± 0.083 | 0.706 ± 0.127 | 0.177 ± 0.048 |
| GPT-5.4 | 0.364 ± 0.058 | 0.394 ± 0.147 | 0.212 ± 0.017 |
| Claude Opus 4.7 | 0.353 ± 0.076 | 0.294 ± 0.178 | 0.118 ± 0.042 |
| Meta-Harness | 0.462 ± 0.096 | 0.540 ± 0.130 | 0.269 ± 0.083 |
| HARDE (Ours) | 0.500 ± 0.117 | 0.984 ± 0.031 | 0.500 ± 0.117 |
Table 5: SHADE-Arena Results with Standard Deviations
HARDE achieves the highest mean Utility and Safe Success, along with the second-highest mean Safety among the compared methods, substantially outperforming the strongest baseline in Safe Success. To further assess the reliability of these improvements despite the small sample size, we conduct a paired bootstrap analysis with 100,000 resamples, estimating 95% confidence intervals for the Safe Success differences between HARDE and Vanilla Monitor, the strongest baseline on SHADE-Arena.
| Monitor | HARDE SS | Vanilla Monitor SS | Paired Difference (95% CI) |
|---|---|---|---|
| DeepSeek-V4-Flash | 50.0% | 28.1% | +21.9 pp [+3.2, +42.0] |
Table 6: Paired Bootstrap Confidence Intervals
HARDE achieves a statistically significant improvement in Safe Success over the strongest baseline, with a 95% confidence interval of [+3.2, +42.0] percentage points.
We will incorporate these experimental results into the revised manuscript
Response to Reviewer SBs1. – Part 1
We sincerely appreciate the reviewer’s thoughtful feedback on this central issue. We agree that distinguishing detection errors from execution and recovery failures is essential for understanding the sources of utility degradation. Below, we first clarify the relationship between the monitor model and the monitor policy, and then address each question with additional experimental evidence.
(1) Clarification of Monitor Model vs. Monitor Policy
We believe there may be a potential misunderstanding regarding the distinction between monitor false positives and harness execution failures, which we would like to clarify.
Specifically, the underlying monitor model and the monitor policy are not equivalent. The monitor policy is an integral part of the risk-aware harness, rather than an independent component whose errors are unrelated to harness design. Even with identical model weights, detection behavior can vary substantially depending on how the harness constructs and presents execution context, when monitoring is invoked, and how previous interventions affect subsequent agent behavior.
Therefore, the two failure modes identified by the reviewer—incorrect risk judgments and inadequate recovery after correct detection—should not be treated as entirely independent of harness design. Both can be influenced by the design and interaction of harness components.
Importantly, HARDE does not aim merely to compensate for a weak monitor model or improve feedback in isolation. Instead, while keeping the underlying monitor model fixed, HARDE jointly optimizes the trigger, monitor, and feedback policies to improve both risk assessment and execution control.
We provide additional quantitative evidence below to support this claim and address the reviewer's specific questions.
(2) False Positive Rate Analysis
To investigate the reviewer's concern regarding the substantial utility degradation with Qwen3-8B on AgentDyn, we conduct an additional false positive rate (FPR) analysis under three settings:
For each setting, we randomly sample 50 execution trajectories and use DeepSeek-V4.1-Flash as a proxy for human annotation to determine whether each evaluated action is benign or malicious. We calculate FPR as (FP/(FP+TN)), where FP denotes benign actions incorrectly blocked and TN denotes benign actions correctly allowed.
Table 1: False positive analysis
First, the comparison between the standalone monitor and Vanilla Monitor illustrates why the monitor model and monitor policy should be distinguished. When Qwen3-8B is integrated into an unoptimized harness, the observed FPR increases from 81.59% to 90.38%, accompanied by a substantial increase in the number of FP.
One important difference is that the monitor's previous decisions can influence subsequent execution contexts. For example, after an action is blocked, the agent may attempt to resume its legitimate task. However, under an unoptimized harness, the subsequent monitor may continue to be influenced by the preceding blocked actions and recovery attempts, potentially resulting in repeated blocking.
Second, HARDE substantially reduces the observed FPR with the same underlying Qwen3-8B monitor model. By jointly optimizing the trigger, monitor, and feedback policies, HARDE invokes monitoring when necessary, provides more relevant and refined context for risk assessment, and guides agents back toward legitimate tasks to avoid repeated blocking.
These results demonstrate that monitoring errors are not solely determined by the underlying model but are also influenced by how it is integrated into the harness--monitor policy. Thus, harness optimization can mitigate both monitor false positives and failures to provide effective recovery paths.