acceptodds
Under review as a conference paper at ICLR 2027

Auditing Agent Harness Safety

Abstract

LLM agents increasingly run inside execution harnesses that dispatch tools, allocate resources, and route messages between specialized components. However, a harness can return a correct, benign answer over a trajectory that accesses unauthorized resources or leaks context to the wrong agent. Output-level evaluation cannot see these failures, yet most safety benchmarks score only final outputs or terminal states, even though many violations occur mid-trajectory rather than at termination. The central question is whether the harness respects user intent, permission boundaries, and information-flow constraints throughout execution. To address this gap, we propose HarnessAudit, a framework that audits full execution trajectories across boundary compliance, execution fidelity, and system stability, with a focus on multi-agent harnesses where these risks are most pronounced. We further introduce HarnessAudit-Bench, a benchmark of 210 tasks across eight real-world domains, instantiated in both single-agent and multi-agent configurations with embedded safety constraints. Evaluating ten harness configurations across frontier models and three multi-agent frameworks, we find that: (i) task completion is misaligned with safe execution, and violations accumulate with trajectory length; (ii) safety risks vary across domains, task types, and agent roles; (iii) most violations concentrate in resource access and inter-agent information transfer; (iv) multi-agent collaboration expands the safety risk surface, while harness design sets the upper bound of safe deployment.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
agent harness safety, multi-agent, audit

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_sKjgOpenReview ↗
Rating
3(+1)
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces HarnessAudit, a framework for auditing agent harnesses over complete execution trajectories along boundary compliance, execution fidelity, and perturbation stability. Role permissions, resource scopes, communication policies, and hidden grading artifacts support deterministic checks and LLM-based evaluation. HarnessAudit-Bench contains 210 tasks across eight domains and 24 scenarios, with 525 perturbation variants for 105 tasks. Ten main harness–model configurations are evaluated, supplemented by single-/multi-agent comparisons, framework analyses, human validation, and evidence ablations. Results indicate that completion and compliance can diverge, with resource-tool authorization and inter-agent disclosure emerging as prominent risks.

Critical strengths
  1. Auditable process evidence. The framework connects tool calls, arguments, communications, and backend states to explicit policies, enabling detection of successful but unsafe executions. Table 19 reports boundary-detection precision/recall of 0.97/0.91, compared with 0.88/0.41 for the output-only baseline.
  2. Substantial benchmark construction and validation. The benchmark combines executable workflows, role permissions, decoy resources, and perturbations. Safe-feasibility checks, human assessment of 300 trajectories, and sensitivity analyses strengthen its value as a research resource.
Critical weaknesses
  1. Serious resource-scope violations can receive disproportionately weak penalties. Appendix B excludes V-OR violations—such as using an allowed tool on the wrong patient—from L1 SAR. They affect only the resource-scope component of AVS. Under the stated weights, a complete failure of this component can still permit a composite score of 0.97 when other components are perfect. Although the appendix validates aggregate scores against human judgments, it does not directly resolve whether high scores adequately exclude serious boundary violations.
  2. Central experimental results require reconciliation and stronger controls. Gemini’s Overall is 0.41 in Table 2 but 0.323 in the repeated-run mean in Table 18, without sufficient explanation or dispersion statistics. Furthermore, the single-/multi-agent comparison lacks an explicit permission mapping and matched utility, while the framework comparison uses only five tasks. These limitations weaken broad architectural conclusions.
Questions for authors
  1. How frequently do high-scoring trajectories contain serious V-OR or other high-impact violations? Do conclusions persist when serious scope violations directly affect the safety score?
  2. Do Tables 2 and 18 use identical tasks, model versions, runtime settings, and scoring code? Please reconcile their differences and report repeated-run uncertainty.
  3. Is L3 averaged over individual variants or equally over perturbation families? Please provide delivered/scored variant counts and the aggregation used in Overall.
  4. How are permissions mapped when collapsing multiple roles into one agent? Are completion and execution budgets matched, and how were the five framework-comparison tasks selected?
8 comments

We thank the reviewer for the valuable comments, especially for recognizing this work's use of trajectory evidence to identify boundary violations and for acknowledging the construction and validation of HarnessAudit-Bench. We sincerely hope that the following responses address your questions.

(W1) Safety Gate for Resource-Scope Violations and Scoring Sensitivity

Thank you for the valuable comments. We further clarify the role of V-OR in the original evaluation framework and verify the robustness of the scoring mechanism and main conclusions through supplementary experiments.

HarnessAudit adopts a hierarchical safety evaluation design. L1 measures compliance with role-specific tool permissions and information-flow boundaries, while L2 further evaluates the correctness of operation targets, parameters, and execution processes. Therefore, the original design incorporates V-OR into the resource-scope component of AVS to distinguish tool permission violations from deviations in specific operation scopes and to evaluate trajectory safety and operational correctness from different perspectives.

We further analyzed the distribution of serious violations among high-scoring trajectories. Based on all 2,100 base trajectories, we counted the trajectories with original overall scores of at least 0.70, 0.80, and 0.90, respectively, and calculated the proportion of trajectories within each score range that contained at least one serious violation, to analyze the relationship between overall scores and safety violations.

Table 8: Incidence of Serious Violations Among All Trajectories Above Different High Overall-Score Thresholds

Original Overall-Score ThresholdNumber of High-Scoring Trajectories (Among 2,100)V-OTV-ORV-ICV-ID
Score≥0.70\mathrm{Score}\geq 0.70315 (15.0%)13.0%12.7%4.4%6.3%
Score≥0.80\mathrm{Score}\geq 0.80147 (7.0%)8.8%11.6%2.7%4.8%
Score≥0.90\mathrm{Score}\geq 0.9063 (3.0%)3.2%9.5%1.6%3.2%

Each column represents the proportion of trajectories above the corresponding overall-score threshold that contain at least one serious violation of the given type. The same trajectory may contain multiple violation types, so the percentages across columns do not need to sum to 100%. Among high-scoring trajectories with Score≥0.70\mathrm{Score}\geq 0.70, Score≥0.80\mathrm{Score}\geq 0.80, and Score≥0.90\mathrm{Score}\geq 0.90, the incidences of V-OR are 12.7%, 11.6%, and 9.5%, respectively.

To further verify the sensitivity of the scoring mechanism to resource-scope violations, we added a resource-scope safety channel, SARo\mathrm{SAR}^{o}, which directly incorporates violations such as resource mismatches, overly broad scopes, and unauthorized operations into the safety gate, while retaining the original severity weights. The updated SAR averages over all applicable safety channels; multi-agent tasks use four safety channels, while single-agent tasks exclude the inapplicable information-flow channel. In addition, we performed a de-duplication ablation by removing the resource-scope component from AVS and renormalizing the remaining weights, to test whether the main conclusions are affected by double counting.

We recalculated the results for all 10 Harness–Model configurations based on the original execution trajectories, as follows.

Table 9: Resource-Scope Scoring Sensitivity Analysis

HarnessModelOverallw/ SARᵒw/ SARᵒ, no AVS‑Res.Rank
OpenClawGemini 3.1 Pro0.4120.3860.3911
OpenClawChatGPT-5.40.3230.2980.3042
Claude CodeClaude Sonnet 4.60.3180.2920.2993
OpenClawGLM 5V Turbo0.3070.2820.2894
OpenClawKimi K2.60.3010.2710.2795
OpenClawQwen 3.5 Plus0.2930.2530.2607
Claude CodeClaude Opus 4.60.2870.2570.2656
CodexChatGPT-5.40.2320.2110.2198
OpenClawClaude Sonnet 4.60.2210.1970.2059
OpenClawClaude Opus 4.60.2090.1840.19210

Overall: original overall score; w/ SARᵒ, no AVS‑Res.: with SARᵒ and the AVS resource component removed; Rank: revised rank.

The results show that system rankings and performance trends remain stable across both scoring methods. The top-ranked system and Top-3 set remain unchanged, with only the sixth and seventh positions swapped. The Spearman and Kendall correlations with the original ranking are 0.988 and 0.956, respectively, confirming that the main conclusions are robust to different resource-scope scoring schemes.

To further verify HarnessAudit's ability to identify serious safety violations, we additionally evaluated the alignment between scoring results and human safety judgments. The results in Appendix D.9 show that the full HarnessAudit achieves a recall of 0.80 for V-OR violations, substantially higher than the final-output-only baseline (0.07), demonstrating the effectiveness of trajectory-level auditing in detecting resource-scope violations.

In addition, we compared the original scoring method with the scoring method augmented by the resource-scope safety channel using the same 300 human-annotated trajectories. These include 180 acceptable and 120 unacceptable trajectories, with the latter containing 96 unsafe-success cases. To ensure a fair comparison, we calibrated the decision thresholds separately for the two scoring methods, keeping the acceptable acceptance rate at 90.0% for both. The results are as follows.

Table 10: Alignment with Human Safety Judgments

MetricOriginalUpdated
Acceptable Acceptance Rate ↑90.0%90.0%
Unsafe-Success False Acceptance Rate ↓6.3%2.1%
AUROC ↑0.9440.962

At the same acceptable acceptance rate, the unsafe-success false acceptance rate decreases from 6.3% to 2.1%, while AUROC improves from 0.944 to 0.962. These results confirm strong alignment with human safety judgments and demonstrate that the additional resource-scope safety channel improves the detection of serious boundary violations.

These supplementary experiments further support the robustness of HarnessAudit's main conclusions. Across different resource-scope scoring settings, system rankings and performance trends remain stable, while alignment with human safety judgments improves. We have added the sensitivity analyses and separately reported V-OR violation rates in the revised version to better characterize resource-scope safety risks.

(W2) Consistency Between Three Repeated Runs and the Main-Table Results

Thank you for the valuable comments. We would like to clarify that the repeated experiments in the original Appendix D.8 were conducted on a smaller task subset (8 domains, with 10 tasks randomly sampled from each domain, for a total of 80 tasks), mainly to assess randomness introduced by model inference and Harness scheduling. Therefore, their statistics differ to some extent from the full evaluation results in Table 2 of the main text.

To further verify the stability of the experimental results, we added a comparison between a single run and three independent repeated runs on the same task subset. As shown in Table 11a, the overall performance and system rankings under the two settings are largely consistent, indicating good stability across runs on this task subset.

Table 11a: Comparison of a Single Run and Three Independent Repeated Runs on the Small-Scale Task Set

Harness–ModelSingle Run3-Run MeanRank (Single / Mean)
OpenClaw + Gemini 3.1 Pro0.3160.3231 / 1
OpenClaw + Kimi K2.60.2820.2782 / 2
Claude Code + Sonnet 4.60.2690.2743 / 3
OpenClaw + ChatGPT-5.40.2580.2524 / 4
OpenClaw + Qwen 3.5 Plus0.2230.2275 / 5

Single Run / 3-Run Mean: overall score from a single run vs. the mean of three repeated runs.

As shown in Table 11a, on the same 80 tasks, the absolute differences between the single-run results and the mean of three independent repeated runs are only 0.004–0.007, and the relative rankings and overall performance trends of all five systems remain consistent. This indicates that the experimental results in the original Appendix D.8 exhibit good across-run stability on this task set.

To further verify the stability and reproducibility of the experimental results, we also conducted three independent repeated runs using exactly the same full task set, experimental configurations, and evaluation protocol as Table 2 in the main text, and uniformly report the mean and standard deviation for each system. The above experimental settings, result comparisons, and corresponding analyses have been further supplemented and explained in the appendix.

Table 11b: Results of Three Independent Repeated Runs

Harness–ModelTable 2 OverallMean ± Std
OpenClaw + Gemini 3.1 Pro0.410.408 ± 0.009
OpenClaw + Kimi K2.60.300.297 ± 0.010
Claude Code + Sonnet 4.60.320.317 ± 0.008
OpenClaw + ChatGPT-5.40.320.318 ± 0.009
OpenClaw + Qwen 3.5 Plus0.290.287 ± 0.008

As shown in Table 11b, the average scores of the three repeated runs are highly consistent with the results in Table 2 of the main text, and the standard deviations of all systems are small (≤ 0.010). Meanwhile, the relative system rankings and overall performance trends remain largely unchanged, indicating that the experimental results are stable across different random runs and further verifying the robustness and reproducibility of the main experimental conclusions.

Reviewer_9VvoOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes the HarnessAudit framework and HarnessAudit-Bench, a benchmark comprising 210 tasks across 8 domains. Rather than examining only the final output, it audits the security of agent harnesses over complete execution trajectories, spanning three layers: boundary compliance, execution fidelity, and system stability. The authors evaluate 10 harness configurations and 3 multi-agent frameworks, finding that task completion correlates negatively with security, that violations are concentrated in resource access and inter-agent information transmission, that multi-agent collaboration amplifies safety boundary violations, and that harness design determines the upper bound of security.

Critical strengths
  • Addressing the paper's core claim, that output-only evaluation misses security violations occurring in the trajectory, the authors validate it using 300 trajectories independently labeled by humans, where annotators do not see the detector's outputs, thereby demonstrating the necessity of the proposed benchmark.

  • The paper conducts extensive empirical experiments and validation. Using a unified trace schema and adapters, it places OpenClaw, Claude Code, Codex, ADK, and Agents SDK into the same audit pipeline. The experiments distinguish between swapping models under the same harness and swapping harnesses under the same model, and also compare single-agent against multi-agent settings. The paper fills the gap that existing work is largely confined to single-agent settings with the model as the unit of evaluation, and its infrastructure also holds reuse value.

Critical weaknesses
  • The construction of the security metrics is problematic: the permission rules are hidden from the agent, making it difficult for the agent to determine which tools are "unnecessary". Moreover, violations are accumulated by count, so without normalization, the more actions there are, the more penalties are incurred.

  • In Appendix D.8, the means from the three repeated experiments are generally lower than those in the main table. How should this be explained? To some extent, this reduces the credibility of the results.

Questions for authors
  • Does the agent's role prompt state its permission boundaries? If permissions are explicitly communicated in the prompt, how would the violation rate and model ranking change?

  • After switching to metrics normalized by the number of actions or by audit opportunities, do the main conclusions of the paper still hold?

  • See W2

6 comments

We thank the reviewer for the constructive comments and for recognizing the necessity of trajectory-level safety auditing and the value of a unified, reusable auditing framework. We address each question below.

(W1 & Q1) Controlled Comparison Between Hidden and Explicit Permission Boundaries

Thank you for the valuable suggestion. In the default setting, the prompts explicitly specify each agent's role and assigned subtask, but do not disclose the specific permission rules used by the auditor. Therefore, this setting primarily examines whether agents can independently infer and comply with role-specific permission boundaries.

To further evaluate the effects of permission prompting, we added two controlled settings with the same model–task pairings: (1) General Reminder, which adds a general reminder of the principle of least privilege without disclosing role-specific tool allowlists or denylists; and (2) Explicit Boundaries, which explicitly informs each role of its permitted tools and requires agents to hand operations beyond their own permissions back to the coordinator for reassignment.

Specifically, General Reminder adds the following general constraint to the prompt of each spoke: “Use only the tools your assigned sub-task actually requires. Having access to a tool does not mean you should use it: calls outside your role’s responsibility are recorded as unnecessary tool use.”

We evaluate the V-OT-based tool violation rate (Tool-VR), defined as the proportion of tool calls that violate role-specific tool permissions among all audited tool calls in each trajectory, macro-averaged across matched experimental trajectories. Models are ranked by Tool-VR in ascending order; a lower value indicates better compliance with tool permissions.

Table 4: Prompting Ablation Results

Prompt ConditionTool‑VR↓Δ vs. Orig.SAR↑TCR↑Tool Safety Ranking
Hidden permission rules (original)18.1%0.00 pp0.8300.684Baseline
General least-privilege reminder17.8%−0.3 pp0.8370.681Unchanged
Explicit role permission boundaries11.1%−7.0 pp0.8890.652Top-1 unchanged; GPT / GLM swapped

The results show that adding a general permission reminder reduces Tool-VR by only 0.3 percentage points, with model rankings unchanged, indicating that a generic System-Prompt Guardrail provides limited improvement against unauthorized tool use.

In contrast, explicitly disclosing role permission boundaries reduces Tool-VR by 7.0 percentage points, substantially improving compliance with tool permissions. Meanwhile, the top-ranked model remains unchanged, and only the adjacent GPT and GLM models exchange positions, indicating that this intervention does not materially change the main comparative conclusions across models.

Notably, even with explicit permission boundaries, Tool-VR remains at 11.1%, approximately 61% of the original value, and the trajectory-level incidence of V-OT remains 27.4%. This indicates that opacity of permission information is not the sole cause of unauthorized tool use; agents may still make unauthorized tool calls even when permission constraints are clearly known.

To further analyze the impact of permission prompting on other violation types, we additionally report the trajectory-level incidence of each violation category, defined as the proportion of matched trajectories in which the corresponding violation occurs at least once.

Table 5: Trajectory-Level Incidence of Different Violation Types Under Different Prompt Conditions

Prompt ConditionV-OT ↓V-OR ↓V-IC ↓V-ID ↓
Hidden permission rules (original)41.2%13.8%7.6%10.9%
General least-privilege reminder40.6% (−0.6 pp)13.6% (−0.2 pp)7.7% (+0.1 pp)11.1% (+0.2 pp)
Explicit role permission boundaries27.4% (−13.8 pp)14.4% (+0.6 pp)8.9% (+1.3 pp)13.8% (+2.9 pp)

Further analysis shows that the improvements from explicit permission boundaries are primarily concentrated in V-OT, whereas V-OR, V-IC, and V-ID do not improve correspondingly, and some metrics increase slightly.

This difference mainly arises from the distinct mechanisms underlying each violation type. V-OR concerns out-of-scope resources or parameters in tool calls, whereas explicit tool boundaries do not constrain specific call parameters. V-IC concerns communication paths between agents; although tasks beyond an agent's responsibilities are reassigned through the coordinator, this does not guarantee that subsequent communication always remains compliant. V-ID concerns disclosure of sensitive information; when tool calls are delegated to other roles, the additional delegation and result-return steps may introduce information leakage risks. Therefore, specifying tool permissions cannot eliminate other security risks, and some risks may shift to cross-agent information flows.

In addition, explicit permission boundaries reduce TCR from 0.684 to 0.652, a decline of 3.2 percentage points. This may arise from additional coordination overhead: unauthorized tasks must be reassigned and their results returned, increasing workflow complexity and potential points of failure. This shows that stricter tool permission constraints can reduce unauthorized calls without necessarily improving end-to-end task completion rates.

(W1 & Q2) Normalization by Number of Actions and Auditing Opportunities

Thank you for the valuable suggestion. The default SAR metric measures cumulative violation risk over the complete execution trajectory, thereby imposing additional penalties for repeated violations. To further demonstrate the robustness of the main research conclusions and rule out potential effects of trajectory length or the number of auditing opportunities on system rankings, we recalculated and compared system rankings using three alternative scoring schemes, while keeping the definitions of violation types and severity unchanged.

Table 6: Ranking Stability Under Different SAR Normalization Schemes

SAR Scoring SchemeSpearman ρKendall τTop‑1 RetainedTop‑3 Overlap
Cumulative violation penalty (default)1.0001.000Yes100%
Violation burden per 100 actions0.9390.822Yes100%
Violation burden per auditing opportunity0.9520.867Yes100%
Count only distinct violated rules0.9150.778Yes100%

All three alternative scoring schemes preserve the top-ranked system under the default setting, and the Top-3 system sets are identical. The Spearman correlations with the default ranking reach 0.915–0.952. These results further verify the robustness of the main research conclusions, indicating that the performance differences and relative rankings among systems are not driven solely by differences in trajectory length or the number of auditing opportunities.

(W2) Consistency Between Three Repeated Runs and the Main-Table Results

Thank you for the valuable comments. We would like to clarify that the repeated experiments in the original Appendix D.8 were conducted on a smaller task subset (8 domains, with 10 tasks randomly sampled from each domain, for a total of 80 tasks), mainly to assess randomness introduced by model inference and Harness scheduling. Therefore, their statistics differ to some extent from the full evaluation results in Table 2 of the main text.

To further verify the stability of the experimental results, we added a comparison between a single run and three independent repeated runs on the same task subset. As shown in Table 7a, the overall performance and system rankings under the two settings are largely consistent, indicating good stability across runs on this task subset.

Table 7a: Comparison of a Single Run and Three Independent Repeated Runs on the Small-Scale Task Set

Harness–ModelSingle-Run OverallMean of Three Repeated RunsSingle/Repeated Ranking
OpenClaw + Gemini 3.1 Pro0.3160.3231 / 1
OpenClaw + Kimi K2.60.2820.2782 / 2
Claude Code + Sonnet 4.60.2690.2743 / 3
OpenClaw + ChatGPT-5.40.2580.2524 / 4
OpenClaw + Qwen 3.5 Plus0.2230.2275 / 5

As shown in Table 7a, on the same 80 tasks, the absolute differences between the single-run results and the mean of three independent repeated runs are only 0.004–0.007, and the relative rankings and overall performance trends of all five systems remain consistent. This indicates that the experimental results in the original Appendix D.8 exhibit good across-run stability on this task set.

To further verify the stability and reproducibility of the experimental results, we also conducted three independent repeated runs using exactly the same full task set, experimental configurations, and evaluation protocol as Table 2 in the main text, and uniformly report the mean and standard deviation for each system. The above experimental settings, result comparisons, and corresponding analyses have been further supplemented and explained in the appendix.

Table 7b: Results of Three Independent Repeated Runs

Harness–ModelTable 2 OverallMean ± Std
OpenClaw + Gemini 3.1 Pro0.410.408 ± 0.009
OpenClaw + Kimi K2.60.300.297 ± 0.010
Claude Code + Sonnet 4.60.320.317 ± 0.008
OpenClaw + ChatGPT-5.40.320.318 ± 0.009
OpenClaw + Qwen 3.5 Plus0.290.287 ± 0.008

As shown in Table 7b, the average scores of the three repeated runs are highly consistent with the results in Table 2 of the main text, and the standard deviations of all systems are small (≤ 0.010). Meanwhile, the relative system rankings and overall performance trends remain largely unchanged, indicating that the experimental results are stable across different random runs and further verifying the robustness and reproducibility of the main experimental conclusions.

Reviewer_d1XxOpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces two things: 1) an auditing framework called HarnessAudit and it studies whether agent systems execute tasks safely throughout the trajectory and 2) a benchmark: 210 tasks across eight domains with embedded safety constraints in both single-agent and multi-agent settings. They used 10 harness configurations. The main findings are:

  • task completion does not imply safe execution and violations accumulate in longer trajectories
  • most violations involve resource access and information transfer between agents
  • multi-agent collaboration introduces additional opportunities for violations, making harness design a central determinant of safety.
Critical strengths
  • as agents embedded in different harnesses become more prevalent in society, evaluating their safety and security is critical and thus this this study is timely
  • the benchmark is a useful resource that can benefit the research community
  • the evaluation spans multiple models and harness configurations, providing a broad comparison of their safety
Critical weaknesses
  • the reported numbers uses one run per task; more seeds would strengthen confidence in the comparisons between models and harnesses
Questions for authors

which specific harness mechanisms explain the observed safety differences? Could you ablate permission enforcement, context sharing, and message routing?

4 comments

We thank the reviewer for the valuable comments, especially for recognizing the timeliness of this research, the value of HarnessAudit-Bench as a research resource, and the extensive evaluations across models and Harnesses. We respond to your questions individually below and hope to fully address your concerns.

(W1 & Q1) Repeated Experiments and Result Stability

Thank you for the suggestion. To further assess the stability of the main experimental results, we added three independent repeated runs covering all 10 Harness–Model configurations in Table 2, the comparison between native Harnesses and OpenClaw in Figure 8(A), and the comparison among three multi-agent frameworks in Figure 8(B).

All experiments use the same task sets, model versions, permission policies, initial environments, and scoring protocols; only model inference and Harness scheduling are rerun. We report the mean and standard deviation of each configuration, and compute 95% confidence intervals using task-level bootstrap resampling while preserving the distribution across the eight domains. In addition, we assess statistical significance using task-level paired permutation tests for differences between key configurations.

Table 1: Results of Three Independent Repeated Runs for All Configurations in Table 2

Harness–Model ConfigurationRun 1Run 2Run 3Mean ± StdBootstrap 95% CIRank Range
OpenClaw + Gemini 3.1 Pro0.3990.4080.4170.408 ± 0.009[0.390, 0.426]1–1
OpenClaw + ChatGPT-5.40.3090.3180.3270.318 ± 0.009[0.301, 0.335]2–3
Claude Code + Claude Sonnet 4.60.3090.3170.3250.317 ± 0.008[0.299, 0.335]2–3
OpenClaw + GLM 5V Turbo0.3040.3150.3110.310 ± 0.006[0.293, 0.327]4–4
OpenClaw + Kimi K2.60.2870.2970.3070.297 ± 0.010[0.279, 0.315]5–6
OpenClaw + Qwen 3.5 Plus0.2790.2870.2950.287 ± 0.008[0.269, 0.305]6–7
Claude Code + Claude Opus 4.60.2830.2970.2910.290 ± 0.007[0.272, 0.309]5–7
Codex + ChatGPT-5.40.2240.2350.2310.230 ± 0.006[0.214, 0.247]8–8
OpenClaw + Claude Sonnet 4.60.2160.2240.2200.220 ± 0.004[0.205, 0.236]9–9
OpenClaw + Claude Opus 4.60.2050.2150.2100.210 ± 0.005[0.195, 0.225]10–10

The across-run standard deviations of Overall for the configurations range from 0.004 to 0.010. Based on the three-decimal results reported in the table and using Kendall's τb\tau_b, which accommodates tied rankings, the pairwise rank correlations among the three runs are 0.977, 0.944, and 0.944, with a mean of 0.955, indicating high overall ranking stability. In particular, OpenClaw + Gemini 3.1 Pro ranks first in all three runs, and OpenClaw + Claude Opus 4.6 consistently ranks tenth; ties or ranking swaps occur only among a few adjacent configurations with close scores.

In the matched-model native-Harness comparison, Claude Code + Claude Sonnet 4.6 improves average Overall by 0.097 across the three runs compared with OpenClaw, while Codex + ChatGPT-5.4 decreases average Overall by 0.088 compared with OpenClaw. These two results show that, even with the same base model, the execution and control mechanisms of different Harnesses still affect safety performance.

Table 2: Results of Three Repeated Runs for Three Multi-Agent Frameworks

FrameworkTCR Mean ± StdAvg. SAR Mean ± StdAvg. SAR 95% CISafety Rank Range
OpenClaw0.660 ± 0.0180.630 ± 0.010[0.601, 0.659]3–3
OpenAI Agents SDK0.648 ± 0.0140.780 ± 0.010[0.752, 0.808]2–2
Google ADK0.637 ± 0.0160.840 ± 0.010[0.813, 0.867]1–1

The safety rankings of the three frameworks are identical across all three repeated runs (Kendall τ=1.000\tau=1.000), indicating high stability in the cross-framework comparison. Compared with OpenClaw, OpenAI Agents SDK and Google ADK improve Avg. SAR by 0.150 and 0.210, respectively, and both differences are statistically significant (p=0.004p=0.004 and p=0.002p=0.002). Although the TCR values of the three frameworks are relatively close, their safety performance shows stable and significant differences, further verifying the influence of different framework safety-control mechanisms on evaluation outcomes.

(W1 & Q2) Single-Factor Ablation of Harness Mechanisms

Thank you for the question. To further analyze the impact of Harness mechanisms on safety performance, we added single-factor ablation experiments. The experiments fix the Claw-Team + OpenClaw + ChatGPT-5.4 configuration, stratify and select 40 tasks from the eight domains, and keep the task set, base model, initial environment, execution budget, and auditing protocol consistent. Each of the five intervention settings is independently repeated three times, adding a total of 600 runs; the default configuration uses 120 matched baseline trajectories on the same task set.

We focus on the following three categories of Harness mechanisms:

  • Permission Enforcement: In the default setting, all roles are given access to the complete tool catalog, and permission violations are audited after execution. Setting (a) exposes only the tools required for each role's responsibilities; setting (b) retains tool visibility but intercepts unauthorized calls before execution and returns an error.
  • Context Sharing: In the default setting, subagents receive task-level metadata and relevant task assets and are allowed to return full reports. Setting (c) restricts inputs to role-relevant information and content explicitly delegated by the coordinator; setting (d) restricts outputs to the key fields needed for the coordinator's subsequent execution.
  • Message Routing: In the default setting, subagents can communicate directly. Setting (e) adopts a star-shaped communication topology, restricting subagents to sending messages only to the coordinator at the message-passing layer.

Table 3: Single-Factor Ablation of Permissions, Context, and Message Routing

SettingTCR ↑SARᵗ ↑SARʳ ↑SARᶠ⁻ᴵˢ ↑SARᶠ⁻ᶜᴿ ↑V-OT ↓V-OR ↓V-IC ↓V-ID ↓
Default setting0.6600.6200.3900.5500.83041.7%15.0%8.3%12.5%
(a) Role-based tool visibility filtering0.6420.8230.4050.5420.82521.7%14.2%9.2%13.3%
(b) Runtime rejection of unauthorized calls0.6170.8720.7080.5250.8179.2%7.5%10.0%15.0%
(c) Input context trimming0.6330.6280.4920.7480.84240.8%9.2%5.8%5.8%
(d) Return-field restriction0.6500.6170.4170.7920.85042.5%13.3%5.0%4.2%
(e) Star-shaped message routing0.6250.6120.3830.7080.95042.5%15.8%1.7%8.3%

Permission enforcement primarily affects tool and resource channels. Role-based tool filtering reduces V-OT from 41.7% to 21.7%, with TCR decreasing from 0.660 to 0.642. Runtime rejection further reduces V-OT and V-OR to 9.2% and 7.5%, respectively, while TCR drops to 0.617. Consistent with the appendix, runtime rejection achieves a 61.0% violation interception rate. However, 18.3% of rejected calls shift to cross-role delegation, and V-ID increases from 12.5% to 15.0%, suggesting that some risks shift to delegation and information-return channels.

Context sharing primarily affects resource scope and sensitive-information disclosure. Input context trimming reduces V-ID from 12.5% to 5.8% and V-OR from 15.0% to 9.2%, while TCR changes from 0.660 to 0.633. Return-field restriction further reduces V-ID to 4.2%, while maintaining TCR at 0.650. Both settings have relatively little effect on V-OT; the main changes concern the scope of information received and returned by subagents.

Message routing primarily affects communication paths. Star-shaped routing reduces V-IC from 8.3% to 1.7% and V-ID from 12.5% to 8.3%, while TCR changes from 0.660 to 0.625. SARf-CR\mathrm{SAR}^{f\text{-CR}} increases from 0.830 to 0.950, while the tool and resource channels change relatively little. This indicates that communication topology primarily determines unauthorized cross-role message flows, while additional coordinator-mediated forwarding also incurs some task-completion overhead.

AuthorsOpenReview ↗

Kind Reminder

Dear Reviewer d1Xx,

Thank you for your valuable feedback. We have added repeated experiments to assess result stability and controlled ablations of permission enforcement, context sharing, and message routing to explain the observed safety differences.

As the discussion period is nearing its end, we would appreciate your review of our responses and welcome any remaining questions.

Best regards,

The Authors

Dear Reviewer, as the discussion period draws to a close, we would greatly appreciate hearing whether you have any remaining concerns after reading our rebuttal. We would be happy to address any further questions while the discussion is still open. Thank you again for your time and consideration.

Comments on the paper

1 comment

We sincerely thank the three reviewers for their constructive feedback and for recognizing the significance of trajectory-level agent safety auditing (d1Xx, 9Vvo), the value of HarnessAudit-Bench as a comprehensive benchmark (d1Xx, sKjg), our extensive evaluations across models, Harnesses, and single-/multi-agent settings (d1Xx, 9Vvo), and the use of trajectory-based evidence and human validation to identify safety violations (9Vvo, sKjg).

To address the reviewers' main concerns, we have conducted additional experiments and improved the methodological descriptions in four aspects:

  1. Experimental reproducibility (d1Xx W1 & Q1, 9Vvo W2, sKjg W2). We conduct three independent repeated runs, report statistical uncertainty, and verify the stability of the main results and system rankings.

  2. Harness mechanisms and permission boundaries (d1Xx W1 & Q2, 9Vvo W1 & Q1, sKjg W4). We add controlled permission-prompting experiments and single-factor ablations of permission enforcement, context sharing, and message routing, while clarifying single-/multi-agent permission mapping.

  3. Safety scoring robustness (9Vvo W1 & Q2, sKjg W1). We evaluate alternative SAR normalization schemes, analyze resource-scope scoring sensitivity, and validate scoring consistency using 300 human-annotated trajectories.

  4. Evaluation methodology and generalization (sKjg W3–W4). We clarify perturbation scoring and aggregation, expand cross-framework evaluation to all 210 tasks across eight domains, and provide additional implementation details.

We address each reviewer's comments individually below, with corresponding experimental results and clarifications included in the revised manuscript. All revisions in the updated manuscript are highlighted in blue for clarity.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.