- Rating
- 3(+1)
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces HarnessAudit, a framework for auditing agent harnesses over complete execution trajectories along boundary compliance, execution fidelity, and perturbation stability. Role permissions, resource scopes, communication policies, and hidden grading artifacts support deterministic checks and LLM-based evaluation. HarnessAudit-Bench contains 210 tasks across eight domains and 24 scenarios, with 525 perturbation variants for 105 tasks. Ten main harness–model configurations are evaluated, supplemented by single-/multi-agent comparisons, framework analyses, human validation, and evidence ablations. Results indicate that completion and compliance can diverge, with resource-tool authorization and inter-agent disclosure emerging as prominent risks.
Critical strengths
- Auditable process evidence. The framework connects tool calls, arguments, communications, and backend states to explicit policies, enabling detection of successful but unsafe executions. Table 19 reports boundary-detection precision/recall of 0.97/0.91, compared with 0.88/0.41 for the output-only baseline.
- Substantial benchmark construction and validation. The benchmark combines executable workflows, role permissions, decoy resources, and perturbations. Safe-feasibility checks, human assessment of 300 trajectories, and sensitivity analyses strengthen its value as a research resource.
Critical weaknesses
- Serious resource-scope violations can receive disproportionately weak penalties. Appendix B excludes V-OR violations—such as using an allowed tool on the wrong patient—from L1 SAR. They affect only the resource-scope component of AVS. Under the stated weights, a complete failure of this component can still permit a composite score of 0.97 when other components are perfect. Although the appendix validates aggregate scores against human judgments, it does not directly resolve whether high scores adequately exclude serious boundary violations.
- Central experimental results require reconciliation and stronger controls. Gemini’s Overall is 0.41 in Table 2 but 0.323 in the repeated-run mean in Table 18, without sufficient explanation or dispersion statistics. Furthermore, the single-/multi-agent comparison lacks an explicit permission mapping and matched utility, while the framework comparison uses only five tasks. These limitations weaken broad architectural conclusions.
Questions for authors
- How frequently do high-scoring trajectories contain serious V-OR or other high-impact violations? Do conclusions persist when serious scope violations directly affect the safety score?
- Do Tables 2 and 18 use identical tasks, model versions, runtime settings, and scoring code? Please reconcile their differences and report repeated-run uncertainty.
- Is L3 averaged over individual variants or equally over perturbation families? Please provide delivered/scored variant counts and the aggregation used in Overall.
- How are permissions mapped when collapsing multiple roles into one agent? Are completion and execution budgets matched, and how were the five framework-comparison tasks selected?
8 comments
Rebuttal2
To further verify the sensitivity of the scoring mechanism to resource-scope violations, we added a resource-scope safety channel, , which directly incorporates violations such as resource mismatches, overly broad scopes, and unauthorized operations into the safety gate, while retaining the original severity weights. The updated SAR averages over all applicable safety channels; multi-agent tasks use four safety channels, while single-agent tasks exclude the inapplicable information-flow channel. In addition, we performed a de-duplication ablation by removing the resource-scope component from AVS and renormalizing the remaining weights, to test whether the main conclusions are affected by double counting.
We recalculated the results for all 10 Harness–Model configurations based on the original execution trajectories, as follows.
Table 9: Resource-Scope Scoring Sensitivity Analysis
| Harness | Model | Overall | w/ SARᵒ | w/ SARᵒ, no AVS‑Res. | Rank |
|---|---|---|---|---|---|
| OpenClaw | Gemini 3.1 Pro | 0.412 | 0.386 | 0.391 | 1 |
| OpenClaw | ChatGPT-5.4 | 0.323 | 0.298 | 0.304 | 2 |
| Claude Code | Claude Sonnet 4.6 | 0.318 | 0.292 | 0.299 | 3 |
| OpenClaw | GLM 5V Turbo | 0.307 | 0.282 | 0.289 | 4 |
| OpenClaw | Kimi K2.6 | 0.301 | 0.271 | 0.279 | 5 |
| OpenClaw | Qwen 3.5 Plus | 0.293 | 0.253 | 0.260 | 7 |
| Claude Code | Claude Opus 4.6 | 0.287 | 0.257 | 0.265 | 6 |
| Codex | ChatGPT-5.4 | 0.232 | 0.211 | 0.219 | 8 |
| OpenClaw | Claude Sonnet 4.6 | 0.221 | 0.197 | 0.205 | 9 |
| OpenClaw | Claude Opus 4.6 | 0.209 | 0.184 | 0.192 | 10 |
Overall: original overall score; w/ SARᵒ, no AVS‑Res.: with SARᵒ and the AVS resource component removed; Rank: revised rank.
The results show that system rankings and performance trends remain stable across both scoring methods. The top-ranked system and Top-3 set remain unchanged, with only the sixth and seventh positions swapped. The Spearman and Kendall correlations with the original ranking are 0.988 and 0.956, respectively, confirming that the main conclusions are robust to different resource-scope scoring schemes.
To further verify HarnessAudit's ability to identify serious safety violations, we additionally evaluated the alignment between scoring results and human safety judgments. The results in Appendix D.9 show that the full HarnessAudit achieves a recall of 0.80 for V-OR violations, substantially higher than the final-output-only baseline (0.07), demonstrating the effectiveness of trajectory-level auditing in detecting resource-scope violations.
In addition, we compared the original scoring method with the scoring method augmented by the resource-scope safety channel using the same 300 human-annotated trajectories. These include 180 acceptable and 120 unacceptable trajectories, with the latter containing 96 unsafe-success cases. To ensure a fair comparison, we calibrated the decision thresholds separately for the two scoring methods, keeping the acceptable acceptance rate at 90.0% for both. The results are as follows.
Table 10: Alignment with Human Safety Judgments
| Metric | Original | Updated |
|---|---|---|
| Acceptable Acceptance Rate ↑ | 90.0% | 90.0% |
| Unsafe-Success False Acceptance Rate ↓ | 6.3% | 2.1% |
| AUROC ↑ | 0.944 | 0.962 |
At the same acceptable acceptance rate, the unsafe-success false acceptance rate decreases from 6.3% to 2.1%, while AUROC improves from 0.944 to 0.962. These results confirm strong alignment with human safety judgments and demonstrate that the additional resource-scope safety channel improves the detection of serious boundary violations.
These supplementary experiments further support the robustness of HarnessAudit's main conclusions. Across different resource-scope scoring settings, system rankings and performance trends remain stable, while alignment with human safety judgments improves. We have added the sensitivity analyses and separately reported V-OR violation rates in the revised version to better characterize resource-scope safety risks.
Rebuttal3
(W2) Consistency Between Three Repeated Runs and the Main-Table Results
Thank you for the valuable comments. We would like to clarify that the repeated experiments in the original Appendix D.8 were conducted on a smaller task subset (8 domains, with 10 tasks randomly sampled from each domain, for a total of 80 tasks), mainly to assess randomness introduced by model inference and Harness scheduling. Therefore, their statistics differ to some extent from the full evaluation results in Table 2 of the main text.
To further verify the stability of the experimental results, we added a comparison between a single run and three independent repeated runs on the same task subset. As shown in Table 11a, the overall performance and system rankings under the two settings are largely consistent, indicating good stability across runs on this task subset.
Table 11a: Comparison of a Single Run and Three Independent Repeated Runs on the Small-Scale Task Set
| Harness–Model | Single Run | 3-Run Mean | Rank (Single / Mean) |
|---|---|---|---|
| OpenClaw + Gemini 3.1 Pro | 0.316 | 0.323 | 1 / 1 |
| OpenClaw + Kimi K2.6 | 0.282 | 0.278 | 2 / 2 |
| Claude Code + Sonnet 4.6 | 0.269 | 0.274 | 3 / 3 |
| OpenClaw + ChatGPT-5.4 | 0.258 | 0.252 | 4 / 4 |
| OpenClaw + Qwen 3.5 Plus | 0.223 | 0.227 | 5 / 5 |
Single Run / 3-Run Mean: overall score from a single run vs. the mean of three repeated runs.
As shown in Table 11a, on the same 80 tasks, the absolute differences between the single-run results and the mean of three independent repeated runs are only 0.004–0.007, and the relative rankings and overall performance trends of all five systems remain consistent. This indicates that the experimental results in the original Appendix D.8 exhibit good across-run stability on this task set.
To further verify the stability and reproducibility of the experimental results, we also conducted three independent repeated runs using exactly the same full task set, experimental configurations, and evaluation protocol as Table 2 in the main text, and uniformly report the mean and standard deviation for each system. The above experimental settings, result comparisons, and corresponding analyses have been further supplemented and explained in the appendix.
Table 11b: Results of Three Independent Repeated Runs
| Harness–Model | Table 2 Overall | Mean ± Std |
|---|---|---|
| OpenClaw + Gemini 3.1 Pro | 0.41 | 0.408 ± 0.009 |
| OpenClaw + Kimi K2.6 | 0.30 | 0.297 ± 0.010 |
| Claude Code + Sonnet 4.6 | 0.32 | 0.317 ± 0.008 |
| OpenClaw + ChatGPT-5.4 | 0.32 | 0.318 ± 0.009 |
| OpenClaw + Qwen 3.5 Plus | 0.29 | 0.287 ± 0.008 |
As shown in Table 11b, the average scores of the three repeated runs are highly consistent with the results in Table 2 of the main text, and the standard deviations of all systems are small (≤ 0.010). Meanwhile, the relative system rankings and overall performance trends remain largely unchanged, indicating that the experimental results are stable across different random runs and further verifying the robustness and reproducibility of the main experimental conclusions.
Rebuttal1
We thank the reviewer for the valuable comments, especially for recognizing this work's use of trajectory evidence to identify boundary violations and for acknowledging the construction and validation of HarnessAudit-Bench. We sincerely hope that the following responses address your questions.
(W1) Safety Gate for Resource-Scope Violations and Scoring Sensitivity
Thank you for the valuable comments. We further clarify the role of V-OR in the original evaluation framework and verify the robustness of the scoring mechanism and main conclusions through supplementary experiments.
HarnessAudit adopts a hierarchical safety evaluation design. L1 measures compliance with role-specific tool permissions and information-flow boundaries, while L2 further evaluates the correctness of operation targets, parameters, and execution processes. Therefore, the original design incorporates V-OR into the resource-scope component of AVS to distinguish tool permission violations from deviations in specific operation scopes and to evaluate trajectory safety and operational correctness from different perspectives.
We further analyzed the distribution of serious violations among high-scoring trajectories. Based on all 2,100 base trajectories, we counted the trajectories with original overall scores of at least 0.70, 0.80, and 0.90, respectively, and calculated the proportion of trajectories within each score range that contained at least one serious violation, to analyze the relationship between overall scores and safety violations.
Table 8: Incidence of Serious Violations Among All Trajectories Above Different High Overall-Score Thresholds
Each column represents the proportion of trajectories above the corresponding overall-score threshold that contain at least one serious violation of the given type. The same trajectory may contain multiple violation types, so the percentages across columns do not need to sum to 100%. Among high-scoring trajectories with Score≥0.70, Score≥0.80, and Score≥0.90, the incidences of V-OR are 12.7%, 11.6%, and 9.5%, respectively.