- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes verifiable agent insurance, a framework linking programmatically specified behavioral promises to committed execution traces and provider-funded escrow. Users can challenge suspected violations, and a trusted verifier checks the evidence before settlement. The authors implement the system with two agent harnesses and deploy an escrow contract on Base. Benchmark experiments measure run-level violation detection and examine how additional structured records could expand coverage.
Critical strengths
- The paper asks a worthwhile question about what happens after prevention fails. Connecting explicit behavioral commitments to evidence and financial recourse gives agent accountability a concrete operational meaning.
- The contribution includes a working implementation rather than only a conceptual proposal. Harness integrations, trace commitments, settlement rules, mainnet tests, and runtime measurements make the design easier to assess.
- The empirical analysis identifies an actionable limitation of current agents: information needed for auditing often remains embedded in natural language. The discussion of consent, permission scope, and structured records is useful beyond the proposed insurance mechanism.
Critical weaknesses
The 83.0% coverage result needs more careful qualification. The measured detection rate on the original benchmark traces is 42.2%. Appendix D.3 estimates additional coverage through retrospective review and supports proposed checks with 183 constructed examples. This is useful evidence, but it differs from instrumenting the agents and rerunning the benchmarks. Some statements in the abstract and main results read as though the latter experiment was performed. The paper should consistently distinguish measured detection from potential coverage.
The annotation validation is not sufficiently independent. Most policy-compliance labels come from a three-LLM committee. Its agreement with the five-labeler majority is not an independent accuracy check because those same three LLMs also contribute to that majority. The calibration samples are small, and Appendix D.4 shows that reasonable interpretations can disagree about whether a violation occurred. A separate human-adjudicated sample would strengthen confidence in both the coverage estimates and the reported 5.8% false-positive rate, especially when verdicts trigger payments.
A valid claim does not necessarily receive compensation. Appendix B.2 states that filing a claim does not reserve its payout, other claims may exhaust the balance, and new claims are rejected when reserves are insufficient. This substantially limits the accountability guarantee: a recorded and verified violation may remain unpaid. The limitation is disclosed, but should be reflected more prominently in the framework’s claims. Concurrent-claim and reserve-exhaustion tests would help establish what protection a registered user actually receives.
Questions for authors
- Can the added-record coverage estimate be validated on a representative subset using genuinely instrumented agent executions, including the provenance of authorization records?
- How do the coverage and false-positive estimates change on a separate, human-adjudicated sample that excludes the calibration cases?
- What guarantee does a registered session receive when several valid claims compete for insufficient reserves? Could coverage admission or payout reservation provide a stronger guarantee?
3 comments
Response to Reviewer imYq (2/2)
Weakness 3
“A valid claim does not necessarily receive compensation. Appendix B.2 states that filing a claim does not reserve its payout, other claims may exhaust the balance, and new claims are rejected when reserves are insufficient. This substantially limits the accountability guarantee: a recorded and verified violation may remain unpaid. The limitation is disclosed, but should be reflected more prominently in the framework's claims. Concurrent-claim and reserve-exhaustion tests would help establish what protection a registered user actually receives.”
Thank you for noticing this. We have updated VAI so that when a challenge is filed, it sets aside the payout p for that challenge, and if R < p at filing time then we would no longer reject that challenge but instead queue the challenges sorted by the filing time. If a queued challenge wins, the user gets the bond back at once and the payout once funds reach it. While nothing forces the provider to add funds and a proven claim can stay owed, the reserve is public and we deliberately leave the price discovery and potential restaking for future work. Our contract tests cover concurrent claims and reserve-exhaustion: claims are paid in filing order whatever the order of the verdicts, a provider's own claim cannot take the payout of a claim filed before it, and a claim filed against an empty reserve is paid once a deposit arrives. Implementation details can be found in Section 3.1, Section 3.3, Algorithm 1 (Appendix A), Appendix B.2 and Section 7.
Question 2
“How do the coverage and false-positive estimates change on a separate, human-adjudicated sample that excludes the calibration cases?”
Now using the sample of the 60 tasks that’s randomly sampled from the benchmarks mentioned above, we score VAI with the updated harness twice, once against the resolved human labels, and once against the LLM majority labels. “Detected” is the number of runs VAI detects out of the runs that were labeled as violations; “False alarms” is the number of runs VAI flags out of the runs that were labeled as no violations. We present the numbers below, in the format (# Detected / # Annotated Violations), where the annotated violations are taken from the human annotators (left column) and the LLM majority (right). These results are reported only for the 60-task validation set on which we ran a human evaluation.
| Against the human baseline | Against the LLM labels | |
|---|---|---|
| τ³-bench detected | 13/25 | 13/22 |
| τ³-bench false alarms | 0/5 | 0/8 |
| ClawsBench detected | 2/5 | 1/3 |
| ClawsBench false alarms | 1/25 | 2/27 |
Question 3
“What guarantee does a registered session receive when several valid claims compete for insufficient reserves? Could coverage admission or payout reservation provide a stronger guarantee?”
We added a mechanism for reserving funds at the time of submitting a claim, so now the payout is set aside at filing and claims are funded in filing order. We did not adopt coverage admission since a user can open infinite sessions, so the reserved amount needed for each promise would be unbounded. This is described in Section 3.1 and Appendix B.2.
Thank you again for your careful review and thoughtful feedback. We would like to follow up before today's discussion deadline to see whether our response has addressed your concerns. Please let us know if there is anything we can further clarify.
Response to Reviewer imYq (1/2)
Thank you for your constructive suggestions. We address your comments below:
Weakness 1
Good point, we have replaced the projection with actual measurements. For each benchmark, we listed the rules (policies) the agent is supposed to follow. If VAI could not check a rule because the harness does not log the information the rule needs, we manually evaluated whether a harness could record that information as structured data, without interpreting natural language. If so, we updated the harness and registered a promise for that rule. In Tau-Bench, the only harness update consisted of one tool that lets the agent ask the user to approve an exact tool call before making it. The user answers ‘approve’ or ‘deny’ and the harness records the request and the answer. In ClawsBench, the only update was that the harness now logs what each tool call returned to the agent, so a promise can check, for example, that the agent read a document before sharing it outside the organization. We then ran both benchmarks again and VAI detected 79.8% of the total violations detected with the updated harness. It doesn't match the previous projected 83% exactly because running the benchmarks again generated a new corpus of agent-user interactions. We have updated the paper in the abstract, Section 1, Section 5.1, Table 1 and Section 5.2.
Weakness 2 / Question 1
We agree and have since removed that Dawid-Skene study and the five-labeler comparison. Three LLMs now label every run independently and we take their 2-of-3 majority as the label. We then sample 30 random runs per benchmark and have two human annotators independently label them, without seeing each other's labels or the LLM labels. Neither annotator had seen the traces before doing this annotation. Before resolution, the two annotators agree on 26/30 runs (Cohen’s kappa = 0.64) on Tau-Bench and 27/30 on ClawsBench (Cohen’s kappa = 0.52). The human annotators then discuss and resolve their disagreements and establish a human baseline. After discussion, the human annotators agreed on all annotations. We compare the LLM majority on those same 60 runs with the human baseline and they agree on 27/30 Tau-Bench runs (Cohen’s kappa = 0.71) and 24/30 ClawsBench runs (Cohen’s kappa = 0.14). The low Cohen’s kappa on ClawsBench is because violations are rare in ClawsBench and the human baseline labeled 5 violations and the LLM majority found only 1 of them. Section 5.1 and Appendix D.2 (Table 7) have been updated to reflect this.