- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes Proof-Carrying Agent Contracts (PACT), a runtime framework that separates an agent’s proposed actions from authorization to execute them. PACT represents execution constraints in typed contracts, checks applicable obligations against current state before releasing side effects, and records authorization evidence and declared recovery paths. Its formal analysis establishes conditional release soundness, monotonicity under contract refinement, and correctness of cached verification. Evaluation uses VeraContract-Bench, which adds deployment constraints to 180 SWE-bench-derived instances. The authors report reduced unsafe releases with similar task success, supported by component ablations and measurements of recovery, verification latency, and operator diagnosis.
Critical strengths
The paper makes a useful distinction between task completion and admissible execution: an agent can produce a correct result through unauthorized actions. Evaluating both outcomes captures deployment failures that task-success metrics alone would miss.
The reported results suggest that PACT reduces unsafe releases without a substantial loss in task success. Ablations separately examine live revalidation, recovery handling, and proof reuse, providing evidence about the roles of individual components, although they do not resolve my concerns about novelty and baseline adequacy.
Critical weaknesses
The technical contribution beyond existing runtime enforcement is unclear. PACT combines policy checks, approval handling, recovery declarations, and evidence recording, but the paper does not establish which capability requires its proposed contract framework. The formal guarantees assume complete policy coverage, sound verifiers, and trusted execution binding, rather than demonstrate how these difficult conditions are achieved. This limits the case for a substantive advance over existing approaches.
The evaluation does not convincingly isolate the framework’s practical advantage. Although a runtime-judge baseline is included, its specification is insufficient to assess whether it provides comparable policy coverage and access to current state. A simpler rule-based router with an independent verifier for ambiguous actions remains a plausible alternative. Without a comparison that matches these factors, the reported gains could reflect more comprehensive checks rather than the proposed architecture, leaving its additional complexity insufficiently justified.
Questions for authors
Could you clarify how the runtime-judge and AgentSpec-style baselines are implemented, particularly their policy coverage, access to current state, and handling of approval expiry and policy changes? Are these factors matched to PACT, so that the reported gains isolate the contribution of the contract framework?
What specific capability does PACT provide beyond deterministic policy checks combined with an independent verifier for ambiguous actions and commit-time state revalidation? Please identify a concrete failure case and explain which proposed mechanism addresses it.
What backbone model, decoding settings, and hardware were used, and what are the paired confidence intervals for the main performance differences? Please also clarify whether the latency comparison includes contract compilation, verification, and repair costs. These are requests to clarify the current evidence, rather than suggestions for future work.
Matched baseline, recovery dependencies, and cost
Thank you. We evaluated your strengthened rule-router alternative. PACT retains substantial gains with separate verification and commit-time revalidation.
1. Strengthened baseline and parity. All methods share source tasks, repository snapshots, task-card policies, backbone, and budgets (40 tool calls, 1800 s). The runtime judge uses a rule-plus-model decision over policy text and the action record; AgentSpec-style applies deterministic rules before tool execution (Appendix E).
The strengthened baseline adds a separate runtime-verifier invocation for judgment-dependent actions, live-state access, approval-expiry and policy-epoch checks, and commit-time revalidation. It retains AgentSpec-style action blocking and agent-driven recovery under the same task-card eligibility/scoring rules: recovery improves from 54.8% to 58.2%. PACT adds the compiled action graph, graph-linked compensation preconditions, proof reuse, and Ω.
Task/UAR/VRR/Recovery/GSR are 62.9/5.6/81.3/58.2/40.6% for the strengthened baseline and 63.5/1.8/95.1/79.4/49.3% for PACT-full. RCR is 93.8% for the strengthened baseline and 98.2% for PACT-full (Table 2). Paired differences (PACT minus baseline), in percentage points with 95% intervals:
PACT retains safety, recovery, and governed-completion gains with similar observed task utility. Table 2 reports this comparison and intervals; Table 1 in Section 6 documents capability matching.
2. What the contract adds. PACT compiles task-specific cross-action dependencies into typed release obligations. The compiler instantiates G=(V,Efwd,Ecomp), checks mandatory-obligation closure, and validates recovery/evidence references. The gate checks obligations in current state and binds the verdict to βt=(da,dC,ds,ρt) (Eq. 9).
Appendix G, Case 2 records an LLMCompiler-style parallel refactor that removes a test file required by a later staging action's compensation path. Current-state access reveals the file's absence; the compiled graph represents its role as a prerequisite of that compensation path and staging release. PACT checks this graph-linked prerequisite at the staging commit (Eqs. 7, 29), alongside Pattern fan-out constraints, and withholds the effect when it fails. The evaluated action-local baseline has no compiled compensation dependency linking that resource to the later release. This illustrates the additional obligation that PACT constructs and enforces.
One-at-a-time ablations retain the remaining PACT configuration: removing compensation edges lowers recovery from 79.4% to 38.5% and GSR from 49.3% to 22.8%; removing critical-subgraph checks raises UAR from 1.8% to 4.4%; removing bounded safe downgrade lowers Task from 63.5% to 58.7%; removing Ω lowers blame localization from 87.4% to 71.9% (Tables 6, 8).
Eq. 13 defines complete dependency tuples. Under tuple equality, deterministic verification, unchanged verifier versions, and valid cache signatures, Theorem 2 equates reuse with recomputation. Across 600 trials mutating absence predicates, expiry, policy epochs, registry/verifier versions, and action/contract digests, cached-path decisions matched full recomputation in 600/600 cases, with correct invalidation of affected entries.
3. Settings and statistics. All methods use claude-sonnet-4-5-20250929 via an OpenAI-compatible chat-completions endpoint. Proposals: max_tokens 4096 and temperature 0.0 as the only sampling parameter sent; matched harness labels {17,42,101}. Full PACT compilation uses temperature 0.0, requests strict JSON-schema responses, and has k=5 total attempts. Local hosts: 4 A100-SXM4-80GB GPUs, dual EPYC 7763 CPUs, 512 GB RAM.
Intervals use 10,000 hierarchical bootstrap replicates with source-issue clusters and nested matched harness runs; primary tests use Holm control. Against original AgentSpec-style, paired 95% intervals are delta UAR -5.3 [-6.8,-3.9], recovery +24.6 [+19.8,+29.4], and GSR +12.1 [+8.4,+15.9].
4. Full latency accounting. First-attempt compilation P50/P99 is 2.9/7.4 s; repair instances average 1.4 steps, with per-step P50 of 1.6 s. Measured compilation and repair contribute under 3% of run wall-clock within the shared 1800 s budget. Verification-path P50/P95/P99 is 612/938/1617 ms cold and 246/378/634 ms with proof reuse, a warm runtime, and speculative preparation. The combined configuration reduces path P99 by 60.8% (Table 9).
Final synchronous gate P99 is 64 ms, versus AgentSpec-style's 96 ms and the rule-verifier's 118 ms (Table 2). Preverified stable obligations reduce repeated work; live-dependent checks remain at release. Section 7 and Appendix I.3 define compilation/repair, verification-path, and synchronous-gate timing boundaries. We would appreciate reconsideration of your assessment and score.