acceptodds
Withdrawn from ICLR 2027

Actions with Receipts: Jointly Binding Claims, Evidence, and Execution for Replayable Tool-Agent Auditing

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Tool-using agents can expose citations and execution logs while leaving a critical association unaudited: whether the claim shown to a user is the claim emitted by the committed execution and supported by the cited source. A valid citation and a valid trace can therefore remain individually well formed while being transplanted across claims, actions, runs, or source versions. We introduce a claim-anchored execution contract that jointly binds the emitted claim, its exact source span, the ordered execution prefix that produced it, and the source version and access state observed by that execution. Each receipt contains an emission anchor that deterministically locates the claim inside a committed answer or claim-bearing action, together with source identifiers, offsets, hashes, quotes, and a domain-separated execution commitment. A deterministic integrity verifier reconstructs these bindings before semantic or task labels are joined. We separate this integrity plane from a pluggable support plane, so structural validity is not used as a proxy for entailment. The contract exposes seven independently testable properties: claim-emission binding, source binding, ordered-execution binding, oracle separation, persisted-object replay, execution-rerun consistency, and version/access binding. This design reports what was emitted, what evidence was bound to it, which execution produced it, whether the association can be replayed, and whether the evidence supports the claim.

Primary area:
reinforcement learning
Keywords:
Tool-Using Agents, Large Language Models, Agent Auditing, Provenance Tracking, Execution Traces, Evidence Grounding, Replayable Systems, Reliable AI

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_6aVfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces claim-anchored receipts that bind an emitted claim to its source span, execution trace, and source state. It separates deterministic integrity checks from semantic support evaluation. Experiments assess substitution detection, semantic verification, agent performance, and robustness to changing sources.

Critical strengths
  1. Explicit claim anchors address the gap between citation records and execution logs, which is whether the cited claim actually appears in the recorded execution.

  2. Separating integrity from semantic support avoids treating a valid receipt as evidence of factual correctness. The lexical-adversarial controls illustrate this distinction clearly.

  3. The evaluation includes targeted binding ablations, matched cost controls, and a factorial study separating receipt instrumentation from stopping and guarding.

Critical weaknesses
  1. Section 3.3 hashes the trace and receipt, but the main text does not specify how the verifier obtains an independently trusted commitment. If an attacker can replace the persisted objects and recompute their hashes, internal consistency alone cannot establish integrity. The threat model should state which commitment remains protected.

  2. The contract does not explicitly establish that cited evidence was observed before the claim was emitted. The emission anchor locates the claim, and the source checks validate the cited span, but the stated verification rules do not require a matching retrieval event preceding that anchor. Binding these objects together therefore may not, by itself, establish the claimed execution provenance.

  3. The coverage definition requires at least one individual span to support the claim, which misses jointly sufficient evidence. For example, “Alice was born in Lyon” and “Lyon is in France” jointly support “Alice was born in France,” although neither alone entails it. How does the method assess coverage for such multi-hop claims?

  4. Reproducibility remains limited. Appendix E distinguishes an executable two-event fixture from reported aggregates and an external corpus bundle. The supplied PDF does not establish access to the per-case evidence needed to verify the headline attack and agent results.

  5. Presentation issues:

    • The introduction makes broad claims about existing benchmarks and citation evaluators without direct citations. Linking these claims to the closest prior work would make the proposed research gap more convincing.
    • The headline substitution result in Section 4.2 lacks attack examples and an explanation of the five undetected cases. These details matter for understanding the contract’s practical failure boundary.
Questions for authors
  1. Which commitment is independently trusted during verification, and what prevents an attacker from modifying the persisted objects and recomputing their hashes?

  2. Does the verifier require each cited span to match an observation event preceding the claim’s emission? Where is this constraint enforced?

  3. How is coverage assessed when multiple spans jointly support a claim, but no individual span is sufficient?

  4. Since lexical support also controls early stopping, how does the policy avoid stopping on high-overlap evidence that reverses the claim’s meaning, such as “Alice defeated Bob” versus “Bob defeated Alice”?

Reviewer_UXMQOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies transplantation ambiguity in tool-agent auditing: individually valid citations and execution traces need not establish that a cited claim belongs to the execution being audited. The paper proposes a joint receipt that binds the exact emitted substring, source spans, an ordered execution prefix, and source version/access-state identities. Deterministic integrity verification is separated from a pluggable semantic-support evaluator, and persisted-bundle replay is distinguished from execution rerun.

The evaluation includes fixed-corpus protocol checks, structural and lexical mutations, 1,280 cross-object substitutions, seven-property ablations, a 384-pair adjudicated semantic study, a 2,550-task three-seed agent evaluation, and dynamic-document/injection tests. The paper reports detecting 1,275 substitutions and semantic-support F1 of 0.8865 on the adjudicated split.

Critical strengths
  1. A concrete auditing contract. The paper makes the association between an emitted claim, its evidence, and its recorded execution explicit and reconstructible. Exact emission anchors address a useful gap left by storing citations and traces independently.

  2. Separation of integrity and semantic support. The paper appropriately avoids treating a valid hash or receipt as proof of entailment. Oracle separation and property-specific ablations provide a useful framework for distinguishing record integrity, support quality, and task outcomes.

Critical weaknesses
  1. Inconsistent central replay accounting. Table 26 (p. 30) defines joint replay as success of both bundle replay and execution rerun, yet all four receipt-enabled rows report joint replay above execution rerun: 0.9987 versus 0.9981, 0.9984 versus 0.9972, 0.9978 versus 0.9967, and 0.9976 versus 0.9964. With the same execution units, denominators, and aggregation weights, an intersection cannot exceed its marginal. Four-decimal rounding cannot explain these gaps. Different populations could explain them, but the paper does not specify such distinctions clearly enough to interpret a central reliability result.

  2. Insufficient identification of the main experimental settings. The paper reports 2,550-task agent results and frozen NLI/LLM comparisons without identifying the concrete agent and evaluator models or the precise task-source composition, sampling, filtering, and adaptation in the manuscript (B.6, D.7, D.10-D.11). B.6 and E.2 defer identities to manifests. These choices directly affect baseline strength, utility, latency, and interpretation of small semantic-score differences. Identifiable configurations and the corresponding manifests are needed to assess the empirical contribution.

The paper's principal empirical conclusions are insufficiently substantiated in the current presentation. The inconsistent replay accounting and incomplete experiment specification motivate the recommendation of clear rejection.

Questions for authors
  1. What are the evaluation unit, denominator, bundle-pass count, rerun-pass count, and intersection count for each receipt-enabled row of Table 26, and how do these reconcile with Tables 6 and 23, including the latter's disjoint first-failure categories?

  2. What are the exact agent, NLI, and LLM-judge model/version identities and the construction and category composition of the 2,550-task and semantic test sets, and which submitted manifests or row-level artifacts contain these details and support reconstruction of the headline results?

Additional suggestions:

  1. Provide minimal cases and reason codes for the five undetected substitutions and reported false rejections, with clean-control denominators and the exact checks disabled in each property ablation.

  2. Report family-level semantic outcomes and paired difference intervals against the NLI/LLM evaluators, including the supported-claim recall and coverage tradeoff.

Reviewer_SyBEOpenReview ↗
Rating
2
Confidence
1
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper seems to be about auditing whether a claim shown to a user is the claim emitted by the committed execution and supported by the cited source.

Honestly speaking, I do not understand this paper. It is perhaps due to my lack of expertise in the area, but the paper should have been written better.

Critical strengths

See above

Critical weaknesses

See above

Questions for authors

See above

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.