- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper introduces claim-anchored receipts that bind an emitted claim to its source span, execution trace, and source state. It separates deterministic integrity checks from semantic support evaluation. Experiments assess substitution detection, semantic verification, agent performance, and robustness to changing sources.
Critical strengths
-
Explicit claim anchors address the gap between citation records and execution logs, which is whether the cited claim actually appears in the recorded execution.
-
Separating integrity from semantic support avoids treating a valid receipt as evidence of factual correctness. The lexical-adversarial controls illustrate this distinction clearly.
-
The evaluation includes targeted binding ablations, matched cost controls, and a factorial study separating receipt instrumentation from stopping and guarding.
Critical weaknesses
-
Section 3.3 hashes the trace and receipt, but the main text does not specify how the verifier obtains an independently trusted commitment. If an attacker can replace the persisted objects and recompute their hashes, internal consistency alone cannot establish integrity. The threat model should state which commitment remains protected.
-
The contract does not explicitly establish that cited evidence was observed before the claim was emitted. The emission anchor locates the claim, and the source checks validate the cited span, but the stated verification rules do not require a matching retrieval event preceding that anchor. Binding these objects together therefore may not, by itself, establish the claimed execution provenance.
-
The coverage definition requires at least one individual span to support the claim, which misses jointly sufficient evidence. For example, “Alice was born in Lyon” and “Lyon is in France” jointly support “Alice was born in France,” although neither alone entails it. How does the method assess coverage for such multi-hop claims?
-
Reproducibility remains limited. Appendix E distinguishes an executable two-event fixture from reported aggregates and an external corpus bundle. The supplied PDF does not establish access to the per-case evidence needed to verify the headline attack and agent results.
-
Presentation issues:
- The introduction makes broad claims about existing benchmarks and citation evaluators without direct citations. Linking these claims to the closest prior work would make the proposed research gap more convincing.
- The headline substitution result in Section 4.2 lacks attack examples and an explanation of the five undetected cases. These details matter for understanding the contract’s practical failure boundary.
Questions for authors
-
Which commitment is independently trusted during verification, and what prevents an attacker from modifying the persisted objects and recomputing their hashes?
-
Does the verifier require each cited span to match an observation event preceding the claim’s emission? Where is this constraint enforced?
-
How is coverage assessed when multiple spans jointly support a claim, but no individual span is sufficient?
-
Since lexical support also controls early stopping, how does the policy avoid stopping on high-overlap evidence that reverses the claim’s meaning, such as “Alice defeated Bob” versus “Bob defeated Alice”?