- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper studies whether a data agent’s final delivery satisfies the requirements accepted over the course of an interaction, including subsequent revisions and cancellations. It proposes CER-VERIFY, which reconstructs an active requirement ledger, resolves terminal artifact evidence, and combines LLM judgments with deterministic checks for schemas, artifact completeness, and namespaces. Evaluation uses 147 trajectories comprising 65 production cases, 31 counterfactual failures, and 51 context-stress variants. The method produces 121 correct verdicts, compared with 101 for the strongest component-matched control.
Critical strengths
- The paper addresses a practical issue: executable SQL and successful tool execution do not establish that an agent delivered what the user agreed to.
- Checking accepted requirements against actual terminal artifacts is useful, and the reported improvement in retaining conformant deliveries is encouraging. This motivates the application, but does not establish the claimed methodological contribution.
Critical weaknesses
- The formalization adds notation without settling the central technical questions.
Section 2 puts the hardest parts of the problem inside the functions Active & Observe: deciding which requirements still apply & finding authoritative terminal evidence. The equations name these operations, but they add little detail about how they work or when their outputs can be trusted. Equations (3) & (4) describe requirement checking & verdict aggregation, while Equation (7) gives deterministic contradictions priority over semantic judgments. These rules could be explained more clearly in plain language.
The implementation details give record schemas and shortened prompts, but key decisions such as partial supersession, ambiguous acceptance, artifact identity and evidence freshness still depend on semantic inference. The behavior of that inference is not specified well enough. I find the methodological novelty limited & the formal presentation goes further than what it actually establishes.
The computational discussion in Section 3.6 has the same weakness. Its linear bounds cover indexing and comparison after requirements, artifacts & schema fields have already been resolved. Those bounds make sense for the operations described, but they leave out the difficult reconstruction and resolution stages that motivate the paper. They say little about the computational difficulty of the proposed verifier. Repeated abstract descriptions of grounding, reconstruction and reconciliation also make the actual algorithmic contribution harder to see.
- The evaluation does not prove the claimed reason for the gains and validation of the reference labels is still incomplete.
The paper argues that recovering the active contract & terminal evidence is the main reason verification improves. Table 2 does not clearly prove that claim. Revision aware target reconstruction raises correct verdicts from 84 to 91, while adding evidence binding lowers them to 79. The large increase to 121 comes from deterministic validators. The paper admits that the earlier transitions also change how information is presented. So these comparisons do not separate the contribution of contract reconstruction from that of evidence binding. The clearest controlled result supports deterministic checks within this pipeline, not the broader explanation put forward in the paper.
The empirical basis is also still preliminary. Appendix D.6 says that independent annotation auditing is pending for part of the corpus. This matters because deciding the binding contract is the main ambiguity being studied: an independently checked reference contract and verdict are needed to judge if the verifier resolves that ambiguity correctly.
Questions for authors
Please try to address the two Weaknesses mentioned above.