acceptodds
Withdrawn from ICLR 2027

When SQL Correctness Is Not Enough: Verifying Contract-to-Delivery Conformance in Data Agents

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Data agents can produce executable and seemingly reasonable artifacts that nevertheless violate the requirements ultimately established with the user. We formulate this distinct failure mode, *silent delivery non-conformance*, as a weak-oracle verification problem in which both the binding contract and the relevant evidence must be reconstructed from evolving proposals, revisions, approvals, cancellations and tool observations. We propose CER-VERIFY (Contract–Evidence Reconciliation for Verification), a framework that decouples revision-aware contract reconstruction from terminal-state evidence collection and performs requirement-level reconciliation using semantic judgment plus deterministic checks for schema consistency, artifact completeness, and namespace correctness. Across 147 data-agent trajectories spanning nine failure families, CER-VERIFY achieves 121/147 correct verdicts, retains 55/57 conformant deliveries, and detects 66/90 non-conformant ones, compared with 93/147 correct verdicts for the strongest standard baseline and 101/147 for the strongest component-matched control. Structural experiments show that, when the verification target evolves during interaction, errors arise primarily before the final correctness decision: the verifier must first infer what remains binding and resolve what was actually delivered. Deterministic checks become reliable only after this inference. Our findings therefore suggest that verification for persistent-state data agents is fundamentally a state-reconstruction problem before it is a correctness-checking problem.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
Data Agents, Agent Verification, Delivery Conformance, Weak-Oracle Verification

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_9QPTOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies whether a data agent’s final delivery satisfies the requirements accepted over the course of an interaction, including subsequent revisions and cancellations. It proposes CER-VERIFY, which reconstructs an active requirement ledger, resolves terminal artifact evidence, and combines LLM judgments with deterministic checks for schemas, artifact completeness, and namespaces. Evaluation uses 147 trajectories comprising 65 production cases, 31 counterfactual failures, and 51 context-stress variants. The method produces 121 correct verdicts, compared with 101 for the strongest component-matched control.

Critical strengths
  • The paper addresses a practical issue: executable SQL and successful tool execution do not establish that an agent delivered what the user agreed to.
  • Checking accepted requirements against actual terminal artifacts is useful, and the reported improvement in retaining conformant deliveries is encouraging. This motivates the application, but does not establish the claimed methodological contribution.
Critical weaknesses
  1. The formalization adds notation without settling the central technical questions.

Section 2 puts the hardest parts of the problem inside the functions Active & Observe: deciding which requirements still apply & finding authoritative terminal evidence. The equations name these operations, but they add little detail about how they work or when their outputs can be trusted. Equations (3) & (4) describe requirement checking & verdict aggregation, while Equation (7) gives deterministic contradictions priority over semantic judgments. These rules could be explained more clearly in plain language.

The implementation details give record schemas and shortened prompts, but key decisions such as partial supersession, ambiguous acceptance, artifact identity and evidence freshness still depend on semantic inference. The behavior of that inference is not specified well enough. I find the methodological novelty limited & the formal presentation goes further than what it actually establishes.

The computational discussion in Section 3.6 has the same weakness. Its linear bounds cover indexing and comparison after requirements, artifacts & schema fields have already been resolved. Those bounds make sense for the operations described, but they leave out the difficult reconstruction and resolution stages that motivate the paper. They say little about the computational difficulty of the proposed verifier. Repeated abstract descriptions of grounding, reconstruction and reconciliation also make the actual algorithmic contribution harder to see.

  1. The evaluation does not prove the claimed reason for the gains and validation of the reference labels is still incomplete.

The paper argues that recovering the active contract & terminal evidence is the main reason verification improves. Table 2 does not clearly prove that claim. Revision aware target reconstruction raises correct verdicts from 84 to 91, while adding evidence binding lowers them to 79. The large increase to 121 comes from deterministic validators. The paper admits that the earlier transitions also change how information is presented. So these comparisons do not separate the contribution of contract reconstruction from that of evidence binding. The clearest controlled result supports deterministic checks within this pipeline, not the broader explanation put forward in the paper.

The empirical basis is also still preliminary. Appendix D.6 says that independent annotation auditing is pending for part of the corpus. This matters because deciding the binding contract is the main ambiguity being studied: an independently checked reference contract and verdict are needed to judge if the verifier resolves that ambiguity correctly.

Questions for authors

Please try to address the two Weaknesses mentioned above.

Reviewer_CUzdOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies silent delivery non-conformance in data agents: cases where the final artifacts are executable but violate the latest requirements accepted during interaction. CER-VERIFY reconstructs the active contract, resolves terminal delivery evidence, and reconciles the two using semantic reasoning plus deterministic checks for schema, artifact completeness, and namespace consistency.

On 147 trajectories, CER-VERIFY achieves 121/147 correct verdicts, retaining 55/57 conformant deliveries and detecting 66/90 non-conformant ones. The paper argues that verification in persistent-state agents requires recovering the binding target and terminal state before final correctness can be reliably assessed.

Critical strengths

1. Important problem formulation.

The distinction between executable output and conformance to the latest accepted requirements is meaningful for persistent-state agents. I found the interaction-derived contract formulation useful and practically relevant.

2. Well-motivated verification decomposition.

Separating contract reconstruction, evidence resolution, and final reconciliation is intuitive and supported by a broad set of controls, robustness experiments, and failure analyses.

Critical weaknesses

1. The Gold-label reliability is not yet fully established.

The paper states that independent double annotation and adjudication are still pending for part of the corpus. If the Gold active contract or final verdict is uncertain on even a modest number of cases, the relative ranking of methods and the failure-locus analysis may change.

I would know how many of the 147 cases have actually completed independent double annotation and adjudication, what the agreement is before adjudication, and whether the main results remain similar on the fully audited subset or after completing the audit.

2. The experiments do not fully isolate the claimed mechanism.

The paper argues that state reconstruction is fundamental to verification, but several structural variants change both the introduced component and the information representation. In addition, only 65 of the 147 cases are production trajectories, with the rest consisting of controlled or derived variants.

Consequently, the transitions in Table 2 do not by themselves isolate the causal contributions of contract reconstruction and evidence binding?

Questions for authors

Q1. How many of the 147 cases have completed independent double annotation and adjudication? Could you report agreement statistics and results on the fully audited subset?

Q2. Was CER-VERIFY, including its prompts, failure taxonomy, and deterministic validators, developed using any of the 147 evaluation cases? Is there an independently held-out evaluation set?

Q3. Can you provide a cleaner causal analysis of the proposed intermediate states, for example using Gold active contracts and/or Gold evidence bindings while keeping the downstream verifier fixed?

Reviewer_bZpPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper addresses the scenario of delivery verification for data agents. The authors identify a problem where data agents may execute tasks successfully but fail to meet user requirements, a phenomenon they term “Silent Delivery Non-conformance.” By proposing the CER-VERIFY framework, the authors extract a set of valid user requirements from multi-round interactions and collect the agent’s deliverables, database schemas, and tool execution evidence to determine whether the agent’s behavior satisfies user requirements. On a custom-built agent trace dataset, CER-VERIFY achieved the best results.

Critical strengths
  1. The research question has clear practical value and differs from the Exact Match evaluation used in traditional SQL.
  2. The method demonstrates a certain degree of cross-model stability and effectively increases the compliance delivery rate.
Critical weaknesses
  1. Regarding “Silent Delivery Non-conformance,” the specific methods for determining (1) “valid under local execution checks” and (2) “the terminal delivery violates at least one active requirement” are not clearly explained; only formal definitions are provided.
  2. The composition of the dataset is complex: some data comes from real production environments, some is artificially constructed, and some is derived from existing tasks. The validity of this composition is questionable. Furthermore, the current dataset has not undergone an independent annotation audit, casting doubt on the reliability of the labels.
  3. There appears to be an inconsistency between the dataset and the “Silent Delivery Non-conformance” issue. Since “Silent Delivery Non-conformance” requires both passing local execution checks and the final delivery being non-compliant, the dataset samples do not explicitly state that they have all passed local execution checks.
  4. The CER-VERIFY process is more complex and resource-intensive than the Baseline; there is no guarantee that it will achieve better results at the same cost. Additionally, some simple Baseline approaches were not compared, such as using large language models to summarize requirements and perform rule checks in Python or SQL.
Questions for authors
  1. Do all samples in the dataset meet the definition of “Silent Delivery Non-conformance”?
  2. Can you provide a robust baseline for calculating budget matching?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.