When SQL Correctness Is Not Enough: Verifying Contract-to-Delivery Conformance in Data Agents

Data agents can produce executable and seemingly reasonable artifacts that nevertheless violate the requirements ultimately established with the user. We formulate this distinct failure mode, *silent delivery non-conformance*, as a weak-oracle verification problem in which both the binding contract and the relevant evidence must be reconstructed from evolving proposals, revisions, approvals, cancellations and tool observations. We propose CER-VERIFY (Contract–Evidence Reconciliation for Verification), a framework that decouples revision-aware contract reconstruction from terminal-state evidence collection and performs requirement-level reconciliation using semantic judgment plus deterministic checks for schema consistency, artifact completeness, and namespace correctness. Across 147 data-agent trajectories spanning nine failure families, CER-VERIFY achieves 121/147 correct verdicts, retains 55/57 conformant deliveries, and detects 66/90 non-conformant ones, compared with 93/147 correct verdicts for the strongest standard baseline and 101/147 for the strongest component-matched control. Structural experiments show that, when the verification target evolves during interaction, errors arise primarily before the final correctness decision: the verifier must first infer what remains binding and resolve what was actually delivered. Deterministic checks become reliable only after this inference. Our findings therefore suggest that verification for persistent-state data agents is fundamentally a state-reconstruction problem before it is a correctness-checking problem.