Action-Conditioned Evidence Contracts for Prospective Evaluation of Scientific Agents
Abstract
Final-answer correctness alone does not reveal whether a scientific agent used admissible evidence or revised its reported belief appropriately. We introduce a prospective evaluation protocol in which an agent commits to a belief and an action before receiving evidence, then reports an updated belief and proposes releasing or withholding its conclusion. A Prospective Scientific Episode links these records; an Action-Conditioned Evidence Contract supplies admissibility constraints. On 160 synthetic episodes, runtime filtering adds holds for a non-checking rule policy but none for aligned policies. Across 480 runs of the non-checking policy, full enforcement blocks 33 releases admitted by provenance-and-quality checks; in 32 of these, the posterior mode matches the oracle label. At matched coverage, contract risk falls from 10.0% to zero, while oracle error risk rises from 14.55% to 15.82%. Reanalysis of 313 historical language-model trajectories shows only one additional hold from runtime filtering; supplied likelihoods and a domain–condition confound restrict these results to diagnostics of recorded responses. A live register and an eight-task executable pilot illustrate the workflow without establishing a capability gain. The linked episode record exposes disagreements between admissibility, outcome correctness, and release decisions that a single “verified” label would hide.
est. 32% chance this paper gets accepted at ICLR 2027.
What do you think this paper will get?
All positions stay anonymous.