acceptodds
Under review as a conference paper at ICLR 2027

Action-Conditioned Evidence Contracts for Prospective Evaluation of Scientific Agents

Abstract

Final-answer correctness alone does not reveal whether a scientific agent used admissible evidence or revised its reported belief appropriately. We introduce a prospective evaluation protocol in which an agent commits to a belief and an action before receiving evidence, then reports an updated belief and proposes releasing or withholding its conclusion. A Prospective Scientific Episode links these records; an Action-Conditioned Evidence Contract supplies admissibility constraints. On 160 synthetic episodes, runtime filtering adds holds for a non-checking rule policy but none for aligned policies. Across 480 runs of the non-checking policy, full enforcement blocks 33 releases admitted by provenance-and-quality checks; in 32 of these, the posterior mode matches the oracle label. At matched coverage, contract risk falls from 10.0% to zero, while oracle error risk rises from 14.55% to 15.82%. Reanalysis of 313 historical language-model trajectories shows only one additional hold from runtime filtering; supplied likelihoods and a domain–condition confound restrict these results to diagnostics of recorded responses. A live register and an eight-task executable pilot illustrate the workflow without establishing a capability gain. The linked episode record exposes disagreements between admissibility, outcome correctness, and release decisions that a single “verified” label would hide.

open until 14 Dec 2026

est. 32% chance this paper gets accepted at ICLR 2027.

Reject 68%Accept 32%

What do you think this paper will get?

All positions stay anonymous.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.