acceptodds
Under review as a conference paper at ICLR 2027

Action-Conditioned Evidence Contracts for Prospective Evaluation of Scientific Agents

Abstract

Final-answer correctness alone does not reveal whether a scientific agent used admissible evidence or revised its reported belief appropriately. We introduce a prospective evaluation protocol in which an agent commits to a belief and an action before receiving evidence, then reports an updated belief and proposes releasing or withholding its conclusion. A Prospective Scientific Episode links these records; an Action-Conditioned Evidence Contract supplies admissibility constraints. On 160 synthetic episodes, runtime filtering adds holds for a non-checking rule policy but none for aligned policies. Across 480 runs of the non-checking policy, full enforcement blocks 33 releases admitted by provenance-and-quality checks; in 32 of these, the posterior mode matches the oracle label. At matched coverage, contract risk falls from 10.0% to zero, while oracle error risk rises from 14.55% to 15.82%. Reanalysis of 313 historical language-model trajectories shows only one additional hold from runtime filtering; supplied likelihoods and a domain–condition confound restrict these results to diagnostics of recorded responses. A live register and an eight-task executable pilot illustrate the workflow without establishing a capability gain. The linked episode record exposes disagreements between admissibility, outcome correctness, and release decisions that a single “verified” label would hide.

Primary area:
datasets and benchmarks
Keywords:
scientific agents, agent evaluation, prospective evaluation, evidence contracts, runtime verification

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_hF4WOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes an evaluation protocol for scientific agents, the agent commits to a belief, a prediction, a falsifier, and an action. this commitment is hashed then to make sure it's irreversible. It then receives only the evidence from the chosen action and returns a revised belief together with a proposal on what to do next. another component checks for the correctness of this flow.

Critical strengths
  • the motivation is on time and there is indeed need for making sure that what agents propose is actually grounded in real, correct evidences
Critical weaknesses
  • The paper is very hard to read. The abstract packs in numbers whose meaning depends on values defined much later. Almost every result sits among several qualifications. Version-history details also appear in the paper, for example “V3.1 displayed 14.4,” the salt mismatch, and recovered-script rounding differences. That material doesn't belongs in a conference submission. The main text spends more space explaining what each result does not show (e.g. this is this not that) than what it does show. The introduction is too small and it doesn't give enough motivation + clear explanation of current challenges and the solution, it just jumps to next section.

  • The novelty is low, the paper explicitly disclaims new hashing, timestamping, preregistration. It also states that ACEC “supplies the binding and admission mechanism,” other works like Preregistered Belief Revision Contracts (Alqithami, 2026) already specify evidence triggers, admissible revision, and verifiable evidence. ResearchLoop (Xia & Wang, 2026) already provides evidence-gated admission rules, it doesn't explicitly show what is novel in the paper and the contributions seems to not be enough

Questions for authors
  • How would you validate falsifiers?
  • How would you make sure the distribution returned as belief distribution is actually correctly formalized by the agent?
Reviewer_pZX1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Prospective Scientific Episodes (PSE) and Action-Conditioned Evidence Contracts (ACEC), linking prior commitments, controlled evidence acquisition, belief revision, and release decisions. Through synthetic tasks, historical model responses, and real-data pilots, it examines the incremental effects of runtime enforcement. The main concerns are whether these highly structured tasks adequately represent scientific work and whether their findings transfer to end-to-end agents.

Critical strengths
  • Clear distinction between procedural compliance and outcome correctness. The paper separately evaluates evidence admissibility, belief revision, and release decisions, avoiding the assumption that a correct answer or a passed check necessarily indicates reliable scientific reasoning.

  • Transparent reporting of limitations. The authors explicitly disclose the cost shortcut, domain confounding, negative results, and implementation boundaries, helping readers assess the scope of the conclusions.

Critical weaknesses
  • The tasks are simplified and have limited ability to distinguish scientific capabilities. The synthetic tasks directly supply hypotheses, priors, likelihoods, and a fixed action menu, primarily testing probability calculations and rule following. Selecting the most expensive feasible action achieves optimal EIG, making it impossible to distinguish information-based reasoning from a price heuristic. Perfect confounding between domain and fault status further limits interpretation of model behavior. The executable pilot, where both conditions succeed on all eight tasks, provides little additional discriminating evidence.

  • Falsifier evaluation remains superficial. Although agents must declare a falsification condition in advance, the implementation only checks whether the text contains at least three whitespace-separated tokens. It does not assess scientific meaning, testability through the selected action, or whether subsequent revisions honor the commitment. Consequently, the experiments do not substantively evaluate this central component of the proposed workflow.

  • The connection between the constrained protocol and end-to-end evaluation is not established. The protocol prescribes the workflow, restricts evidence flow, and uses a separate call for the revision stage. These conditions differ substantially from autonomous task execution followed by trajectory analysis. The paper does not establish whether performance, failure modes, or intervention effects transfer to end-to-end settings. Moreover, issues such as premature access to future data may also be prevented through environment-side access controls. The paper should clarify which problems require its protocol and which can be addressed by ordinary environmental constraints.

Questions for authors
  • What does this method add beyond end-to-end trajectory analysis? Which failures specifically require prior commitments and staged evidence disclosure to identify, and which could also be addressed through trajectory auditing or environment-side access controls?

  • Do findings transfer between the two evaluation settings? Could the authors compare staged and end-to-end execution using the same tasks and models, examining whether model rankings and failure modes remain consistent and distinguishing assistance provided by the protocol from the models’ own capabilities?

Reviewer_jeZpOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

In this paper, the authors propose a new way to test scientific AI agents that looks at more than whether the final answer is correct. The agent first states its belief and picks which evidence to collect. It then sees only that evidence, updates its belief, and says whether its conclusion should be released or held back. A set of rules, called a contract, checks whether the evidence was allowed, for example whether it came too early or from a bad source, and a runtime check can block a release that breaks the rules. On synthetic episodes and past model runs, the authors show that these checks help only when the agent does not already check itself, and that blocking rule-breaking releases does not make outcomes more correct. They also find that simply picking the most expensive action matched the best action every time, which exposes a weakness in how action quality is scored.

Critical strengths
  1. The experiments are controlled and one of the findings is informative. Choosing the most expensive allowed action reached the best information gain on every menu, because costs and informativeness were paired by rank.
Critical weaknesses
  1. The evidence is mostly synthetic and built to produce the results. The main experiments use 160 generated episodes where faults and rules are designed by the authors. Oracle truth comes from the same generator, and the gates trust the flags the generator provides.
  2. The language-model results are confounded and thin. Only two models from one provider were tested, there is no same-task control, and several endpoints (information gain, matched coverage) were added after looking at the data.
  3. The real-world tests are too small to show anything. The live forecasts have one properly timed batch, correlated predicates, and gaps from the baseline that include zero. A simple “always no” forecast beat both models. The eight-task pilot succeeded in every case in both arms, so it shows neither a benefit nor equivalence.
  4. The figures are also very rudimentary.
  5. The paper is difficult to follow
Questions for authors

N/A

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.