acceptodds
Under review as a conference paper at ICLR 2027

Scores Alone Do Not Prove Discovery: The Discovery Certification Protocol for Auditing AI Research Agents

Abstract

AI research agents combine public information and experimental feedback to produce measurable results. The Discovery Certification Protocol (DCP) turns an outcome claim into an executable audit under a registered model, information boundary, and budget. Gate 1 validates useful improvement. Gate 2 tests recovery by matched agents given the starting information and observed Web content, with run history and new measurements withheld. Core requires adequate registered controls, zero recoveries, and a finite-sample recovery bound. Optional Gate 3 compares truthful and neutral feedback from a shared checkpoint; Evidence adds a supported effect and a null-policy equivalence check. Controlled SQLite and virtual catalyst audits pass both decision kernels. On real-data response surfaces, Yacht and Ionosphere pass the Core kernel after zero recoveries in 96 attempts, with an upper bound of 0.0468. Each target combines ten observed utilities and six predictions into a 16-entry data product. Yacht scores 0.7677 on reconstruction of all 32 switch effects, with utility-prediction MAE 0.0315 on its six unmeasured configurations. Fresh truthful continuations recover the target level in 9/30 and 16/30 trials, respectively, separating achieved utility from process repeatability. A deterministic verifier reproduces these local decisions from frozen records.

Primary area:
datasets and benchmarks
Keywords:
AI research agents, automated scientific discovery, research evaluation, discovery certification, outcome recovery, feedback attribution, causal evaluation, reproducibility

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_7s4pOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The authors present Discovery Certification Protocol (DCP) that provides an empirical non-recovery result for AI research agents. The authors establish low recoverability under a single challenger distribution.

Critical strengths

I believe there are some interesting finding in the paper and some good methodological points such as separating outcome from process. Same thing for the recoverability witnesses.

Critical weaknesses

There are multiple notable concerns I have after reading this paper regarding the validity of the certification. I put these into two main categories:

  1. the certification seems too weak. Under the challenger, 96 trials detect a witness 61.9% of the time if the true recovery probability is 1% which is hidden in the 0/96 claim, and the positive controls only show that the challenger can access files and submit outputs, not that it is actually adequate. As the challenge being sufficiently adequate is a key component to the certificate's validity, this is especially concerning. In addition, the target and challenger are not directly comparable as the target seems to get multiple sessions and the challenger doesn't. Most critically, several verifier classifications are seem to change after problems are observed, which goes against the claim about precommitment.

  2. the certification itself, even taken at face value, seems much narrower than what the paper suggests. To my understanding, the claim seems to be that something is "not easily recoverable by model X under setup Y at time T" rather than something more broader due to the configurations of the certificate.

In general, the paper seems to add a bit too much formalism around what is essentially a simple statistical bound and auditing infrastructure.

Questions for authors
  1. What do you define as "discovery"? The proposed protocol seems to measure recoverability under QBQ_B, not typical concepts associated with discovery such as novelty or originality.
  2. Why would inserting values count as evidence of adaptive research?
  3. Can you report Gate 1 and Gate 3 excluding direct insertion, so the effect isolates prediction/reasoning?
  4. Why UCI toy response surfaces as the main “real-data research” validation over alternatives?
  5. How do you establish independence of 96 provider-generated episodes?
  6. Why is formal_certificate_issued=false?
Reviewer_oUAuOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes the Discovery Certification Protocol (DCP), a process for checking what an AI research result actually demonstrates. It separates three questions: Did the result improve on a baseline? Could a comparable AI system reach a sufficiently good alternative without the original run's new experimental measurements? Did useful experimental feedback help produce the result? The first check uses an evaluation kept hidden from the system. The second gives a challenger comparable resources and the starting information, while withholding the original run's history and new measurements. The protocol's Core decision requires suitable checks, no successful recovery of an alternative, and a statistical limit on how often that particular challenger might succeed. An optional third check compares continuations that receive real feedback with those that receive feedback containing no useful information. This supports a separate Evidence decision about the effect of feedback. Controlled software-optimization and virtual-catalyst examples pass both checks locally. Two real-data tasks, Yacht and Ionosphere, pass Core after no successful recoveries in 96 attempts each, but their Evidence results remain inconclusive. The paper also reports successful recoveries, incomplete audits, corrections, and verification tools. It distinguishes these local results from certificates issued by an independent authority.

Critical strengths
  1. The protocol turns several different claims into separate, checkable questions.

  2. The protocol also accounts for resources carefully. If a challenger tries several candidates and keeps the best, the whole procedure counts as one attempt with its full cost. Uncertainty about whether a candidate qualifies as a recovery is kept separate from uncertainty about how often recovery happens. This makes the conclusions easier to interpret without claiming to prove universal originality or that no other system could find a solution.

  3. The examples include success, failure, and unresolved outcomes. It also preserves original decisions and distinguishes a correction to the implementation from changes to analysis rules made after seeing results (Table 7).

Critical weaknesses
  1. The main real-data audits do not establish that the challengers are strong enough. Their 45 control tests ask the system to copy a supplied marker into a file and finish. That checks whether information and files can move through the process. It does not check whether the challenger can reason its way to an alternative solution when one is available—repeating one fixed prompt measures the performance of that particular setup. Another prompt or strategy using the same model and resources could perform better. The paper acknowledges this limit, and it does not make the statistical calculation wrong. But someone seeking a favorable certificate could benefit from choosing a weak challenger. The approval rules therefore need to show how an independent authority would reject an uncompetitive setup. Controls in the simpler demonstrations test more relevant behavior, but do not settle this concern for the real-data tasks.

  2. The real-data examples leave uncertainty about the agent's added value and independent use of the protocol. Each scored table combines ten measured entries with six predicted entries. Table 6 shows that adding the predictions improves the score beyond simply inserting the new measurements. This is useful evidence. However, the comparison leaves unmeasured entries at an average based on only two starting measurements. A simple prediction method using all ten observations could be much more competitive. Comparing against a basic fitted model or interpolation using the same measurements would show more clearly what the agent adds. This concern limits the evidence for its added value; it does not invalidate the paper's stated comparison of the total effect of gathering and using feedback. Neither main real-data example passes the Evidence check, and all reported passes come from local evaluations rather than independently issued certificates. The paper states these limits honestly. Still, they leave open how well the protocol works when an independent auditor must evaluate a claimant who might select favorable evidence or settings.

Questions for authors
  1. What rules, set before the audit, would let an authority reject a challenger that is too weak? Are there existing real-data checks where an alternative solution is known to be available and finding it requires meaningful reasoning?
  2. Using the same ten measurements in Yacht and Ionosphere, how well does a simple fitted model or reasonable interpolation method perform? Please report both errors on the six unmeasured entries and the paper's score for reconstructing the 32 effects. This would help assess the agent's added prediction value while retaining the original inconclusive decisions and the stated interpretation of the feedback test.
  3. How would an independent auditor verify which model was actually used and that every attempt was recorded, without relying solely on the person producing the evidence? Please distinguish what is taken on trust in the local demonstrations from the independent records needed for formal certification.
Reviewer_7CTWOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper introduces the Discovery Certification Protocol (DCP), a framework of converting AI research agents’ numerical scores into an executable audit consisting of a series of gates (test), which operate on two levels informing the decisions. This question is importatn because a good score may have come from genuine utility of the program, search using provided data and other available information, or truthful experimental observations. Across the different tasks, the verifier appears to be sound and reproducible and producing the expected results.

Critical strengths
  • The experimental setup appears to be disciplined and mostly appropriate controls have been applied to avoid information leakage.
Critical weaknesses
  • The presentation was disorganized and it is not immediately clear e.g. what the experimental set up was and how to interpretate the quantitative results conceptually in the context of the scientific questions that the manuscript sought to address.
  • The experiments do not seem to have substantial bearing on how the research agents would perform in real-world tasks. One could argue that the experiments were more proof-of-concept than measuring performances in real-world deployement, in that the tested workflows have contours of research workflows, but the lack of direct evidence represents a major shortcoming. In particular, the tasks have definite "ground truth" (albeit "sealed") whereas real research is inherently uncertain and the results are only partially known.
Questions for authors
  • L321-322 "Each scored artifact A∗A^* combines two public anchors, eight measured utilities, and six predictions over four binary interventions": What are these listed items actually?
  • Similarly, explain what the listed items are in L360-361 "Each audit registered 96 challengers, 45 file-channel controls, 30 feedback pairs, and 60 null pairs"?
  • L392-393 "The surface selects measured configuration 1110, combining target transformation, scaling, degree-two features, and Ridge α\alpha = 1": there seem to be a some internal nomenclature in this, and do "degree-two features" and "Ridge α\alpha" refer to some regression analysis?
Reviewer_8ZjNOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes the Discovery Certification Protocol (DCP), an auditing framework for claims made by AI research agents. The protocol separates three questions: whether an output provides a useful improvement, whether a matched agent can recover a comparable outcome without the original research history or new measurements, and whether truthful experimental feedback improves subsequent outcomes relative to a neutral feedback policy. These questions are implemented through three gates, leading to the Core and Evidence decisions. The paper demonstrates the protocol on controlled tasks and two response-surface reconstruction tasks using real datasets.

I agree with the motivation. A strong final result does not, by itself, tell us how much the research process contributed. This distinction matters as more systems are described as scientific discovery agents. However, I am less convinced that the current experiments establish the practical value of the proposed certification framework. My main concern is the gap between demonstrating that new information helps and demonstrating that an agent conducts an effective research process.

Critical strengths
  1. The paper asks an important evaluation question.

Many agent papers report the quality of a final artifact while providing limited evidence about why the system succeeded. The result could come from prior knowledge, additional sampling, engineering effort, or information acquired through experiments. Separating these explanations is worthwhile. I particularly appreciate the distinction between an achieved outcome, the repeatability of the process producing it, and the effect of subsequent feedback. These quantities are often treated as interchangeable.

  1. The recovery criterion allows alternative solutions.

Gate 2 does not require a challenger to reproduce the original implementation. Any valid artifact reaching the registered score threshold counts. This is a useful choice: if a simpler or substantially different method can reach the same outcome under the stated constraints, that should affect how we interpret the original claim. This question is broader than a component ablation within one implementation.

  1. Several statistical and reporting choices are careful.

The paper explicitly conditions its recovery bound on a registered challenger distribution and budget. It also distinguishes a recovery witness from a recovery probability, and does not treat zero observed recoveries as proof of impossibility. Reporting fresh truthful continuation rates separately is helpful. Similarly, the real-data Evidence decisions remain Inconclusive when the required checks are not satisfied. I appreciate that the authors report these outcomes rather than relaxing the criteria to obtain a cleaner success story.

Critical weaknesses
  1. The real-data experiments establish the value of measurement access more directly than the value of an agent’s research strategy.

This is my central concern. In the Yacht and Ionosphere tasks, the target artifact contains two public anchors, eight newly measured utilities, and six predictions. The evaluator directly inserts the authorized measurements into the final table. The challenger and neutral arms receive only the two anchors.

Under this construction, a substantial advantage can arise simply because one arm receives task-specific information that the other arm is denied. This is consistent with the stated information-boundary question, but it provides limited evidence that the agent is good at choosing experiments or using them to guide research.

The insertion-only diagnostic in Table 6 makes this issue concrete. Inserting the eight measurements while leaving the remaining entries at the anchor mean already clears the registered minimum improvement for both tasks. The agent’s predictions provide additional gains, which is encouraging, but this comparison does not establish whether the agent outperforms straightforward alternatives given the same observations.

I would like to see two complementary controls:

Under the same measurement budget, compare the agent’s query choices against random selection, a fixed coverage design, and an appropriate simple optimization or experimental-design baseline.

Given the same acquired observations, compare the agent’s completion against simple interpolation or regression methods.

The first comparison tests acquisition quality; the second tests what the agent contributes after acquisition. DCP can reasonably audit the total acquisition-and-completion outcome, but the current evidence leaves these contributions largely unresolved.

  1. The empirical study demonstrates the mechanics of the protocol more convincingly than its scientific usefulness.

The controlled cases show that the verifier can produce passes, recovery vetoes, and inconclusive outcomes. This is useful validation of the implementation. However, I wanted a stronger demonstration of what we learn by applying DCP to an actual research agent.

For example, does the audit overturn a seemingly persuasive success claim? Does it reveal that an agent’s improvement mostly comes from additional sampling rather than feedback? Does one research strategy pass while another fails under comparable conditions? Such findings would make the framework much more compelling.

The current real-data tasks are small response-surface reconstruction problems, and the controlled tasks are deliberately structured environments. Neither real-data task obtains an Evidence pass. This is not itself a methodological failure, but it leaves the strongest part of the proposed framework without a convincing demonstration in a realistic research workflow.

I do not think the paper needs a major scientific discovery to be valuable. It does need an application where the audit changes our understanding of an agent’s contribution.

  1. The meaning of Core is narrower than the phrase “discovery certification” suggests.

A Core pass supports a bounded statement: fresh episodes from a particular registered challenger distribution rarely recover the achieved level under a specified information boundary and budget. It does not establish historical novelty, broad irrecoverability, or the necessity of the original research trajectory.

The paper acknowledges several of these limitations, which I appreciate. Nevertheless, the framing still invites a stronger interpretation than the test supports. An outcome may be difficult for the chosen challenger to recover because the challenger has a weak search policy, because the task requires private information, or because the allowed budget is restrictive. These explanations have different implications for a discovery claim.

The positive controls do not fully resolve this issue. In the real-data audits, the controls test a file read/write and submission channel. Passing these controls establishes interface functionality, but says little about whether the challenger is a competitive solver of the underlying problem.

I would find the Core interpretation more convincing with sensitivity analyses across challenger policies and budgets, or task-relevant controls demonstrating that the challenger can recover comparable outcomes when sufficient starting information is available.

  1. Gate 3 identifies a feedback-policy effect, but does not isolate adaptive scientific reasoning.

The paper correctly defines the estimand as the total effect of truthful versus neutral feedback conditional on a checkpoint. That is a defensible causal quantity. However, the effect includes direct access to measured values, improved completion of the output, and any downstream changes in action selection.

These mechanisms should not be interpreted interchangeably. In particular, the real-data tasks query a batch of eight configurations, and the final artifact directly incorporates the measurements. A positive effect can therefore arise without showing that the agent makes better sequential research decisions in response to intermediate evidence.

A useful additional comparison would provide the same amount of truthful information through a fixed or non-adaptive acquisition policy. This would help distinguish the benefit of receiving measurements from the benefit of deciding what to measure next.

The null-policy equivalence check is thoughtful, but its interpretation also remains local to the known-answer task family. As the paper acknowledges, passing it does not bound neutral-policy distortion on the target task. I therefore view it as a diagnostic rather than a general validation of the counterfactual feedback condition.

  1. The methodological contribution needs a clearer empirical payoff.

The framework brings together preregistration, sealed evaluation, matched controls, finite-sample recovery bounds, randomized feedback comparisons, and equivalence testing. A contribution based on integrating established tools can be valuable; I do not require a new statistical estimator.

However, the paper should make clearer what this integration enables that a careful experimental evaluation would otherwise miss. At present, much of the technical detail concerns decision rules, eligibility requirements, provenance, and versioned audit handling. These details may be necessary for deployment, but their practical importance would be easier to assess through a substantive audit finding.

My concern is therefore not simply that the components are familiar. It is that the current experiments do not yet show a sufficiently strong return from the additional machinery.

Questions for authors

Under an equal eight-query budget, how does the agent compare with random acquisition, a fixed coverage design, and a simple task-appropriate acquisition strategy? Given identical observations, how does its completion compare with simple predictors?

How sensitive are the Core decisions to the challenger policy and budget? Can a task-relevant competence control be added beyond checking the submission channel?

Can the authors demonstrate a setting in which sequential adaptation matters, and distinguish it from receiving an informative fixed batch of measurements?

Is there an existing research agent or published outcome claim for which applying DCP produces a materially different conclusion from the original evaluation?

What substantive claim should a reader infer from a Core pass beyond low recovery probability under the registered challenger distribution? Would more restrained terminology communicate this scope more accurately?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.