acceptodds
Under review as a conference paper at ICLR 2027

When Statistical Discoveries Become State: Reliability in Continual Research Agents

Abstract

When scientific discovery agents operate continually, statistical discoveries can become operational state that guides subsequent search rather than remain terminal outputs. This creates a Decision-to-State Gap: guarantees for local admission decisions do not characterize the reliability of the persistent state that drives future search. We study this gap in a controlled benchmark with exact ground truth, 128 hypotheses, and a frozen stateful proposer, distinguishing currently usable discoveries (CURRENT) from immutable admission history (EVER). Across three preregistered positive-dependence stress conditions, permanent append yielded terminal Global-Null active-state FDRs of 0.410–0.422, compared with 0.044–0.064 under cumulative recertification; the approximate 95% simultaneous bands for all three paired gaps lay entirely above zero. Controlled interventions on proposal-visible state changed the next candidate in 20% of selected planted-signal snapshots, while equal-size states with different membership yielded different next candidates in 37.5% of planted pairs. A frozen real-LLM transport pilot further found state-conditioned proposal-distribution shifts beyond same-state repeat instability (, 95% stability interval [0.158, 0.413]). An exploratory downstream-error signal motivated an independently seeded, preregistered 320-pair confirmation, but its primary hypothesis, an increase in distinct future false admissions, was not confirmed. Together, these results identify reusable statistical state as a distinct reliability object and motivate a governance principle: preserve admission history while separately governing future action capability.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
scientific discovery agents, continual research agents, statistical reliability, false discovery rate, multiple hypothesis testing, agent memory, AI scientists

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Usx3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies reliability when statistical discoveries become persistent state used by a continual research agent. It distinguishes local admissions, currently usable discoveries (CURRENT), and immutable admission history (EVER). In a synthetic benchmark with 128 candidates and known truth, it compares permanent retention after per-batch BH testing with cumulative BH recertification of frozen p-values.

The principal experiment reports substantially higher terminal Global-Null active-state FDR under permanent append. Separate interventions show that changing proposal-visible membership alters subsequent search, including when state cardinality is held fixed. A single-model LLM pilot measures state-conditioned changes in candidate selection after accounting for repeated-call instability. Finally, an independently seeded, preregistered 320-pair experiment does not confirm the exploratory increase in distinct future false admissions.

Critical strengths
  1. The separation of CURRENT capability from EVER provenance provides a useful operational perspective on agent reliability. The controlled interventions preserve history, catalog, and keyed randomness while changing proposal-visible state, and the size-matched comparison helps distinguish membership effects from cardinality alone. This makes the state-to-search question more precise than a generic comparison of agents with and without memory.

  2. The paper carefully separates exploratory findings from independent confirmation. The downstream-error hypothesis is tested on a separately seeded, prospectively selected 320-pair sample with a frozen primary endpoint, and its nonconfirmation is reported prominently. The manuscript also distinguishes marginal, simultaneous, and descriptive intervals and explicitly limits the interpretation of its calibration screen and LLM pilot. This transparency makes the negative result informative and avoids treating behavioral change as automatic evidence of statistical harm.

Critical weaknesses
  1. The main reliability result does not sufficiently distinguish the proposed Decision-to-State Gap from established accumulation of error across repeatedly tested batches. The registered experiment fixes a score-blind 100-candidate sequence, so its FDR gap occurs without state-dependent proposal feedback. Under the Global Null, permanent append measures whether any batch ever rejects. For independent batches each having rejection probability 0.05, that probability after ten batches is 1 - 0.95^10, approximately 0.401, already close to the reported 0.410–0.422. This calculation is illustrative rather than an exact model of the experiment. Moreover, the comparison changes the testing family from separate batches to the cumulative family, alongside changing retention. Zrnic et al. (2020), already cited, explicitly demonstrate the inflation caused by naively composing fixed-level BH tests. A suitable online/batched-control baseline and a clearer separation of testing-family effects from permission effects are needed to establish what additional reliability insight the agent-state formulation contributes.

  2. The agent-specific evidence establishes state sensitivity more convincingly than a reliability benefit from governing that state. The controlled proposer is explicitly designed to select parents and mutation neighborhoods from visible identities, so membership sensitivity is partly built into its mechanism. The LLM pilot extends this observation to one model choosing from finite catalogs, but does not measure closed-loop statistical reliability. The independent downstream experiment estimates only +0.01875 additional false admissions, with a 95% interval of approximately [-0.0446, 0.0821], and does not confirm the proposed increase. Reporting this negative result is a strength. However, taken together, the experiments leave the practical reliability contribution beyond ordinary multiple-testing control insufficiently demonstrated. Stronger evidence could concern a consequential failure mode, a utility–reliability trade-off, or a more precise characterization of when state governance matters.

Questions for authors
  1. In the fixed-candidate experiment, what part of the reported gap remains after accounting for the ordinary accumulation of rejection events across batches? Can you compare against an appropriate level-allocating procedure, such as Batch-BH under verified assumptions, while preserving permanent retention? This would help separate the contribution of cumulative testing from that of state governance.

  2. Is there an experiment that holds the statistical admission procedure fixed while varying operational retention and demonstrates a consequential reliability or utility difference? The capability-only rollout is relevant, but its primary downstream-error effect was not confirmed. Please clarify which additional conclusions are supported by that experiment.

  3. The reported 95% interval has an upper endpoint of 0.0821, below the prespecified planning effect of 0.10. Without treating non-rejection as equivalence, what does this imply about increases of practical interest within the selected-source population and three-cycle horizon? The discussion currently emphasizes unresolved amplification but gives less attention to the effect sizes the interval constrains.

Reviewer_Wr9zOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

I recommend that this manuscript be desk rejected due to significant incompleteness. The core empirical results remain unfinished, marked by multiple unresolved placeholders and unconfirmed findings (e.g., explicit "NOT Confirmed" markers regarding the primary hypotheses). Key experimental conclusions and downstream statistical claims are left open and incomplete, preventing a proper evaluation of the work's methodological soundness and technical contributions. Hence, this paper is not ready for formal evaluation in its current state.

Critical strengths

I recommend that this manuscript be desk rejected due to significant incompleteness. The core empirical results remain unfinished, marked by multiple unresolved placeholders and unconfirmed findings (e.g., explicit "NOT Confirmed" markers regarding the primary hypotheses). Key experimental conclusions and downstream statistical claims are left open and incomplete, preventing a proper evaluation of the work's methodological soundness and technical contributions. Hence, this paper is not ready for formal evaluation in its current state.

Critical weaknesses

I recommend that this manuscript be desk rejected due to significant incompleteness. The core empirical results remain unfinished, marked by multiple unresolved placeholders and unconfirmed findings (e.g., explicit "NOT Confirmed" markers regarding the primary hypotheses). Key experimental conclusions and downstream statistical claims are left open and incomplete, preventing a proper evaluation of the work's methodological soundness and technical contributions. Hence, this paper is not ready for formal evaluation in its current state.

Questions for authors

I recommend that this manuscript be desk rejected due to significant incompleteness. The core empirical results remain unfinished, marked by multiple unresolved placeholders and unconfirmed findings (e.g., explicit "NOT Confirmed" markers regarding the primary hypotheses). Key experimental conclusions and downstream statistical claims are left open and incomplete, preventing a proper evaluation of the work's methodological soundness and technical contributions. Hence, this paper is not ready for formal evaluation in its current state.

Reviewer_21v8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper explores the Decision-to-State Gap in continual scientific discovery agents. The authors point out that the agent local statistical decisions will be transformed into the persistent system state later consumed by search. The paper defines two states: currently usable discoveries(CURRENT) and immutable admission history(EVER). By constructing a synthetic ground-truth environment and testing it on DeepSeek, the paper compares two state governance strategies: permanent append and cumulative recertification. The results show that “permanent append” can cause the FDR to 0.410–0.422, while “cumulative recertification” can keep it around 0.044–0.064. At the same time, intervention experiments demonstrate that changing the state content can significantly alter the agent's future search direction. Despite the validity of the mechanism, the paper also honestly reports that the independent downstream-harm primary leaves the hypothesized increase unconfirmed.

Critical strengths
  1. The paper provides carefully controlled experiments, transparent reporting, and an independent confirmatory evaluation of an exploratory finding, including the reporting of a non-significant result.
  2. The distinction between changes in search behavior and downstream statistical consequences is valuable.
Critical weaknesses
  1. The central statistical problem substantially overlaps with existing research. Zrnic et al. [1] already demonstrated cumulative FDR inflation under naive batch-wise BH and developed methods with formal FDR guarantees under specified assumptions. This directly challenges the novelty of the primary statistical phenomenon demonstrated in Figure 2. Related problems have also been studied by Fisher [2], who considers revisable decisions in online FDR control, and Katsevich et al. [3], who examine FDR control after filtering rejection sets. Although neither directly addresses the proposed agent-memory framework, both provide relevant theoretical context that is missing from the manuscript. The proposed problem does not appear specific to AI agents. The same statistical framework applies whether hypotheses are generated by an LLM, a human scientist, or another adaptive procedure. The manuscript does not clearly establish what new statistical problem the Decision-to-State Gap introduces beyond existing sequential and adaptive inference frameworks.

  2. The paper lacks innovation at the algorithm level. The core method is based on classical statistical theory, merely repeatedly running BH tests with frozen p-values. The paper does not propose any new network structures, representation learning methods, or innovations in memory architectures for LLMs.

  3. The paper fails to confirm downstream hazards, which may significantly diminish the significance of the research. Furthermore, compared to complex real-world interaction environments that consume millions of tokens, the experimental environment design is simple and lacks scale. It is insufficient to demonstrate the mechanism universality in real and complex scientific research scenarios.

[1] Zrnic, Tijana, et al. "The power of batching in multiple hypothesis testing." International Conference on Artificial Intelligence and Statistics. PMLR, 2020. [2] Fisher, Aaron J. "Online Control of the False Discovery Rate Under "Decision Deadlines"." International Conference on Artificial Intelligence and Statistics. PMLR, 2022. [3] Katsevich, Eugene, Chiara Sabatti, and Marina Bogomolov. "Filtering the rejection set while preserving false discovery rate control." Journal of the American Statistical Association 118.541 (2023): 165-176.

Questions for authors
  1. What does the Decision-to-State Gap contribute beyond established online and batched FDR frameworks, particularly Zrnic et al. [1]?
  2. Which properties make this a specifically agent-related problem rather than a general adaptive statistical inference problem?
  3. In real-world engineering, why should we address a problem that exists in the mechanism but has insignificant consequences?
  4. Assuming that the historical records span several years and the data volume is extremely large, what is the time complexity of the global p-value recertification? How can computational feasibility be ensured in large-scale systems?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.