- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies reliability when statistical discoveries become persistent state used by a continual research agent. It distinguishes local admissions, currently usable discoveries (CURRENT), and immutable admission history (EVER). In a synthetic benchmark with 128 candidates and known truth, it compares permanent retention after per-batch BH testing with cumulative BH recertification of frozen p-values.
The principal experiment reports substantially higher terminal Global-Null active-state FDR under permanent append. Separate interventions show that changing proposal-visible membership alters subsequent search, including when state cardinality is held fixed. A single-model LLM pilot measures state-conditioned changes in candidate selection after accounting for repeated-call instability. Finally, an independently seeded, preregistered 320-pair experiment does not confirm the exploratory increase in distinct future false admissions.
Critical strengths
-
The separation of CURRENT capability from EVER provenance provides a useful operational perspective on agent reliability. The controlled interventions preserve history, catalog, and keyed randomness while changing proposal-visible state, and the size-matched comparison helps distinguish membership effects from cardinality alone. This makes the state-to-search question more precise than a generic comparison of agents with and without memory.
-
The paper carefully separates exploratory findings from independent confirmation. The downstream-error hypothesis is tested on a separately seeded, prospectively selected 320-pair sample with a frozen primary endpoint, and its nonconfirmation is reported prominently. The manuscript also distinguishes marginal, simultaneous, and descriptive intervals and explicitly limits the interpretation of its calibration screen and LLM pilot. This transparency makes the negative result informative and avoids treating behavioral change as automatic evidence of statistical harm.
Critical weaknesses
-
The main reliability result does not sufficiently distinguish the proposed Decision-to-State Gap from established accumulation of error across repeatedly tested batches. The registered experiment fixes a score-blind 100-candidate sequence, so its FDR gap occurs without state-dependent proposal feedback. Under the Global Null, permanent append measures whether any batch ever rejects. For independent batches each having rejection probability 0.05, that probability after ten batches is 1 - 0.95^10, approximately 0.401, already close to the reported 0.410–0.422. This calculation is illustrative rather than an exact model of the experiment. Moreover, the comparison changes the testing family from separate batches to the cumulative family, alongside changing retention. Zrnic et al. (2020), already cited, explicitly demonstrate the inflation caused by naively composing fixed-level BH tests. A suitable online/batched-control baseline and a clearer separation of testing-family effects from permission effects are needed to establish what additional reliability insight the agent-state formulation contributes.
-
The agent-specific evidence establishes state sensitivity more convincingly than a reliability benefit from governing that state. The controlled proposer is explicitly designed to select parents and mutation neighborhoods from visible identities, so membership sensitivity is partly built into its mechanism. The LLM pilot extends this observation to one model choosing from finite catalogs, but does not measure closed-loop statistical reliability. The independent downstream experiment estimates only +0.01875 additional false admissions, with a 95% interval of approximately [-0.0446, 0.0821], and does not confirm the proposed increase. Reporting this negative result is a strength. However, taken together, the experiments leave the practical reliability contribution beyond ordinary multiple-testing control insufficiently demonstrated. Stronger evidence could concern a consequential failure mode, a utility–reliability trade-off, or a more precise characterization of when state governance matters.
Questions for authors
-
In the fixed-candidate experiment, what part of the reported gap remains after accounting for the ordinary accumulation of rejection events across batches? Can you compare against an appropriate level-allocating procedure, such as Batch-BH under verified assumptions, while preserving permanent retention? This would help separate the contribution of cumulative testing from that of state governance.
-
Is there an experiment that holds the statistical admission procedure fixed while varying operational retention and demonstrates a consequential reliability or utility difference? The capability-only rollout is relevant, but its primary downstream-error effect was not confirmed. Please clarify which additional conclusions are supported by that experiment.
-
The reported 95% interval has an upper endpoint of 0.0821, below the prespecified planning effect of 0.10. Without treating non-rejection as equivalence, what does this imply about increases of practical interest within the selected-source population and three-cycle horizon? The discussion currently emphasizes unresolved amplification but gives less attention to the effect sizes the interval constrains.