acceptodds
Under review as a conference paper at ICLR 2027

Replay the Scientist: Calibrated Evidence After Interactive LLM Hypothesis Search

Abstract

Large language models increasingly search for scientific claims through interaction. They adapt hypotheses, analyses, repairs, and measurements before reporting a result, but testing only the final hypothesis discards the outcome-dependent path that selected it. We introduce Transcript Replay, a black-box protocol that reruns the discovery procedure under randomized outcomes and ranks the selected score against counterfactual discoveries. Under a declared exchangeable null, the discovery program is the inferential unit and retains its search policy. In controlled studies, final-only rejection reaches 96.3%, while Transcript Replay remains calibrated through adaptive search and acquisition. We evaluated Qwen and Llama on rule search, generated Python features, and external equation discovery. In full-data rule search, Transcript Replay has 3.0% null rejection versus 64.8% for final-only testing and gains 24.0 power points over held-out testing. At fixed full-data scoring, a larger search sample improves power by 11.5 points and planted-mask recovery by 12.2 points. We reuse exact reference scores, reducing required executions by 91.9% in the primary validation batch for repeated queries to a fixed scientist on outcomes related by the specified transformations. The replay *p*-value evaluates the discovery procedure behind the selected rule or program.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Adaptive data analysis, selective inference, randomization tests, large language models, automated scientific discovery

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_csM6OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates the following question: after an AI repeatedly searches, modifies, and filters hypotheses on the same dataset, how should we determine whether its final finding reflects statistically significant structure beyond random noise, after accounting for the search process? The method proposed by the authors, Transcript Replay, can be roughly divided into four steps:

  1. Run the AI on the original data.
  2. Permute the outcome variable.
  3. Have the same AI search procedure run again from scratch for each permutation, including its outcome-dependent proposals, feedback, repairs, stopping, and selection.
  4. Compare the selected score on the original data with the selected scores from the permuted data, thereby obtaining a search-corrected p-value under the declared null.

The experiments cover controlled rule search, LLM-generated rules and Python features, and external equation discovery.

Critical strengths

The paper addresses an important problem: how to assess statistical evidence after adaptive hypothesis search by AI systems.

Critical weaknesses
  1. I find this paper very difficult to follow.

  2. The experiments show that Replay controls false positives, but the benefits beyond this are limited. As the authors acknowledge in Section 3.4, a significant result does not mean that the selected rule or program is the correct scientific explanation. The main gains are in rule search, where all possible rules can be enumerated and exhaustive search has a higher estimated power than LLM Full Replay. For Python programs, Full Replay gives little or no improvement over held-out testing, and a simple linear model has similar or higher power. It is therefore unclear when LLM search with Replay is more useful than these simpler alternatives.

  3. The method has a limited range of applications. Most experiments use 4B or 8B models, datasets with 256 rows, and synthetic or semi-synthetic outcomes. The larger controlled study uses a hand-written search program rather than an LLM. To replay adaptive data collection, the method also needs either outcomes for every sample in the measurement pool or a simulator that can provide outcomes under the null hypothesis. This is a strong requirement for real experiments. A new task with 39 reference runs still needs 40 complete searches. The reported 91.9% saving comes from sharing reference runs across a specific batch of compatible queries. It does not reduce this cost for a new single-query task.

  4. The method does not seem very novel to me. Permuting outcomes and rerunning the learner was already used in Permutation Tests for Studying Classifier Performance (Ojala & Garriga, JMLR 2010). Information Mapping with Pattern Classifiers: A Comparative Study (Pereira & Botvinick, NeuroImage 2011, Section 2.3) also explicitly recommends rerunning the whole procedure, including training candidate classifiers and choosing the best. The authors acknowledge this earlier work, and Proposition 1 follows the standard permutation test argument. What they add is mainly how to replay an interactive LLM workflow and when calculations can be reused. These details are useful, but I do not think they add enough new knowledge or insight, especially given the limited experimental gains above.

Questions for authors

I have no additional questions for the authors. My main concerns are described in the Weaknesses section.

Reviewer_4BVnOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces Transcript Replay, a protocol for calibrated inference after interactive LLM hypothesis search. It reruns the complete discovery procedure under randomized outcomes, including proposals, repairs, feedback, acquisition, and stopping, and compares the observed score with these counterfactual runs. The statistical guarantee follows classical randomization testing; the main contribution is its operationalization for adaptive LLM workflows. Experiments cover controlled search, local LLM rule and Python-feature discovery, and external equation discovery. The results provide convincing evidence of calibration, although power improvements and computational savings are more setting-dependent.

Critical strengths
  1. The inferential target and assumptions are carefully stated. The paper clearly distinguishes evidence against a global null from verification of the selected scientific explanation, and identifies which outcome-dependent decisions must be replayed. This makes the protocol technically sound and helps prevent overinterpretation.

  2. The evaluation includes informative failure controls, such as frozen search paths and incomplete acquisition replay. The rule-search results are particularly strong: full replay maintains calibration and improves power over held-out testing, while matched scoring comparisons and planted-mask recovery help explain the benefit of using more search data.

Critical weaknesses
  1. The practical efficiency gains depend on restrictive reuse conditions: the scientist, covariates, and outcome orbit must remain compatible across queries. The reported 91.9% execution reduction therefore does not establish comparable savings for independent scientific tasks; a cold query still requires 40 executions in the primary setup. Although acknowledged, this limits the demonstrated practical applicability. A comparison of power and wall-clock cost under equal compute budgets would strengthen the case.

  2. The evidence for improved discovery beyond rule search is limited. Python-feature experiments show little or no power improvement over held-out testing, and simple linear baselines remain competitive or stronger. The external equation-discovery study covers only two tasks and two seeds, with original-query p-values at the minimum attainable value. These results support feasibility and calibration more clearly than a broad improvement in scientific discovery.

Questions for authors
  1. For independent queries that cannot share reference scores, how does Transcript Replay compare with held-out testing at a matched computational budget?

  2. Do the authors have evidence from a broader program or equation-discovery setting where adaptive generated features improve on simple baselines and replay provides a meaningful power gain?

Reviewer_ipkROpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Transcript Replay, a statistical inference framework for calibrating evidence produced by interactive LLM-based scientific discovery. The central problem is that adaptive hypothesis generation, feedback, repair, data acquisition, and stopping decisions introduce selection bias that cannot be corrected by testing only the final selected hypothesis. The proposed method treats the entire discovery procedure as a test statistic. Under a specified exchangeable null, it reruns the scientific agent on randomized outcome vectors, allowing the complete search trajectory and selected hypothesis to change. A Monte Carlo randomization test then compares the observed selected score against scores obtained from these counterfactual searches. The paper provides a finite-sample validity result under null invariance and evaluates the method through controlled hypothesis-search simulations, Qwen and Llama agents performing rule and Python-feature search, and an external equation-discovery workflow. Results show that Transcript Replay maintains approximately nominal false-positive rates where final-only testing becomes severely inflated. In rule search, it also improves statistical power over held-out evaluation, while exact reference reuse reduces execution costs for compatible query batches.

Critical strengths
  1. The paper addresses an important problem in LLM-based scientific research: hypotheses are often selected through many adaptive interactions, making conventional final-hypothesis significance testing unreliable. Transcript Replay provides a conceptually clean solution by incorporating the complete discovery procedure into the randomization statistic. The controlled experiments are compelling, showing that final-only false-positive rates can reach 96.3% while complete replay remains approximately calibrated. This is the strongest consideration supporting acceptance.

  2. The paper goes beyond demonstrating calibration. It compares full replay, incomplete replay, held-out testing, and exhaustive correction, and separates the effects of using more observations for search from those of using more observations for scoring. In local rule-search experiments, full replay improves power by 24.0 percentage points over held-out testing, while the score-matched comparison attributes an additional 11.5-point gain to increased search data. The reference-reuse analysis also provides a practical optimization for repeated inference on compatible outcome orbits.

Critical weaknesses
  1. The theoretical novelty is primarily an application of established randomization inference, and its practical validity depends on restrictive assumptions. The central finite-sample guarantee follows from the standard permutation-test rank argument applied to a complete algorithmic statistic. The methodological contribution therefore lies mainly in identifying and executing the appropriate replay unit, rather than developing a fundamentally new statistical inference principle.

  2. Generalization beyond controlled rule search is not yet convincingly established. The strongest statistical-power improvements occur in relatively small, structured rule-search problems. In Table 2, the Python-program scientists show only a 1.6-point improvement for Qwen and no improvement for Llama over held-out testing, with uncertainty intervals that do not establish a consistent advantage. The external equation-discovery study also uses a limited number of synthetic systems and seeds, and alternative methods achieve better out-of-distribution prediction in several comparisons.

Questions for authors
  1. Clarification on exchangeability and practical validity: Proposition 1 requires the null distribution to be invariant under the chosen transformation group. How should practitioners determine whether this assumption is appropriate for observational scientific datasets with confounding, correlated observations, or structured measurement noise? Are there identifiable failure modes in which an inappropriate randomization group produces misleadingly calibrated-looking results?

  2. Clarification on incomplete replay: The proposed framework requires all outcome-dependent decisions to be regenerated. What happens when a workflow contains nondeterministic external tools or historical states that cannot be reproduced exactly? Does the validity result extend to independently resampled tool randomness, and what conditions are required for that extension?

  3. Clarification on scientific discovery power: The power improvements are substantial for rule search but limited for Python-program search. Can the authors explain whether this difference is primarily due to the search policy, hypothesis-space complexity, or the strength of the underlying linear baseline? How should this affect the generality of the paper's empirical conclusions?

  4. Clarification on computational practicality: The reported 91.9% execution reduction relies on reference reuse across compatible queries. How commonly would such compatibility arise in realistic scientific-agent applications where datasets, hypothesis spaces, and tool outputs vary across tasks? What is the expected cost for a new, independent discovery query without reusable references?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.