- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper investigates the following question: after an AI repeatedly searches, modifies, and filters hypotheses on the same dataset, how should we determine whether its final finding reflects statistically significant structure beyond random noise, after accounting for the search process? The method proposed by the authors, Transcript Replay, can be roughly divided into four steps:
- Run the AI on the original data.
- Permute the outcome variable.
- Have the same AI search procedure run again from scratch for each permutation, including its outcome-dependent proposals, feedback, repairs, stopping, and selection.
- Compare the selected score on the original data with the selected scores from the permuted data, thereby obtaining a search-corrected p-value under the declared null.
The experiments cover controlled rule search, LLM-generated rules and Python features, and external equation discovery.
Critical strengths
The paper addresses an important problem: how to assess statistical evidence after adaptive hypothesis search by AI systems.
Critical weaknesses
-
I find this paper very difficult to follow.
-
The experiments show that Replay controls false positives, but the benefits beyond this are limited. As the authors acknowledge in Section 3.4, a significant result does not mean that the selected rule or program is the correct scientific explanation. The main gains are in rule search, where all possible rules can be enumerated and exhaustive search has a higher estimated power than LLM Full Replay. For Python programs, Full Replay gives little or no improvement over held-out testing, and a simple linear model has similar or higher power. It is therefore unclear when LLM search with Replay is more useful than these simpler alternatives.
-
The method has a limited range of applications. Most experiments use 4B or 8B models, datasets with 256 rows, and synthetic or semi-synthetic outcomes. The larger controlled study uses a hand-written search program rather than an LLM. To replay adaptive data collection, the method also needs either outcomes for every sample in the measurement pool or a simulator that can provide outcomes under the null hypothesis. This is a strong requirement for real experiments. A new task with 39 reference runs still needs 40 complete searches. The reported 91.9% saving comes from sharing reference runs across a specific batch of compatible queries. It does not reduce this cost for a new single-query task.
-
The method does not seem very novel to me. Permuting outcomes and rerunning the learner was already used in Permutation Tests for Studying Classifier Performance (Ojala & Garriga, JMLR 2010). Information Mapping with Pattern Classifiers: A Comparative Study (Pereira & Botvinick, NeuroImage 2011, Section 2.3) also explicitly recommends rerunning the whole procedure, including training candidate classifiers and choosing the best. The authors acknowledge this earlier work, and Proposition 1 follows the standard permutation test argument. What they add is mainly how to replay an interactive LLM workflow and when calculations can be reused. These details are useful, but I do not think they add enough new knowledge or insight, especially given the limited experimental gains above.
Questions for authors
I have no additional questions for the authors. My main concerns are described in the Weaknesses section.