- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper asks how we can trust a statistical conclusion produced by an AI. It proposes a protocol with three parties. The scientist posts the problem (design matrix, model, hypothesis to test, and declared assumptions such as bounds on the errors) but keeps the actual data hidden. The AI then proposes a test statistic (an "e-value") together with a proof in Lean that the test is valid under those assumptions. The Lean proof checker verifies the proof. Only then is the data revealed and the result computed.
The key idea is that the AI writes a new proof for each specific problem, instead of relying on general results fixed in advance, as existing tools (StatWhy, Tea, Tisane, etc.) do. The paper also formalises in Lean two well-known statistical paradoxes (Bahadur-Savage and Lindley) and uses them to justify its design choices. It demonstrates the protocol on thirteen classic datasets. Main findings: the certified tests give strong evidence in several cases; the choice of assumptions matters a lot; and the very small p-values usually reported for these datasets, based on large-sample approximations, cannot be justified at their actual sample sizes without extra assumptions.
Critical strengths
-
A new certified proof for each problem. Existing tools prove a test valid once, in advance, so they cannot adapt to a scientist’s own assumptions. Here the AI writes a new proof for each specific problem, and Lean checks it. This is an original idea that shows what AI can add to statistics, and it is the main reason I think the paper should be published.
-
An important finding about common p-values. The paper shows that the very small p-values usually reported for several classic datasets cannot be justified at their real sample sizes without extra assumptions. This matters for everyday statistical practice, and together with point 1 it keeps my score above the acceptance line.
Critical weaknesses
-
The experiments do not follow the protocol. The protocol requires that the AI has not seen the data and makes only one proposal. In the paper, the datasets were likely in the AI’s training data, the authors already knew the data, and there were several proposals with repeated human help. So the experiments show that the proofs can be written, not that the protocol works in real use. This is the main reason my score is borderline and not a clear accept.
-
The AI’s ability is never measured. The paper says the protocol depends on how capable the AI is, but reports no success rate, no failed attempts, no cost and no measure of human help. Without this, a reader cannot tell whether a scientist could use the protocol with a standard AI model, or whether it needs months of expert work.
Questions for authors
- Can you run the protocol exactly as specified on at least one dataset the AI cannot have seen, with one proposal and no human help?
- How many attempts did the AI make per dataset, how many failed, and how much human help was needed? Please report this for each AI model.
- For two or three datasets, how does the e-value change across different assumption choices?
- How was the simulation search set up, and how much of the assumption class didit explore?
- Can the abstract state exactly which parts are machine-checked?
- Can you show one example where existing tools cannot express the scientist's assumptions but your protocol can?
- Please cite the Lean 4 version of Mathlib.
- Were the assumptions fixed before any e-value was computed? Can the repository history show this?
Pre-registered experiment
Thank you for your review. We have run a pre-registered experiment to address your question on strong versus weak provers, for which we would value your feedback (if your time allows) before conducting a second run with more data. See top-level comments ("Pre-registered clean run on held-out data" and "Results of the pre-registered run"). Replies to your other points to follow.