acceptodds
Under review as a conference paper at ICLR 2027

Proof-carrying hypothesis testing: certified statistical inference by AI

Abstract

Progress in formalisation technology, such as Lean, has allowed advanced mathematical proofs to be confirmed mechanically by computers. As well as providing greater assurance around existing, human, mathematical proofs, an application of this technology is to validate mathematical proofs by AI. In this paper, we consider the corresponding problem of certifying statistical inference by AI. We focus on testing a parameter (or regression contrast) under the linear model. We propose a statistical protocol, proof-carrying hypothesis testing, in which an AI produces a test with a formal proof of validity against the specific null hypothesis, design matrix, and assumption class set by the scientist, who withholds the data. An important aspect of the design of the protocol is to confront major tensions and statistical paradoxes which are pertinent to this hypothesis testing problem, namely the Bahadur–Savage theorem, Lindley's paradox, and the notion that “all models are wrong”. We demonstrate the protocol on thirteen datasets across science, policy, medicine and AI, with finite-sample p-values and e-values derived and machine-checked by AI, finding strong evidence in several cases under declared variance and error bounds. At those sample sizes, asymptotic p-values based on the CLT cannot be supported without further assumptions.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
formal verification, Lean, statistical inference, e-values, hypothesis testing, linear regression, model misspecification

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_xPeAOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks how we can trust a statistical conclusion produced by an AI. It proposes a protocol with three parties. The scientist posts the problem (design matrix, model, hypothesis to test, and declared assumptions such as bounds on the errors) but keeps the actual data hidden. The AI then proposes a test statistic (an "e-value") together with a proof in Lean that the test is valid under those assumptions. The Lean proof checker verifies the proof. Only then is the data revealed and the result computed.

The key idea is that the AI writes a new proof for each specific problem, instead of relying on general results fixed in advance, as existing tools (StatWhy, Tea, Tisane, etc.) do. The paper also formalises in Lean two well-known statistical paradoxes (Bahadur-Savage and Lindley) and uses them to justify its design choices. It demonstrates the protocol on thirteen classic datasets. Main findings: the certified tests give strong evidence in several cases; the choice of assumptions matters a lot; and the very small p-values usually reported for these datasets, based on large-sample approximations, cannot be justified at their actual sample sizes without extra assumptions.

Critical strengths
  1. A new certified proof for each problem. Existing tools prove a test valid once, in advance, so they cannot adapt to a scientist’s own assumptions. Here the AI writes a new proof for each specific problem, and Lean checks it. This is an original idea that shows what AI can add to statistics, and it is the main reason I think the paper should be published.

  2. An important finding about common p-values. The paper shows that the very small p-values usually reported for several classic datasets cannot be justified at their real sample sizes without extra assumptions. This matters for everyday statistical practice, and together with point 1 it keeps my score above the acceptance line.

Critical weaknesses
  1. The experiments do not follow the protocol. The protocol requires that the AI has not seen the data and makes only one proposal. In the paper, the datasets were likely in the AI’s training data, the authors already knew the data, and there were several proposals with repeated human help. So the experiments show that the proofs can be written, not that the protocol works in real use. This is the main reason my score is borderline and not a clear accept.

  2. The AI’s ability is never measured. The paper says the protocol depends on how capable the AI is, but reports no success rate, no failed attempts, no cost and no measure of human help. Without this, a reader cannot tell whether a scientist could use the protocol with a standard AI model, or whether it needs months of expert work.

Questions for authors
  1. Can you run the protocol exactly as specified on at least one dataset the AI cannot have seen, with one proposal and no human help?
  2. How many attempts did the AI make per dataset, how many failed, and how much human help was needed? Please report this for each AI model.
  3. For two or three datasets, how does the e-value change across different assumption choices?
  4. How was the simulation search set up, and how much of the assumption class didit explore?
  5. Can the abstract state exactly which parts are machine-checked?
  6. Can you show one example where existing tools cannot express the scientist's assumptions but your protocol can?
  7. Please cite the Lean 4 version of Mathlib.
  8. Were the assumptions fixed before any e-value was computed? Can the repository history show this?
Reviewer_ffN6OpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper gives a protocol by which to use AI and a proof checker to analyze statistical data. Specifically, the idea is to start with a dataset, have a human produce a null hypothesis, and then have the AI produce an e-value function from outcomes to nonnegative real numbers, such that the expected value of the e-value under any probability distribution consistent with the null hypothesis and declared assumptions is less than or equal to 1. The idea is to have the AI propose an e-value function given assumptions, and then prove that it is a proper e-value from the perspective of the null hypothesis. Then you evaluate the e-value on the data to see whether one should reject the null-hypothesis. The authors demonstrate this protocol across 13 different datasets, and they justify their use of e-values instead of p-values.

Critical strengths

This is immediately applicable as a technique to anyone reading the paper. It helps me understand potential solutions for the central issue of how to best use AI in conjunction with Lean and real data/statistics. It uses 13 different examples, sufficiently demonstrating the feasibility of the approach.

Critical weaknesses

The paper is somewhat unfocused in establishing a protocol and also picking a preferred statistical quantity. But in this instance, I am receptive to this paper being somewhat opinionated, because the writers convincingly argue their preference for e-values.

Questions for authors

Do you think there is a way you could make Box 1 clearer and easier to interpret on first reading? I understood it only after seeing much of the rest of the paper.

Reviewer_RiRpOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a protocol in which AI constructs a statistical test for a hypothesis testing problem defined by a statistician, including the design matrix, null hypothesis, and assumption class, and provides a machine-verifiable proof of its validity in Lean. The authors demonstrate the protocol on 13 datasets.

Critical strengths
  • The problem is interesting, as Lean-style verification has worked for mathematical proofs, and how to adapt similar idea to statistical inference is a meaning topic.
  • The main idea is understandable. The workflow seperates the judgement part done by statistician with the mathematical part done by AI.
Critical weaknesses
  • The authors propose a question at the very begining "how can we trust what an AI has determined to be true based on data?", but after reading the paper it is still not clear to me how this question is actually answered or addressed.
  • The protocol restricts the target to testing regression coefficients. I understand one cannot ask for a general one that covers every test, but for other common problems, say testing a variance ratio or testing independence, would the same framework apply with minor changes, or would it require substantial new theory and formalisation? If the latter, the framework could be somewhat limited.
  • What is the practical advantage of the protocol over a pure statistical test?
  • Is there any practical guidance for the proposed protocal? I would like to know how a statistician should impletement the protocal when a testing problem is define.
  • Is there any basic guarantee on power? Validity alone seems of limited use.
  • Would different AI models make a difference, say a strong versus a weak prover? The paper says power depends on the AI’s capability but gives no evidence.
Questions for authors

See above

1 comment
AuthorsOpenReview ↗

Pre-registered experiment

Thank you for your review. We have run a pre-registered experiment to address your question on strong versus weak provers, for which we would value your feedback (if your time allows) before conducting a second run with more data. See top-level comments ("Pre-registered clean run on held-out data" and "Results of the pre-registered run"). Replies to your other points to follow.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Second pre-registered run: two further datasets

Since there are no comments, we hope it is ok to proceed with a second pre-registered run, adding two further datasets; again, appearing sufficiently recently they can't have been seen in training, no human help, one proposal per run, and the same four models, conditions and instructions as before.

Dataset 1: the COVALENT trial (JAMA Network Open 9(8), published online 3 August 2026; patients enrolled March 2022 to February 2025; no results on the trial registry and no preprint found, so the results were first public on 3 August 2026), first-pass success (0/1) of tracheal intubation, direct laryngoscopy versus hyperangulated video laryngoscopy. Posting: two groups, n0 = 799 (direct), n1 = 812 (video); null hypothesis: equal group means; assumption class as before: independent outcomes with |Y_i - E Y_i| <= 1 and Var Y_i <= 1/4.

Dataset 2: Mwezi, Kipanyula, Vianney and Stephano, Frontiers in Toxicology, published 9 September 2026 (no preprint), 7-day mortality (0/1) of Drosophila melanogaster at five concentrations of a plant extract, 180 flies per concentration. Posting: regression of the outcome on dose (0, 0.1, 0.5, 1, 2 % w/v) with an intercept; null hypothesis: slope zero; same assumption class, with independence between flies.

Runs: one per model, condition and dataset, 16 in all, unattended.

Recorded, and reported afterwards, as before: whether a proposal is produced; whether it is accepted by the Lean kernel with the standard axioms only; the method chosen; the number of compile attempts; then the counts from the published papers and the e-value of each accepted proposal.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.