acceptodds
Withdrawn from ICLR 2027

N×N E-VALUATION: Hypothesis Certification Without Per-Hypothesis Conditional Resampling Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

High-throughput discovery pipelines produce hypotheses faster than a conditional null can be designed for each, making certification a bottleneck. Held-out valida- tion shows whether a hypothesis predicts but not how: the same score can come from population prevalence, a pattern shared across the scope, a unit-matched re- striction, or an association carried by a minority. Conditional randomization tests must specify, for each hypothesis, its explanatory variables, conditioning set and resampling law. We introduce N×N e-valuation, in which the units governed by one hypothesis serve as references for one another. It tests whether held- out success is carried by the hypothesis’s scope, by its restriction, or by which unit receives which restriction, with no resampling model written per hypothesis, and reports which of these carries the prediction. E-values give finite-sample, per-hypothesis control of false certification under stated exchangeability nulls, and an asymptotically valid equivalence test certifies sharedness only when re- striction differences above a declared noise ceiling are rejected. On planted and randomized-intervention data, N×N certifies shared associations that a CRT and an exact permutation test reject; on these and on real data with expert-audited la- bels, it rejects minority-carried associations they certify: their nulls ask whether any unit-level dependence exists, not whether an association holds across its scope. Removing any of its statistics merges a verdict that certifies with one that rejects. The certificates concern predictive association, not causality.

Primary area:
causal reasoning
Keywords:
certification, CRT, hypothesis verification, causal reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_qbTnOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

Rather than testing a hypothesis in order to solely reject or accept it, the authors suggest to test more meaningful aspects of the hypothesis such as whether it is unit specific or shared across the population. They suggest to test these aspects through a framework called NxN E-valuation. They evaluate how well each unit’s predicted restriction fits other unit’s outcomes and derive an NxN matrix. This matrix, combined with other statistical tests, determines whether predictions are unit-specific or shared and whether the hypotheses are meaningful and sufficiently widespread.

Critical strengths

The problem the authors aim to solve is important and well-motivated. Instead of testing a hypothesis in order to reject or accept it, they test whether the hypothesis is meaningful. They distinguish between unit-specific associations, shared population-level associations, base-rate effects, and subgroup-driven associations. This can further enable automated scientific discovery. The method is intuitive and useful. Creating the NxN matrix and using it to test the different aspects of the hypothesis is very appealing.

Critical weaknesses
  1. As the authors acknowledge, the baselines that are considered (CRT, permutation test) do not target the same statistical problem, thus the comparison seems somewhat unfair.

  2. Overall, the paper is well structured, but not written very clearly. Some explanations are difficult to follow, making the proposed methodology harder to understand.

  3. The experiments are somewhat limited in their scope. All the experimental settings are based on recommendation systems. Including other types of hypotheses, such as continuous outcomes or other scientific associations would strengthen the paper.

  4. It is unclear how to choose the coverage threshold and the parameter k, which determines how similar the restrictions must be in order to be considered shared. Although the authors provide sensitivity analyses, they do not suggest any data driven way to choose these parameters in order to achieve reliable diagnostic.

Questions for authors

None

Reviewer_H4xNOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes N*N e-valuation to certify machine-proposed hypotheses without the per-hypothesis resampling model that a model-X CRT requires. Each in-scope unit's restriction is scored on every in-scope outcome, and four statistics read from this N*N matrix and an out-of-scope reference (two e-values, an equivalence test, and a coverage test) map each hypothesis to one of seven verdicts. Experiments on a synthetic recommendation generator, a planted intervention on Amazon Video-Games, and 101 LLM-written hypotheses compare it with a model-X CRT, a pairing permutation test, and held-out validation.

Critical strengths
  1. The validity results are correct, and the paper separates the exact parts from the asymptotic one.
  2. The proposal and certification data are kept disjoint, which avoids obvious leakage.
Critical weaknesses
  1. The machine-automatable requirement does not separate N*N from the CRT. The CRT's computation is fully mechanical, and what a machine cannot verify is its resampling law. N*N has the same problem. Its guarantee rests on exchangeability nulls that the user must check, and a wrong one silently voids it, just as a wrong resampling law does. The real difference is the null. The CRT tests conditional independence, which needs a model of the covariates as Shah and Peters showed in 2020, while N×N tests a weaker marginal null that needs none. Automatability thus comes from the null and not from the method. It also does no work in the claim of being the first to meet all three requirements, since the CRT already fails the diagnostic one, and no experiment shows a machine-written CRT model failing.

  2. The attributable requirement does not motivate the N*N construction. It asks that the hypothesis, not prevalence, carry the prediction across the scope. The significance e-value and the coverage test already meet this, and neither uses the off-diagonal entries of the matrix. The held-out problem it points to is also easy to fix by scoring each restriction on users outside the scope instead of on random items, which is exactly what the significance e-value does. The matrix only matters for the uniqueness e-value and the invariance call, and those serve the diagnostic requirement. The paper still argues attributability through the pairing of units and restrictions, which its own definition never mentions. So it is unclear what problem the matrix is meant to solve.

  3. The diagnostic requirement is the only one that motivates the matrix, and it is never defined or tested. The paper gives no formal statement of the certification problem. What should be certified is fixed by the method itself, since Matched and Shared are defined by the four statistics. Diagnostic only asks that the output help the proposer, with no criterion for what counts as helping. The verdict names are outside the guarantee, the paper calls its readings examples rather than definitions, and no experiment feeds a verdict back to a proposer. So the part of the method that needs the N*N matrix rests on the weakest of the three requirements.

  4. state that temporal drift breaks the significance null and that your permutation check tests the estimator, not this assumption. In the Amazon data each outcome is a user's last three purchases, so a scope built from history, such as recent PS5 buyers, puts every in-scope outcome after 2020 while reference outcomes can come from earlier years. How would a machine check the significance null for each such scope, and was it checked for the 17 real certificates?

Questions for authors
  1. The authors state that temporal drift breaks the significance null and that their permutation check tests the estimator, not this assumption. In the Amazon data each outcome is a user's last three purchases, so a scope built from history, such as recent PS5 buyers, puts every in-scope outcome after 2020 while reference outcomes can come from earlier years. How would a machine check the significance null for each such scope, and was it checked for the 17 real certificates?
  2. All planted base rate cases use a scope that is independent of the outcomes, so their significance e-value is near 1 by construction. Consider the best sellers of 2021 claimed for recent PS5 buyers. It is a base rate for one period, yet it would give a large significance e-value and a shared restriction. Which statistic rejects it, and how does it differ from the planted Shared case of holiday shoppers buying gift wrap?
  3. The authors suggest that their four misses on the real corpus may be right and the human labels wrong. If the labels cannot show that a verdict is wrong, and the verdict names are outside Theorem 1, what evidence in the paper could show that a verdict is wrong?
  4. The permutation test patched with the other three statistics already reaches 404 of 408 in Table 5, and Section F applies e-BH only to the significance e-value. What does the uniqueness e-value gain over an exact permutation p-value, which needs no threshold of 20?
Reviewer_uaTCOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The key insight in this paper is that if you have some hypothesis and empirical data to test it; instead of using hold-out prediction error, you can certify a hypothesis by leveraging how well you would predict other individuals in the dataset using your prediction on a given individual.

Critical strengths

This presents a practical and principled solution to a fairly deep problem: can we leverage other individual outcomes in a dataset to help evaluate whether our predictive hypotheses are good. I haven't seen such a setup (with the NxN cross eval) before and I can see that it's theoretically grounded and practical. It's cool that it is designed to disambiguate individually predictive information from base-rate effects etc...

Critical weaknesses

This paper is quite hard to follow; it introduces the notion of a conditional randomization test in paragraph 2 sentence 1 without really formally defining what it means. This made it quite hard to follow as someone without a background in hypothesis testing. I also think the statistical tools are fairly standard.

Questions for authors

Is it a practical assumption to make that you can always compute r_i(y_j)? Or is this specific to the recommender setting? Can you give some discussion of when you can do this in a scientific setting?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.