acceptodds
Under review as a conference paper at ICLR 2027

Feedback Loops Invalidate Significance Tests in LLM Alpha Mining

Abstract

LLM alpha mining, like other automated discovery systems, proposes candidate formulas and scores them on historical data. The scores are fed back to guide the next proposals. The system then presents its highest scoring formula as a discovery. It backs that claim with a multiple testing correction, which promises at most a 5% chance that the discovery is false. That promise assumes the candidates were chosen without looking at the data. A feedback loop breaks this assumption, because every new proposal depends on how earlier ones scored. As a result, the more these systems search, the less their discoveries can be trusted. Prior work established this risk in theory, but its remedies change how the search runs. We measure the problem in real LLM discovery loops and fix it without changing them. First, we prove that no correction computed from the chosen candidates can keep its promise for every feedback loop. For a natural class of loops, the chance of a false discovery tends to 100% as the search grows. Second, we measure the effect on stock market data shuffled so that no formula can predict returns. There, the strongest standard test for data snooping declares an LLM alpha miner's best formula significant 15% of the time. The Benjamini-Hochberg procedure does so 29% of the time. A search that never sees its scores, and the same loop with its scores shuffled, both stay near 5%. The feedback is therefore the cause. Four times the budget lifts the two rates to 28% and 70%, while both controls stay flat. The gap widens as the search grows. The same failure appears in symbolic regression, outside finance. Third, we give a fix. Counting candidates more carefully does not help. Instead, we rerun the unchanged search many times on shuffled data and set the significance threshold from those runs. This keeps the 5% promise for any searcher. On later market data the search never saw, 92% of the formulas it accepts remain significant. Its main cost is rerunning the search, and a simple statistical shortcut cuts the reruns needed from about 100 to about 20. Practitioners can therefore attach a trustworthy significance claim to any discovery loop, at modest cost and without redesigning it.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
significance testing, multiple testing, adaptive data analysis, data snooping, LLM agents, automated discovery, alpha mining

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_jXdSOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes full-procedure null calibration to address significance-test failures caused by feedback loops in LLM alpha mining. The method reruns the unchanged search many times on shuffled data and sets the significance threshold based on those runs.

Critical strengths
  1. The authors study an important problem of significance test failure in LLM alpha mining.
  2. The authors provide solid theoretical analysis of why feedback-driven search invalidates significance tests.
  3. The authors conduct multi-setting experiments to verify the impact of feedback-based search on significance testing.
Critical weaknesses
  1. The proposed method relies on an appropriate null-hypothesis dataset, which may be difficult to construct for real-world financial time series.
  2. Existing experiments are conducted using GPT, while the performance of the method across different LLMs has not yet been evaluated.
  3. The paper did not provide ablation studies on the granularity and frequency of the feedback provided to the LLM.
Questions for authors
  1. How do the authors prevent errors and hallucinations generated by LLMs from affecting the results of statistical inference?
  2. Could the authors evaluate additional LLMs, such as Qwen and DeepSeek, to demonstrate the robustness of the proposed method?
1 comment
AuthorsOpenReview ↗

Response to Reviewer jXdS: the null construction, other LLMs, feedback ablations, and LLM errors

Thank you for the review. We go through each point below, and we say plainly where the paper has a limit.

W1. The method needs an appropriate null dataset. That's the method's one assumption, and we agree its scope should be stated clearly. The Discussion and the Limitations (H.8) now do that. In the setting of the paper the requirement is mild, because we never model the time series. The within-date permutation only reshuffles, on each day, which stock gets which return. Every feature's time series, each day's return distribution and the dependence among candidates that runs through the features stay exactly as they are, and for rank statistics the null is exact by exchangeability. It does destroy each stock's return autocorrelation. The forward-shift null keeps it and reproduces the ordering for the per-candidate procedures, and on the real market the i.i.d. and Newey-West statistics differ by a median factor of 1.05 (H.8). We also check the null instead of assuming it: on it, the non-adaptive control sits at nominal with uniform pp-values (Table 1, KS p=0.30p=0.30). And we report the constructions that failed that check, and why (Appendix E): block resampling and a circular shift leak the target into the features, so we excluded them. The real limit is that a strategy on a single time series has no cross-section to permute, so its nulls are approximate. The paper now says so.

W2 / Q2. Other LLMs such as Qwen and DeepSeek. The language-model evidence is one model, gpt-oss-20b. We didn't have enough time in the response period to run another one. The smallest useful check, the feedback arm and its shuffled control at R=100R=100, takes 200 full searches with the model in the loop. So we leave other models to future work, and the Limitations say plainly that the evidence is one model. We did try two larger models before submission, and H.8 explains why neither was usable. What we can tell you is which claims depend on the model.

  • The remedy doesn't. Proposition 2 holds for any searcher, so the guarantee is the same whatever model proposes the formulas. In our runs the remedy's measured size is the same on every arm, including the two with no language model. A different model would change how many formulas pass the bar, not whether the bar is valid.
  • The failure isn't specific to language models. Genetic programming, with no language model, reaches 64% under Benjamini-Hochberg by a budget of 10,000 (Table 7). The simulation, with no model at all, reaches 43% under SPA and 86% under Benjamini-Hochberg (Table 6).
  • The magnitudes are this model's. The 15% and 29% in Table 1 could be larger or smaller for another model, and we can't say which without running it. The Limitations now say this.

W3. Ablations on the granularity and frequency of feedback. You're right, this is missing for the language model. On that arm the feedback format is fixed. Each round the model proposes 20 formulas, then sees the 15 best so far and 10 random others, with scores. We didn't have time to vary the table size or the batch size in the response period, and the paper lists this as a limitation.

The paper has three related experiments. They aren't the ablation you ask for, but point the same way: the more the search learns from its scores, the more the tests over-reject.

  • What the feedback contains. With no scores, or with scores shuffled across formulas, SPA stays near 5% (3.8% and 4.8%). With real paired scores it is 15.2% (Table 1).
  • How long the loop runs. Four times the budget takes SPA to 28% and Benjamini-Hochberg to 70%, while both controls stay flat (Figure 1, Table 7).
  • How strongly proposals follow the scores. In the simulation we turn this up from 0 to 0.95, and SPA goes from 6% to 43% (Table 6).

Conversely, the ladder shows a score only when it beats the best so far (0.38% of queries). Under Benjamini-Hochberg that brings genetic programming back to nominal on the US panel, though it halves what the search finds (Section 5.4).

Q1. LLM errors and hallucinations. The model never computes or reports a statistic. It only writes formula strings. Each string is parsed against a fixed grammar, and if it doesn't parse it's discarded and never scored. A valid formula is scored on the data by a deterministic evaluator. So a hallucination can only be an invalid formula, which is dropped, or a valid but poor one, which is just one more candidate. Both are part of the procedure, and the same model with the same prompt is re-run on every null dataset (prompts and grammar are hashed, with no mismatch in 2,000 replicates), so whatever errors the model makes show up in the null distribution too. That's what "for any searcher" in Proposition 2 means. Appendix D reports the one such failure we found on the reported model, which we fixed before any reported run: a reasoning model's scratchpad was being parsed as proposals.

Thank you again for the time and effort you put into this review.

Reviewer_Azz2OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper fixes the issue that a feedback loop introduces in an LLM alpha mining system, i.e., breaking the assumption that the candidates were selected without using the data. The paper both measure the problem in LLM discovery and prove that no correction based on the candidates can fix the problem in the feedback loop. Instead of recounting the candidates, the paper shuffles the data to accordingly set the significance threshold. The result is demonstrated in several simulations.

Critical strengths
  • The non-technical parts of the paper are presented very well and are easy to understand for a general reader.
  • The problem itself is investigated empirically and theoretically. Then a solution is provided which is tested empirically.
Critical weaknesses
  • The technical parts are very difficult to read for a non expert audience, especially Section 3.
Questions for authors

Can the authors provide an illustrative example for Section 3 to make the concepts and the first two propositions all clear?

1 comment
AuthorsOpenReview ↗

Response to Reviewer Azz2: a worked example now opens Section 3.1

Thanks for the suggestion. Section 3.1 now opens with a worked example, and Appendix H.1 has the full version with a table.

Setting. Take data with no signal at all, so every formula's score is pure noise, just a standard normal number. A "discovery" is any score above the bar that a correction sets.

The search. First, score K=1000K=1000 unrelated formulas. By luck alone the best one scores about 3.2. Then evaluate N=1000N=1000 variants of that winner, each correlated ρ=0.7\rho=0.7 with it. Each variant inherits 0.7×3.20.7\times3.2 from its parent and adds noise of its own, and the best of 1000 such draws is worth about 2.3. So the best variant scores about 4.6.

The correction. Bonferroni for the 2000 formulas evaluated sets the bar at 4.06. The search clears it on 95% of signal-free datasets, even though the correction promises 5%.

The comparison (Proposition 1). Now change just one thing: pick the parent blindly instead of picking the best. A blind parent scores about zero, its best variant about 2.3, and the same bar is cleared only 3.5% of the time.

blind parentbest parent
best of the 1000 parents3.243.24
best of the 1000 variants2.324.58
Bonferroni bar for 2000 tests4.064.06
false discovery, Bonferroni3.5%95.4%
bar from re-running the search3.965.21
false discovery, re-run bar5.0%5.0%

Both searches evaluate the same number of formulas, with the same correlations among them. A correction that only looks at "how many formulas, and how correlated" sees the same inputs, so it has to set the same bar for both. That bar is right for the blind search and wrong for the one that reads its scores. That's Proposition 1: such a correction can't be exact for the first search and valid for the second, because the thing that separates them, how the parent was chosen, isn't in its inputs. Corollary 1 adds that for this kind of search the gap grows with its size: with the same Bonferroni rule the false-discovery rate is 53%, 95% and 100% at K=102K=10^2, 10310^3 and 10410^4.

The repair (Proposition 2). Shuffle the data so that no signal can exist, and re-run the whole search on it, including the step "pick the best and try its variants". The re-runs also reach about 4.6, because the selection step happens inside them too. The 95th percentile of their best scores, 5.21, becomes the bar, so a signal-free dataset clears it 5% of the time by construction. For the blind search the same recipe gives 3.96. In other words, the bar adapts to the search, and nobody has to describe how the search works.

Proposition 1 is now stated this way in the paper: two searches whose outputs look the same to a correction, one adaptive and one not. We also shortened the densest passages of Sections 4.2 and 5.3. If any step is still unclear, please don't hesitate to let me know.

Thank you again for the time and effort you put into this review.

Reviewer_mMEROpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies discovery loops where a proposer (an LLM, or genetic programming) emits formulas, each one is scored on a fixed market panel, and the scores go back into the next round. The usual corrections for the best of many strategies (Bonferroni, Holm, Benjamini-Hochberg, Benjamini-Yekutieli, White’s Reality Check, Hansen’s SPA, deflated Sharpe) assume the candidate set was chosen without looking at the data, and the loop breaks that. Proposition 1 says no correction computed from the realised candidates is valid for every adaptive searcher, and Corollary 1 says its size goes to one as the search grows for a cluster construction. On within-date permutation nulls of CSI 300 the LLM arm rejects under SPA 15.2% of the time and under Benjamini-Hochberg 28.5% against 5% nominal, while a random search, the same model with no feedback table, and the same loop with scores shuffled across formulas sit at nominal (Table 1). The rates grow with budget (Figure 1, Table 7). Counting trials better through an effective trial count does not help, and the reusable holdout and the ladder restore size but cost yield or run out of queries (Section 5.4). The fix is to rerun the unchanged search on R permuted panels and take the 95th percentile of the null maximum as the hurdle (Proposition 2), with a Gumbel fit to bring R from about 100 to about 20 (Appendix H.9). A simulated null, a forward shift null, CSI 500, a CRSP panel and a symbolic regression domain are in the appendices.

Critical strengths

The experimental hygiene is the best part of this paper. The llm against llm shuffled contrast is the right causal control, since the prompt, the table shape and the score distribution are identical and only the pairing is destroyed (Section 4.4, Appendix C). Every arm runs on the same null panels so the contrasts are paired, the controls are checked for uniform p-values and not just size (Figure 6), and period C was scored once by a script which refuses to run twice (Section 4.5). Also, Appendix A lists three earlier bugs which inflated the effect and were caught because the control stopped sitting at nominal. So I trust Table 1 more than I usually trust numbers of this kind.

Critical weaknesses

“The feedback is therefore the cause” is in the abstract, but the causal contrast holds on CSI 300 and the simulation and nowhere else. On CSI 500 the shuffled control is itself at 10% on SPA with non-uniform p-values, and the paired test does not separate llm from it (p = 0.63, Table 13). On symbolic regression the set conditioned on the full budget gives llm against llm shuffled p = 0.18 (Appendix H.6). And on the simulated null the random control is above nominal on SPA with Kolmogorov-Smirnov p = 2.3e-4 (Table 10), which the paper’s own recommended practice in Appendix H.1 says indicates an implementation error. So the marginal result replicates everywhere and the causal one replicates on one market, and the abstract should say that. Also, Appendix H is full of passages marked “moved verbatim” with stale numbers, for example 13% and 26% for llm in Appendix H.7 against 15.2% and 28.5% in Table 1, and “the same 100 panels” in Appendix H.8 for a table at R = 400, so it took me real effort to work out which number is current. This can be fixed.

Questions for authors

Who is the practitioner at risk? Table 7 says the practitioner rule (IC ≥ 0.02, |t| ≥ 2, Sharpe ≥ 1) never fires on a null at any budget up to 10,000, and Section 2 says none of the LLM alpha miners cited audits its own selection statistics. So “It backs that claim with a multiple testing correction” in the abstract describes a practice the paper itself says does not exist. If the target is the factor-zoo hurdles being applied to an LLM loop in future, please say so, because it changes how much the title promises.

1 comment
AuthorsOpenReview ↗

Response to Reviewer mMER: causal claim scoped, Appendix H cleaned, and the user at risk stated

Thanks for the careful report. We agree with both weaknesses and with how you read the target. We come back to the title at the end.

1. Scope of the causal claim. The abstract now says: "On this market the feedback is therefore the cause. [...] The inflation reappears on a second market and in symbolic regression, outside finance, though the shuffled control separates less cleanly there." Section 5.2, the Discussion and the Limitations (H.8) say the same.

  • CSI 500. On SPA, llm doesn't separate from llm_shuffled (p=0.63p=0.63), and the shuffled control is itself at 10%. We don't have an explanation for that 10%. Your comment made us run the paired test on the other procedures too, which we hadn't reported. On Benjamini-Hochberg the shuffled control is at nominal on CSI 500 (7%, [3.4, 13.7]) and the contrast does separate: 22 against 3 discordant panels, p=1.6×10−4p=1.6\times10^{-4}. These tests are post hoc, across six procedures, and only this one survives a Bonferroni adjustment for the six (p≈10−3p\approx10^{-3}). So CSI 500 replicates the causal contrast on Benjamini-Hochberg, post hoc, and not on the bootstrap tests. The paper says it with that label.
  • Symbolic regression. We had only reported the set conditioned on the full budget (p=0.18p=0.18). The primary set is now there too: 10 against 3 discordant datasets, p=0.09p=0.09; against random p=0.013p=0.013 and against zero-shot p=0.001p=0.001 (H.6). So the marginal result replicates, and the causal contrast isn't established at R=100R=100.
  • Simulated null. You're right: a control with KS p=2.3×10−4p=2.3\times10^{-4} contradicts our own sentence that the control "must sit at nominal". The rule we actually follow is narrower, and "Recommended practice" (H.1) now spells it out. On a null that is exact for the statistic (the within-date permutation), a non-nominal control means an implementation error. On a null that is only approximately valid (the parametric simulation), the control measures the null's own miscalibration. There, the three arms without paired feedback share the inflation (7.5-7.8% on SPA), which is our evidence that it belongs to the null. So absolute levels can't be read there; only the paired contrast on shared panels can. The introduction now quotes that null's numbers next to its control's.

H.8 says that on the bootstrap tests the causal contrast is established on CSI 300 and nowhere else. We didn't run CSI 500 at R=400R=400, and more replicates may not settle it: at the same R=100R=100, CSI 300 already separated the two on SPA (p=0.003p=0.003).

2. Stale numbers in Appendix H. Fixed.

  • Every "moved verbatim" heading is gone, and passages duplicated within the appendix are merged or removed.
  • The paragraph in H.7 that read 13% and 26% now gives the R=400R=400 values, with Wilson intervals recomputed from the per-panel indicators: SPA 15.2% [12.1, 19.1], Bonferroni 12.5% [9.6, 16.1], BY 8.5% [6.1, 11.6], BH 28.5% [24.3, 33.1].
  • The old values were the first 100 of the same 400 panels. We keep them in one labelled paragraph, because the budget sweep and the CSI 500 comparison are at R=100R=100.
  • "The same 100 panels" in H.8 is corrected to 400. Appendix H now opens by saying which RR each table uses, and Section 5.1 says that sizes are measured at R=400R=400 and hurdles calibrated at R=100R=100.
  • Sentences that said "every correction" breaks now except the deflated Sharpe ratio, which is conservative on every arm.

3. Who is the practitioner at risk? You read it correctly, and the abstract was wrong to say the system "backs that claim with a multiple testing correction". Sorry about this mistake. None of the LLM alpha miners we cite reports a selection adjustment. The target is the one you name, the hurdles the finance literature prescribes for mined strategies, applied to the output of a feedback loop. That's the step a careful user, a risk committee or a referee would ask for next, and what we show is that it doesn't deliver the guarantee it states. The paper now says this.

  • Abstract: "The standard way to back such a claim is a multiple testing correction".
  • Introduction: "Today's alpha miners report their winner with no correction at all (Section 2). The repair the factor-zoo literature prescribes is to add one, and whether that survives a real adaptive search has not been measured."

On the practitioner rule: it never fires on a null at any budget, and the paper still says so. We're not claiming that practitioners who use it are being fooled today. It's a bar on economic magnitude, not a test, and it states no error rate. The title is about significance tests: anyone who wants an error-rate statement for a loop's output has to go beyond that rule, and the standard way of doing that is what fails. We think the title promises that and no more, but we'd welcome your view if you still read it more broadly.

Separately, Proposition 1 is now restated more narrowly and proved in full (H.1; see our response to Reviewer zYfW).

Thank you again for the time and effort.

Reviewer_zYfWOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies LLM alpha mining with automated searches that propose candidate formulas, evaluate them on a fixed panel of market data, and repeat the loops. It shows that standard multiple-testing methods violates the assumption of a fixed candidate set before data are seen. The authors propose to rerun the complete search on transformed datasets and using the resulting maxima to calibrate the right hurdle. Extensive simulation examples and real market analyses are used to support the proposed method.

Critical strengths
  1. The problem is important.
  2. The experimental design is strong.
Critical weaknesses

The theory is insufficient and contain gaps.

  1. The proof of Proposition 1 is wrong. The proof of proposition 1 in H.1 states: “A procedure calibrated on (M,Σ^)(M,\hat{\Sigma}) takes as its null the law of max over these columns with the block anchored on an unconditional representative.” This statement assumes that every allowed correction uses the quantile obtained by treating the selected parent as fixed without seeing the scores. However, specifying the inputs to a function does not determine the function itself. The argument thus does not establish the failure for every function of these inputs.
  2. Even the above problem is corrected, the construction in this proof and the proof of corollary 1 is still wrong. The proof starts with KK initial candidates, each with KK variants. The search evaluates 2K2K candidates. However, every evaluated score belongs to a family of K+K2K+K^2 possible scores, and the multiplicity should be K+K2K+K^2 instead of 2K2K.
Questions for authors

If the proofs of proposition 1 and corollary 1 are corrected, which specific fixed-design calibrations are covered by them?

1 comment
AuthorsOpenReview ↗

Response to Reviewer zYfW: Proposition 1 restated and proved in full; the K+KN count

Thank you for reading the proofs closely. Your first point found a real gap, and we've fixed it. On your second point, we agree with your count but not with the conclusion drawn from it. Both are now in the revised PDF (Section 3.1, Appendix H.1).

  1. "Specifying the inputs to a function does not determine the function." This is right. The proposition talked about every functional of (M,Σ^)(M,\hat\Sigma), but the sketch only handled the one that returns the fixed-design quantile. The statement was too strong as written: a conservative enough functional is valid on our construction. So we replaced both.

    Revised Proposition 1. Fix ρ∈(0,1)\rho\in(0,1) and ϵ>0\epsilon>0. For all large KK there are a Gaussian candidate class and two searchers of the form (1), a non-adaptive A0\mathcal A_0 and an adaptive A1\mathcal A_1, whose M=2KM=2K realised candidates have the same field covariance Σ\Sigma, such that every real cc satisfies

    Pr⁡A1(max⁡iSD(fi)>c)≥1−ϵorPr⁡A0(max⁡iSD(fi)>c)≤ϵ.\Pr_{\mathcal A_1}\big(\max_i S_D(f_i)>c\big)\ge 1-\epsilon\quad\text{or}\quad\Pr_{\mathcal A_0}\big(\max_i S_D(f_i)>c\big)\le\epsilon .

    Proof (full version in H.1). Take KK parents with i.i.d. N(0,1)N(0,1) scores ZkZ_k, each with N=KN=K variants scoring ρZk+1−ρ2εk,n\rho Z_k+\sqrt{1-\rho^2}\varepsilon_{k,n}. Both searchers evaluate every parent, then every variant of one parent JJ. A1\mathcal A_1 picks J=arg⁡max⁡kZkJ=\arg\max_k Z_k; A0\mathcal A_0 picks JJ uniformly, blind to the scores. The realised Σ\Sigma is the same matrix up to relabelling the parents, so any functional of (M,Σ)(M,\Sigma) that doesn't read candidate labels gives both the same cc. With bK=2log⁡Kb_K=\sqrt{2\log K}, the statistic is bK(1+oP(1))b_K(1+o_P(1)) under A0\mathcal A_0 and (ρ+1−ρ2)bK(1+oP(1))(\rho+\sqrt{1-\rho^2})b_K(1+o_P(1)) under A1\mathcal A_1, and ρ+1−ρ2>1\rho+\sqrt{1-\rho^2}>1. A threshold halfway between separates the two, and every cc falls on one side of it.

    So a correction that rejects with probability above ϵ\epsilon on the fixed design, for instance one that is exact at a level α>ϵ\alpha>\epsilon, has size at least 1−ϵ1-\epsilon on A1\mathcal A_1. One that is valid on A1\mathcal A_1 rejects with probability at most ϵ\epsilon on the design it was built for. Estimating Σ\Sigma doesn't let you tell the two apart either. For Gaussian rows and mean scores, Σ^\hat\Sigma is independent of the scores and has the same law under both. So a critical value that concentrates below the halfway threshold, as a consistent estimate of the fixed-design quantile does, has size near one on A1\mathcal A_1. A randomised one is not covered.

  2. "The multiplicity should be K+KNK+KN instead of K+NK+N." We agree with the count. Bonferroni over K+KNK+KN is valid on A1\mathcal A_1, and in this construction it's close to the best valid bar. But that doesn't make the construction wrong. It's the second case of the dichotomy. A0\mathcal A_0 produces the same MM and Σ\Sigma, and for it K+NK+N is enough. So which count applies can't be read off the realised candidates. It depends on what the kernel could have proposed across the score histories it might have seen.

    In numbers (K=N=1000K=N=1000, ρ=0.7\rho=0.7, one-sided 5%; H.1), the bar for K+KNK+KN tests is 5.33. On A1\mathcal A_1 that is valid (size 2.9%). On A0\mathcal A_0, whose exact bar is 3.96, it rejects 0.01% of the time. For a language-model proposer, the potential family is every formula the model might write in response to every table of scores it might be shown. At best that's the whole grammar, and counting it is the second case at its most conservative. Re-running the search on null data (Proposition 2) integrates over that family without listing it. Its bar on A1\mathcal A_1 is 5.21, and it's exact.

  3. Which fixed-design calibrations are covered? H.1 now lists them. Each makes its first rejection at or below a threshold of 2log⁡M(1+o(1))\sqrt{2\log M}(1+o(1)), while the statistic on A1\mathcal A_1 is (ρ+1−ρ2)2log⁡M(\rho+\sqrt{1-\rho^2})\sqrt{2\log M}, so the size of each goes to one (Corollary 1):

  • the exact fixed-design quantile of the maximum under N(0,Σ)N(0,\Sigma) (Westfall-Young max-tt with fixed columns), which the Reality Check and SPA bootstraps estimate;

  • Bonferroni and Holm;

  • Benjamini-Hochberg, which rejects whenever Bonferroni does, and Benjamini-Yekutieli;

  • Bonferroni with any effective trial count N′≤MN'\le M.

    The limits: the result is asymptotic and the family is constructed. At K=1000K=1000 and ρ=0.7\rho=0.7 the sizes on A1\mathcal A_1 are 98% for the exact quantile, 95% for Bonferroni and at least 53% for Benjamini-Yekutieli. The two bootstrap tests are covered only as far as the bootstrap recovers the fixed-design quantile; we check that by simulation (54% against 2.7% at K=50K=50, N=200N=200, ρ=0.9\rho=0.9) but don't prove it. A correction whose bar reads the realised scores is outside the statement. The deflated Sharpe ratio is one, and in our experiments it's conservative on every arm.

Thank you again for the time and effort you put into this review.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.