acceptodds
Under review as a conference paper at ICLR 2027

Acceptance-Side Multiple Testing for Proxy Discovery

Abstract

Proximal causal learning (PCL) identifies causal effects under unmeasured confounding from valid proxies of the confounder. When proxies must be discovered from high-dimensional data, thousands to millions of candidate proxy sets are screened at once, and the consequential error is false validation: retaining an invalid candidate that then biases the causal estimate. We formulate proxy discovery as an acceptance-side multiple-testing problem. A joint tetrad test gives one -value per candidate triplet, a two-group model converts these into posterior validity scores , and accepting has oracle false validation rate (FVR) at most . The plug-in rule inherits no guarantee: where invalid -values retain mass near one, the upward bias of Storey's , conservative for the FDR, is anti-conservative for the FVR, and no rule based on the -values alone can correct it. An exact identity equates the accepted set's mean absolute bias to its empirical FVR times the mean bias of the accepted invalid triplets. Simulations show where each layer holds and fails; we illustrate the pipeline on 5.7 million candidate triplets from TCGA–BRCA.

Primary area:
causal reasoning
Keywords:
proximal causal learning, proxy discovery, negative controls discovery, multiple testing

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_EPPXOpenReview ↗
Rating
3(+1)
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper is about picking valid proxy variables for proximal causal inference when you have to search through millions of candidate gene triplets. The key point is that the costly mistake here is accepting a bad proxy, not rejecting a good one. So the authors flip the usual multiple-testing view to the acceptance side. They give each triplet one p-value from a joint tetrad test, turn it into a posterior probability this triplet is valid with a standard two-group model, and accept triplets above a threshold. They show this controls contamination in the oracle case, explain why the plug-in version can fail when weak violations look like nulls, and show that the bias of the accepted set equals contamination times severity. Simulations and a TCGA-BRCA example illustrate the pipeline.

Critical strengths
  1. Framing proxy validation as controlling false acceptances is the right way to think about it, and the paper shows clearly why rejection-side corrections like Bonferroni or BH make acceptance more permissive, not less.
  2. Proposition 2 explains exactly when and why the plug-in rule becomes anti-conservative, and the worked bounds match the simulations closely.
  3. The FVR × severity identity separates "how often bad proxies get in" from "how much damage they do," and the same-sign benchmark design avoids flattering the method through bias cancellation.
Critical weaknesses
  1. Most of the theory is a direct consequence of well-known multiple-testing results. The oracle guarantee, the non-identifiability result, and the bias identity are each short derivations of standard facts, and the paper itself admits this. What's really new is the framing, which is useful but thin for a main contribution.
  2. The proposed rule accepts exactly the same triplets as a simple p-value cutoff, it only chooses where to put that cutoff. And the cutoff depends on estimating how many candidates are valid, which is the very estimate the paper shows is biased in the unsafe direction.
  3. The guarantee only holds when bad proxies are easy to spot, which is when any reasonable rule would work anyway. When violations are subtle, contamination ends up several times higher than the target. The TCGA sample size falls in exactly that weak regime, so the method can't promise much on its own showcase application, and the paper offers no fix for this case.
  4. Dependence between candidates is mostly left open. In TCGA, millions of triplets are built from a few hundred genes, so they overlap heavily. The dependence experiment is a stylized p-value simulation, and even there, the target is missed about half the time once correlation is moderately high.
  5. The real-data example doesn't really support the method. Cancer stage is treated as a continuous outcome in a linear model, null calibration is never checked on this data, and the two "consistency checks" are, by the authors' own admission, guaranteed to pass for any triplet that passes the test. On top of that, genes in the same co-expression cluster are quite likely to be directly connected, which would break the proxy assumptions from the start.
Questions for authors
  1. Since the accepted set is {P ≥ ĉ}, can you show precision–coverage curves of your rule against fixed raw-p cutoffs at matched coverage, across regimes? Does the gain come only from choosing the cutoff?
  2. Is there a conservative variant, such as a bound on θ from estimated noncentralities or a lower bound on π0, that restores FVR control in the weak regimes, even at a coverage cost?
  3. For TCGA, can you run a semi-synthetic check (plant known DNCTs into the real covariance structure at N = 507) to estimate the realized FVR?
  4. Can you check null calibration on TCGA with a permutation or sandwich covariance on a subset of triplets?
  5. How variable is π̂0 on TCGA under a bootstrap over tumours, given the heavy overlap between triplets?
6 comments
AuthorsOpenReview ↗

Response to Reviewer EPPX (1 of 6): W1, W2 and Q1

We thank the reviewer for a careful reading.

Terms. With τ=0.9\tau=0.9, at most 10% of accepted triplets should be invalid (the target). Invalid share: the fraction of accepted triplets that are invalid (the paper's FVR); a rule fails when it exceeds the target. Coverage: the fraction of valid triplets accepted.

W1. The theory follows from known results; the new part is the framing.

We agree that the proofs are short; Appendix E.1 says that Proposition 1 "is not a new theorem about multiple testing". The contribution is the acceptance-side question. Existing proxy-discovery methods, including DANCE, Proxy-Rank, Proxy-GIN and generalized-tetrad procedures, accept a candidate set when fixed-level tests do not reject it; to our knowledge none controls or reports what fraction of the accepted sets is invalid. New since submission: which invalid triplets pass and how much bias they cause (W3; Concern 2), and a comparison with fixed cutoffs (below; Concern 1).

W2 and Q1. The rule accepts the same triplets as a p-value cutoff, and the cutoff rests on a biased estimate. Is there any gain beyond the choice of cutoff?

In short: no gain in ranking; the gain is in where the cutoff goes. Within one dataset the rule accepts {p≥c^}\lbrace p\ge\hat c\rbrace, so at equal coverage it selects the same triplets as a fixed cutoff and the precision–coverage curves coincide. Across datasets the right cutoff changes, and a fixed one cannot follow it.

How we tested it (Concern 1). We start from the valid five-variable model: one hidden confounder drives the three proxies, treatment and outcome. We then move into the space of alternatives by adding one or two extra arrows, a chain of two correlated confounders, or a second confounder, and vary the strength of each violation. The strength sets how far the alternative is from the null, dd, and so its detectability ν=Nd\nu=Nd. We used 98 such violations, from barely to clearly detectable, at five sample sizes from 500 to 10,000. For each, 5,000 simulated datasets give 5,000 p-values, which we mix with uniform p-values of valid triplets at valid shares 0.1, 0.2 and 0.5: 1,470 populations with known truth. Our rule, using its own π^0\hat\pi_0 and so its overestimate, and 13 fixed cutoffs see the same p-values. We record the invalid share, the coverage and the harmful share: the fraction of accepted triplets that bias the causal estimate by more than 10%.

Results. Coverage and failures in the 874 populations where some cutoff can meet the target; harmful share over all 1,425 arrow and chain populations, where our rule keeps 50% of valid triplets:

RuleValid triplets keptFailsHarmful share
Our rule0.706%2.4%
Cutoff 0.050.050.9533%13.3%
Cutoff 0.300.300.7020%9.5%
Cutoff 0.500.500.5014%8.2%
Cutoff 0.700.700.3010%7.2%
Cutoff 0.900.900.105%6.3%
Cutoff 0.950.950.053%6.0%

Our rule's failures count populations where it accepts (it abstains in 4%). A fixed cutoff can only trade valid triplets kept for fewer failures; our rule sits off that trade-off. No fixed cutoff keeps as many valid triplets and fails as rarely, overall or at any single sample size or valid share, and none accepts as small a harmful share. Over the full catalogue (3,190 alternatives, 10% valid, 7,752 populations where the target can be met) the picture is the same: our rule keeps 74% and fails in 2.6%; the cutoff 0.300.30 keeps 70% and fails in 10%.

The bias of π^0\hat\pi_0. We agree (Proposition 2), and the comparison above includes it. It is a property of the p-values, not of our rule: any acceptance region on the p-value, a fixed cutoff included, has an invalid share of at least 1−π0/π0⋆1-\pi_0/\pi_0^\star.

Conclusion. The rule is a p-value cutoff, and the revision says so. Its contribution is that the cutoff is placed from the data. Despite the overestimate of π^0\hat\pi_0, this gives a third as many failures and a third of the harmful acceptances of a fixed cutoff of equal coverage, and seven times the coverage of one of equal safety.

AuthorsOpenReview ↗

Response to Reviewer EPPX (2): W3

Response to Reviewer EPPX (2): W3

The point. The guarantee holds only when violations are easy to detect, which is when any reasonable rule would work anyway; TCGA, with 507 patients, is in the weak regime, and the paper offers no fix.

1. Detectability is a distance. Each invalid triplet sits some distance from the null, in units of sampling noise, growing with the strength of the violating edges and the number of patients.

2. Near violations look valid. A near violation's p-values are nearly uniform, like valid ones, up to p=1p=1. Raising the cutoff removes both together, so the FVR, the invalid share of the accepted set, stays above a positive limit even at the strictest cutoff.

3. So there is a limit for every rule. If violations are near enough, this limit is above the 10% target, and no rule that sees only p-values can do better: part of the invalid p-value distribution is uniform, like the valid one, and the data cannot separate the two (Proposition 2). For the same reason π^0\hat\pi_0 is too high, counting that part as valid: harmless for rejection rules, but on acceptance it lets invalid triplets in. The limit comes from the data, not our rule, and is not at easy violations: there the 5% test still misses them with probability between a third and two thirds.

4. Above the limit, our rule picks the cutoff from the data. Both accept the largest p-values in the same order; a fixed cutoff stops at the same cc whatever the alternative. Our rule accepts a triplet when its estimated probability of being valid, π^0/f^(p)\hat\pi_0/\hat f(p), is at least 90%; this increases with pp, so the rule is a cutoff set by the alternative through the p-values. Far from the null, invalid p-values pile up near 0 and the cutoff is low; close to it, they spread towards 1 and the cutoff rises. The right cutoff, the lowest with FVR at most 10%, changes with the alternative, which the analyst does not know. So "any reasonable rule would work" is wrong.

In simulation (Concern 1): close to the null, the weakest violations for which some cutoff still meets the target (105 populations); far from the null, four or more times as detectable (510). Medians; ours where it accepts (it abstains in 16% and 1%).

Right cutoffOur cutoffInvalid share accepted: ours / 0.050.05 / 0.950.95Valid accepted: right cutoff / ours / 0.050.05 / 0.950.95
Close to the null0.880.7910.3 / 61 / 7%12 / 20 / 95 / 5%
Far from the null0.000.080 / 0 / 0%100 / 91 / 95 / 5%

Each fixed cutoff suits one row: close to the null 0.050.05 accepts sets 61% invalid; far from it 0.950.95 discards 95% of valid triplets. Close to the null even the right cutoff accepts only 12% of valid triplets. Ours accepts 20%, because π^0\hat\pi_0 is overestimated (step 3), so its invalid share exceeds 10% in half the populations, but by little: at most 15% in 90% of them, 19% at worst. These are the weakest violations: they bias the causal effect estimate by 0.9% averaged over accepted triplets (median population), against 4.3% for 0.050.05.

5. Below the limit, accepted invalid triplets mostly cause small bias. For extra arrows and chains of confounders, detectability and the bias in the causal effect estimate both grow with violation strength, so the invalid triplets that pass are weak. Ours passes fewer harmful triplets (bias above 10%) than all 13 fixed cutoffs (Concern 2, part 2): 2.4% of its accepted set, against 8.2% for the cutoff 0.50.5 at the same 50% of valid triplets, and 5.7% even for 0.990.99. The exception is a second confounder acting on the proxies almost like the first: no tetrad test detects it, yet its bias can be large (Concern 3).

In one sentence. Below a certain detectability no rule meets the target; above it, our rule reads the right cutoff from the p-values, and even with π^0\hat\pi_0 overestimated, no fixed cutoff matches it on both precision and coverage over the range of alternatives, from weak to strong.

AuthorsOpenReview ↗

Response to Reviewer EPPX (3): Q1, Q2 and Q4

Q1. Precision–coverage against fixed cutoffs; does the gain come only from choosing the cutoff? Yes, by design. Fix one alternative, i.e. one type and strength of violation at one sample size. Our rule accepts {p≥c^}\lbrace p\ge\hat c\rbrace, so it is one point on that alternative's precision–coverage curve (precision =1−=1-FVR), and the fixed cutoff c^\hat c does exactly as well. But the analyst does not know which alternative generated the invalid triplets, and the right cutoff ranges from about 0.9 for alternatives close to the null to almost 0 for those far from it (comment 2, step 4). A fixed cutoff commits to one point for every alternative; our rule reads the alternative from the p-value distribution and moves along the curve to the right point. The relevant comparison is therefore over the range of alternatives, at matched coverage. Over the 874 populations of Concern 1 in which some cutoff can meet the target (our rule's precision where it accepts, 96% of them):

Valid triplets acceptedMean precisionPopulations with FVR above 10%
Our rule70%97.9%6%
Cutoff 0.300.30, same coverage70%94.4%20%
Cutoff 0.700.70, same precision30%97.7%10%

Pooled over the alternatives, the fixed cutoffs trace one precision–coverage curve as cc varies, and our rule lies above it: at the same coverage its accepted sets are less than half as contaminated and miss the target a third as often; matching its precision costs a fixed cutoff more than half the valid triplets. Its accepted sets also hold fewer harmful triplets than all 13 fixed cutoffs (Concern 2, part 2). We will add the curves, per alternative and pooled, to the appendix.

Q4. Null calibration on TCGA. Done with the sandwich covariance, on all 5.7 million triplets. The test's only normal-theory step is the covariance of the sample covariances; replacing it by empirical fourth moments (Bollen's distribution-free test) barely changes the selection: π^0\hat\pi_0 goes from 0.157 to 0.168, the implied cutoff stays at 0.950, and 94% of the accepted set is accepted again. Non-normal data therefore do not produce the large p-values.

Q2 answered in Response to Reviewer LDhe, part 2 of 2 (Q1, Q2)

Reviewer_t94pOpenReview ↗
Rating
2(−1)
Confidence
1
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The focus of the paper is on Proximal causal learning. The required proxy discovery of the cofounder is formulated as a multiple hypothesis testing problem.

Critical strengths

None (because of lack of expertise).

Critical weaknesses

The paper is highly technical and hard to follow. It is written for experts of the field, without much discussion about the practical interest of the approach. I was quickly lost.

There is no reference about the phenomenon of false validation mentioned in the abstract to motivate this work.

Questions for authors

Please provide examples of application of the method.

Please provide references about the phenomenon of false validation mentioned in the abstract; how serious is this in practice?

Reviewer_LDheOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The paper formulates proxy discovery in proximal causal learning (PCL) as an acceptance-side multiple-testing problem. It replaces separate per-tetrad tests with a single candidate-level joint Wald test (Bollen's test) for candidate proxy triplets. A two-group mixture model converts these p-values into posterior local validity scores L(p)=P(H=0∣P=p)L(p) = \mathbb{P}(H=0 \mid P=p), and accepting candidates where L(p)≥τL(p) \ge \tau controls the false validation rate (FVR) at an oracle level of at most 1−τ1 - \tau. The authors demonstrate that Storey's tail estimator π^0\hat{\pi}_0, which is conservative for the false discovery rate (FDR) on the rejection side, becomes anti-conservative on the acceptance side due to upward bias when invalid p-values retain mass near one (Proposition 2). Furthermore, an exact identity equates the mean absolute PCL bias of an accepted set to its empirical FVR times the severity (mean absolute bias) of the accepted invalid triplets. The framework is evaluated through extensive simulations and illustrated on 5.7 million candidate triplets from TCGA-BRCA gene expression data.

Critical strengths
  • The paper shifts the perspective of multiple testing from rejection-side error control (FDR/FWER) to acceptance-side error control (FVR/pFNR), which directly matches the operational requirements of PCL where candidate proxy sets must be retained rather than discarded.
  • By establishing the exact factorization identity connecting empirical FVR and severity to the mean absolute population PCL bias, the work bridges a critical gap between statistical screening errors and downstream causal inference validity.
Critical weaknesses
  • In the broader multiple-testing and empirical Bayes literature (e.g., Storey, 2002, 2003; Efron, 2010), upward biases in tail estimators like π^0\hat{\pi}_0 are considered safe because they lead to conservative (over-controlled) rejection-side FDR rules. However, as Proposition 2 highlights, this behavior completely reverses on the acceptance side: an overestimated π^0\hat{\pi}_0 inflates every local validity score L^(p)\hat{L}(p), expanding the accepted set and producing anti-conservative FVR control. As the authors note, unidentifiable "impure" null mass from weak violations or faithfulness failures cannot be separated from the true null distribution using p-values alone, exposing a fundamental limitation of p-value-based acceptance methods.
  • When viewed alongside broader proxy-discovery frameworks e.g. Proxy-Rank and Proxy-GIN (Xie et al., 2024), which accommodate multiple confounders using rank or non-Gaussian independence constraints, or generalized tetrads (Ying et al., 2025) for nonlinear latent relations—the method's reliance on linear single-confounder SEMs and tetrad faithfulness is quite restrictive. Tetrad-compatible invalid triplets caused by proportional loading failures remain entirely invisible regardless of sample size, meaning the pipeline's theoretical guarantees depend heavily on untestable structural assumptions
Questions for authors
  • How does the method's strict requirement for a single latent confounder and tetrad faithfulness compare in practice to multi-confounder approaches like Proxy-Rank (Xie et al., 2024) when applied to complex settings (such as TCGA data in genomics) where multiple regulatory pathways likely interact?
  • Given that Proposition 2 shows the plug-in rule can be anti-contaminated under weak signals, why did the authors not pursue lower confidence bounds on π0\pi_0 or explicit bounds on the impurity parameter θ\theta to provide a more robust finite-sample safeguard?
2 comments
AuthorsOpenReview ↗

**Response to Reviewer LDhe, part 1 of 2 (W1, W2)*

Response to Reviewer LDhe, part 1 of 2 (W1, W2)

We thank the reviewer for a precise reading and for recognising the acceptance-side framing. Fuller tables are in Concerns 2 and 3.

Terms used below. A triplet is invalid if it breaks the assumed model. Its bias is how far the causal effect estimate would be from the truth if that triplet were used, in percent of the true effect; it is harmful if the bias exceeds 10%, and mild otherwise. π^0\hat\pi_0 is our estimate of the fraction of valid triplets.

W1. An overestimated π0\pi_0 inflates every validity score, and the impure null mass cannot be separated using p-values alone.

In short: we agree (Proposition 2). New simulations show what this mass consists of and what it costs.

What we did. We built 3,144 invalid models with a known violation (one or two extra arrows, a chain of two confounders, or a second confounder) and computed the bias of each exactly. For each model we mixed 4,500 of its p-values with 500 valid ones, so the true valid fraction is 0.1, and recorded how far π^0\hat\pi_0 exceeds 0.1.

What we found.

  1. The overestimate depends on how detectable the violation is. Write ν=Nd\nu=Nd for the noncentrality of the test (NN patients; dd the distance of the model from the null). When ν\nu is at most 1 the median overestimate is 0.87; above 10 it is 0.00. So π^0\hat\pi_0 is misled only by violations the test has little power against.
  2. Those violations are mostly mild. For arrows and chains, a weaker violation is both harder to detect and less biasing. Of the 1,114 harmful arrow and chain models, 28% inflated π^0\hat\pi_0 by more than 0.1 at N=500N=500, 4% at N=2,000N=2{,}000 and 0.2% at N=10,000N=10{,}000.
  3. So an accepted set can contain many invalid triplets and few harmful ones. With 20% valid triplets and one-arrow violations of mixed strength, the accepted set was 47% invalid but 3.9% harmful at N=500N=500, 13.6% and 0.5% at N=2,000N=2{,}000, and 0.3% and 0.0% at N=10,000N=10{,}000. Two-arrow violations are harder: 34.5%, 15.9% and 1.4% harmful. (Over batches in which the rule accepted something; at N=2,000N=2{,}000 it accepted nothing in 14 and 24 of 30 batches.)
  4. Our rule still does better than a fixed p-value cutoff. Over 1,425 simulated settings without the near-proportional family, 2.4% of the triplets accepted by our rule were harmful, against 8.2% for the cutoff p≥0.5p\ge0.5, which keeps the same share of valid triplets. None of 13 fixed cutoffs reached 2.4%.

Limits. This does not restore the nominal contamination level, and every rule fails for a second confounder with nearly proportional loadings (W2).

W2. Reliance on a linear single-confounder model and on tetrad faithfulness; proportional loadings are invisible at any sample size.

In short: the test needs less than "one confounder", but it has two blind spots, which the revision states and quantifies.

What the test needs. The tetrads check one pattern: the three proxies are related to each other, and to treatment and outcome, through one shared factor. Several latent causes may lie behind that factor. For the causal estimate to be right, it must also carry the confounding of treatment and outcome. With two latent causes:

CaseTetradsCausal effect estimate
The proxies measure the factor that confounds treatment and outcomezerounbiased
The proxies depend on the two latent causes in different proportionsnot zerorejected, given enough patients
The proxies' factor misses part of the confounding, or the proportions are equalzerocan be biased

Evidence.

  • Case 1. Valid chains of two confounders are accepted as they should be: the 5% test rejects them 3.4% to 5.9% of the time.
  • Case 2. Of 23 harmful chains with the proxies split between the two confounders, five passed p≥0.05p\ge0.05 in most datasets at N=500N=500 and none from N=1,000N=1{,}000.
  • Blind spot A: proxies that miss confounding. If the proxies measure a factor only correlated with the confounder, or TT and OO share a further confounder that no proxy measures, the tetrads are exactly zero at every NN. The bias is 19% at correlation 0.99 and 100% at 0.9.
  • Blind spot B: proportional loadings (Section 2.1). For each proxy take the ratio of the second confounder's loading to the first's; the separation is the difference between neighbouring ratios. The median overestimate of π^0\hat\pi_0 is about zero at separation 0.2 from N=500N=500; at 0.03 it falls from 0.81 at N=500N=500 to 0.09 at N=10,000N=10{,}000; at 0.003 it stays at 0.86 or more at every tested NN.

In our Gaussian constructions of both blind spots, the five variables have the covariances of a valid model with a different causal effect, so no method using only these covariances can tell them apart.

AuthorsOpenReview ↗

Response to Reviewer LDhe, part 2 of 2 (Q1, Q2)

Response to Reviewer LDhe, part 2 of 2 (Q1, Q2)

Q1. How does this compare in practice with Proxy-Rank on data such as TCGA, where several pathways interact?

In short: the two methods answer different questions and could be combined. We have not run Proxy-Rank on TCGA.

  • What Proxy-Rank does that we do not. It can adjust for several confounders of the same treatment and outcome at once, using rank constraints on suitable proxy sets under its identification conditions. Our triplet test handles one confounding factor per treatment and outcome. With one confounder the two share rank-one (tetrad) constraints, although their full procedures differ.
  • What we do that Proxy-Rank does not. Proxy-Rank, like other existing methods, accepts a candidate set when fixed-level tests do not reject it. To our knowledge no existing method controls what fraction of the accepted sets is invalid. With 5.7 million candidates this question arises whichever test is used, and it is the question our paper addresses.
  • What neither can do. Neither resolves the two blind spots of W2: the covariances match those of a valid model with a different effect, so every covariance-based method accepts the invalid set.
  • Combining them. Our acceptance step needs one calibrated p-value per candidate set and the matching causal estimator. Given those from Proxy-Rank's test, it could be applied on top of it (Appendix E.4). We have not evaluated this, and its guarantees would have to be re-established.
  • Interacting pathways in our TCGA analysis. Each triplet has its own shared factor, so different treatment genes can have different confounders. For each treatment gene, proxies come from a cluster in which every pair of genes has absolute correlation of at least 0.35 (Appendix F). This keeps the search among mutually correlated genes; that the cluster's factor carries all the confounding is an assumption.

Q2. Why not use a lower confidence bound on π^0\hat\pi_0, or bound the impurity explicitly?

In short: a confidence bound does not remove the error, because the error includes a systematic bias. A bound on the impurity would, but it needs an assumption that p-values cannot check.

Why a confidence bound does not help. A confidence bound corrects for having too few p-values. Here invalid triplets with large p-values are counted as valid however many p-values there are. Example: a weak arrow between two proxies that the test detects poorly (ν≈1.9\nu\approx1.9), with 10% valid triplets. With 1,000 p-values the average estimate is 0.53; with 5,000 it is still 0.53, and only its standard deviation falls, by more than half. The estimate becomes more precise around a wrong value. (This violation is mild: it biases the causal effect estimate by 5.5%.)

What would help. Valid p-values fall above a point λ\lambda with probability 1−λ1-\lambda; the estimate is too high because some invalid p-values fall there too. Suppose every invalid triplet lands above λ\lambda at most a fraction r<1r<1 as often as a valid one. Then

π0≥Pr⁡(P>λ)/(1−λ)−r1−r,\pi_0\\ \ge\\ \frac{\Pr(P>\lambda)/(1-\lambda)-r}{1-r},

and using this lower value in the rule keeps the contamination at or below the target, at the price of accepting fewer triplets.

Where rr could come from, and why we do not claim it. The p-values cannot supply rr (Proposition 2(i)); it needs an assumption that every invalid triplet is at least some distance from the null.

  • For harmful violations this is partly available. In our simulations every single-arrow or chain violation with bias above 10% had ν\nu of at least 6.8 at N=2,000N=2{,}000, enough to bound rr.
  • It fails for two-arrow violations, where two effects can cancel in the tetrads (smallest ν\nu of a harmful model: 0.8 at N=2,000N=2{,}000), and for near-proportional loadings.
  • It is weak at small NN, and it says nothing about mild invalid triplets, which can be arbitrarily close to the null.

So such a bound could protect against accepting harmful triplets under stated assumptions, not against contamination in general. A usable version must also account for estimating the density, the selection step and dependence between triplets.

In the revision. We state the bound as a sensitivity analysis with these limits, not as a guarantee.

Summary of changes prompted by this review. The paper now reports invalid and harmful acceptances separately (W1); states both blind spots of the tetrads, with the measured sensitivity near proportional loadings (W2); describes Proxy-Rank's broader scope and the conditions for combining the methods (Q1); and explains why a confidence bound does not address the bias, giving the conditional bound with its limits (Q2).

Comments on the paper

5 comments
AuthorsOpenReview ↗

Commom response to all reviewers- We thank the reviewers for their comments and address all concerns one by one Concern 1

Concern 1 (Reviewer EPPX, W2 and Q1). The validity score orders candidates exactly as the p-value does. Does the method add anything beyond a choice of cutoff, especially when that choice rests on a biased estimate?

Answer. The method does not improve the ranking within a dataset; it improves the choice of cutoff across populations, and the improvement is measurable. Within one dataset the fitted score is a monotone function of the p-value, so a raw p-value rule with the same cutoff selects exactly the same triplets. We agree with the reviewer: we claim no improvement in ranking. The contribution is practical: which cutoff should be used before one knows which triplets are invalid?

Measured benefit. Raising a fixed p-value cutoff lowers contamination only by discarding valid triplets. In our experiment (below), over 874 simulated populations, the fitted rule reaches a combination that none of the 13 fixed cutoffs matches: 70% mean valid coverage, 6% of settings above the 10% contamination target, and 1.1% pooled contamination. The gain comes from adapting the cutoff to each population, despite the estimation error in π^0\hat\pi_0.

Experiment. From a valid five-variable model (A,B,C,T,O)(A,B,C,T,O) with one hidden confounder, we took 98 alternatives: one or two additional arrows, a chain of two confounders, or a second latent factor, over violation strengths from negligible to strong. Each alternative has a population distance dd from the null; dd and NN together decide how often the test wrongly accepts it.

Settings. For each alternative and each NN (500, 1,000, 2,000, 5,000, 10,000) we simulated 5,000 datasets, one p-value each. A setting is one alternative, one NN and one share of valid triplets π0\pi_0 (0.1, 0.2 or 0.5): a batch of 5,000 p-values, the valid ones independent and uniform on (0,1)(0,1), the rest from the alternative. This gives 1,470 settings.

Rules and outcomes. To each batch we applied the fitted rule (τ=0.9\tau=0.9), which estimates its cutoff from that batch, and 13 fixed cutoffs, and recorded

contamination=invalid acceptedall accepted,coverage=valid acceptedall valid.\text{contamination}=\frac{\text{invalid accepted}}{\text{all accepted}},\qquad\text{coverage}=\frac{\text{valid accepted}}{\text{all valid}}.

Reachable settings. A setting is reachable when, under the noncentral chi-square approximation to the test, some cutoff gives a population contamination of 10% or less. This depends only on the population, not on how any rule performs. 874 of the 1,470 settings are reachable; the other 596 are reported below.

Result over the 874 reachable settings (10 of the 13 fixed cutoffs shown; the others lie between).

RuleMean valid coverageSettings above 10% contaminationPooled contamination
p≥0.01p\ge0.0199%39%23.8%
p≥0.05p\ge0.0595%33%15.1%
p≥0.1p\ge0.190%27%11.2%
p≥0.2p\ge0.280%23%7.5%
p≥0.3p\ge0.370%20%5.5%
p≥0.5p\ge0.550%14%3.3%
p≥0.7p\ge0.730%10%2.1%
p≥0.9p\ge0.910%5%1.3%
p≥0.95p\ge0.955%3%1.1%
p≥0.99p\ge0.991%3%0.9%
Fitted rule, τ=0.9\tau=0.970%6%1.1%

Pooled contamination is all invalid triplets accepted divided by all triplets accepted, across settings. The fitted rule accepted nothing in 4% of settings; its 6% is among the rest.

The fitted rule has the coverage of p≥0.3p\ge0.3, exceeds the target about as rarely as p≥0.9p\ge0.9, and has the pooled contamination of p≥0.95p\ge0.95. In each of the 15 combinations of sample size and valid share, no tested fixed cutoff matches the fitted rule on both mean valid coverage and the share of settings above target. On the full simulation grid at π0=0.1\pi_0=0.1 (7,752 reachable settings) the rule keeps 74% of valid triplets with 2.6% of settings above target; a fixed cutoff interpolated to that coverage is above target in 11%.

Scope. Each setting has one type of violation. In 52 reachable settings that mix violation types, pooled contamination still favours the fitted rule (5.0% against 7.7% for a fixed cutoff interpolated to the same coverage), while the share of settings above target is similar (28% against 29%).

On the biased estimate. The comparison uses the fitted π^0\hat\pi_0 and the fitted density, not their true values, so their estimation errors are already in the numbers above. Invalid triplets with large p-values can inflate π^0\hat\pi_0 and can pass fixed cutoffs; fixed cutoffs avoid the first problem but not the second. Point 2 quantifies this bias and gives a safeguard. Separately, in the 596 unreachable settings no cutoff attains the target: the fitted rule exceeded 10% in 89% (accepting nothing in 8%), and all 13 fixed cutoffs exceeded it in 95%.

Conclusion. The method addresses cutoff selection, not ranking. One threshold τ\tau fixed in advance yields a cutoff that adapts to each population, and no tested fixed cutoff matches it on coverage and contamination together.

AuthorsOpenReview ↗

Commom response to all reviewers- We thank the reviewers for their comments and address all concerns one by one; concern 2

Concern 2, part 1 of 2 (LDhe W1; EPPX W3): overestimating π0\pi_0 is anti-conservative Answer. We agree: invalid triplets with large p-values inflate the estimated valid fraction, so the fitted rule can accept more invalid triplets than intended (Proposition 2). (i) What does this cost downstream: do the violations that cause the overestimate also bias the causal effect estimate? Mostly not. The overestimate comes from violations the test cannot detect, and, with one exception (a near-failure of tetrad faithfulness), such violations barely bias the causal effect estimate. (ii) Even with the overestimate, the fitted rule accepts a smaller share of harmful triplets than any of the 13 fixed p-value cutoffs we tested (part 2).

We quantify two errors for the same invalid model. Error of the estimated valid fraction: how far π^0\hat\pi_0 exceeds the known true fraction. Bias in the causal effect estimate: how far PCL with that invalid triplet is from the true causal effect.

Step 1: create one known violation. Start from the valid model: one hidden confounder UU drives three proxies, treatment and outcome; the true effect is 0.5. We built 3,144 invalid models, each violating the DNCT structure in one way and at a chosen strength (arrow coefficients from 0.001 to 1), of four types:

  • One arrow (316): a direct effect between two observed variables, e.g. A→BA\to B.
  • Two arrows (2,663): two such effects.
  • Chain (39): a second hidden confounder, correlated with UU, drives some of the five variables in place of UU.
  • Near-proportional second confounder (126): an independent second confounder whose effects on the three proxies are almost proportional to those of UU.

Step 2: compute the exact relative bias in the causal effect estimate under the invalid model. A model is harmful if this exceeds 10% (1,223 models) and mild otherwise (1,921).

Step 3: generate p-values. We simulate datasets of NN patients from the invalid model and apply the tetrad test to each, at five NN from 500 to 10,000. How easily the test detects the model is summarised by ν=Nd\nu=Nd, the noncentrality of the test statistic (dd: squared standardised distance of the model's tetrads from zero).

Step 4: measure overestimation in π^0\hat\pi_0. A batch has 500 uniform p-values and 4,500 p-values from the invalid model, so the true valid fraction is 0.10. We apply the paper's fitted estimator and record the error π^0−0.10\hat\pi_0-0.10.

1. The overestimate is set by detectability. Grouping all 15,720 batches by ν\nu:

Detectability ν\nuup to 11 to 22 to 55 to 10above 10
Median error of π^0\hat\pi_00.870.470.180.030.00
Batches with error of π^0\hat\pi_0 above 0.10100%100%90%1.5%0%

This matches Proposition 2 (predicted excess 0.9 e−ν/20.9\,e^{-\nu/2}): π^0\hat\pi_0 is misled only by violations the test has low power against.

2. The models that cause the overestimate are mostly mild.

NNMedian error of π^0\hat\pi_0, mild modelsMedian error of π^0\hat\pi_0, harmful modelsAmong models with π^0\hat\pi_0 error above 0.10, percent mild
1,0000.780.0183%
2,0000.690.0091%
10,0000.220.0094%

Main result: a harmful violation rarely inflates π^0\hat\pi_0. Of the 1,114 harmful arrow and chain models, π^0\hat\pi_0 was overestimated by more than 0.10 for 18% at N=1,000N=1{,}000, 4% at N=2,000N=2{,}000 and 0.2% at N=10,000N=10{,}000. Those that remain lie just above the 10% line: among the 486 models with relative bias in the causal effect estimate above 20%, the shares are 0.4%, 0% and 0%. The reason: weakening a violation lowers both its detectability and its bias in the causal effect estimate, which grows like ν/N\sqrt{\nu/N}.

3. Largest relative bias in the causal effect estimate among models with error of π^0\hat\pi_0 above 0.10:

TypeN=1,000N=1{,}000N=2,000N=2{,}000N=10,000N=10{,}000
One arrow or chain12%8%3%
Two arrows35%18%10%
Near-proportional second confounder209%209%209%

The exception. If a second confounder's effects on the proxies are exactly proportional to those of UU, the tetrads are zero although the model is invalid: a failure of tetrad faithfulness, not a generic second confounder. Near proportionality, ν\nu stays near zero while the bias in the causal effect estimate is large. Conclusion of part 1. The excess contamination comes from violations that are hard to detect. For one arrow and chains these barely bias the causal effect estimate, and for two arrows the bias shrinks as NN grows. The exception is a second confounder at or near proportional loadings, where tetrad faithfulness fails or nearly fails. We will add a graph showing pi0 overestimation along distance from null vs bias in causal effect estimates along distance from null, in the updated paper.

AuthorsOpenReview ↗

Commom response to all reviewers- We thank the reviewers for their comments and address all concerns one by one; concern 2 part 2

Concern 2, part 2 of 2 (EPPX W3; LDhe W2): does the fitted rule improve selection despite the overestimate?

4. EPPX W3: the guarantee holds only "when any reasonable rule would work anyway". Part 1 holds for any rule that accepts large p values. But even with the overestimated valid fraction, the fitted rule accepts a smaller share of harmful triplets than any of the 13 fixed p-value cutoffs we tested.

The comparison. We selected 95 arrow and chain configurations, excluding the near-proportional second confounder, and evaluated each at five patient sample sizes and three valid fractions: 1,425 settings. A setting is reachable if, under the noncentral chi-square working model, some p-value cutoff retains candidates with population contamination of at most 10%, and unreachable if no cutoff does. We include both (851 reachable, 574 unreachable) and ask how many harmful triplets each rule accepts, including where the 10% target cannot be met.

RuleMean valid coveragePooled harmful fraction of accepted
Fitted rule, τ=0.9\tau=0.950%2.4%
p≥0.50p\ge0.5050%8.2%
p≥0.05p\ge0.0595%13.3%
p≥0.99p\ge0.991%5.7%

Coverage is averaged over all 1,425 settings; over the reachable ones the fitted rule's coverage is 70%, as in Concern 1. At the same 50% coverage the fixed cutoff has more than three times the harmful fraction. None of the 13 fixed cutoffs reaches the fitted rule's value, not even the strictest, which keeps 1% of valid candidates. The gap is largest at smaller samples: at N=1,000N=1{,}000 the harmful fractions are 1.2% (fitted), 6.5% (p≥0.5p\ge0.5) and 11.0% (p≥0.05p\ge0.05); at N=10,000N=10{,}000 they are 0.1%, 0.2% and 1.4%.

Why this is possible despite identical rankings. A batch is the collection of p-values analysed together, 5,000 here. Within a batch, the fitted rule is equivalent to a p-value cutoff. Across batches it chooses different cutoffs, because it estimates the valid fraction and the p-value density of each batch:

L^(p)=min⁡{1,π^0f^P(p)},\widehat L(p)=\min\left\lbrace 1,\\ \frac{\widehat\pi_0}{\widehat f_P(p)}\right\rbrace,

and accepts candidates with L^(p)≥0.9\widehat L(p)\ge0.9. Where invalid p-values sit near zero, the density reaches its flat level early and the cutoff is low; where they reach far into the upper tail, the cutoff is high or the rule accepts nothing (6% of settings). In the 128 settings where p≥0.5p\ge0.5 accepts more than 10% harmful triplets, the fitted rule's median cutoff was 0.93 and it kept 12% of valid candidates instead of 50%. In the other 1,297 settings both rules accepted 0.2% harmful triplets and the fitted rule kept slightly more valid ones (54%).

The estimates can be wrong: invalid candidates contribute to the apparent uniform component and inflate the scores. The table therefore evaluates whether adapting the cutoff helps despite this estimation error. Across the tested settings it does.

5. LDhe W2: where does adapting the cutoff not help? When a second confounder affects the three proxies in almost the same proportions as the first. For each proxy, take the ratio of the second confounder's loading to the first's; the separation is the difference between neighbouring ratios. A small nonzero separation becomes detectable with enough patients; exact proportionality never does. In this family about 75% of accepted triplets are harmful under every rule, fitted or fixed (details in Concern 3).

Conclusion (both parts). The simulations quantify both the overestimate of the valid fraction and its causal consequences. Substantial overestimation came predominantly from mild violations, those biasing the causal effect estimate by less than 10%. Across the comparison settings, which exclude near-proportional second confounders, the fitted rule had a smaller pooled harmful fraction than every tested fixed cutoff, despite the estimation error. This shows a practical benefit of adapting the cutoff; it does not certify the accepted set in an application such as TCGA.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.