- Rating
- 3(+1)
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper is about picking valid proxy variables for proximal causal inference when you have to search through millions of candidate gene triplets. The key point is that the costly mistake here is accepting a bad proxy, not rejecting a good one. So the authors flip the usual multiple-testing view to the acceptance side. They give each triplet one p-value from a joint tetrad test, turn it into a posterior probability this triplet is valid with a standard two-group model, and accept triplets above a threshold. They show this controls contamination in the oracle case, explain why the plug-in version can fail when weak violations look like nulls, and show that the bias of the accepted set equals contamination times severity. Simulations and a TCGA-BRCA example illustrate the pipeline.
Critical strengths
- Framing proxy validation as controlling false acceptances is the right way to think about it, and the paper shows clearly why rejection-side corrections like Bonferroni or BH make acceptance more permissive, not less.
- Proposition 2 explains exactly when and why the plug-in rule becomes anti-conservative, and the worked bounds match the simulations closely.
- The FVR × severity identity separates "how often bad proxies get in" from "how much damage they do," and the same-sign benchmark design avoids flattering the method through bias cancellation.
Critical weaknesses
- Most of the theory is a direct consequence of well-known multiple-testing results. The oracle guarantee, the non-identifiability result, and the bias identity are each short derivations of standard facts, and the paper itself admits this. What's really new is the framing, which is useful but thin for a main contribution.
- The proposed rule accepts exactly the same triplets as a simple p-value cutoff, it only chooses where to put that cutoff. And the cutoff depends on estimating how many candidates are valid, which is the very estimate the paper shows is biased in the unsafe direction.
- The guarantee only holds when bad proxies are easy to spot, which is when any reasonable rule would work anyway. When violations are subtle, contamination ends up several times higher than the target. The TCGA sample size falls in exactly that weak regime, so the method can't promise much on its own showcase application, and the paper offers no fix for this case.
- Dependence between candidates is mostly left open. In TCGA, millions of triplets are built from a few hundred genes, so they overlap heavily. The dependence experiment is a stylized p-value simulation, and even there, the target is missed about half the time once correlation is moderately high.
- The real-data example doesn't really support the method. Cancer stage is treated as a continuous outcome in a linear model, null calibration is never checked on this data, and the two "consistency checks" are, by the authors' own admission, guaranteed to pass for any triplet that passes the test. On top of that, genes in the same co-expression cluster are quite likely to be directly connected, which would break the proxy assumptions from the start.
Questions for authors
- Since the accepted set is {P ≥ ĉ}, can you show precision–coverage curves of your rule against fixed raw-p cutoffs at matched coverage, across regimes? Does the gain come only from choosing the cutoff?
- Is there a conservative variant, such as a bound on θ from estimated noncentralities or a lower bound on π0, that restores FVR control in the weak regimes, even at a coverage cost?
- For TCGA, can you run a semi-synthetic check (plant known DNCTs into the real covariance structure at N = 507) to estimate the realized FVR?
- Can you check null calibration on TCGA with a permutation or sandwich covariance on a subset of triplets?
- How variable is π̂0 on TCGA under a bootstrap over tumours, given the heavy overlap between triplets?
6 comments
Response to Reviewer EPPX (2): W3
Response to Reviewer EPPX (2): W3
The point. The guarantee holds only when violations are easy to detect, which is when any reasonable rule would work anyway; TCGA, with 507 patients, is in the weak regime, and the paper offers no fix.
1. Detectability is a distance. Each invalid triplet sits some distance from the null, in units of sampling noise, growing with the strength of the violating edges and the number of patients.
2. Near violations look valid. A near violation's p-values are nearly uniform, like valid ones, up to . Raising the cutoff removes both together, so the FVR, the invalid share of the accepted set, stays above a positive limit even at the strictest cutoff.
3. So there is a limit for every rule. If violations are near enough, this limit is above the 10% target, and no rule that sees only p-values can do better: part of the invalid p-value distribution is uniform, like the valid one, and the data cannot separate the two (Proposition 2). For the same reason is too high, counting that part as valid: harmless for rejection rules, but on acceptance it lets invalid triplets in. The limit comes from the data, not our rule, and is not at easy violations: there the 5% test still misses them with probability between a third and two thirds.
4. Above the limit, our rule picks the cutoff from the data. Both accept the largest p-values in the same order; a fixed cutoff stops at the same whatever the alternative. Our rule accepts a triplet when its estimated probability of being valid, , is at least 90%; this increases with , so the rule is a cutoff set by the alternative through the p-values. Far from the null, invalid p-values pile up near 0 and the cutoff is low; close to it, they spread towards 1 and the cutoff rises. The right cutoff, the lowest with FVR at most 10%, changes with the alternative, which the analyst does not know. So "any reasonable rule would work" is wrong.
In simulation (Concern 1): close to the null, the weakest violations for which some cutoff still meets the target (105 populations); far from the null, four or more times as detectable (510). Medians; ours where it accepts (it abstains in 16% and 1%).
| Right cutoff | Our cutoff | Invalid share accepted: ours / / | Valid accepted: right cutoff / ours / / | |
|---|---|---|---|---|
| Close to the null | 0.88 | 0.79 | 10.3 / 61 / 7% | 12 / 20 / 95 / 5% |
| Far from the null | 0.00 | 0.08 | 0 / 0 / 0% | 100 / 91 / 95 / 5% |
Each fixed cutoff suits one row: close to the null accepts sets 61% invalid; far from it discards 95% of valid triplets. Close to the null even the right cutoff accepts only 12% of valid triplets. Ours accepts 20%, because is overestimated (step 3), so its invalid share exceeds 10% in half the populations, but by little: at most 15% in 90% of them, 19% at worst. These are the weakest violations: they bias the causal effect estimate by 0.9% averaged over accepted triplets (median population), against 4.3% for .
5. Below the limit, accepted invalid triplets mostly cause small bias. For extra arrows and chains of confounders, detectability and the bias in the causal effect estimate both grow with violation strength, so the invalid triplets that pass are weak. Ours passes fewer harmful triplets (bias above 10%) than all 13 fixed cutoffs (Concern 2, part 2): 2.4% of its accepted set, against 8.2% for the cutoff at the same 50% of valid triplets, and 5.7% even for . The exception is a second confounder acting on the proxies almost like the first: no tetrad test detects it, yet its bias can be large (Concern 3).
In one sentence. Below a certain detectability no rule meets the target; above it, our rule reads the right cutoff from the p-values, and even with overestimated, no fixed cutoff matches it on both precision and coverage over the range of alternatives, from weak to strong.
Response to Reviewer EPPX (3): Q1, Q2 and Q4
Q1. Precision–coverage against fixed cutoffs; does the gain come only from choosing the cutoff? Yes, by design. Fix one alternative, i.e. one type and strength of violation at one sample size. Our rule accepts , so it is one point on that alternative's precision–coverage curve (precision FVR), and the fixed cutoff does exactly as well. But the analyst does not know which alternative generated the invalid triplets, and the right cutoff ranges from about 0.9 for alternatives close to the null to almost 0 for those far from it (comment 2, step 4). A fixed cutoff commits to one point for every alternative; our rule reads the alternative from the p-value distribution and moves along the curve to the right point. The relevant comparison is therefore over the range of alternatives, at matched coverage. Over the 874 populations of Concern 1 in which some cutoff can meet the target (our rule's precision where it accepts, 96% of them):
| Valid triplets accepted | Mean precision | Populations with FVR above 10% | |
|---|---|---|---|
| Our rule | 70% | 97.9% | 6% |
| Cutoff , same coverage | 70% | 94.4% | 20% |
| Cutoff , same precision | 30% | 97.7% | 10% |
Pooled over the alternatives, the fixed cutoffs trace one precision–coverage curve as varies, and our rule lies above it: at the same coverage its accepted sets are less than half as contaminated and miss the target a third as often; matching its precision costs a fixed cutoff more than half the valid triplets. Its accepted sets also hold fewer harmful triplets than all 13 fixed cutoffs (Concern 2, part 2). We will add the curves, per alternative and pooled, to the appendix.
Q4. Null calibration on TCGA. Done with the sandwich covariance, on all 5.7 million triplets. The test's only normal-theory step is the covariance of the sample covariances; replacing it by empirical fourth moments (Bollen's distribution-free test) barely changes the selection: goes from 0.157 to 0.168, the implied cutoff stays at 0.950, and 94% of the accepted set is accepted again. Non-normal data therefore do not produce the large p-values.
Q2 answered in Response to Reviewer LDhe, part 2 of 2 (Q1, Q2)
Response to Reviewer EPPX (1 of 6): W1, W2 and Q1
We thank the reviewer for a careful reading.
Terms. With τ=0.9, at most 10% of accepted triplets should be invalid (the target). Invalid share: the fraction of accepted triplets that are invalid (the paper's FVR); a rule fails when it exceeds the target. Coverage: the fraction of valid triplets accepted.
W1. The theory follows from known results; the new part is the framing.
We agree that the proofs are short; Appendix E.1 says that Proposition 1 "is not a new theorem about multiple testing". The contribution is the acceptance-side question. Existing proxy-discovery methods, including DANCE, Proxy-Rank, Proxy-GIN and generalized-tetrad procedures, accept a candidate set when fixed-level tests do not reject it; to our knowledge none controls or reports what fraction of the accepted sets is invalid. New since submission: which invalid triplets pass and how much bias they cause (W3; Concern 2), and a comparison with fixed cutoffs (below; Concern 1).
W2 and Q1. The rule accepts the same triplets as a p-value cutoff, and the cutoff rests on a biased estimate. Is there any gain beyond the choice of cutoff?
In short: no gain in ranking; the gain is in where the cutoff goes. Within one dataset the rule accepts {p≥c^}, so at equal coverage it selects the same triplets as a fixed cutoff and the precision–coverage curves coincide. Across datasets the right cutoff changes, and a fixed one cannot follow it.
How we tested it (Concern 1). We start from the valid five-variable model: one hidden confounder drives the three proxies, treatment and outcome. We then move into the space of alternatives by adding one or two extra arrows, a chain of two correlated confounders, or a second confounder, and vary the strength of each violation. The strength sets how far the alternative is from the null, d, and so its detectability ν=Nd. We used 98 such violations, from barely to clearly detectable, at five sample sizes from 500 to 10,000. For each, 5,000 simulated datasets give 5,000 p-values, which we mix with uniform p-values of valid triplets at valid shares 0.1, 0.2 and 0.5: 1,470 populations with known truth. Our rule, using its own π^0 and so its overestimate, and 13 fixed cutoffs see the same p-values. We record the invalid share, the coverage and the harmful share: the fraction of accepted triplets that bias the causal estimate by more than 10%.
Results. Coverage and failures in the 874 populations where some cutoff can meet the target; harmful share over all 1,425 arrow and chain populations, where our rule keeps 50% of valid triplets:
Our rule's failures count populations where it accepts (it abstains in 4%). A fixed cutoff can only trade valid triplets kept for fewer failures; our rule sits off that trade-off. No fixed cutoff keeps as many valid triplets and fails as rarely, overall or at any single sample size or valid share, and none accepts as small a harmful share. Over the full catalogue (3,190 alternatives, 10% valid, 7,752 populations where the target can be met) the picture is the same: our rule keeps 74% and fails in 2.6%; the cutoff 0.30 keeps 70% and fails in 10%.
The bias of π^0. We agree (Proposition 2), and the comparison above includes it. It is a property of the p-values, not of our rule: any acceptance region on the p-value, a fixed cutoff included, has an invalid share of at least 1−π0/π0⋆.
Conclusion. The rule is a p-value cutoff, and the revision says so. Its contribution is that the cutoff is placed from the data. Despite the overestimate of π^0, this gives a third as many failures and a third of the harmful acceptances of a fixed cutoff of equal coverage, and seven times the coverage of one of equal safety.