- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces MIRAGEBENCH, which varies latent confounding while preserving the underlying direct lagged causal structure. Its POLE protocol separately evaluates pair ranking, operating points, lag localization, and error control. Experiments with 22 methods show that strong ranking performance can coexist with substantial false discoveries. Increasing the number of observed variables improves error control for some methods, whereas longer time series do not provide comparable benefits.
Critical strengths
- Important problem and clear evaluation. Separating ranking quality, structural recovery, and error control avoids characterizing reliability through a single metric.
- Informative empirical findings. Spectral deconfounding achieves an AUPR of 0.996 while exhibiting a mean FDP of 0.200, illustrating that excellent ranking does not guarantee reliable edge selection. Reporting power alongside FDP and explicitly stating the scope of the findings are additional strengths.
Critical weaknesses
- The contribution of shared confounding remains insufficiently isolated. In Table 13, spectral deconfounding has an FDP of 0.272 under independent latent forcing, compared with 0.204 under shared forcing. False discoveries may therefore also reflect temporal dependence or model misspecification. Their contributions to the observed resource-dependent trends remain unclear.
- Resource variation changes other problem characteristics. Different values of N correspond to independently generated systems with changing graph properties. Increasing H also increases total latent influence without energy normalization. The effects of observation count, network complexity, and latent influence are therefore not fully separated.
- Method and scenario coverage is limited. Experiments focus on linear-Gaussian systems. Methods specifically designed to accommodate latent confounding, such as LPCMCI, are not scored because compatible ground truth is unavailable, limiting the benchmark’s coverage.
- Recovery-threshold robustness requires further validation. Classification depends on 30 repetitions, a t-test, and multiplicity correction. Identical data can yield different thresholds under different correction families. Some settings with mean FDP displayed as 0.00 in Figure 9 are nevertheless classified as undecided and require further explanation. (Appendix A.1; Table 15)
Questions for authors
- Do the main resource-dependent trends persist when the shared-versus-independent latent-forcing comparison is extended across different (N,T) settings?
- Do the recovery boundaries remain stable when the core causal structure is fixed and latent input energy is normalized across different values of H?
- Could an evaluation track with compatible ground truth be developed for methods such as LPCMCI to improve coverage of approaches that accommodate latent confounding?
- Why are settings with near-zero mean FDP classified as undecided? What are their selection sizes and power, and do the thresholds remain stable with more repetitions or alternative inference procedures?
4 comments
Response to Reviewer QGRE (2/2)
Latent-aware methods (W3, Q3). We thank the reviewer for this suggestion. A compatible ground-truth track for methods such as LPCMCI is feasible in principle and would broaden the benchmark's coverage, because the generator retains the observed and latent structural metadata needed to derive a latent-projected reference. However, LPCMCI estimates a partial ancestral graph (PAG) whose adjacencies can include relations induced by latent common causes and whose endpoint marks encode ancestral constraints rather than one-step direct effects. Scoring this output against our direct lagged-edge truth could therefore count some correct PAG adjacencies as false direct-edge discoveries. A valid track requires a lag-window-matched latent-projected reference together with a mark-aware scoring rule. We nonetheless recognize this as a valuable extension and have added it as a concrete direction in Appendix E.6.
Recovery-threshold robustness (W4, Q4). To address the reviewer's question, we answer its two parts in turn. (1) The undecided cells that print a mean FDP of 0.00 in Figure 9 are the five Knockoffs cells at , , one for each number of latent drivers . As stated in Section 3.3 and Appendix A.1, a mean FDP below does not by itself establish recovery, and a cell in which every run has zero FDP has an undefined statistic and is left undecided. In these five cells the screen selects on average only 0.8 to 7.3 lagged edges per run, at mean power below 0.01, so the printed 0.00 accompanies an almost empty selection, and recovery remains undecided under the zero-variance rule. We now state this link between Figure 9 and the classification rule in Appendix D.2.
(2) We evaluated separately the three sources of sensitivity the reviewer raises: the test statistic, the number of repetitions, and the multiplicity family. The table counts the changes relative to the reported classification:
| Variation | Runs per cell | Cell verdicts changed | Slice thresholds changed |
|---|---|---|---|
| Wilcoxon signed-rank test | 30 | 0 of 450 | 0 of 75 |
| Studentized bootstrap | 30 | 0 of 450 | 0 of 75 |
| Holm within each record length | 30 | 1 of 450 | 1 of 75 |
| Benjamini–Hochberg | 30 | 9 of 450 | 6 of 75 |
| Original runs, threshold cells only | 30 | – | 0 of 13 |
| Doubled runs, threshold cells only | 60 | – | 1 of 13 |
The choice of test statistic changes no verdict or threshold. Narrowing the Holm family to each record length moves only the spectral-deconfounding threshold at , , by one grid step from 80 to 60, the cell that Section 4.3 already names as sensitive to the correction family. Doubling the runs at the 13 numeric spectral-deconfounding thresholds, classified in a Holm family of those 26 cells, moves the same threshold to 60 and leaves the other 12 unchanged. The original 30 runs, classified in that same family, leave all 13 unchanged, so this shift reflects the added runs rather than the narrower family. Benjamini–Hochberg moves the same threshold and additionally resolves five slices that the Holm rule leaves unresolved, one at and four as right-censored beyond the grid. The numeric thresholds therefore hold within one grid step, while whether a few boundaries can be resolved depends on the multiplicity family. We have added this sensitivity analysis to Table 20 of Appendix D.2.
Thank you again for your positive assessment and for supporting our work. Please feel free to let us know if you have any further questions. Thank you!
Thank you for the author's response; my concerns have been resolved, and I will keep my score unchanged.
Thank you for your valuable suggestions and comments, which have helped us strengthen the paper.
Response to Reviewer QGRE (1/2)
We thank the reviewer for the constructive comments and for recognizing the importance of the problem, the separation of ranking quality, structural recovery and error control, and the informativeness of the empirical findings. We address each of your concerns point by point below.
Shared versus independent forcing across (N,T) (W1, Q1). To address the reviewer's question, we have run additional experiments that extend the shared-versus-independent comparison along both N and T, and as a further control we have added variance-matched, temporally independent private forcing. For spectral deconfounding and Knockoffs, the FDP trends along both resource axes persist under independent (private) AR(1) forcing, although their endpoint changes are smaller than under shared forcing. The table gives the mean FDP at the endpoints of each axis:
Across the two methods and both axes, the endpoint changes under private AR(1) forcing are 53–71% of those under shared forcing, whereas temporally independent private forcing stays close to the clean reference. The Table 13 ordering is a crossing of the N-response curves: for spectral deconfounding, shared forcing starts higher at N=20 but declines more steeply, leaving private forcing higher at N=40. Sharing has a distinct effect on marginal ranking: SIS pair AUPR collapses under shared forcing at every evaluated (N,T), but remains high under both private controls. We report the full comparison, including CPSS, in Table 14 of Appendix D.1 and summarize this distinction in Section 4.2.
Fixed core and energy-normalized driver count (W2, Q2). We thank the reviewer for raising this concern. We examined each of the two sources of variation with its own control: (1) a fixed causal core across N, and (2) energy-normalized loadings across H. The main spectral-deconfounding pattern remains in both: FDP on the fixed core decreases with N, and the recovery boundary still moves outward with H.
(1) Fixed core across N. We added a nested control in which a fixed 20-variable core is observed together with additional variables, so that the core causal structure and the truth of its edges are the same at every N. Mean FDP / power on the core edges at H=1, c=2, and T=600 are:
Spectral deconfounding reduces FDP from 0.533 to 0.094 while power stays above 0.97. The decline is smaller for Knockoffs and CPSS, and SIS moves in the opposite direction. Thus, observing more variables of the same system lowers FDP on fixed targets for spectral deconfounding, while the size and direction of the effect remain selector-specific. We have added this comparison to Table 19 of Appendix D.2 and summarized it in Section 4.2.
(2) Energy-normalized driver count across H. The outward shift of the spectral recovery boundary also remains after controlling for total latent input energy. As described in Appendix C.3, the direct structure and the earlier drivers are held fixed across H. We additionally rescaled the loadings so that each variable has the same latent forcing variance as at H=1, and classified the (N,H) grid at T=600 as its own 30-cell family, of the same width as the family behind the benchmark-loading thresholds:
Under energy normalization, the spectral threshold still moves from 60 to 80 and then 120. Thus, within this generator, the effect of driver count is not explained by total latent energy alone. We report the full comparison in Table 21 of Appendix D.3 and now state this result in Section 4.3.