- Rating
- 3
- Confidence
- 3(+1)
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper introduces PoET, a novel approach for graph estimation in SEMs with Poisson likelihoods. PoET uses a test-based approach, starting from an empty graph and successively adding edges if they approximately satisfy a dispersion-based criterion (which is satisfied for actual Poisson DAGs). The test is carried out using a nonparametric approach that allows the likelihood to depend nonlinearly on previous nodes. Empirical results are given showing that the PoET approach tends to outperform competitors at very large numbers of samples
Critical strengths
- The paper is clear and presents the PoET method in an easy-to-understand and clear way. The general argument, that the method is a good approach for structure learning in Poisson DAGs, is generally well-made and seems correct. The overall approach is a nice combination of different existing methods, and the story is convincing.
- The empirical arguments are convincing (at least for high numbers of data points, see weaknesses) and the theoretical consistency arguments are correct, at least as far as I can tell.
Critical weaknesses
- The empirical results could be more compelling. The results for the synthetic experiments don't have any results for n <= 1000, and the results at n=1000 generally show a pretty severe drop compared to 3k. As far as I understand, in these SEM cases we are often quite concerned with relatively small number of data points (e.g. the standard Sachs dataset has 853 observational samples). Similarly, the real-world-data examples are the NBA example (rejected for not being Poisson) and the lending club data, which is only 5-dimensional and could presumably be studied with direct enumeration.
- As far as I understand, the advance is somewhat incremental. The technique consists of combining several previously-known ideas, and it's not obvious there is a huge novelty. However, the careful treatment of the approach including proofs and empirical results means it's, as far as I can tell, about in-line with the novelty of previous DAG papers at ICLR in previous years, so this is not a blocker for acceptance. There is perhaps some over-selling in the paper, e.g. the repeated claims about testing being better than scoring. I'm not entirely sure I understand why we can't just threshold our score to give a test
Questions for authors
- Can you either give results showing PoET's performance at lower n, or explain why the high-n regime is important? Compared to previous work, this n seems quite high (e.g. NOTEARS gives results for n=20, n=1000).
- Can you comment on the distinction between scoring and testing, given that we can presumably construct a test by thresholding the score? Is there a question around the calibration/consistency of such a score-driven test?
4 comments
Response to Reviewer odih - W2 & Q2
(W2/Q2) Distinction from score-based methods.
We thank the reviewer for raising this point. We agree that PoET's distinction from score-based methods was not sufficiently clear in the original manuscript. We have added how PoET is related to score thresholding in Section 3.1.
To construct a test by thresholding the overdispersion score using a normal approximation, we would need to establish the asymptotic distribution of . For a naive plug-in score, this expansion can involve a first-order conditional-mean estimation-error term of the form . With nonparametric estimation, which converges at a rate slower than , this term need not be asymptotically negligible and can even diverge, so consistency of the estimated score alone does not justify a normal approximation. We therefore use , whose residual-based construction and cross-fitting control the first-order estimation error, leaving a quadratic bias that is negligible under the stated conditions.
PoET can nevertheless be viewed as a score-thresholding method in a broader sense by interpreting as a new score. Its contribution is the statistical justification of the threshold. For the equidispersion test, we use
Under the stated conditions, Lemma 2 establishes for a fixed equidispersion null, giving an asymptotic rejection probability of . This statistical justification is the distinction we would like to emphasize.
Thanks for your replies. Although I expected PoET to have worse performance at lower , it's good to see that the performance doesn't crater massively. For the comment about the score vs thresholding, your response makes sense--essentially arbitrarily thresholding a score isn't statistically justified while the approach in PoET is well-specified.
I don't think the paper is necessarily at the 'clear accept' level but your replies have increased my confidence in recommending a weak acceptance.
Thank you for taking the time to review our paper and follow up on our responses. We are glad you found the lower results reassuring and that our explanation clarified how PoET’s statistically justified thresholds differ from arbitrary score thresholding. We appreciate your continued support.
Response to Reviewer odih - W1 & Q1
(W1/Q1) Small-sample performance and the role of larger samples.
We thank the reviewer for pointing out that the original manuscript did not adequately evaluate small-sample performance. We have added experiments in Appendix A.3.8 at n=500 and p=5, close to the real-data settings, comparing PoET with all eight baselines over 100 replications. PoET's mean F1 ranges from 0.558 to 0.689, while ODS achieves the highest mean F1 in all six settings.
Average graph-recovery F1 scores (standard errors) at n=500 and p=5. Each method uses 100 replications.
Graph density: g=2
Graph density: g=4
Larger samples are important for PoET for two reasons.
First, PoET checks model assumptions while learning the graph. Unlike score-based methods, which always return a best-ranked graph decision, PoET estimates local null distributions and withholds further decisions when assumption violation is detected. Its local tests require sufficiently accurate conditional-mean estimates [1]. At small sample sizes, estimation error and limited test power can affect both graph recovery and model diagnostics. Larger samples help support these testing decisions throughout learning.
Second, broader functional flexibility carries a nonparametric estimation cost. PoET estimates unspecified multivariate conditional means, allowing nonlinear effects and interactions beyond prespecified parametric forms. This flexibility makes estimation demanding, especially as the number of conditioning variables increases [2]. The low-dimensional p=5 setting may also favor ODS's exact-cell estimation.
These results illustrate the trade-off between flexible, test-integrated learning and small-sample performance. We have clarified this trade-off and PoET's small-sample limitations in the revised manuscript.
[1] Shah, R. D., and Peters, J. The hardness of conditional independence testing and the generalised covariance measure. The Annals of Statistics, 48(3):1514–1538, 2020.
[2] Bühlmann, P., Peters, J., and Ernest, J. CAM: Causal additive models, high-dimensional order search and penalized regression. The Annals of Statistics, 42(6):2526–2556, 2014.