- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper defines realized observational epistemic potency as the reduction in sharp-bound width obtained from auxiliary observational variables. They introduce Potency GNN, a query-conditioned relational graph neural network, together with a dual-certificate network that provides certified lower bounds on potency. This method outperforms several baselines, and provides speedups over exact LP solving.
Critical strengths
The GNN architecture seems well formulated with their ablation results showing the need for their relation specific message passing. Using a separate dualcertificate network to provide conservative potency lower bounds, allows them to not solely rely on the point prediction of the GNN. As far as I know, this is a novel application of these existing techniques.
Critical weaknesses
As stated in the abstract: Exact computation requires four responsefunction linear programs and becomes rapidly expensive as parent counts grow, then claiming PotencyGNN makes the problem more scalable. However in the experimental setup, K is set for K ≤ 256 training, and in the results (section 5.5 and appendix H), when K is outside the training range, the prediction accuracy deteriorates significantly. So the speedups are significant but the quality in their predictions drop as well. If i understand correctly the method is performing best (accuracy wise) when the exact LP is still easily solvable.(roughly 10 ms for K=256 from figure 2). The experimental setup is on graphs that aren’t significant in size "We use ADMGs with three to five observed binary variables, at most two observed parents per variable in C, and K ≤ 256 joint response types; ", so if the motivation is scalability I would like to see results on larger graphs.
Questions for authors
Regarding the amortization analysis, the authors state "for PotencyGNN (Table 5), the training cost is amortized after approximately 21,700 subsequent queries when exact labels are already available, or 47,800 queries when label generation is included." Are there practical applications involving graphs of this size, that would require such a large number of queries?
Thank you for the assessment and for stating the main limitation precisely. Results marked "new" were obtained during the discussion and are in the revised PDF.
W1. Your reading of the submitted results is correct, and the revision responds in two ways. The claim is corrected. In the original stress test, models trained with at most 256 response types reach R² of 0.580 and 0.668 at 512 and 1,024 and fall to −9.579 at 262,144 (Section 5.5). The revised abstract states that exact computation takes milliseconds per query on the primary benchmark, that its cost grows with the number of response types, and that PotencyGNN loses accuracy beyond 256 and that retraining improves it. Section 1 states the aim as a lower cost for each additional computation when many are needed, not solving problems too large for exact computation.
Retraining improves accuracy. New: adding cores with 512 to 2,048 response types to training raises held-out R² at 512, 1,024 and 2,048 from 0.795, 0.744, 0.652 to 0.872, 0.832, 0.859, while R² at up to 256 is 0.865 on the original held-out cores (Table 13). On held-out cores like those used for retraining, accuracy at 2,048 is thus close to that within the original training range. On stress-test graphs from other causal structures (overlapping cores left out, predictions averaged over five seeds), R² rises from 0.571, 0.742, −0.184 to 0.769, 0.800, 0.490; the last graphs have a three-parent confounded variable, unlike any training graph. The added cores also differ structurally, the analysis was not prespecified, and it stops at 2,048 (Appendix H). Cost at these sizes: cached exact computation takes 11.7, 21.9 and 28.5 ms per query, 17.4, 32.2 and 43.4 times as long as PotencyGNN (Table 14). Exact computation remains feasible there, so the gain is in cost per query, and the saving grows with response-space size: the estimated 402.2 s for labeling the added instances and one fit, assuming the earlier labels exist, is recovered after 36,441, 18,916 and 14,443 queries.
W2. Prediction is evaluated up to six observed variables: models trained on six-variable cores reach held-out R² of 0.779 on six-variable cores absent from training (Table 1, Appendix B). Beyond six, neither the definition of potency nor PotencyGNN is tied to graph size. The linear programs are defined for any number of variables, and their size is set by the response types of the confounded set, not by the number of variables; under parental closure and the support assumption, a larger graph reduces to its causal core without changing the bounds or potency (Section 5.1, Appendix C; checked on graphs with up to seven variables); and PotencyGNN applies the same maps at every node, so it takes larger graphs without new parameters. Cores with more than six variables are not tested, but two results point the same way: training on six-variable cores gave the 0.779 above, and retraining on five-variable cores with larger response spaces raised R² at 2,048 from 0.652 to 0.859. In both cases PotencyGNN learned the larger setting once it was represented in training, which suggests that larger cores can be learned in the same way once their exact labels are computed (Appendix G). The certificate network accepts at most five observed variables, because its inputs have 33 constraint rows (Appendix F).
Question. Yes, on graphs of this size one application would reach such counts directly: quantifying the sampling uncertainty of potency. Potency is computed from an estimated distribution and does not measure sampling uncertainty (Section 3); a bootstrap interval recomputes it once per resample. Even with 1,000 resamples, one tenth of the 10,000 replicates the paper uses for its model-comparison intervals (Appendix D), an interval needs 1,000 computations per query, so 22 queries exceed the 21,700 break-even and 48 the 47,800 that includes label generation. Screening candidate queries and comparing auxiliary sets multiply the counts further (Section 1). For scale, the empirical evaluation has 19 queries on five real datasets (Section 5.6), and the published-network study enumerates 1,811 candidates on subgraphs of at most five variables (Appendix E). No evaluated study is large enough to recover the training cost (Section 5.4); Table 5 gives the per-query saving for any study size, and at larger response spaces the break-even is 14,443 to 36,441 queries (Table 14).
Strengths. Thank you for noting both. Removing all graph edges at the same parameter count lowers held-out R² from 0.829 to 0.277, and symmetrizing the directed aggregates lowers it to 0.764 (Appendix D). The certificate is separate from the point prediction and, given the assumed graph, the support assumption, known P(V) and full-program feasibility, cannot overstate potency: across 79,200 held-out evaluations, all certificates satisfy the numerical validity criteria, and the 43.4% that give a positive bound have precision 1.0 (Section 5.7).