- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how to evaluate graph construction methods for point clouds. The authors propose a training-free diagnostic that compares graph construction methods using the Wasserstein-1 distance between discrete Ricci curvature distributions of labeled populations. They compute this distance within bins defined by point count and composition and use complete graphs as a control.
The authors evaluate the diagnostic on QM9 and JetSet and compare model performance with and without curvature features. On QM9, Ollivier-Ricci (ORC) separation between hydrogen and heavy atoms correlates strongly with bond-recovery quality, whereas Forman-Ricci (FRC) separation does not show this correlation. On JetSet, radius graphs retain substantial ORC separation after controlling for point count and composition. In the QM9 prediction experiments, the construction selected by the diagnostic also yields the largest gains from curvature features. On JetSet, separation does not show a consistent association with gains from curvature features in the GNN and transformer.
Critical strengths
-
The paper proposes a diagnostic for graph construction that requires no downstream model training and tests it against known physical structure. On QM9, ORC separation between hydrogen and heavy atoms uses only element labels and the constructed graph, without accessing the true bonds, yet correlates strongly with bond-recovery F1 across 19 constructions (Spearman ρ = 0.88, or 0.87 excluding the complete graph; Table 9). This independent reference provides initial support for a practical tool to assess graph constructions in small scientific point clouds before model training.
-
By comparing different constructions of the same point clouds, the paper shows how edge-building rules affect the class differences detected by curvature. On JetSet, using ORC distributions pooled over all nodes and conditioning on total track count and secondary-vertex track count, the default radius graph retains a separation ratio of 0.68. The default kNN and unique-3 constructions retain 0.33 and 0.34, respectively, close to the complete graph's 0.38 (Figure 2a). This comparison offers a concrete interpretation of construction behavior: some curvature signals largely follow point count, while others continue to distinguish labeled populations after conditioning. It provides useful empirical insight into which data characteristics different construction rules emphasize.
Critical weaknesses
-
The scope of applicability needs clearer support.
The paper proposes using curvature separation to choose graph constructions and their parameters without training a model, but tests the diagnostic only on QM9 and JetSet. Both applications are useful, but it is unclear which properties of a new dataset would make the score a reliable guide. When should greater separation indicate better preservation of the structure relevant to the task? The paper discusses some domain-specific limitations, but does not give clear guidance on when this criterion should be used for other small scientific point clouds.
-
The benefit of conditioning for construction choice is not sufficiently tested.
QM9 provides a reference bond graph, but the main validation uses unconditioned separation (as in Table 9). Both atom populations come from the same molecules, and conditioning on size changes the per-molecule-mean scores by less than 3%, except near the estimation bias floor (Sec 5.1). The correlation supports agreement with bonding structure, but this setting does little to test the benefit of conditioning under substantial size confounding.
JetSet has size confounding but no reference graph. The radius graph gives the best mean GNN baseline performance among the default sparse constructions on b-versus-light, which is encouraging. However, jointly conditioned ORC separation correlates only weakly with GNN baseline performance (Spearman ρ = 0.23, pooling within-task ranks over nine sparse defaults and three tasks; Appendix G.8). Model performance is only an indirect check of structural preservation. The complete-graph control shows that conditioning suppresses a pure size signal, but does not test whether the conditioned ranking agrees with independently known structure when size and structural differences coexist.
-
Node–edge misalignment in the GNN implementation calls the predictive evidence into question.
In
utils/diminishing_returns_deep_models.py,build_padded_tensorssorts rows only by graph ID and fills the feature tensor by row position (lines 23–40), without ensuring that within-graph node order matches the original node indices. Meanwhile,build_padded_edgesretains the originalsrc/dstindices (lines 79–94), whichSimpleGNN.forwarduses directly to retrieve node features (lines 160–167). When sorting changes the within-graph node order, message passing can therefore connect features belonging to unintended node pairs. Until node–edge correspondence in the reported experiments is established, the GNN comparisons across constructions and the gains from curvature features remain difficult to interpret as reliable evidence about the intended graph constructions.
Questions for authors
-
What guidance do the current results provide for deciding when curvature separation is a useful criterion for choosing a graph construction?
-
Beyond the observed label separation, what evidence supports interpreting higher conditioned ORC separation as better preservation of task-relevant structure on JetSet?
4 comments
Thank you for the response. I maintain my score for the following reasons: (1) the bug fix changes the GNN results substantially (e.g., the radius-graph baseline falls from best to second-lowest), which the response does not discuss; (2) on JetSet, the pooled and per-flavour results disagree on whether conditioning helps; (3) singling out the radius family relies on a cross-family comparison, which the paper itself calls less reliable.
Follow-up to Reviewer U2Jp (1/2)
Thank you for your follow-up comments. We address the three points as follows.
the bug fix changes the GNN results substantially (e.g., the radius-graph baseline falls from best to second-lowest), which the response does not discuss
You are correct that our first response did not discuss how the bug fix changed the GNN results. The corrected runs were still in progress at the time, and we did not want to describe the change from partial results. All reruns are now complete, and every GNN entry that the bug could affect comes from the corrected code.
Three GNN results on jets changed:
- The GNN baseline no longer favors the radius graph. On b vs light, the corrected GNN's AUROC without curvature on the radius graph is 0.7192. The value 0.7192 is the second lowest among the sparse default constructions, which range from 0.7140 to 0.7247. We withdrew the claim that the GNN baseline on jets follows the diagnostic. The paper now states that the GNN's accuracy without curvature is nearly the same on every construction.
- The gains from curvature are smaller. On b vs light, curvature improves the GNN's AUROC by only +0.0007 to +0.0075, depending on the construction. On vs light, curvature lowers the GNN's AUROC significantly on eight of the ten sparse default constructions that enforce a minimum degree.
- The degree control experiment is complete. In the degree control experiment, we gave the GNN each node's degree in place of the node's curvature. Node degree alone recovers half or more of the curvature gain on all eight constructions that we tested. On the radius graph, the two gains are almost equal: +0.0101 from degree and +0.0098 from curvature. The degree control result confirms what the initial submission reported, namely that node degree mostly matches the curvature gain. We do not claim that curvature gives the GNN information beyond node degree on jets.
The diagnostic itself did not change. The separation scores and the transformer results do not involve the GNN, so the separation scores and the transformer results are identical to the results in the original submission. The permutation nulls and the vertex recovery test, which we added in the first revision, also do not involve the GNN, so the bug fix did not affect them either.
On QM9, where a reference graph exists, the bug fix brought the GNN into agreement with the diagnostic. In the original submission, the GNN's error without curvature on the 1.8 \AA\ graph was 0.0219 Ha, which was higher than the GNN's error on the fully connected graph (0.0159 Ha). With the corrected code, the GNN's error on the 1.8 Å graph is 0.0082 Ha, which is the lowest error among all 22 constructions.
The model results therefore support the diagnostic on QM9 but not on jets.
- On QM9, the GNN is most accurate, and the transformer gains most from curvature, on the graph that the diagnostic selects.
- On jets, the models gain about as much from curvature on the fully connected graph, which has no structure, as on the radius graph that the diagnostic selects. A model's gain therefore cannot tell us which construction is better.
For this reason, we compare constructions on jets by conditioned separation and not by model gain.
Follow-up to Reviewer U2Jp (2/2)
This comment continues our reply:
on JetSet, the pooled and per-flavour results disagree on whether conditioning helps
You are correct that the pooled and per-flavor results disagree. The table below gives the Spearman correlation, across the 36 constructions, between a construction's separation and the construction's same-vertex F1. The same-vertex F1 measures how well the edges of a construction match the pairs of tracks that come from the same decay vertex (Appendix G.4).
| Jets | With conditioning | Without conditioning |
|---|---|---|
| Three flavor pairs pooled | 0.60 | 0.66 |
| b vs light | 0.58 | 0.51 |
| c vs light | 0.44 | 0.27 |
| vs light | 0.72 | 0.66 |
Pooled, conditioning lowers the correlation slightly (0.66 to 0.60). Per flavor, conditioning raises the correlation. With only 36 constructions, neither change is large enough to be meaningful, so we do not claim that conditioning improves the correlation with same-vertex F1.
The table shows that constructions with a higher separation also connect more of the track pairs that come from the same decay vertex, with or without conditioning on jet size. Every correlation in the table is positive, and the pooled correlation with conditioning (0.60) has a 95% bootstrap interval of [0.31, 0.80]. When we replace curvature with node degree or triangle count and compute the separation in the same way, the correlations with same-vertex F1 are negative (−0.57 and −0.59).
We condition for a different reason. Jet flavors differ in the number of tracks per jet, and curvature depends on the number of tracks. Without conditioning, two flavors can therefore look different even on a graph with no structure. Conditioning compares only jets of the same size. Two tests show that conditioning removes the size effect.
On jets, the fully connected graph has no structure, yet without conditioning the fully connected graph separates the flavors. With conditioning, 0.38 of the fully connected graph's separation remains, which is the same fraction that remains when the flavor labels are shuffled (0.38).
On QM9, the bonds are known. When we induce a size difference between the two populations, the correlation between separation and bond recovery falls from 0.88 to 0.81. Conditioning restores the correlation to 0.88.
singling out the radius family relies on a cross-family comparison, which the paper itself calls less reliable
Our earlier wording made the selection of the radius family unclear, and we thank you for identifying the problem.
To be precise about terms: a family is a construction rule, such as kNN or the radius graph, and a construction is a family with one parameter setting, such as the radius graph at = 0.3.
We do not select the radius family by comparing raw scores across families. We select the radius family because the radius graphs are the only constructions that stay well above their own shuffled-label baseline.
For each construction we compute two numbers: the fraction of the construction's separation that survives conditioning, and the same fraction when the flavor labels are shuffled. The paper calls the difference the excess (step 5 of Appendix D.1, and the "excess" column of Table 2a). Three examples:
| Construction | Survives conditioning | With shuffled labels | Excess |
|---|---|---|---|
| Radius graph ( = 0.3) | 0.68 | 0.38 | 0.30 |
| Laman graph | 0.39 | 0.38 | 0.01 |
| Fully connected graph | 0.38 | 0.38 | 0.00 |
Across all 36 constructions, the four radius graphs have an excess of 0.19 to 0.38. The largest excess for any other construction is 0.03 (Section 5.2, Table 2a).
The excess can be compared across families because both numbers are fractions of the construction's own separation. The procedure in Appendix D.1 now states this explicitly.
The within-family restriction that you mention applies to a different step. Step 6 of the procedure compares conditioned separations within one family in order to choose a parameter. Step 5, which computes the excess, tests each construction only against the construction's own shuffled labels.
The vertex recovery test that we added in response to your review ("does not test whether the conditioned ranking agrees with independently known structure") points to the same family. The construction with the highest same-vertex F1 is the radius graph at = 0.2, and the same construction has the largest conditioned separation among all 36 constructions (Section 5.1, Figure 2b, Appendix G.4).
We do not compare the remaining families with one another. We reworded the limitation in Section 6 to state that the raw scores, and not the shuffled-label test, are less comparable across families.
Thank you again for the careful reading. These points made the paper more precise about what it does and does not claim.
Response to Reviewer U2Jp: scope, conditioning tests, and the node-edge fix
Weakness 1 and Question 1. We stated when the score is a reliable guide (Section 3.2). The labeled populations must differ in local connectivity in the true structure, each bin must hold enough point clouds, and parameters must be compared within one family of constructions. We added a six-step procedure for a new dataset (Appendix D.1). The procedure tests the assumption itself, because the fully connected graph must not exceed its permutation null and at least one construction must.
Weakness 2, QM9. We tested the size control where a size difference and a reference graph coexist. We induced the size difference on QM9 by keeping hydrogens mostly from large molecules and heavy atoms mostly from small ones, and we left the bonds unchanged. Without size control, the fully connected graph separates the two populations and the Spearman ρ with bond recovery falls from 0.88 to 0.81. Conditioning on molecule size returns both to their control values. Conditioning on a variable that does not carry the size difference does not restore the agreement (0.83). These results are in Section 5.2 and Appendix G.3.
Weakness 2, JetSet, and Question 2. We added independent structural evidence on jets. JetSet records the decay vertex of each track, so we scored the edges of each construction against same-vertex track pairs, which the diagnostic never uses. Conditioned ORC separation tracks this recovery across the 36 constructions (ρ = 0.60 overall, and 0.58, 0.44, and 0.72 within b, c, and τ jets). The unconditioned separation, which uses no vertex information, agrees as well (ρ = 0.66). On jets the benefit of conditioning shows in the permutation null, where the fully connected graph and most constructions no longer exceed chance once size is fixed (Section 5.2). These results are in Section 5.1, Figure 2b, and Appendix G.4. We agree that model performance is only an indirect check. With the corrected GNN, the correlation between conditioned ORC separation and the GNN baseline is 0.34 across the nine sparse default constructions and three tasks, and we report it as an observation (Appendix H.8).
Weakness 3. We confirmed the node-edge misalignment. The node table was sorted with an unstable sort while the edge index kept the original node positions. We fixed the sort with a stable one and added an alignment check that runs before training. The diagnostic, the QM9 control, and the transformer results are not affected. The corrected GNN runs are ongoing. So far we replaced the GNN results for the 12 default jet constructions, for 18 sweep constructions on b vs light, and for 15 of the 22 QM9 constructions (14 for the dipole moment), and the remaining cells are marked as pending. Among the finished QM9 runs, the corrected GNN is most accurate on the graph that the diagnostic selects (0.0082 Ha). We withheld the claims that rested on the GNN degree control until its rerun finishes, and we will update the paper when the runs complete.