- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper contributes:
- A cross-domain graph generation benchmark, spanning five graph distributions in two domains, molecular and synthetic graphs. Metrics and splits are taken from the original benchmarks, the contribution is to project the samples onto a shared domain.
- Baseline results in the proposed benchmark achieved by training a large range of existing graph generation models from scratch on the cross-domain dataset
- A model that performs well on the cross-domain benchmark empirically
- An analysis showing that their model can extrapolate from domains
Critical strengths
- The benchmark measures something interesting: How well do graph generation models work on more general tasks?
- The model works on the new benchmark
- Very cool result regarding extrapolation
Critical weaknesses
The comparison against existing baselines in Table 1 is weak. The baseline uplift recipe e.g. does not mask the baseline to the respective domain in intermediate stages, and the compute cutoff is conveniently chosen so that their model just finishes while e.g. SPECTRE is terminated prematurely (implied from Table 6). The paper's main claims are not about a comparison with prior models but it should still be noted that the comparison may not be fair.
Questions for authors
- How many epochs did each baseline training actually complete before the 10-day cutoff?
- The vocabulary masking mechanism is applied to both nodes and edges, right? Applying it to nodes should not improve structural if I understand correctly, so the edge-masking should be load-bearing. Would you mind publishing results with only node masking and another with only edge masking?
- I am curious how this scales in the number of domains covered. How does performance look e.g. when only training on 3 or 4 instead of 5 domains? Or if you add a 6th domain like moses or guacamol? What happens if you only consider molecule or only consider graph domains?
- If you train your implementation of prior models as well as your own model on only one domain, what are the results? It would be good to compare against the results reported by the original papers?
4 comments
Rebuttal Part 2/2
Q3 continued.
We also include the MMD ratio results from training DiG$^2$ on a combined LGGM+DiG$^2$ benchmarking protocol with 15 total datasets, as compared to LGGM. As shown below, DiG$^2$ exceeds LGGM performance on 14 of the 15 tested datasets and is within 0.04 of LGGM on the Econ dataset, demonstrating the capacity of DiG$^2$ to scale on additional datasets.
| Model | FB | ASN | Web | Road | Power | Bio | Econ | RT | Col | Eco | Citation | Planar | SBM | Tree | Average MMD ratios (↓) | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| DiG^2 | 1.62 | 2.08 | 1.50 | 2.18 | 3.70 | 1.48 | 1.39 | 1.15 | 2.37 | 3.31 | 2.86 | 2.07 | 92.5%/1.20 | 85%/2.30 | 97.5%/1.92 | 2.08 |
| LGGM | 2.15 | 5.01 | 2.27 | 3.05 | 4.93 | 3.78 | 1.63 | 1.11 | 21.94 | 13.58 | 4.52 | 2.81 | 0.0%/137 | 5.0%/11.3 | 0.0%/12.94 | 15.23 |
Q4: single-domain results? compare against original papers?
Q4. We include single-domain results for DiG$^2$, DeFoG, and DiGress from the per-node budget pipeline and the results for each respective model, as derived from their reported paper results and indicated with a '(paper)' label. The 'DiG$^2$ (paper)' results are from the multi-domain setting. As demonstrated below, the single-domain setting for this study's pipeline enables models to converge towards performance comparable to originally reported results. Notably, the DiG$^2$ pipeline even improves the reported accuracy values in the DiGress paper, making DiGress competitive with a single-domain DiG$^2$ and DeFoG.
| Model | Planar | SBM | Tree | QM9 | Zinc |
|---|---|---|---|---|---|
| DiG^2 | 97.5/1.8 | 80/2.2 | 100/1.9 | 99.45/0.27 | 95.80/2.5 |
| DiG^2(paper) | 100/2.3 | 85/1.8 | 97.5/1.56 | 99.92/0.29 | 95.46/2.7 |
| DeFoG | 95.0/1.6 | 85/3.1 | 95/1.8 | 99.23/0.27 | 95.35/2.2 |
| DeFoG(paper) | 99.5/1.6 | 90/4.9 | 96.5/1.6 | 98.9/0.26 | 96.65/2.1 |
| DiGress | 85/1.2 | 80/2.3 | 95/2.0 | 99.4/0.3 | 94.16/2.5 |
| DiGress(paper) | 77.5/5.1 | 60.0/1.7 | 90.0/1.6 | 99.0/-- | --/-- |
Super cool results on the scaling regarding the scaling in the number of domains: It seems like sometimes the results benefit from other datasets? Add some more ablations and nice graphs regarding this to the paper and I will raise to 3! Very cool! Would be good to understand whether one dataset actually helps performance on the other!
Response to Reviewer v4nL – new DiG Figure 3
Thank you for the encouraging response, we also believe that the results indicate the cool capabilities of DiG^2!
We conduct an investigation of the internal structural encoding for DiG$^2$, as shown in the updated PDF manuscript's Figure 3. Within the figure, we find that once training is scaled to include the 12 LGGM domains, the separation between internal Planar and SBM representations improves -times, increasing the gap from 0.03 to 0.32. Additionally, to determine if scaling provides a downstream benefit for Planar and SBM, we then collapsed the fused prototype embedding during sampling towards a per-domain mean, inducing a scenario where DiG$^2$ must rely solely on domain labels for internal model separation. We find this reduces SBM accuracy by points, from 85% to 45%, even after training with the LGGM datasets. Therefore, indicating that DiG$^2$ sampling benefits from it's structural encoding schema for separating internal per-domain signals and maintaining high-quality performance.
We hope this addresses your concerns and appreciate the excited feedback.
Rebuttal Part 1/2
Thank you for your review, we appreciate your evaluation on our extrapolation results and are intrigued by your questions on scaling. We include our rebuttal below and hope it addresses all of your weaknesses and questions. We summarized your questions and weaknesses to mark our separate answers, please inform us if any specific weaknesses or questions require further addressing.
W1A. We note that DiG$^2$'s benchmarking pipeline does constrain node- and edge-vocabularies during all training and sampling steps for each baseline. We have updated the manuscript to say this directly.
W1B, Q1. We note that evaluating models at the 10-day window was determined to ensure consistency in evaluation standards. 10-days was also selected as a relative cut-off, given that 5 other models (DeFoG, DiGress, LGGM, GraphBSI, and GraphRNN) reached the 10,000 epoch cut-off for the paper's Table 1 benchmarking. However, the benchmarking study did attempt to train slower models past the 10-day window, as shown below in the table. We also note that SPECTRE reached epoch 1770 by day 10, with signs of mode-collapse where generated samples consist entirely of hub-structures, instead of successfully generating reference data.
To determine if additional training for SPECTRE can recover signal for multi-domain graph generation, we have trained SPECTRE for an additional five-days and evaluated the model at epoch 2,699. As shown below, SPECTRE with 15 total-days of training has not converged towards a signal for generating graphs across 5 domains and still generates samples with hub-structure.
Q2: Thank you for the suggestion, this rebuttal ran a targeted ablation on node- and edge-vocabulary masking for DiG$^2$ across Planar, SBM, and Tree. As shown below, ablating node-masking for Planar and Tree retains accuracy for the two datasets within the estimated ~8% noise limits on performance. The results also indicate your intuition is correct. The removal of edge-masking is load-bearing on SBM and Planar, reducing performance by 12.5% for SBM and 35.0% for Planar, both of which fall past noise-limits on performance from the paper results.
Q3. We include updated results demonstrating that DiG$^2$ maintains strong performance across 4 and 6 domains relative to the paper's results. We also include results from molecule-only and synthetic-only training to show that DiG$^2$ is not dependent on comparisons between synthetic and molecular datasets to achieve strong results.