- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
In this work, the authors attempt to improve the performance of flow-based graph generation methods where the flow acts on a latent space representation of the graph rather than an explicit representation (such as the adjacency matrix). They hypothesize that the embedding dimension have a varying importance and effect in reconstruction error. To this end, they propose applying a linear reweighting for the errors when learning the denoising model. The proposed weights are given by variance-weighting the Jacobian of the decoder with respect to the latent dimensions. To support this hypothesis they consider two experiments; (1) comparing the reconstruction of elements from the tree and planar graph datasets using the top 10% of the embedding dimensions with reconstruction using the full embedding and the bottom 90% of the embedding dimensions, and (2) evaluating the performance of two different latent-space flow-based generation models (LG-Flow and VGAE-Flow) on a larger version of the tree, planar, and stochastic block model data sets as well as two molecular data sets (Guacamol and MOSES).
Critical strengths
- Overall the paper is accessibly written and addresses an interesting and important question.
Critical weaknesses
-
The evaluation of the impact of the importance measure is extermely limited, exploring only a single cutoff value between
useful'' andleast useful'' and a single embedding model. In light of Figure 3, the evaluation method chosen would seem to be the least challenging case of the four models described. -
The latent dimension (128) is large relative to the size of the size of the graphs in the experiments. Indeed, trees on 128 vertices can be exactly encoded exatly (with minimal packing) in 63 characters. With a little cleverness, it is certainly possible to encode all the planar graphs in 24 short ints by explicitly enumerating the edges and packing 64 edges into a single short int. Clearly, this is not what the autoencoder is doing, but it does make it signficantly less surprsing that there is significant noise in the embedding.
-
The AI use statement does not align with this paper. The authors claim to have used generative AI to formulate mathematical claims, but not provide proofs or assist in proof generation. However, there are no mathematical claims or proofs in this article. Further, they list a variety of potential generative AI use cases as ``not relevant to the paper'', including editting, formating references, etc., while the paper clearly contains references and is organized as a research paper.
Questions for authors
-
How do the results in Figure 4 change when the auto-encoder is changed? Or a different dataset is used? Or a different cut-off?
-
It appears from Figure 3 that the importance scores are fairly close in general with a precipitous drop-off. Providing the normalized scores explicitly would give more insight into the behavior.
-
How do the variance and importance measures perform independently as reweighting schemes?
-
How is the accuracy of the SBM model reconstruction evaluated? Since SBMs are edge-independent models with non-zero edge probabilities, any generated graph is a valid SBM (albeit rare).
-
What does the coloring in Figures 9, 10, and 11 indicate?
Minor Notes:
-
I think the expectation in equation (6) should depend on the encoder not the decoder.
-
In 6.1, evaluation metrics there are numerous typographical issues.
3 comments
Response of Raised Questions [Part 2/3]
The latent dimension (128) is large relative to the size of the size of the graphs in the experiments....With a little cleverness, it is certainly possible to encode all the planar graphs in 24 short ints by explicitly enumerating the edges and packing 64 edges into a single short int.
While the capacity argument is correct, we don't think it's in tension with our contribution. This is merely a artificat of 2/5 datasets (Planar & Tree) used in our paper. However, it doesn't apply to SBM and both molecular graphs. This is further borne out by the fact that LIR consistently improves performance on those 3 other datasets below. We show the mean increase across both latent models across those 3 datasets. This proves that the benefit of LIR is not only restricted to the characteristics of the Planar/Tree datasets.
| Dataset | Acc/Valid% |
|---|---|
| SBM | +18.5% |
| MOSES | +2.5% |
| Guacamol | +4.25% |
In conclusion, we agree that Tree and Planar are simpler datsets. However, they are merely used as a guidepost to analyze the capabilities of graph generative models. Furthermore, even though the other datasets are much more complex, LIR is still helpful, showing that this is still a problem on more complicated graphs.
The AI use statement does not align with this paper. The authors claim to have used generative AI to formulate mathematical claims, but not provide proofs or assist in proof generation. However, there are no mathematical claims or proofs in this article.
This is an error in communication on our part. By mathematical claims, we simply mean that we used AI for generating the latex code for our equation. We have amended the statement to reflect this.
Further, they list a variety of potential generative AI use cases as ``not relevant to the paper'', including editting, formating references, etc., while the paper clearly contains references and is organized as a research paper.
It's unclear to us what you are referecing when you quote "not relevant to the paper". The AI use statement included in our work is based on the official version included in the ICLR 2027 template. We modified it for our specific use case to denote which areas we did and did not use AI. We used AI solely for generating equations in Latex. We did not use it for formatting references or editing non-mathematical equations. Our current statement reflects this fact and is consistent with our usage and final paper.
Note that we obviously agree that this is a potential use of generative AI as a whole (though not for graph generation). But the AI use statement is merely to communicate where we used AI in our paper, not to reflect what our work can be used for. This is per the conference requirements (see here).
It appears from Figure 3 that the importance scores are fairly close in general with a precipitous drop-off. Providing the normalized scores explicitly would give more insight into the behavior.
The scores in Figure 3 are already the normalized importances from Eq. 7 (, summing to 1 across ). Rather Figure 3 shows the cumulative sum of importance scores in the top k% of latents. It is not the raw unnormalized scores.
Specifically, we order all importance scores from highest to lowest, where the top % of scores are the top % importances scores (e.g., =10% are the top 10% of importance scores). For each value of we sum the importance scores of those latent dimensions and plot them in Figure 3. E.g., for VGAE on Planar the top 12 of 128 dims carry 98.8% of the mass (see Figure 3a)
How do the variance and importance measures perform independently as reweighting schemes?
Thank you for the suggestion. We included these experiments on the Tree dataset for VGAE+Flow (we were unable to run more experiments due to time). All results are over 5 seeds. They can be found in the table below.
We can clearly see that LIR does the best. While variance-only and Jacobian-only can lead to a modest boost in performance, their combination leads to the largest performance increase by far, thus validating our design.
We have included these results at the end of Section 6.2 (see Ablation Study).
Tree (VGAE, steps=1000, 5-fold × 1024 samples/fold)
| Condition | Acc | Ratio |
|---|---|---|
| Baseline (no reweighting) | 17.3% ± 0.8 | 10.62 ± 0.17 |
| variance-only reweighting | 17.9% ± 1.3 | 10.64 ± 0.48 |
| Jacobian-only reweighting | 19.5% ± 1.1 | 9.92 ± 0.27 |
| LIR | 23.2% ± 2.5 | 8.75 ± 0.67 |
Response of Raised Questions [Part 3/3]
How is the accuracy of the SBM model reconstruction evaluated? Since SBMs are edge-independent models with non-zero edge probabilities, any generated graph is a valid SBM (albeit rare).
The SBM "accuracy" metric tests if the generated graph falls in the general distribution used to generate the SBM graphs. We follow the common procedure introduced by SPRECTRE [1] (you can see the original description in Appendix C in their paper). Specifically, the train/valid/test graphs are generated using a predefined intra/edge probabilities. The number of communities and nodes are drawn randomly from a predefined ranges. During inference, a generative models generated N graphs. For each generated graph, we estimate both the number of communities and the intra-/inter-block edge probabilities using Bayesian inference . A Wald test is then used to determine the probability that the probabilities of the generated graph match the original parameters. If the p-value >= 0.9, we consider it a valid SBM graph. So although any edge-independent graph has non-zero probability under some SBM, it is only credited if it recovers both the reference block-size regime and the reference edge probabilities to high statistical confidence.
We included a reference to this explanation in [1] on lines 436-437.
[1] "SPECTRE: Spectral Conditioning Helps to Overcome the Expressivity Limits of One-shot Graph Generators". ICML 2022.
What does the coloring in Figures 9, 10, and 11 indicate?
We apologize for the confusion. The color is the the fiedler value connectivity (i.e., vector). This is used to extenuate graph substructures. We would like to note that this is the default coloring used for representing generated graphs in literature. For example, it can be found here in the code for SPECTRE [1] and here for DeFoG. We have included a note of this in Appendix D.1.
I think the expectation in equation (6) should depend on the encoder not the decoder.
Thank you for bringing this to our attention. You are correct. We have updated the equation.
In 6.1, evaluation metrics there are numerous typographical issues.
We apologies for any issues. We want to ensure that our paper is as readable as possible. However, we are unsure of which typographical issues you are referring to. We'd be happy to fix those you have noticed.
Response of Raised Questions [Part 1/3]
We appreciate that you took the time to review our paper and give us constructive feedback. We respond to each point below:
In addition to the original results for VGAE on Planar, we further include LG-VAE on Planar and VGAE on Tree (due to time constraints we were unable to run LG-VAE on Tree). The results can be found in the Table below.
In conclusion, we can observe that for VGAE on Tree sample accuracy falls to 0. This is similar to VGAE on Planar (Figure 4). This is expected, since the importance for VGAE on tree is highly concentrated in few dimensions. For LG-VAE on Planar, the results is more modest but still exists. This is because the importance concentration is much flatter. As such, this supports our own analysis, as this is exactly where we would expect the effect to be smaller.
These results are included in Appendix C.6.
The cutoff is only a diagnostic, not part of our method. LIR itself reweights all latent dimensions continuously (Eq. 7–8), it never discards any. The cutoff experiment (Fig. 4) exists only to validate that our importance score ranks dimensions correctly. Furthermore, the cutoff k is not hand-picked, it's derived the same way for every dataset/autoencoder we test. Specifically, k is chosen for the top k dims whose importance weight exceeds the mean. This gives different k per setting (e.g., 12/128 for VGAE+Planar, 22/128 for VGAE+Tree, 10/24 for LG-Flow+Planar), adapting to how concentrated that latent's importance actually is.
Nonetheless, we agree with your suggestion and include the results for VGAE on Planar and Tree (we omit LG-Flow due to time constraints). For different values of k, we perform the same experiment for the top-k dimensions ("Active") and the rest of the dimensions ("Inactive").
The results can be found below. We can make two conclusions: (a) Active performance saturates at the original k we chose (default). This validates our chose of k used in our experiments. (b) The active performance is nearly always better than the inactive performance. The one exception is 6 dimensions on Tree where it does slightly worse on Sample Acc%. This is expected as 6 dimensions for VGAE on Tree contains <50% of the cumulative latent importance.
Planar (VGAE)
Tree (VGAE)