acceptodds
Under review as a conference paper at ICLR 2027

Not All Latents Are Equal: Importance-Weighted Latent Graph Generation

Abstract

Graph generation has shown tremendous effectiveness in modeling complex scientific tasks ranging from molecule generation to optimized circuit design. Most recent advancements have developed models that learn to denoise noisy graphs using either discrete diffusion or flow matching. However, discrete graph diffusion scales quadratically with the number of nodes, making it difficult to use such models on larger graphs. On the other hand, latent graph generation instead seeks to denoise noisy graph latents, which scale better to more nodes due to their lower dimension. Latent generation works by first learning a set of latents via an autoencoder trained on graph reconstruction. The learned latents are then denoised in the continuous space, where the final denoised latent is then mapped back to the discrete graph. However, in practice, latent graph generation models lag behind their discrete counterparts in performance. We study this problem, finding that it is due to the fact that the importance of specific latent units (i.e. dimensions) in a graph latent vary tremendously in their impact on graph reconstruction. That is, the vast majority of impact is contained in only a smaller number of latent units. However, existing work places equal emphasis on reconstructing the entire latent in the loss function, regardless of how much influence each unit has on reconstructing the graph. We propose to tackle this by simply weighting the latent units differently as per their importance to final graph reconstruction. The importance is calculated as the relative influence of a single latent unit on graph reconstruction, which is represented via the Jacobian of the decoder relative to the latent. Our method is model-agnostic, and we show that it can both boost performance and lead to faster training convergence when applied to two different autoencoders on various benchmark datasets.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
Graphs, diffusion, flow matching

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_jCTROpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

In this work, the authors attempt to improve the performance of flow-based graph generation methods where the flow acts on a latent space representation of the graph rather than an explicit representation (such as the adjacency matrix). They hypothesize that the embedding dimension have a varying importance and effect in reconstruction error. To this end, they propose applying a linear reweighting for the errors when learning the denoising model. The proposed weights are given by variance-weighting the Jacobian of the decoder with respect to the latent dimensions. To support this hypothesis they consider two experiments; (1) comparing the reconstruction of elements from the tree and planar graph datasets using the top 10% of the embedding dimensions with reconstruction using the full embedding and the bottom 90% of the embedding dimensions, and (2) evaluating the performance of two different latent-space flow-based generation models (LG-Flow and VGAE-Flow) on a larger version of the tree, planar, and stochastic block model data sets as well as two molecular data sets (Guacamol and MOSES).

Critical strengths
  1. Overall the paper is accessibly written and addresses an interesting and important question.
Critical weaknesses
  1. The evaluation of the impact of the importance measure is extermely limited, exploring only a single cutoff value between useful'' and least useful'' and a single embedding model. In light of Figure 3, the evaluation method chosen would seem to be the least challenging case of the four models described.

  2. The latent dimension (128) is large relative to the size of the size of the graphs in the experiments. Indeed, trees on 128 vertices can be exactly encoded exatly (with minimal packing) in 63 characters. With a little cleverness, it is certainly possible to encode all the planar graphs in 24 short ints by explicitly enumerating the edges and packing 64 edges into a single short int. Clearly, this is not what the autoencoder is doing, but it does make it signficantly less surprsing that there is significant noise in the embedding.

  3. The AI use statement does not align with this paper. The authors claim to have used generative AI to formulate mathematical claims, but not provide proofs or assist in proof generation. However, there are no mathematical claims or proofs in this article. Further, they list a variety of potential generative AI use cases as ``not relevant to the paper'', including editting, formating references, etc., while the paper clearly contains references and is organized as a research paper.

Questions for authors
  1. How do the results in Figure 4 change when the auto-encoder is changed? Or a different dataset is used? Or a different cut-off?

  2. It appears from Figure 3 that the importance scores are fairly close in general with a precipitous drop-off. Providing the normalized scores explicitly would give more insight into the behavior.

  3. How do the variance and importance measures perform independently as reweighting schemes?

  4. How is the accuracy of the SBM model reconstruction evaluated? Since SBMs are edge-independent models with non-zero edge probabilities, any generated graph is a valid SBM (albeit rare).

  5. What does the coloring in Figures 9, 10, and 11 indicate?

Minor Notes:

  1. I think the expectation in equation (6) should depend on the encoder not the decoder.

  2. In 6.1, evaluation metrics there are numerous typographical issues.

3 comments
AuthorsOpenReview ↗

Response of Raised Questions [Part 1/3]

We appreciate that you took the time to review our paper and give us constructive feedback. We respond to each point below:

How do the results in Figure 4 change when the auto-encoder is changed? Or a different dataset is used?

In addition to the original results for VGAE on Planar, we further include LG-VAE on Planar and VGAE on Tree (due to time constraints we were unable to run LG-VAE on Tree). The results can be found in the Table below.

In conclusion, we can observe that for VGAE on Tree sample accuracy falls to 0. This is similar to VGAE on Planar (Figure 4). This is expected, since the importance for VGAE on tree is highly concentrated in few dimensions. For LG-VAE on Planar, the results is more modest but still exists. This is because the importance concentration is much flatter. As such, this supports our own analysis, as this is exactly where we would expect the effect to be smaller.

These results are included in Appendix C.6.

DatasetAutoencoderdzd_zConditionSample AccEdge Acc
TreeVGAE128All dims (ref.)23.8%99.14%
TreeVGAE128Useful (top 22)30.8%99.85%
TreeVGAE128Least useful (bottom 106)0%93.58%
DatasetAutoencoderdzd_zConditionSample AccEdge Acc
PlanarLG-VAE24All dims (ref.)90.6%99.99%
PlanarLG-VAE24Useful (top 12)0%94.8%
PlanarLG-VAE24Least useful (bottom 12)0%84.94%

The evaluation of the impact of the importance measure is extremely limited, exploring only a single cutoff value between useful'' and least useful'' and a single embedding model. In light of Figure 3, the evaluation method chosen would seem to be the least challenging case of the four models described.

The cutoff is only a diagnostic, not part of our method. LIR itself reweights all latent dimensions continuously (Eq. 7–8), it never discards any. The cutoff experiment (Fig. 4) exists only to validate that our importance score ranks dimensions correctly. Furthermore, the cutoff kk is not hand-picked, it's derived the same way for every dataset/autoencoder we test. Specifically, kk is chosen for the top kk dims whose importance weight exceeds the mean. This gives different kk per setting (e.g., 12/128 for VGAE+Planar, 22/128 for VGAE+Tree, 10/24 for LG-Flow+Planar), adapting to how concentrated that latent's importance actually is.

Nonetheless, we agree with your suggestion and include the results for VGAE on Planar and Tree (we omit LG-Flow due to time constraints). For different values of kk, we perform the same experiment for the top-k dimensions ("Active") and the rest of the dimensions ("Inactive").

The results can be found below. We can make two conclusions: (a) Active performance saturates at the original kk we chose (default). This validates our chose of kk used in our experiments. (b) The active performance is nearly always better than the inactive performance. The one exception is 6 dimensions on Tree where it does slightly worse on Sample Acc%. This is expected as 6 dimensions for VGAE on Tree contains <50% of the cumulative latent importance.

Planar (VGAE)

k (active dims)Active Sample Acc%Active Edge Acc%Inactive Sample Acc%Inactive Edge Acc%
632%99.9%0%99%
12 (default)98%100%0%50%
2695%100%0%50%
6430%99.2%0%50%

Tree (VGAE)

k (active dims)Active Sample Acc%Active Edge Acc%Inactive Sample Acc%Inactive Edge Acc%
611%99%26%99%
22 (default)33%99.7%0%91%
2632%99%0%50%
6430%99%0%50%
AuthorsOpenReview ↗

Response of Raised Questions [Part 2/3]

The latent dimension (128) is large relative to the size of the size of the graphs in the experiments....With a little cleverness, it is certainly possible to encode all the planar graphs in 24 short ints by explicitly enumerating the edges and packing 64 edges into a single short int.

While the capacity argument is correct, we don't think it's in tension with our contribution. This is merely a artificat of 2/5 datasets (Planar & Tree) used in our paper. However, it doesn't apply to SBM and both molecular graphs. This is further borne out by the fact that LIR consistently improves performance on those 3 other datasets below. We show the mean increase across both latent models across those 3 datasets. This proves that the benefit of LIR is not only restricted to the characteristics of the Planar/Tree datasets.

DatasetAcc/Valid%
SBM+18.5%
MOSES+2.5%
Guacamol+4.25%

In conclusion, we agree that Tree and Planar are simpler datsets. However, they are merely used as a guidepost to analyze the capabilities of graph generative models. Furthermore, even though the other datasets are much more complex, LIR is still helpful, showing that this is still a problem on more complicated graphs.

The AI use statement does not align with this paper. The authors claim to have used generative AI to formulate mathematical claims, but not provide proofs or assist in proof generation. However, there are no mathematical claims or proofs in this article.

This is an error in communication on our part. By mathematical claims, we simply mean that we used AI for generating the latex code for our equation. We have amended the statement to reflect this.

Further, they list a variety of potential generative AI use cases as ``not relevant to the paper'', including editting, formating references, etc., while the paper clearly contains references and is organized as a research paper.

It's unclear to us what you are referecing when you quote "not relevant to the paper". The AI use statement included in our work is based on the official version included in the ICLR 2027 template. We modified it for our specific use case to denote which areas we did and did not use AI. We used AI solely for generating equations in Latex. We did not use it for formatting references or editing non-mathematical equations. Our current statement reflects this fact and is consistent with our usage and final paper.

Note that we obviously agree that this is a potential use of generative AI as a whole (though not for graph generation). But the AI use statement is merely to communicate where we used AI in our paper, not to reflect what our work can be used for. This is per the conference requirements (see here).

It appears from Figure 3 that the importance scores are fairly close in general with a precipitous drop-off. Providing the normalized scores explicitly would give more insight into the behavior.

The scores in Figure 3 are already the normalized importances from Eq. 7 (IiI_i, summing to 1 across dzd_z). Rather Figure 3 shows the cumulative sum of importance scores in the top k% of latents. It is not the raw unnormalized scores.

Specifically, we order all importance scores from highest to lowest, where the top kk% of scores are the top kk% importances scores (e.g., kk=10% are the top 10% of importance scores). For each value of kk we sum the importance scores of those latent dimensions and plot them in Figure 3. E.g., for VGAE on Planar the top 12 of 128 dims carry 98.8% of the mass (see Figure 3a)

How do the variance and importance measures perform independently as reweighting schemes?

Thank you for the suggestion. We included these experiments on the Tree dataset for VGAE+Flow (we were unable to run more experiments due to time). All results are over 5 seeds. They can be found in the table below.

We can clearly see that LIR does the best. While variance-only and Jacobian-only can lead to a modest boost in performance, their combination leads to the largest performance increase by far, thus validating our design.

We have included these results at the end of Section 6.2 (see Ablation Study).

Tree (VGAE, steps=1000, 5-fold × 1024 samples/fold)

ConditionAccRatio
Baseline (no reweighting)17.3% ± 0.810.62 ± 0.17
variance-only reweighting17.9% ± 1.310.64 ± 0.48
Jacobian-only reweighting19.5% ± 1.19.92 ± 0.27
LIR23.2% ± 2.58.75 ± 0.67
AuthorsOpenReview ↗

Response of Raised Questions [Part 3/3]

How is the accuracy of the SBM model reconstruction evaluated? Since SBMs are edge-independent models with non-zero edge probabilities, any generated graph is a valid SBM (albeit rare).

The SBM "accuracy" metric tests if the generated graph falls in the general distribution used to generate the SBM graphs. We follow the common procedure introduced by SPRECTRE [1] (you can see the original description in Appendix C in their paper). Specifically, the train/valid/test graphs are generated using a predefined intra/edge probabilities. The number of communities and nodes are drawn randomly from a predefined ranges. During inference, a generative models generated N graphs. For each generated graph, we estimate both the number of communities and the intra-/inter-block edge probabilities using Bayesian inference . A Wald test is then used to determine the probability that the probabilities of the generated graph match the original parameters. If the p-value >= 0.9, we consider it a valid SBM graph. So although any edge-independent graph has non-zero probability under some SBM, it is only credited if it recovers both the reference block-size regime and the reference edge probabilities to high statistical confidence.

We included a reference to this explanation in [1] on lines 436-437.

[1] "SPECTRE: Spectral Conditioning Helps to Overcome the Expressivity Limits of One-shot Graph Generators". ICML 2022.

What does the coloring in Figures 9, 10, and 11 indicate?

We apologize for the confusion. The color is the the fiedler value connectivity (i.e., vector). This is used to extenuate graph substructures. We would like to note that this is the default coloring used for representing generated graphs in literature. For example, it can be found here in the code for SPECTRE [1] and here for DeFoG. We have included a note of this in Appendix D.1.

I think the expectation in equation (6) should depend on the encoder not the decoder.

Thank you for bringing this to our attention. You are correct. We have updated the equation.

In 6.1, evaluation metrics there are numerous typographical issues.

We apologies for any issues. We want to ensure that our paper is as readable as possible. However, we are unsure of which typographical issues you are referring to. We'd be happy to fix those you have noticed.

Reviewer_YRMUOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the mismatch between the unequal influence of individual latent dimensions on graph reconstruction and the uniform weighting of these dimensions in standard denoising objectives for latent graph generation. The authors compute an influence score for each dimension using the decoder's Jacobian sensitivity and the empirical variance of the encoder's posterior means. They propose Latent Influence Reweighting (LIR), which uses these precomputed scores to reweight the noise prediction loss for flow matching after freezing the pretrained autoencoder. Experiments on two autoencoder architectures and five datasets show improvements in generation metrics across multiple settings.

Critical strengths
  1. The method is simple: its main modification is dimension-wise reweighting of the existing training loss, without changing the generative model architecture or adding model parameters at inference time.
Critical weaknesses
  1. The molecular generation experiments cover only LG-Flow, leaving it unclear whether the gains generalize across autoencoders on molecular tasks. Although the authors acknowledge computational constraints, evaluating a second backbone in a smaller, representative setting would help substantiate the method's applicability.
  2. The benefits of LIR clearly depend on the backbone, evaluation metric, and sampling budget, with trade-offs between structural validity and distribution matching. Claims of steady or consistent improvements in the main text should be qualified, and the negative results should be analyzed.
  3. The current presentation gives the impression that the paper's novelty lies in discovering that latent dimensions are not equally important. However, this is not a new finding: similar observations have already been made in ARD-VAE (Saha et al., 2025).
Questions for authors
  1. The current results support the effectiveness of the proposed reweighting method relative to a uniformly weighted baseline, but they do not isolate the contributions of Jacobian sensitivity, variance correction, and the correct assignment of weights to dimensions. Could the authors include variance-only, Jacobian-only, and shuffled-weight controls to establish the value of the specific influence metric?
  2. Why does accuracy decrease from 73.5% to 63.2% for Planar with LG-Flow? This performance degradation warrants further analysis.
2 comments
AuthorsOpenReview ↗

Response of Raised Questions [Part 1/2]

We thank the reviewer for the constructive feedback. We respond to each of your comments below.

The molecular generation experiments cover only LG-Flow, leaving it unclear whether the gains generalize across autoencoders on molecular tasks. Although the authors acknowledge computational constraints, evaluating a second backbone in a smaller, representative setting would help substantiate the method's applicability.

We thank the reviewer for this suggestion and have addressed it directly: we now report molecular results for VGAE+Flow. Beyond adding the backbone, we also upgraded the evaluation protocol for both backbones from a single seed to a 3-seed cross-validation (the original seed plus two additional seeds).

The results are in the table below. We can observe that LIR improves every single reported metric for VGAE+Flow on both molecular datasets, with substantially larger margins than we observed for LG-Flow on the same datasets. This directly supports the claim that LIR's benefit is not an artifact specific to one autoencoder: on a second, architecturally distinct to a single backbone but is more general.

Together with the synthetic graph results in Table 2 this shows that LIR can consistently help across both autoencoders and types of datasets, demonstrating LIR's applicability and generality. We have revised Table 2 in our paper to reflect this new set of experiments.

ModelGuacaMol V.U.N. ↑GuacaMol FCD ↑GuacaMol NSPDK ↓MOSES Val. ↑MOSES Unique ↑MOSES Novelty ↑MOSES FCD ↓MOSES NSPDK ↓
LG-Flow80.9 ± 0.278.9 ± 0.10.49 ± 0.00187.6 ± 1.099.94 ± 0.0392.9 ± 0.32.44 ± 0.090.529 ± < 0.001
+ LIR82.9 ± 0.380.6 ± 0.20.48 ± < 0.00186.4 ± 0.699.97 ± 0.0194.0 ± 0.12.38 ± 0.060.521 ± < 0.001
% Latent Difference+2.5%+2%+2%−1%+0.03%+1%+3%+2%
VGAE+Flow75.0 ± 0.383.8 ± 0.20.48 ± 0.00280.3 ± 0.199.95 ± 0.0292.2 ± 0.21.87 ± 0.060.524 ± 0.001
+ LIR79.6 ± 0.284.3 ± 0.30.47 ± 0.00285.0 ± 0.199.98 ± 0.0193.4 ± 0.11.58 ± 0.010.517 ± 0.001
% Latent Difference+6%+1%+2%+6%+0.03%+1%+15%+1%

Mean ± std over 3 seeds; 10,000 (MOSES) / 25,000 (GuacaMol) generated molecules, 1,000 Euler steps. GuacaMol FCD is the GuacaMol score 100·exp(−0.2·FCD).

The benefits of LIR clearly depend on the backbone, evaluation metric, and sampling budget, with trade-offs between structural validity and distribution matching. Claims of steady or consistent improvements in the main text should be qualified

We agree that benefit of LIR does depend on several factors. To better show this, we have revised our manuscript to better discuss when LIR best helps. Please see Appendix C.5 for our extended analysis. For reference, we include a brief disccusion of those results here as well.

Specifically, we measure how LIR impact performance across multiple axes, including by: (a) latent model; (b) dataset; (c) metric. We show the results for the synthetic datasets below. We can observe a steady increase of 20-30%. Please see the end of our rebuttal for a discussion of specific failure cases (this is also included in Appendix C.4).

Synthetic — By Model (Planar/Tree/SBM for Acc+Ratio):

ModelMean
LG-Flow+23.2%
VGAE+Flow+27.7%

Synthetic — By Dataset (VGAE-Flow/LG-Flow for Acc+Ratio):

DatasetMean
Planar+19.3%
Tree+35.0%
SBM+22.0%

Synthetic — By Metric (Planar/Tree/SBM for VGAE-Flow/LG-Flow ):

MetricMean
Accuracy+22.0%
Ratio+28.8%

We also include the results for both molecular datasets below. While molecular gains are more modest than synthetic (consistent with our hypothesis in the text that rich node features already give better-conditioned latents), they remain positive for both backbones and both datasets.

Molecular — By Model (GuacaMol + MOSES across all metrics):

ModelMean
LG-Flow+1.4%
VGAE+Flow+4.0%

Molecular — By Dataset (LG-Flow/VGAE-Flow across all metrics):

DatasetMean
GuacaMol+2.6%
MOSES+2.8%
AuthorsOpenReview ↗

Response of Raised Questions [Part 2/2]

The current results support the effectiveness of the proposed reweighting method relative to a uniformly weighted baseline, but they do not isolate the contributions of Jacobian sensitivity, variance correction, and the correct assignment of weights to dimensions. Could the authors include variance-only, Jacobian-only, and shuffled-weight controls to establish the value of the specific influence metric?

Thank you for the suggestion. We included these experiments on the Tree dataset for VGAE+Flow (we were unable to run more experiments due to time). All results are over 5 seeds. They can be found in the table below.

We can clearly see that LIR does the best. While variance-only and Jacobian-only can lead to a modest boost in performance, their combination leads to the largest performance increase by far, thus validating our design.

We have included these results at the end of Section 6.2 (see Ablation Study).

Tree (VGAE, steps=1000, 5-fold × 1024 samples/fold)

ConditionAccRatio
Baseline (no reweighting)17.3% ± 0.810.62 ± 0.17
variance-only reweighting17.9% ± 1.310.64 ± 0.48
Jacobian-only reweighting19.5% ± 1.19.92 ± 0.27
Shuffled LIR13.0% ± 0.611.66 ± 0.44
LIR23.2% ± 2.58.75 ± 0.67

Why does accuracy decrease from 73.5% to 63.2% for Planar with LG-Flow? This performance degradation warrants further analysis.

We thank the reviewer for noticing this and we apologize for missing the explanation. We explored this, finding that it is due to two reasons: (1) the latent dimension size; (2) characteristics of the Planar dataset.

In their original paper, the LG-Flow authors used a latent dimension of dz=24d_z=24. To remain faithful to the original paper, we followed this suggestion. We further tried running with a latent dimension of dz=128d_z=128 like is done with other datasets. We can observe the results below when comparing different latent sizes. We observe that for the acc% decrease is much more modest (while also increasing in general) while the ratio still decreases significantly when using LIR.

Planar 512 nodes(lg-flow)Baseline+LIRChange
dz=24d_z=24 — Accuracy73.563.2−14.0%
dz=24d_z=24— Ratio16.75.5+67.1%
dz=128d_z=128 — Accuracy77.274.9−2.9%
dz=128d_z=128 — Ratio41.735.1+15.8%

Interestingly, we can see that while overall Acc% is better for dz=128d_z=128 the ratio increases dramatically (but still decreases when applying LIR). We find that this is actually a common property of some datasets, where there seems to be a trade-off between acc% and ratio. For example, DeFoG [1] find a trade-off between both metrics on Planar like we do for certain hyperparameters (see Figures 3a and 3b in [1]). They also observe similar relationships for SBM (Figures 9 and 10 in [1]). A similar observation can be found for HSpectre [2], as they observe a similar trend for SBM scoring 15 points higher V.U.N. than DiGress at roughly 6× worse Ratio (see Table 1 in [1]). On the other hand, DiGress achieves a lower ratio than DeFoG while having a considerably lower acc%. Overall, this points to an overall trend that is not isolated to our paper. We have included a discussion of this in the in Appendix C.4.

[1] "DeFoG: Discrete Flow Matching for Graph Generation". ICML 2025.
[2] "Efficient and scalable graph generation through iterative local expansion". ICLR 2024.

Reviewer_fvmtOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper studies latent graph diffusion/flow matching , specifically the utilization of different latent latent dimensions ziz_i by the decoder D:Z→ GD: Z \to  G. A delightful and appreciated detail is that it is very obviously either hand written, or if llms were used, crafted with much love.

The authors use  the jacobian J(Z)≔δD(Z)δZJ(Z)\coloneqq \frac{\delta D(Z)}{\delta Z} to compute the influence  of each latent dimension ii  as the raw jacobian w.r.t. to that dimension Si≔EZ∼ D(⋅∣G)[∥J(Z(i))∥2]S_i \coloneqq E_{Z \sim  D( \cdot | G)} \left[ \Vert J(Z^{(i)}) \Vert_2 \right] scaled by the variance of that embedding dimension vi=var(μi)v_i=var(\mu_i), Ii=vi⋅Si∑jvj⋅SjI_i=\frac{v_i \cdot S_i }{\sum_j v_j \cdot S_j}

They collect the influence  across latents on the Tree dataset of Bergmeister et al 2023, Planar of martinkus et al 2022, using a modified VGAE and a modified LG-VAE/LG-FLOW, using a a hidden dimension of 128 dims. They find that for VGAE ca. 20% of   latent dims carry almost all of the signal for VGAE, with LG-Flow having a similar profile for planar and a  flatter profile for tree (flatter in the sense of 80% of dims being required to saturate the influence mass).

To alleviate this concentration, the paper proposes to weight the latent dimensions by their infleunce in the loss of the latent flow matching objective, which they deem as “Latent Importance Reweighting” or LIR as  $\mathcal{L}{FM}^w\coloneqq \mathbb{E}{G,Z,t}\left[1/dx \sum{d=1}^{d_x} I_d \cdot \Vert \hat{Z}_0^{(d)} - Z_0^{(d)} \Vert_2 \right] $ . IdI_d is  computed once on the full dataset per auto encoder and reused throughout.

The authors evaluate LIR on MOSES and Guacamol, as well as extended versoins of Tree,Planar and SBM with 8192 training samples, as well as 256 validatoin and test samples, and sizes extended to 512 (planar), 128 (tree) and 125 (on average, for sbm). They use VGAE and LG-VAE as the auto-encoder, using a GIN encoder for both, the original encoder for LG-VAE and a transformer for VGAE. For the denoising model, they use DiT. All are trained on the predicted noise again.

On the extended synthetic data, authors observe a consistent improvement of edge accuracy and MMD ratios for VGAE+flow vs vgae+flow+LIR, while for LG-FLOW accuracy suffers on planar and mmd ratio suffers on sbm.

On guacamol, LG-FLOW+LIR has a modest uplift of ~2% on average, while on MOSES validity suffers, depressing the V.U.N score, while FCD and NSPDK enjoy a modest uplift of ~2% on average again. Convergence time of validation accuracy improves markedly and strongly for both LG-FLOW  and VGAE-flow.  

Via ablations the authors show that LIR outperforms simply removing less used dimensions (70% vs 82% edge accuracy of 16 dim unweighted vs 128 dim weighted on planar)

Critical strengths
  1. The convergence benefits are very clear and are worth studying in and of themselves as a form of preconditioning any optimization making use of latent diffusion I think.
  2. The presentation, data derived  and execution is very clear,clean and focused for the most part (see caveat in the weaknesses, which are all ameliorable) and as a building block for working towards latent methods, I believe this paper presents a promising first step if framed and contextualized appropriately.

Combined, I'm using these to judge the paper as "incremental, but worthwhile" because demonstrating opportunities for improvement, even via transfer from other fields or only as a building lock like this this is valuable.

I also independently from the score want to thank the authors for not dropping a slopped up manuscript, it was a delight to read this after 8/10 papers in my recent batches being dripping of llmisms.

Critical weaknesses

Post rebuttal update: the authors have resolved all of my concerns during rebuttals


  1. The biggest weakness of the paper is a conceptual one, in that it does not sufficiently engage with the limitations of graph autoencoders, including edge independence https://proceedings.neurips.cc/paper/2021/file/cc9b3c69b56df284846bf2432f1cba90-Paper.pdf , which limits the value of any single step auto encoder based method. This shows in the papers results, with specific structured graphs (guacamol, moses) experiencing little benefit, while conceptually simple models (sbm, planar, tree) which can be reduced to simple probabilistic and local rules are able to be improved. Discussing this in more detail and discussing incremental and hierarchical methods like sparsediff, EDGE, bergmeister et al, higen or mrgrad would ameliorate this (I think the method can serve as a building block to improve these models)

  2. The 2nd biggest flaw is that of insufficiently rigorous assessment and contextualization of the claims:

    1. no error bars on the large molecule presentation, despite the improvements there being the most tenuous and plausibly wiped out by sampling noise (as well as seed to seed training variatoins, but computational expense is a thing)
    2. the ablation appears gimped to me: 16 to 128 is very tight based on the cumulative mass plots in figure 3, due to the beneficial nature of degreesof freedoms I would like to see ablations to 32 and 64 on how well that can close the gaps
    3. the figure 1 acccuracy plot of defog and lg-flow doesn’t match the papers reported accuracies, no explanation given
    4. the authors cite Saha et al 2025 and describe their measure as “similar”, however their IiI_i is exactly the score used by ARD-VAE, just without squaring.
    5. while the authors hint at explaining a gap to discrete models (line 49ff), the paper doesn’t really engage enough to justfy this: LG-FLOW is competetive wit digress and disco already, and on the synthetic numbers where they could justify the claim, no direct comparison is reported (which would be possible if using MRGrad,EDGE, bergmeister et all, sparsediff etc)
    6. the vision community has already intensely studied the issue of capacity balancing, e.g. in https://arxiv.org/abs/2510.11690 https://arxiv.org/abs/2501.01423 and https://arxiv.org/abs/2502.14831 which would all be worth discussing and contrasting against
Questions for authors

Clarifications (could change my assessment):

  1. Why did you not run any of the incremental approaches I noted? Could you run (some of) them on the synthetic methods to help ground your papers results (matched for parameter count/architecture ideally)
  2. What is the generator's hidden width per node token? RAE (https://arxiv.org/abs/2510.11690 ) shows that a model narrower than the token dimension cannot reach the minimum of the flow loss

Minor nits/suggestions (unrelated to the score):

  1. The paper claims to be following Siraudin & Morris 2026 in predicting the noise Z0Z_0 but on checking the original paper, they predict Z−Z0Z-Z_0  ?
  2. The loss typeset in eq4 and eq 8 is not MSE, I asume this is a typo?
  3. How do you think you could work around the issue of edge independence/incorporate this into already edge dependent models to scale them?
3 comments
AuthorsOpenReview ↗

Response of Raised Questions [Part 1/3]

Thank you for your review. We respond to each of your comments below.

does not sufficiently engage with the limitations of graph autoencoders, including edge independence, which limits the value of any single step auto encoder based method.

Thank you for raising this interesting problem. However the bounds in Chanpuriya et al. (2021) are for models that output a single fixed edge-probability matrix. In our work, the flow model learns to sample a latent ZZ which is not edge-independent since it is done over T denoising steps. Essentially our model learns to generate a mixture of many ZZ's over TT steps. This allows us to side-step this problem and ensure some edge-dependence.

discussing incremental and hierarchical methods like sparsediff, EDGE, bergmeister et al, higen or mrgrad would ameliorate this (I think the method can serve as a building block to improve these models)

Why did you not run any of the incremental approaches I noted? Could you run (some of) them on the synthetic methods to help ground your papers results (matched for parameter count/architecture ideally)

Thank you for the suggestion. We have included a discussion of this in the related work on lines 107-128. We would like to note that our method, LIR, can't directly be applied to these methods since LIR operates in the latent space while those methods operate in the discrete space.

Furthermore, we were unable to implement and properly tune these within the rebuttal window. The issue is one of time. Due to the discrete nature of each method, they take an exorbiant amount of time to run.

To demonstrate this, we tried running SparseDiff [1] and measured the amount of time needed to run one training epoch on an 80Gb H100 GPU. We chose SparseDiff as they showed in their paper that they are more efficient that other methods like EDGE or Hi-Gen (see Table 6 in their TMLR camera-ready verison). For Tree, Planar, and SBM we ran SparseDiff for a short period of time and estimated the amount of time needed to run 50K epochs. Note that in their official code, they used 200K epochs for SBM and 300K for Planar (code). We furthermore used the max possible batch size that fit onto our GPU.

The results are in the table below. We can clearly observe that training each for 50K epochs would take an exorbiant mount of time. The fastest running dataset, Tree, would still take ~115 days to run 50K epochs. This shows the limitations of discrete methods as they struggle to scale to larger graphs, inhibiting real-world use.

DatasetBatch sizeEpoch timeEpochs so farTime so farTime for 50,000 epochs
Tree(128 nodes)32198.19 s2,101 (115.7 h)115.7 h (4.82 d)2,752.6 h ≈ 114.7 days
Planar(512 nodes)412,960.0 s21 (75.6 h)75.6 h (3.15 d)180,000 h ≈ 7,500 days (≈ 20.5 years)
SBM(average 125 nodes)32177.59 s73 (3.60 h)3.60 h2,466.5 h ≈ 102.8 days

[1] "SparseDiff: Sparse Discrete Diffusion for Scalable Graph Generation."" TMLR 2025.

How do you think you could work around the issue of edge independence/incorporate this into already edge dependent models to scale them?

We agree that this is an interesting future direction. One way of achieving this would be combining latent and discrete methods which would allow for the use of LIR with these methods. For example a latent prior over coarse cluster graphs.

AuthorsOpenReview ↗

Response of Raised Questions [Part 2/3]

no error bars on the large molecule presentation, despite the improvements there being the most tenuous and plausibly wiped out by sampling noise (as well as seed to seed training variatoins, but computational expense is a thing)

We have added error bars in the Table below with three different seeds (0,1,2). It's not a standard practice to report error bars because (a) the confidence is high for large molecular datasets; (b) they are very expensive to run for discrete models. Nonetheless, we can observe below that the variance when running over three seeds is very low.

We have added revised the paper to include these results (see Table 2).

ModelGuacaMol V.U.N. ↑GuacaMol FCD ↑GuacaMol NSPDK ↓MOSES Val. ↑MOSES Unique ↑MOSES Novelty ↑MOSES FCD ↓MOSES NSPDK ↓
LG-Flow80.9 ± 0.278.9 ± 0.10.49 ± 0.00187.6 ± 1.099.94 ± 0.0392.9 ± 0.32.44 ± 0.090.529 ± < 0.001
+ LIR82.9 ± 0.380.6 ± 0.20.48 ± < 0.00186.4 ± 0.699.97 ± 0.0194.0 ± 0.12.38 ± 0.060.521 ± < 0.001
% Latent Difference+2.5%+2%+2%−1%+0.03%+1%+3%+2%
VGAE+Flow75.0 ± 0.383.8 ± 0.20.48 ± 0.00280.3 ± 0.199.95 ± 0.0292.2 ± 0.21.87 ± 0.060.524 ± 0.001
+ LIR79.6 ± 0.284.3 ± 0.30.47 ± 0.00285.0 ± 0.199.98 ± 0.0193.4 ± 0.11.58 ± 0.010.517 ± 0.001
% Latent Difference+6%+1%+2%+6%+0.03%+1%+15%+1%

Mean ± std over 3 seeds; 10,000 (MOSES) / 25,000 (GuacaMol) generated molecules, 1,000 Euler steps. GuacaMol FCD is the GuacaMol score 100·exp(−0.2·FCD).

the ablation appears gimped to me: 16 to 128 is very tight based on the cumulative mass plots in figure 3, due to the beneficial nature of degrees of freedoms I would like to see ablations to 32 and 64 on how well that can close the gaps.

We agree the original ablation for planar with 512 nodes (D=16 and D=128 endpoints) leaves the interior of the curve under-specified, especially given Figure 3's cumulative-mass plot shows most of the importance mass concentrating in a small, non-trivial subset of dimensions. We ran the same 5-fold k-fold evaluation (1,024 samples/fold, 1,000 sampling steps) at dzd_z ∈ {16, 32, 64, 128} for the VGAE + flow backbone.

We can observe that the performance still lacks behind dz=128d_z=128+LIR, maintaining our original assertion that simply lowering the dimension doesn't help. Furthermore we can observe a large instability in performance across dimensions sizes, including a large drop in performance for 64 latent dimensions. This suggests that training w/o LIR produces additional instability that can harm performance.

We have updated Figure 5 in our paper to reflect these new results.

d_zAcc (%)Avg Ratio
1670.5 ± 1.98.94 ± 0.92
3277.1 ± 1.56.13 ± 0.87
6439.9 ± 1.921.03 ± 3.25
12872.9 ± 0.013.10 ± 0.6
128 + LIR82.9 ± 0.012.8 ± 0.4

the figure 1 acccuracy plot of defog and lg-flow doesn’t match the papers reported accuracies, no explanation given

LG-FLOW is competetive wit digress and disco already

We apologize for not making these points clear. Several points explain this:

  • Figure 1 used the original Tree dataset (64 nodes) created by bergmeister et al. However, it only considers 200 total graphs across train/valid/test. This is the same as SBM and Planar which also consider 200 total graphs. We have revised the figure caption to better reflect this distinction.
  • The authors of LG-Flow argue that 200 graphs is too few and extend the dataset to 8192 training graphs. This aids in learning better latents. However, this leads to overly easy benchmarks, as the small size of each graph allows for nearly perfect performance resulting in a highly saturated benchmark evaluation. As such, it is not a good barometer for comparing the capabilities of different models.
  • To account for this, we later create a new dataset in Section 5 that considers graphs with more nodes (e.g., 512 for Planar) which creates a harder task that allows us to better analyze and compare models.
AuthorsOpenReview ↗

Response of Raised Questions [Part 3/3]

the authors cite Saha et al 2025 and describe their measure as “similar”, however their is exactly the score used by ARD-VAE, just without squaring.

We apologize for any confusion. We were heavily inspired by the score used in ARD-VAE. However, there are two other important distinction.

(1) In their paper they estimate they weight the raw score (for us SiS_i; for them wσ^w_{\hat{\sigma}}) by the estimated variance of a prior distribution while we consider the observed variance across the posterior mean. Specifically, they consider the estimated variance of a Gamma distribution prior that considers two prior parameters such that σ^=bL/aL\hat{\sigma} = b_L / a_L. We consider the variance over the posterior mean ui\mathbf{u}_i. This is done as the variance over the posterior mean is more indicative of the actual sampled latents rather than just the prior.

(2) While squaring (ARD-VAE) vs. euclidean norm (LIR) might seem small, it can have a profound impact on the final weights, as squaring mangnifies the difference between latent dimensions. This can have the effect of concentrating the importance in a much smaller number of dimensions.

We have modified our paper to include a description of how our approach differs from that in ARD-VAE in Appendix B.

the vision community has already intensely studied the issue of capacity balancing, e.g. in https://arxiv.org/abs/2510.11690 https://arxiv.org/abs/2501.01423 and https://arxiv.org/abs/2502.14831 which would all be worth discussing and contrasting against.

We appreciate you bringing this to our attention. We include a discussion of this line of work on lines 144-155 in our revised manuscript. We include the additonal discussion below:

Autoencoders for Latent Diffusion: A closely related line of work in the vision community studies how to improve autoencoders for use in diffusion and flow models. REPA (Yu et al., 2024) shows that aligning a diffusion transformer’s internal representations with a pretrained self-supervised encoder dramatically eases training in this regime. Zheng et al. (2026) pair pretrained encoders (DINO/SigLIP/MAE) with trained decoders to expand latent capacity. Furthermore, Yao et al. (2025) align the tokenizer’s latent space with a vision foundation model (VA-VAE) to tame the resulting optimization difficulty and Skorokhodov et al. (2025) instead regularize the decoder for scale-equivariance to remove excess high frequency latent content that clashes with diffusion’s coarse-to-fine synthesis. We note that these methods focus on changing the autoencoder itself. On the other hand, our proposed approach takes an orthogonal approach where we leave already-trained autoencoder completely untouched, making it applicable post-hoc to any existing frozen encoder-decoder pair.

What is the generator's hidden width per node token? RAE (https://arxiv.org/abs/2510.11690 ) shows that a model narrower than the token dimension cannot reach the minimum of the flow loss.

We consider a width of either 384 (synthetic) or 768 (molecule). The maximum token dimension is 128. As such the hidden dimension is also larger than the input, ensuring that this is not a concern.

The paper claims to be following Siraudin & Morris 2026 in predicting the noise Z0Z_0 but on checking the original paper, they predict Z−Z0Z - Z_0?

We agree that the original paper mentions predicting Z−Z0Z - Z_0. However, in their code they seem to always predict Z0Z_0. We can observe on lines 125-130 (official code) they have options for either predicting ZZ, Z0Z_0, or Z−Z0Z - Z_0. This is determined by the param option in the config files. This is always set to Z0Z_0. Given that this is what is done in the official code repo, we followed this.

The loss typeset in eq4 and eq 8 is not MSE, I asume this is a typo?

Yes this is a typo. We have fixed this in the revised version. Thank you for pointing this out.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Last Day of Discussion

Dear Reviewers,

Thank you again for taking the time to provide valuable feedback to our work.

Given that today is the last day of the author-reviewer discussion, we kindly request that you consider our rebuttal in your assessment of our work. We look forward to any questions you may have and hope that our current rebuttal and manuscript updates address all of your concerns.

Thank You,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.