- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how the dynamics of deep Graph Transformer denoisers affect graph generation. The authors propose SGT, which replaces standard attention-based node mixing with a Cayley-transform-based orthogonal operator while preserving permutation equivariance. A damping parameter allows controlled comparisons between orthogonal and contractive transport. Within the DeFoG framework, experiments on Planar, SBM, ZINC250k, and MOSES show that SGT remains effective at greater depths where the standard backbone degrades. Representation and spectral analyses provide evidence linking these improvements to reduced representation collapse and better signal propagation.
Critical strengths
-
The paper offers a useful perspective on why increasing denoiser depth can hurt graph generation. The orthogonal transport construction, damping experiments, and representation analysis connect architectural choices to observable generation behavior. This makes the work informative beyond the proposed model.
-
The empirical gains are meaningful and extend beyond preventing failure at extreme depths. On ZINC250k, SGT reduces FCD from 1.43 to 0.94 at the same eight-layer depth, with further improvement at greater depths. Its ability to sustain effective generation at 32 layers, where the DeFoG backbone collapses on the synthetic benchmarks, supports the practical value of the architecture.
Critical weaknesses
-
Theorem 3.1 explicitly assumes fixed attention scores, edge inputs, and global inputs, but the relevance of this conditional result to the actual model dynamics remains unclear. Fixing attention establishes orthogonality of the transport operator while excluding gradients arising from the dependence of attention on node representations. The spectral and representation analyses provide useful evidence, but their connection to the conditional Jacobian is not sufficiently explained. Since gradient preservation is central to the explanation of the depth gains, the paper needs a clearer justification for why this conditional analysis captures the behavior of the complete model.
-
The ZINC250k results contain an inconsistency that affects the depth comparison. Table 2 reports an FCD of 1.81 for the 16-layer DeFoG baseline, whereas Table 4 reports 1.30; both tables report 1.43 for the eight-layer baseline. These values imply opposite trends as depth increases. The results need to be reconciled before the claim that DeFoG deteriorates with depth on this dataset can be assessed.
Questions for authors
-
Please generate more graphs on the synthetic benchmarks or repeat the evaluation with a few sampling seeds. The differences in the damping experiment are fairly small, so repeated sampling would make the trend more convincing. These evaluations can use the existing checkpoints.
-
Table 7 only reports V.U.N. I would also like to see the corresponding Ratio values. Similar validity rates can hide differences in the generated distributions, which may reveal more about the effect of damping.
-
The eight-layer results are interesting: SGT already improves on ZINC, but its Planar Ratio is 10.22 versus DeFoG’s 2.41. What explains this difference? Any insight from the existing training curves would be useful here.
4 comments
Thank you for adding the four-seed results, damping Ratios, and QM9 experiments. These strengthen the empirical support. The FCD discrepancy and checkpoint-selection issue have also been addressed, resolving my concerns on those points. I also appreciate the clarification in another response that Figure 3(b) measures the full-layer Jacobian, providing useful evidence about the complete architecture.
My remaining concern is the revised Theorem 3.1 and its proof. The statement now explicitly says that attention scores are computed from the input representations, but Appendix A.1 still uses J=Cᵀ⊗R. This expression includes only the derivative with R held fixed. For F(X)=R(X)XC, the full differential also contains (dR(X))XC. Evaluating the forward pass at a determined input does not remove this term.
Please restore the fixed-score and fixed-conditioning qualifications and identify the guarantee as concerning the conditional transport Jacobian. If the implementation uses stop-gradient, this should be stated in the model definition. The empirical evidence for well-conditioned full-layer Jacobians is valuable, but needs to be distinguished from this conditional theoretical result. I am maintaining my score of 5 for now; correcting this issue, together with the additional experiments, would support reconsideration toward weak acceptance.
We thank the Reviewer for the constructive feedback and for acknowledging that the additional experimental results strengthen the empirical support of our work.
We have now revised Theorem 3.1 and its proof in Appendix A.1 to explicitly account for the dependence of the attention scores on the input representations, rather than assuming them to be fixed, as we do not employ stop-gradient in our experiments.
Indeed, our objective is not to enforce exact orthogonality of the full Jacobian, but rather to maintain near-orthogonal dynamics that substantially mitigate dissipation and preserve information propagation. The revised theorem formalizes this by expressing the Jacobian as an orthogonal component plus a drift term, showing that its singular values remain close to one when this drift is sufficiently small.
This theoretical characterization is also supported by the empirical evidence in Figure 3(a), where the Jacobian eigenvalues of the trained SGT remain close to the unit circle, consistent with the intended near-non-dissipative regime. Figure 3(b) further provides evidence of well-conditioned gradient propagation through the complete architecture.
We hope these revisions address your remaining theoretical concern and, together with the additional experiments, support reconsideration of your score toward acceptance.
Dear Reviewer MuUH,
As the rebuttal discussion period is approaching its end (less than 24h remaining), we would like to kindly follow up on our response.
We hope that the theoretical concern has now been addressed, and we would appreciate any further feedback. While we would be happy to clarify any remaining points, we hope that you might reconsider your score towards acceptance in light of these improvements.
Best regards,
The Authors
We thank the Reviewer for their thoughtful comments and constructive feedback. We appreciate their recognition that our work offers a useful perspective on why increasing denoiser depth can hurt graph generation, and that the orthogonal transport construction, damping experiments, and representation analysis connect architectural choices to observable generation behavior, making the work informative beyond the proposed model. We also appreciate their acknowledgment that the empirical gains are meaningful and extend beyond preventing failure at extreme depths, with SGT achieving improvements even at shallow depths and sustaining effective generation at greater depths. We address their concerns below and hope these clarifications will help improve their assessment towards acceptance.
W1 We thank the Reviewer for this comment. We clarify that, for a given layer (\ell), the attention scores, edge inputs, and global inputs are already determined when evaluating the transport operator at a specific point in the forward pass. Indeed, the input representations do not change within the evaluation of a single layer. More specifically, the attention scores are determined by the input representations X(ℓ) and the corresponding projection weights W(ℓ,h). Both are already determined (i.e., fixed) when evaluating the transport operator at a specific point in the forward pass. Therefore, the attention scores are well-defined and fixed for that evaluation, allowing us to analyze the orthogonality of the resulting Cayley transform. This does not imply that the attention scores are independent of the input representations, but rather that our analysis considers the transport operator induced by a given input. Therefore, these are not additional assumptions imposed on the model, but simply the quantities defining the transport operator at that point. We have revised the statement to avoid introducing unnecessary assumptions and clarify this distinction.
W2. We thank the Reviewer for spotting this typo. The correct FCD value for DeFoG at L=16 is 1.81. We have corrected the corresponding entry in the table. Therefore, our original conclusions remain unchanged, confirming our claims regarding the negative effect of increasing depth on the baseline.
Q1. Following the reviewer’s suggestion, we trained our model with three additional random seeds. Table 1 now reports the mean and standard deviation of V.U.N. and Ratios across four seeds for each layer configuration. We sampled 40 graphs from each independently initialized model, yielding 160 graphs per configuration.
Q2. We thank the Reviewer for the insightful suggestion. Table 7 (now Table 9 in the revised manuscript) now reports the Ratios of the generated graphs, helping to better clarify the damping effect of γ0. We have therefore updated Table 9 to include these ratios for each tested value of γ0 in the L=32 configuration. We are currently extending this ablation to the remaining layer configurations and will update the manuscript once these experiments are complete.
Q3. We thank the reviewer for pointing this out. During our investigation, we identified a genuine error in the checkpoint-selection step of our evaluation pipeline, which selected an early, undertrained checkpoint. The reported ratio of 10.22 was therefore computed using an undertrained model. We have corrected the selection procedure and repeated the evaluation. The corrected Average Planar ratio, averaged across four training seeds, is 2.63, and we have updated the manuscript accordingly. This was an isolated issue, and no other runs reported in the manuscript were affected. Thank you again for your support in spotting this error.
We hope these clarifications and revisions address the Reviewer’s concerns, and we would be grateful if they would consider reassessing their evaluation and score based on the updated text and the above comments.