- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The text proposes EvolveODE, a higher order Graph Neural Network that aims to resolve underlying dynamics directly via attractive and repulsive node interaction terms. Significant subprocesses of the latent dynamics are linked to a complete subgraph topology. The method augments the energy surface implied by the interaction terms by corresponding subprocess energy wells. Supplied theory is empricially evaluated on short- and longterm forecasting tasks with varying degree of data sparsity and node homophily.
Critical strengths
1. Interdependence of coarser and more local structure. Conceptually, I find the paper's recognition of the interdependence between scales important. Moreover, the idea to use attraction-repulsion terms in combination with Graph Neural Networks is sound and reasonable as, e.g., already verified by the cited ACMP.
Critical weaknesses
1. Lack of Clarity and Consistency. Overall, the text does not sufficiently describe how the proposed method is actually implemented. For instance, Definition 1 introduces sub-processes as the interaction between nodes that form a complete subgraph but it never defines what is meant with "an interaction between nodes". The same definition further introduces a "noisy" graph, without alluding to a "noisy" graph later on in the text. Furthermore, in what sense, is time relevant for evaluated datasets like Cora and Squirrel that have no inherent time axis? In general, the text does not clarify over what dimensionality the coarsening takes place. That is, whether links are established across spatial, temporal or other scales.
2. Lack of Reproducibility. The text describes their experiment setup only partially. While dropout and learning rate are given, optimizer the number of epochs etc., but most importantly a comprehensive description of the EvolveODEs GNN architecture is missing. Furthermore, the supplied code does not provide a requirements, environment or similar file that would facilitate reproducibility and portability. More importantly, the code does not run anyhow as it is incomplete.
Experiment scripts import modules (e.g., baselines or profiling_utils) that are not part of the supplied code. A smaller faux pas is the README describing the code as part of a NeurIPS submission.
Minor:
3. Citation style. The authors use a number citation style despite the author guidelines prescribing an author-year format. The used number style leads to significant more available space for this submission compared to other ones that adhered to the guidelines.
4. Writing.
- l. 071-072: "... in EvolveODE breaks monotonically reduces error." is probably missing a word or two
- l. 172-173: "challange" should be "challenge"
- l. 216-217: "...with the form from Equation 3 for is well established" probably got an extra "for"
- l. 366: "... variables, whereas in weither forecasting." ends abruptly
Questions for authors
-
Could the authors substantiate their definition 1 (see Critical Weaknesses)?
-
Could the authors explain over what feature dimension a coarsening is achieved conceptually (see Critical Weaknesses)?
-
Could the authors provide a full description of their experiment setup (see Critical Weaknesses)?
-
What is meant with the observational level of dynamics? (e.g. l. 032-033) In what way does the method allow for / encode hidden states?
-
In what sense can GNNs capture spatio-temporal interactions via physical networks? (l. 093-094)
-
In what sense is EvolveODE "enabling the first marriage of ideas between NeuralODEs and multiscale learning"?(l. 106-107) A quick web search shows already various papers on the topic (e.g. Xu et al., "Multi-scale Neural ODEs for 3D Medical Image Registration" in MICCAI 2021).
-
What is an edge index variable (l. 291)? Is here meant the actual edge index or the edge features? How do individual coupling matrixes look like?
-
The authors, at points, state phrases that are rather grand or general but are not further explained:
- "Behind all naturally occurring observations are underlying coarse-scale processes governing the evolution of these local phenomena." (l. 133-135)
- "Graph topology describes intricate real-world relationships, yet confounding factors entangle their features." (l. 164-165)
- "From an information-theoretic perspective, such feature revitalization injects previously invisible contextual cues into the disentanglement process" (l. 313-315) Could the authours explain what exactly is meant or, at least, have a reference provided?
-
Experiment variance: Experiment tables mostly do not report a metric for spread. Why is that and could it be added, e.g., over training seeds? Conversely, why does Table 1 which would have a natural spread over input buffer sizes not report it? In contrast, over what is the spread reported in Table 3?
-
What do the line colors in Figure 4 imply? Most of the panels (all but "Actor") do device two lines, orange and blue. The figure does not indicate what the different colors imply (and why there is none for "Actor").
-
Can the authors provide information on limitations and future works? On the one hand, this section is missing, on the other the authors refer to it in lines 944-945, where they would want to discuss the "finite split problem" further.
We thank the reviewer for their time and feedback. We address each point below.
Weaknesses
W1. Please see our responses to the questions below.
W2 (Reproducibility). We apologize for the difficulty in running our code. The issue can be resolved by removing
baselinesandprofile_utilsfrom the dependencies, which is a straightforward change. We have tried to make our work as reproducible as possible: we have released almost our entire codebase, the.shfiles used to run our experiments, and instructions in the README on how to start new experiments. Given this, we would kindly ask the reviewer to reconsider the reproducibility score.W3 (Writing). Thank you for pointing out these issues. They have been fixed in the updated paper.
Questions
Q1. The second paragraph of the introduction explains why variables need to be sorted accurately into sub-processes. As stated just before the definition, we formalize the mismatch between variables and sub-processes discussed in the introduction to make it concrete. EvolveODE aims to avoid such mismatches, so the definition serves to motivate the subsequent examples by making the intuition from the introduction precise.
Q2. We believe there may be a small misunderstanding here. The kernel does not apply to a single dimension; it applies to a given input graph. All main-paper experiments have a known graph. The weather experiments (in the Appendix) learn an N×T graph, but this graph still does not occupy a single dimension.
Q3. Yes. The exact hyperparameter setups, including learning rate and weight decay, are available at the code link, and the README contains the commands we used to run our experiments.
Q4. The observational scale refers to the level of granularity of the observations. For example, for the Max Planck weather dataset (Appendix), the scale is the sampling frequency. As is standard, the observations are embedded into a latent space via a learnable embedding map.
Q5. The physical network represents spatial characteristics of the system dynamics, which allows the GNN to learn the temporal evolution of the observations. For example, in a road network, traffic bottlenecks in the graph will be reflected in the measurements at certain times, such as peak hours. Extracting spatiotemporal trends via a physical network is well established in the spatiotemporal GNN literature.
Q6. In context, the phrase refers to GNN research not having explored NeuralODEs specifically for this purpose. The citation at the end of our statement (reference [35], which was not included in the quoted text) supports this. Regarding the paper suggested by the reviewer, we believe it differs from our setting in several respects: it is (a) not a NeuralODE paper, (b) not a GNN paper, and (c) applied to medical imaging. It solves known ODEs across scales by coarsening target variables, but it does not find couplings in multiscale systems, which our work does.
On the other points: "edge index" refers to the
edge_indexvariable in PyTorch Geometric, the most widely used library for GNN research and implementation. We define Q as assignment matrices with entries in {0,1}.Q7. The first two of these three phrases build on earlier context in the paper:
Q8. Reporting results across multiple random seeds is not standard practice in forecasting. The variance is also very small, being one standard deviation less than the mean.
Q10. Thank you for the suggestion; we have added this section. It discusses how performance cannot be improved indefinitely by adding equations, since the number of such relationships is finite. Future work should therefore focus on the operations performed on the uncovered dynamical system. This is a promising direction, as it may allow performance to scale, an approximation of the underlying system having been learnt.