- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper deals with the, well studied but still highly contentious, problem of long-range dependencies in graph neural networks. Although there has been a heap of theoretical studies and architecture innovations, there is still no clear consensus as to whether simple or modified GNNs can or cannot make use of long-range context. In this work, authors argue that the problem comes from a divide between the properties one can prove such models have (how far they can reach), versus how these models actually tap into these properties to solve a given task (or even if they tap into such properties at all for a given task). The paper therefore suggest to compute how a given model actually makes use of long-range context, versus it's idealized behaviour and finds out several failure modes related to either a model's structure or computational approximations. These failure modes are then revealed and tested on controlled benchmarks and for different recent and classical architectures, illustrating discrepancies between realized and idealized behaviour.
Critical strengths
- clear scope, does one thing but does it well: in this case clearly identifies how architecture that optimise reachability of distant information actually realise their potential when trained.
- Proposes well-motivated diagnostic tools (interaction profiles) that are architecture agnostics
- simple but clear theoretical developments: provides clear theoretical insights
- convincing illustrations on controlled benchmarks allow to distinguish various failure modes on simple tasks and for different architectures.
Critical weaknesses
. re-wiring approaches are not dealt, the theoretical machinery being linked to diffusion approaches. But the diagnostic tools introduced in the paper (interaction profiles) seem much more architecture agnostic. Is it possible to use them to compare diffusion and re-wiring approaches ? Or compare how re-wired models perform by comparing performances wrt to original and re-wired graph distances ? . good on controlled benchmarks and ECHO but does that tell us much for benchmarks closer to real world ? In particular how "long-range" does effectively occur in practice ? I worry that all this could be nice on paper but that in practice models don't need to access regimes of long-range interactions that make such a difference. In its present form the paper convincingly explores actual model performances as function of the distance variable. It does not tell us if real-world graph learning tasks really require interaction ranges that goes beyond what current architectures actually realize.
A more modest but still interesting claim should be : Existing architectures (within a certain class) often fail to implement the communication range they theoretically permit.
Questions for authors
. what can you say about re-wiring approaches based on your proposed interaction profiles ? . are there real-world tasks where such long range interactions do make a clear difference (and are not solved by simply adjusting the receptive field) ?
Minor comments for the manuscript :
please add a bit of context at the beginning of Section 2, it reads like a list of definitions
Theorem does not tell us anything about the fixed point H^star, in particular one may want to stop strictly before this point because it is non-informative ?
section 3.3 and theorem 2 really need more context. it is difficult to understand how the various quantities involved relate to the paper