- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper investigates the predictive performance of Transformer Neural Processes (TNPs) under clustered observations. The authors argue that fixed local attention windows may limit the model's ability to leverage long-range information. To address this limitation, they propose Growing-Swin-TNP (G-Swin-TNP), which progressively increases the attention window size across Swin Transformer layers within the Gridded TNP framework, followed by a final global-attention Transformer layer. The authors claim that larger receptive fields are particularly beneficial for clustered observations and that the proposed progressive local-to-global architecture outperforms purely local or global attention architectures on real-world climate data.
Critical strengths
1.The paper studies Neural Processes under clustered observations, a practically important setting for spatial and environmental data where measurements are often highly non-uniform. The proposed G-Swin-TNP introduces a clear local-to-global inductive bias by progressively enlarging the attention window across processor layers while preserving the fixed grid-token structure of Gridded TNPs. The design integrates naturally with the existing framework and provides a conceptually straightforward way to incorporate multiple receptive-field scales without introducing a substantially more complicated architecture. 2.The proposed architecture performs competitively on the synthetic Gaussian random-field experiments and achieves the best NLL and RMSE among the evaluated Transformer-based models on the E-OBS climate dataset. These results provide evidence that G-Swin-TNP as an overall architecture is effective in the evaluated settings.
Critical weaknesses
- The global-layer ablation does not isolate the effect of global receptive fields. In the 2D experiments, the entire grid has a spatial resolution of (32\times32). Similarly, in the climate experiments, the final local attention window has a size of (4\times32\times32), covering the entire grid. Consequently, the model already performs full-grid self-attention before the final global Transformer layer. The ablation in Appendix D.3, which compares models with and without the global layer, therefore evaluates the effect of adding another global-attention block rather than isolating the contribution of global versus local receptive fields.
- From the Introduction onward, the paper motivates its approach through clustered observations, where context points are concentrated in a few regions while target points lie in sparsely observed areas between clusters. This motivates the need for larger receptive fields. However, the experiments do not establish whether the benefits of growing receptive fields are specific to, or substantially more pronounced under, clustered observations compared with other observation distributions.
- G-Swin-TNP modifies not only the processor but also the decoder: the number of nearest neighbors used by the decoder varies with the processor's attention window size, whereas TNP and Swin-TNP consistently use a fixed (3^d) nearest neighbors. As a result, the reported performance differences conflate changes in the processor's receptive field with changes in the decoder's locality bias. The current experiments do not establish whether the improvements primarily result from the growing-window processor or the modified decoder inductive bias.
- The experiments primarily compare fixed local windows, progressively growing windows, and full attention. However, important baselines are missing, including fixed medium or large windows, reversed window-size schedules, and compute- or parameter-matched controls. Consequently, the results support the conclusion that small fixed windows can be insufficient, but do not establish that progressively growing windows are superior to alternative multiscale configurations or larger fixed windows.
- The computational complexity analysis requires clarification and correction. Table 1 reports the complexity of fixed-window Swin-TNP as (O(2L\omega^2)), without explicitly accounting for the total number of tokens (T). For window-based attention with (m) tokens per window, there are approximately (T/m) windows, yielding an overall attention complexity of (O(Tm)), rather than (O(m^2)) for a single window. A similar issue appears to affect the geometric-series derivation for local layers in Eq. (3). Furthermore, the climate experiments enlarge attention windows along only two spatial dimensions, whereas the theoretical derivation assumes scaling across all (d) input dimensions. This discrepancy between the theoretical analysis and the implemented architecture should be addressed.
- The proposed progressive local-to-global attention mechanism is closely related to existing approaches that combine local, global, and multiscale attention. The paper would benefit from a more thorough discussion of relevant prior work, particularly:
- Focal Attention for Long-Range Interactions in Vision Transformers
- CrossFormer: A Versatile Vision Transformer Hinging on Cross-scale Attention
- CrossFormer++: A Versatile Vision Transformer Hinging on Cross-scale Attention A clearer comparison with these methods is needed to establish the distinctive methodological contribution of G-Swin-TNP beyond adapting existing multiscale attention strategies to the Neural Process setting.
Questions for authors
Please refer to Weaknesses.
3 comments
I would like to thank the authors for their hard work. The additional E-OBS regions, the Swin+Global-TNP baseline, and the corrected complexity analysis have addressed several of my concerns. However, some issues remain unresolved, particularly the confounding between the processor window schedule and the decoder neighborhood schedule, as well as the difficulty of isolating the effect of the growing receptive-field design itself. I would like to raise my score from 1 to 2.
Acknowledgement and Gratitude
We sincerely thank you for taking the time to review our work and for providing thoughtful and constructive feedback. We also appreciate your decision to raise your score.
We highly appreciate your detailed feedback on our work. We have addressed your comments below.
Critical Weaknesses:
References: