acceptodds
Under review as a conference paper at ICLR 2027

Transformer Neural Processes with growing receptive field for uncertainty prediction over clustered observations

Abstract

Neural Processes (NPs) are neural-network-based and Gaussian Process (GP)-inspired models for uncertainty prediction, yet most NP backbones are designed and evaluated under approximately uniform observations. Real-world data are rarely so ideal: measurements collected across the Earth are often clustered, with dense sampling near accessible regions and gaps elsewhere. Recent transformer NPs restrict attention to small local windows for efficiency and report little loss in accuracy; we show, however, the size and structure of the receptive field can affect predictive performance. We propose a growing-window processor that enlarges the attention window layer by layer and appends a global layer to capture both short-range and long-range dependencies. On synthetic Gaussian random fields and real-world climate data (ECA&D), our model achieves better predictive log-likelihood on clustered tasks than the baselines with local windows, indicating that clumped observations benefit from combining local and global receptive fields.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
neural process, probabilistic machine learning, clustered observations, transformer, shifted-window transformer, translation equivariance

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_EGgvOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates the predictive performance of Transformer Neural Processes (TNPs) under clustered observations. The authors argue that fixed local attention windows may limit the model's ability to leverage long-range information. To address this limitation, they propose Growing-Swin-TNP (G-Swin-TNP), which progressively increases the attention window size across Swin Transformer layers within the Gridded TNP framework, followed by a final global-attention Transformer layer. The authors claim that larger receptive fields are particularly beneficial for clustered observations and that the proposed progressive local-to-global architecture outperforms purely local or global attention architectures on real-world climate data.

Critical strengths

1.The paper studies Neural Processes under clustered observations, a practically important setting for spatial and environmental data where measurements are often highly non-uniform. The proposed G-Swin-TNP introduces a clear local-to-global inductive bias by progressively enlarging the attention window across processor layers while preserving the fixed grid-token structure of Gridded TNPs. The design integrates naturally with the existing framework and provides a conceptually straightforward way to incorporate multiple receptive-field scales without introducing a substantially more complicated architecture. 2.The proposed architecture performs competitively on the synthetic Gaussian random-field experiments and achieves the best NLL and RMSE among the evaluated Transformer-based models on the E-OBS climate dataset. These results provide evidence that G-Swin-TNP as an overall architecture is effective in the evaluated settings.

Critical weaknesses
  1. The global-layer ablation does not isolate the effect of global receptive fields. In the 2D experiments, the entire grid has a spatial resolution of (32\times32). Similarly, in the climate experiments, the final local attention window has a size of (4\times32\times32), covering the entire grid. Consequently, the model already performs full-grid self-attention before the final global Transformer layer. The ablation in Appendix D.3, which compares models with and without the global layer, therefore evaluates the effect of adding another global-attention block rather than isolating the contribution of global versus local receptive fields.
  2. From the Introduction onward, the paper motivates its approach through clustered observations, where context points are concentrated in a few regions while target points lie in sparsely observed areas between clusters. This motivates the need for larger receptive fields. However, the experiments do not establish whether the benefits of growing receptive fields are specific to, or substantially more pronounced under, clustered observations compared with other observation distributions.
  3. G-Swin-TNP modifies not only the processor but also the decoder: the number of nearest neighbors used by the decoder varies with the processor's attention window size, whereas TNP and Swin-TNP consistently use a fixed (3^d) nearest neighbors. As a result, the reported performance differences conflate changes in the processor's receptive field with changes in the decoder's locality bias. The current experiments do not establish whether the improvements primarily result from the growing-window processor or the modified decoder inductive bias.
  4. The experiments primarily compare fixed local windows, progressively growing windows, and full attention. However, important baselines are missing, including fixed medium or large windows, reversed window-size schedules, and compute- or parameter-matched controls. Consequently, the results support the conclusion that small fixed windows can be insufficient, but do not establish that progressively growing windows are superior to alternative multiscale configurations or larger fixed windows.
  5. The computational complexity analysis requires clarification and correction. Table 1 reports the complexity of fixed-window Swin-TNP as (O(2L\omega^2)), without explicitly accounting for the total number of tokens (T). For window-based attention with (m) tokens per window, there are approximately (T/m) windows, yielding an overall attention complexity of (O(Tm)), rather than (O(m^2)) for a single window. A similar issue appears to affect the geometric-series derivation for local layers in Eq. (3). Furthermore, the climate experiments enlarge attention windows along only two spatial dimensions, whereas the theoretical derivation assumes scaling across all (d) input dimensions. This discrepancy between the theoretical analysis and the implemented architecture should be addressed.
  6. The proposed progressive local-to-global attention mechanism is closely related to existing approaches that combine local, global, and multiscale attention. The paper would benefit from a more thorough discussion of relevant prior work, particularly:
    • Focal Attention for Long-Range Interactions in Vision Transformers
    • CrossFormer: A Versatile Vision Transformer Hinging on Cross-scale Attention
    • CrossFormer++: A Versatile Vision Transformer Hinging on Cross-scale Attention A clearer comparison with these methods is needed to establish the distinctive methodological contribution of G-Swin-TNP beyond adapting existing multiscale attention strategies to the Neural Process setting.
Questions for authors

Please refer to Weaknesses.

3 comments

We highly appreciate your detailed feedback on our work. We have addressed your comments below.

Critical Weaknesses:

  1. We agree with this comment that our ablation study on the effect of the global layer is not a fair comparison, but the last local attention window of full size has the Swin layer structure and hence is different in its structure from the global layer. So it is expected that they may have different effect on performance. We will consider it in our future work and thank the reviewer for the comment.
  2. Although not provided in the submitted manuscript, we confirmed through experiments that the performances of TNP, Swin-TNP, and G-Swin-TNP on tasks with uniformly distributed contexts are quite similar, with Swin-TNP and G-Swin-TNP not underperforming TNP. These experiments were run on the synthetic GRF dataset with length scale 0.10.1 and the E-OBS dataset over latitude/longitude [35,50]/[10,25][35,50]/[10,25]. The performance gap under clustered contexts can be attributed to the sparsity of observations: clumped contexts are concentrated in a few regions and extremely sparse elsewhere, so targets in the sparse regions must rely on distant contexts. A larger receptive field is therefore needed to reach the relevant observations, which is precisely where Swin-TNP's small fixed window falls short. Moreover, we do not know the distributions of clusters in reality, our growing receptive fields can capture diverse ranges of context data gradually and hence have some benefits for variously clustered real datasets like E-OBS.
  3. As commented by the reviewer, G-Swin-TNP's decoder adopts different neighbor counts according to the window sizes, while Swin-TNP and TNP adopts the fixed neighbor count, 3d3^d. Please note that this coupling is intended since it is natural that the decoder considers the neighbors in the corresponding receptive window of the processor it follows. Also we note that the number of neighbors in the G-Swin-TNP never exceeds 3d3^d by design. Nonetheless, we agree that isolating the processor's effect by adopting the varying neighboring count to Swin-TNP or TNP is an appropriate control, but due to time limitation for rebuttal we plan to include the results in the final version if we are allowed to submit it.
  4. Additional controls--fixed medium/large windows, reversed window-size schedules, and parameter- or compute-matched variants--would further characterize the effect of the receptive-field schedule, and we plan to add them as further study. We emphasize, however, that G-Swin-TNP, Swin-TNP, and TNP already have the same number of parameters, so our comparison is parameter-matched (parameter counts are reported in Appendix A.4 of the revised manuscript). We do not consider compute-matched variants because matching compute would change the number of layers, which can itself substantially affect performance.
  5. We thank the reviewer for pointing it and have corrected and generalized the complexity analysis in the revised manuscript.
  6. We compare G-Swin-TNP with the suggested models in terms of multi-scale attention. The Focal Attention (ref 1.) and CrossFormer (ref 2.) capture long- and short-range interactions by merging tokens at multiple scales, and CrossFormer++ (ref 3.) uses coarse attention for long-range interactions. In contrast, G-Swin-TNP uses full attention at each window size. This full attention design avoids merging or loosely attending to tokens, which could otherwise over-smooth the representation or propagate information from under-informed tokens through long-range dependencies. The clustered contexts in our setting are sparse and unevenly distributed. So, when computing per-location grid token representations, information is more likely to be drawn from tokens that do not yet encode sufficient contexts, which we expect to degrade performance. For this reason we think our growing receptive fields with full attention for each window size can have some benefits. Since Focal Attention, CrossFormer, and DrossFormer++ have lower computational cost, empirically verifying whether their approximations harm the performance would be valuable, which we leave for future work.

References:

  1. Yang, Jianwei, et al. "Focal attention for long-range interactions in vision transformers." Advances in neural information processing systems 34 (2021): 30008-30022.
  2. Zhang, Yunhao, and Junchi Yan. "Crossformer: Transformer utilizing cross-dimension dependency for multivariate time series forecasting." The eleventh international conference on learning representations. 2023.
  3. Wang, Wenxiao, et al. "Crossformer++: A versatile vision transformer hinging on cross-scale attention." IEEE Transactions on Pattern Analysis and Machine Intelligence 46.5 (2023): 3123-3136.

I would like to thank the authors for their hard work. The additional E-OBS regions, the Swin+Global-TNP baseline, and the corrected complexity analysis have addressed several of my concerns. However, some issues remain unresolved, particularly the confounding between the processor window schedule and the decoder neighborhood schedule, as well as the difficulty of isolating the effect of the growing receptive-field design itself. I would like to raise my score from 1 to 2.

AuthorsOpenReview ↗

Acknowledgement and Gratitude

We sincerely thank you for taking the time to review our work and for providing thoughtful and constructive feedback. We also appreciate your decision to raise your score.

Reviewer_BC77OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates how the receptive field of Transformer Neural Processes affects predictions when observations are spatially clustered rather than uniformly distributed. The authors propose Growing-Swin-TNP, which gradually increases the attention window size across transformer layers and adds a final global attention layer. The idea is to combine local information with longer-range dependencies while avoiding the computational cost of using global attention throughout the network.

Critical strengths

I think the main strength of the paper is that it investigates a relevant and somewhat overlooked aspect of neural processes. In practice, observations are often highly clustered, and it is interesting to study how this affects the choice of attention mechanism.

Critical weaknesses

My main concern is that the proposed architectural modification is relatively simple, and the experiments do not clearly establish which part of it is responsible for the improvements. In particular, the model combines progressively growing attention windows with a final global attention layer, making it difficult to determine whether the growing schedule itself provides a benefit over simpler combinations of local and global attention.

Questions for authors

Could the authors provide a quantitative comparison against a model using fixed-size local attention windows followed by a global attention layer? Since the largest growing window already covers the entire token grid, I am particularly interested in whether progressively increasing the window sizes provides a measurable benefit beyond simply combining local and global attention.

1 comment

We greatly appreciate the time and effort you have dedicated to this review. We answer to your comments in the below.

Critical weakness:

We conducted an additional ablation study on a processor consisting of five fixed-size Swin layers followed by a global attention layer, included in Appendix D.3 of the revised manuscript. As representative settings, we evaluated it on the synthetic GRF dataset with length scale 0.10.1 and on the E-OBS daily maximum-temperature dataset over latitude/longitude [35,50]/[10,25][35,50]/[10,25]. The comparison of this model with G-Swin-TNP can show the isolation effect on whether the growing-window schedule is beneficial beyond simply combining local and global attention as the reviewer requested. The results show that fixed local windows with a single final global layer underperform our G-Swin-TNP, indicating that the growing-window schedule itself contributes to performance improvement, even though the final window covers the full grid as commented by another reviewer.

Question:

We addressed this under Critical weakness.

Reviewer_H8cEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies transformer-based Neural Processes under clustered (non-uniform) observations and asks when a local receptive field becomes problematic and when a global one helps. The authors propose Growing-Swin-TNP (G-Swin-TNP), a processor that stacks Swin-Transformer layers with window sizes growing layer by layer and appends a single global Transformer layer, on top of the Gridded TNP encoder/decoder. The intuition is to combine short-range and long-range dependencies more cheaply than stacking global layers. The method is evaluated on a 2D synthetic Gaussian random field and on real E-OBS daily-temperature data.

Critical strengths

The paper has a few strengths that I will outline below.

Strengths:

  1. The motivating question - whether attending only to local observations hurts prediction when the observations are clumped - is well-posed and relevant, and the analogy to local/sparse GP approximation is reasonable.
  2. The growing-window processor is a sensible and cost-aware design, and the computational-complexity analysis (showing the local layers add only a constant factor) is clear.
  3. The experimental protocol for the clustered setting is thoughtfully constructed (controlled number/spread of clusters, Mahalanobis/Euclidean distance analysis of target-to-context density).
Critical weaknesses

However, the paper has a few major and minor weaknesses.

Major weaknesses:

  1. The central claim has thin empirical support. On the synthetic GRF (Table 2), the proposed G-Swin-TNP is not the best model - the globally-attending TNP matches or outperforms it at both length scales - and the method only comes out ahead on the single real-world E-OBS dataset (Table 3). With just one dataset supporting the main thesis, and no second clustered real-world task, it is hard to tell whether the advantage is general or specific to the geography of the chosen region (which the authors themselves raise as a confound).
  2. The contribution is incremental and not clearly delineated. The encoder, decoder and grid-token machinery are taken from Gridded TNP, the Swin layers from Liu et al. (2021), and the TE-bias from Ashman et al. (2024a); the genuinely new element is the growing-window stacking plus one appended global layer. The paper would be much stronger if it isolated the effect of the growing schedule itself (e.g., growing vs. fixed-window vs. a few plain global layers at matched cost), rather than comparing only against TNP and Swin-TNP.
  3. The gains on E-OBS are modest (NLL −0.361 -> −0.395, RMSE 0.206 -> 0.201) and reported without a clear significance assessment relative to the architectural and training-cost overhead.

Minor weaknesses:

  1. The paper is not easy to read: the construction is described largely by reference to prior work (Appendices D.1/D.2), so the main body does not fully convey what is new versus inherited.
  2. Several design choices (window-growth factor, number of local vs. global layers, nearest-neighbor counts) are fixed without ablation, so it is unclear how sensitive the results are to them.
Questions for authors

I would like to see especially the responses to the issues mentioned as weaknesses. Besides, I have a few additional questions:

  1. Could you clarify precisely which component is the contribution of this work, relative to Gridded TNP and the Swin-Transformer, and quantify its effect in isolation?
  2. Can the main result be confirmed on at least one more clustered real-world dataset, so the conclusion does not rest on a single table?
  3. Is there an ablation on the window-growth schedule (growing vs. fixed window, and with/without the final global layer) at matched compute?
3 comments

We appreciate your valuable time and feedback. Here are our answers to your comments.

Major weaknesses:

  1. We conducted additional experiments on two other regions of the E-OBS dataset. Although they are also daily maximum-temperature E-OBS data, the station distributions and underlying geography differ, making them a meaningful validation on our thesis. The outcome is consistent: NLL is the smallest for G-Swin-TNP, with Swin-TNP and TNP following. Even though our model, G-Swin-TNP, is not the best on the synthetic GRF, its performance is comparable even for the synthetic GRF. Furthermore, G-Swin-TNP still attains the smallest NLL for all theree regions on the real-world E-OBS dataset including our new experiments, confirming its advantage across regions. The new experimental results are reported in the revised manuscript, Table 3 of Section 4.2.
  2. As the reviewer agrees, one of our contributions is the introduction of growing receptive fields for local windows. Another contribution that we find is that the performance of a model can depend on the sampling distribution of the context data, e.g., clustered or not, hence growing receptive fields would be beneficial because they can capture the relations of context data in diverse regions gradually. Since two baselines (Swin-TNP and TNP) adopt the same TE-bias and grid encoders/decoders as our model (G-Swin-TNP), we think that our experimental results can show the isolated effect of our growing-window stacking plus one appended global layer.
  3. Although the gains on E-OBS are modest, they are consistent across all regions we tested, as noted in our response to Critical weakness 1. Training time is reported in Table 1 (Section 3.2), and we additionally report the parameter counts in Appendix A.4, which are identical for all models, G-Swin-TNP, Swin-TNP, and TNP.

Minor weaknesses:

  1. We have added a brief summary of the grid encoder, the paragraph in red color in Section 3.1 of the revised manuscript, so that the readers can grasp the overall structure without relying on the appendices.
  2. Owing to the limited rebuttal period, we could not consider all the factors that the reviewer mentioned. These design choices can be examined quantitatively in future work.

Questions:

  1. We clarify in the introduction of the revised manuscript that our structural contribution is the growing-window schedule. Please also refer to our response to Critical weakness 2.
  2. We address this under Critical weakness 1.
  3. A window-growth comparison at matched compute will be considered in future work. We note, however, that G-Swin-TNP, Swin-TNP, and TNP are set to have the same number of parameters (provided explicitly in the revised manuscript), since the parameter count can affect performance, whereas matching computational complexity would instead require differing parameter counts, which results in significant changes in the architectures of all models and hence we do not consider it in the current manuscript.

I thank the authors for their clarifications on some of my questions and found weaknesses. However, after reading the responses and the issues indicated by other reviewers, I decided to keep my score.

AuthorsOpenReview ↗

Acknowledgement and Gratitude

We appreciate the time and effort you have taken to review our manuscript. We once again thank you for your constructive comments, which will help us further improve our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.