acceptodds
Withdrawn from ICLR 2027

Not All Interactions Need to Be Global: Query-Centered Multi-Scale Spatial Observation for Neural Operators

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Neural operators provide resolution-independent surrogates for partial differential equations, but accurately modeling spatial dependencies across different ranges remains challenging. While local structures and broad spatial configurations jointly determine the solution, existing operators typically rely on global representations or extensive spatial interactions to acquire long-range context. We propose theMulti-Scale Spatial Observation Operator (MSO), which reformulates long-range dependency modeling as a query-centered observation problem. MSO constructs a hierarchical spatial representation and propagates information independently along high-to-low and low-to-high resolution paths. For each query coordinate, it extracts spatially aligned features from all resolution levels and propagation directions, and adaptively aggregates these observations through scale-wise attention. This design enables broad spatial observation while maintaining sparse query-side interaction, avoiding dense query-to-token mixing. Experiments on diverse PDE benchmarks covering incompressible and compressible fluid dynamics, shallow-water flows, and diffusion-reaction dynamics demonstrate strong predictive performance with a substantially smaller parameter footprint than representative neural operators. Further analyses show that different query locations favor different observation scales, supporting the effectiveness of query-adaptive multi-scale spatial observation.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Neural Operators, Operator Learning, Partial Differential Equations, Multi-Scale Spatial Observation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_HpciOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MSO is a neural operator that combines a multi-scale feature pyramid, local attention, and a query-based decoder. The paper reports improvements on five 2D PDE benchmarks, with different levels of improvement across the tasks.

The experiments show that this combination of multi-scale features and sparse query decoding can achieve good accuracy on these 2D PDE tasks.

Critical strengths
  1. Each query only looks at 2(L+1)2(L+1) features, instead of all the spatial tokens.

  2. The design makes sense. It puts together some common methods, like multi-scale features, bidirectional propagation, Window Attention, and Cross-Attention. The results are quite good.

Critical weaknesses

MSO does not really have lower computational complexity than single-scale Local Attention. Instead, it adds multi-scale encoding and feature propagation to get a larger receptive field. This makes sense from an architecture design perspective, but it also brings extra computational cost.

The experimental results are quite good, but I think there are still some important things missing. First, there is no fair comparison under the same number of parameters, FLOPs, and training budget. The paper also does not provide enough information about the actual training and inference efficiency. Most of the complexity analysis focuses on the Query Decoder, rather than the whole model. Also, some datasets require millions of training updates, so having fewer parameters does not necessarily mean the model is more efficient.

Another concern is the generalization ability. The paper does not test the model across different resolutions, so it's hard to say whether it is really resolution-independent. All experiments are also based on regular 2D grids, without testing more complex geometries, unstructured meshes, or 3D problems. Besides, there are no experiments with multiple random seeds, so we don't know how stable these improvements are.

One more thing is that the multi-scale hierarchy itself is fixed. The model does not actually learn how to divide the spatial scales. What is adaptive is only how the model combines features from different scales through attention. So I think the paper should be more careful when discussing the advantages of its adaptive multi-scale design.

Questions for authors

Would MSO still perform better than a deeper or wider single-scale Window Attention model if they have the same FLOPs, GPU memory usage, and number of parameters?

Reviewer_QcRuOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper presented an architecture called Multi-Scale Spatial Observation Operator (MSO) to learn solution operators of spatiotemporal PDE problems. MSO incorporated a bidirectional course-to-fine and fine-to-course convolutional feature pyramid with local window self-attention, then uses coordinate-centered cross-attention to select among co-located features from various scales. There are also several extra embeddings used in the information processing. Overall 3 benchmarks (Incompressible Navier-Stokes, Compressible Navier-Stokes, and Diffusion-Reaction), the proposed model demonstrated substantial improvement in terms of empirical performance. Overall I found this work mostly engineering efforts.

Critical strengths
  1. The proposed work showed clear improvements over a few SOTA models empirically on challenging benchmarks.
  2. The paper has a clear motivation which separates the processing/interaction of multi-scale information from the extraction of multi-scale information.
Critical weaknesses
  1. I find the choice of benchmarks a bit confusing, especially since the main point of the paper is to show how the processing and interaction of multi-scale spatial information affect performance. The chosen benchmarks all involve autoregressive rollouts, where temporal error accumulation can also have a large effect on the final performance. The authors even acknowledge that this is not addressed by the proposed method. I think it would have been easier to understand the contribution if they had also tested on problems where the effect of spatial processing could be isolated more clearly, or at least separately reported one-step performance.

  2. Despite the authors’ efforts in conducting ablation studies, it is still unclear to me where the substantial performance gain comes from. Using local and multi-scale information is not itself new; previous approaches such as WNO and CNO already incorporate similar ideas in different ways. However, the improvement reported by this paper is very large, and even the simplified variants remain stronger than most of the baselines. This makes me wonder how much of the gain comes from the proposed query-centered multi-scale interaction and how much comes from having a stronger base architecture and training setup.

  3. The paper does not discuss the training or inference time of the proposed approach. Although the final query only attends to a small number of tokens, the multi-scale representations are processed sequentially across resolution levels and along two propagation directions. This sequential pipeline may introduce a nontrivial computational cost that is not captured by the query-side complexity analysis. I think an end-to-end runtime comparison is important for understanding whether the proposed method is also efficient in practice.

Questions for authors
  1. Can the authors include a training and inference time comparison and discussion?

  2. What is the “closest” baseline model? Based on the ablation table, even the substantially simplified versions still perform significantly better than most other baselines. Adding a comparison with the closest base model would help decouple how much of the gain comes from the newly proposed interaction processing and how much comes from the strength of the base model itself.

  3. Does this approach require the input points to be uniformly sampled on a regular grid? The method accepts arbitrary output coordinates, but the convolution, pooling, upsampling, and window operations seem to assume regular-grid inputs.

  4. Can the authors add experiments on zero-shot super-resolution? The introduction places the work in the resolution-independent neural-operator setting, and the method accepts arbitrary output coordinates in principle. However, the current experiments use fixed-resolution inputs and full-grid evaluation.

Suggestion for future work:

If possible, please report multi-seed results in the revised version.

Reviewer_rH63OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents a multi-scale neural operator, which uses a series of downsampled inputs to learn both short- and long-range interactions together through the same query-centered observation unit. The model uses windowed attention covering a 4x4 block of pixels. On downsampled scales, this 4x4 block corresponds to a much larger area in the original pixel space. The model has several other components, which encode temporal history, encode information from low to high resolutions, encode information from high to low resolutions, and an aggregator for features aligned at the same spatial location. The authors train a suite of models on three problems, showing competitive results.

Critical strengths
  1. The authors have strong results, beating much larger foundation models with many fewer parameters.
  2. Attention across large numbers of tokens can indeed become very expensive -- the authors show how this may be amortized.
Critical weaknesses
  1. The authors' central claim is that global information is not necessary. Their own experiments, however, demonstrate that global or near-global information is necessary to achieve optimal results. This is evident in Figure 4, where error decreases with every additional coarser grid size--until coarsest size 4x4, which effectively captures global information.
  2. The benchmark problem results match with those reported in the cited references. The task here is to predict the next 10 time-steps given the previous 10 timesteps. The authors report a very low error; however, figures 8 and 9 show quite a large amount of error already by step 4. Can the authors verify the 1.18% rL2 error is for the entire trajectory and fairly evaluated against the baselines? It seems like this might be a single-step prediction error.
  3. The architecture shares many similarities with a U-Net. It produces features across many levels and fuses them with attention. Despite this, there are no U-Net baselines. U-NO is even cited in the related work, but not tested as a baseline.
  4. The authors only evaluate the model against other baselines on 3 problems. This set is far too limited to make any general claims about the performance of this architecture.
Questions for authors
  1. Is there a path forward to apply this to non-euclidean / non-grid domains?
  2. Global token fusion should, in theory, be able to supersede the results of this approach. In Table 4, we see that it loses clearly. What do the authors attribute this to?
  3. How does this approach scale with larger model sizes and larger datasets?
  4. Is this approach suitable for pre-training a large model to target efficient fine-tuning for new tasks?
  5. Should we expect this approach only to be suitable for locally-dominated PDEs? How does it perform on PDEs with global interactions?
  6. Line 157 states that the temporal history is mapped to a base spatial representation. Are the history steps in Table 5 consistent across all models/baselines?
  7. How does this model perform on PDEs with strong (stronger than INS) nonlocal coupling?
  8. I don't see any off-grid or cross-resolution tests -- is resolution independence affected by the multiscale interactions?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.