acceptodds
Under review as a conference paper at ICLR 2027

Flow-Adaptive Neural Processes with multiscale Spatial Bias for Irregular Inputs

Abstract

Learning expressive function generation from irregular observations is challenging. Irregular observations can be random in available locations, order, and receiving numbers. To handle this problem, we introduce the Structured Flow Process (SFP), a novel generative neural process specifically designed to learn function priors through flow matching over finite-dimensional marginals and achieve efficient posterior inference. Our key innovation is Flow-adaptive Multihead Attention, which introduces multiscale spatial bias and flow-time conditioning into the velocity model to capture evolving correlations along the generative flow. SFP supports both prior modeling and posterior inference on irregular input sets and preserves the desired permutation invariance of the induced distributions. Empirically, SFP achieves strong performance across diverse prior-learning and posterior-inference tasks among synthetic stochastic processes, physical dynamical systems, and real geophysical fields. We further evaluate SFP on downstream applications, including tabular regression and Bayesian optimization with inputs of up to seven dimensions, demonstrating its scalability and broader applicability.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Neural Process, Flow Matching

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_6TL1OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Structured Flow Process (SFP), a flow-matching framework for learning function distributions from irregular observations. Its Flow-adaptive Multihead Attention combines time-conditioned interactions with multiscale spatial biases, enabling flexible prior generation, guided posterior inference, and permutation invariance.

Critical strengths
  • The paper addresses probabilistic function modeling under irregular observations, an important challenge for scientific machine learning.
  • FaMA combines flow-time modulation and multiscale spatial biases to capture evolving spatial dependencies.
  • SFP achieves competitive prior modeling and posterior inference results across synthetic, physical, and real-world datasets.
Critical weaknesses
  • The posterior sampling mechanism largely follows existing guidance-based methods, with limited methodological innovation.
  • Theoretical results establish permutation invariance but do not guarantee marginal consistency across different input sets.
  • Iterative flow-based inference is expensive, which makes it hard for practical deployment.
Questions for authors

See Weaknesses and below:

  • How does FaMA compare against established relative positional attention mechanisms under matched model capacity and computational budgets?
  • Would standard time conditioning combined with multiscale spatial bias achieve comparable performance without the proposed time-dependent QKV modulation?
5 comments
AuthorsOpenReview ↗

Response to Reviewer 6TL1

We sincerely thank the reviewer for the thoughtful review and for the time invested in evaluating our work. We are glad that the reviewer finds the problem important, the combination of flow-time modulation and multiscale spatial bias well suited to evolving spatial dependencies, and our results competitive across synthetic, physical, and real-world datasets.

We appreciate the reviewer's constructive comments. We address each point below.

Weakness

W1: Novelty of Posterior Sampling

We thank the reviewer for this comment. We clarify that posterior sampling is not claimed as a main contribution (Section 1); our contributions are SFP as a flow-based generative NP over irregular inputs and the FaMA block. Building on training-free guidance (Chung et al., 2023; Pokle et al., 2024) is a deliberate choice that decouples the prior from the observation model:

  • One prior, arbitrary contexts. A single unconditional model handles any context–query partition and enables generation of prior samples, yet matches FMNP (Hamad & Rosenbaum, 2025), which is trained specifically for conditional prediction.
  • New likelihoods without retraining. To verify this during the rebuttal, we conducted additional binary classification experiments on three benchmarks, each with three context sizes and 10 random datasets. Incorporating a Bernoulli likelihood into the guidance of the same trained prior, SFP achieves lower NLL than GP classification in all nine settings with better accuracy. We kindly refer the reviewer to our response to Reviewer ZWEN, W2, Table R1.

We have clarified the attribution of the introduced guidance in Section 3.1.

W2: Marginal Consistency

We appreciate the reviewer’s valuable comment on MC. We clarify the scope of our theoretical results: SFP guarantees permutation invariance of its finite-dimensional distributions but not exact MC; MC is encouraged approximately by training on randomly sampled subsets. Many expressive generative NPs, such as NDP (Dutordoir et al., 2023), GeoNDP (Mathieu et al., 2023), and FMNP (Hamad & Rosenbaum, 2025), share this limitation. Exact MC by construction is available only for restricted model classes such as GPs and latent-variable NPs with conditionally independent decoders (Garnelo et al., 2018), whose structural assumptions limit the dependencies they can represent.

We would like to further clarify two points:

  • (1) The learning target of SFP is exactly consistent. The source N(0,I)\mathcal{N}(0, I) is i.i.d. across input points, and the conditional OT path interpolates each point independently. Hence, for nested sets XS⊂XLX_S \subset X_L, the target path on XSX_S is exactly the marginal of the target path on XLX_L. Under the regularity conditions of Assumption C.1, the exact flow-matching velocity therefore induces a consistent model, and any inconsistency of SFP comes only from the approximation error of vθv_\theta. We have added this result to Section 3.3, with the formal statement and proof in Appendix C.2 (Proposition C.5).

  • (2) Empirically, the remaining inconsistency is small. Appendix F.4 (Table 11) compares samples drawn directly on XSX_S with samples drawn on XLX_L and then restricted to XSX_S. SFP (102×10^2\timesMMD 4.15, SWD 0.11) is comparable to SPL-OFM (4.02, 0.13), close to the exact GP (2.87, 0.08), and well below NDP (11.92, 0.20). To further assess this gap, we conducted new MC tests on the real-world wind-field task and on larger GP input sets. SFP stays close to the finite-sample reference in all settings, and the gap does not grow with the input-set size. We kindly refer the reviewer to our response to Reviewer ZWEN (Q1, Table R3) for details.

We thank the reviewer for this important comment, which helped us clarify the scope of our theoretical results and strengthen the empirical evidence for marginal consistency.

W3: Inference Cost

We thank the reviewer for this comment, which we also acknowledge in our Limitations. Iterative sampling is the standard inference mechanism of continuous generative NPs such as NDP, FMNP, and SPL-OFM. It is the price of posteriors that capture correlated and non-Gaussian structure (e.g., the bimodal posterior in Appendix F.12), which single-pass NPs with Gaussian predictives cannot represent. Within this class, SFP improves the cost–quality trade-off:

  • Few steps suffice. Most of the gain is reached within 20–50 Euler steps (Appendix F.6, Table 13). With only 20 steps, SFP already attains lower NCRPS than TNP, the strongest single-pass baseline.
  • Each step is efficient. The FlashAttention-compatible spatial bias reduces peak memory (Table 6), and posterior guidance is computed via an efficient vector–Jacobian product (VJP) computation.
AuthorsOpenReview ↗

Response to Reviewer 6TL1 (Continued)

  • SFP is cheaper than other generative NPs. SFP uses 100 Euler steps by default, compared with 1,000 diffusion steps for NDP and 22–4×1044 \times 10^4 SGLD iterations for SPL-OFM posterior sampling. We kindly refer the reviewer to our responses to Reviewer ZWEN (Q4, Table R4) for the full cost table and Reviewer ynWL (W4, Table R2) for large-scale memory profiling. Since SFP is trained with standard flow matching, few-step samplers and distillation for flow models are directly applicable, which we leave for future work.

We thank the reviewer for raising this point, which motivated us to profile SFP at larger scales and further demonstrate its scalability and cost advantages over other generative NPs. We have added these results to Appendix D.3.

Questions

Q1: Comparison with Relative Positional Attention

We thank the reviewer for this question. FaMA's spatial bias differs from relative positional attention in three respects:

  • Input type. Relative position encodings such as Shaw et al. (2018), T5 biases (Raffel et al., 2020), and ALiBi (Press et al., 2022) are designed for discrete token indices in 1D sequences. FaMA operates on continuous, multi-dimensional coordinates of irregular sets.

  • Inductive bias. Since exp⁡(Bij(h))=exp⁡(−∥xi−xj∥2/dℓh2)\exp(B^{(h)}_{ij}) = \exp(-\|x_i - x_j\|^2 / d\ell_h^2) is an RBF kernel, each FaMA head computes content attention reweighted by an RBF kernel with length scale ℓh\ell_h: Aij(h)∝exp⁡(⟨qi,kj⟩/Dh) kℓh(xi,xj)A^{(h)}_{ij} \propto \exp(\langle q_i, k_j \rangle / \sqrt{D_h})\, k_{\ell_h}(x_i, x_j). This matches the distance-decaying correlations of stationary processes, and the log-spaced heads cover several length scales. RoPE (Su et al., 2024) instead rotates queries and keys by position, which modulates content similarity but does not impose a distance decay.

  • Efficiency. FaMA's squared-distance bias is decomposable: −∥xi−xj∥2-\|x_i - x_j\|^2 expands exactly into inner products of augmented queries and keys (See Eq. 4 in manuscript). FaMA therefore runs on standard FlashAttention without materializing the N×NN \times N bias, which keeps memory linear in NN and scales to 65,536 points (see our response to Reviewer ynWL, W4, Table R2). By contrast, a linear distance bias ∥xi−xj∥\|x_i - x_j\| on continuous multi-dimensional coordinates is not decomposable and requires the full N×NN \times N bias matrix.

  • Existing evidence. The ablations in our paper isolate the effect of the spatial bias: adding the multiscale bias on top of temporal modulation improves MMD and NCRPS on mixture GPs and NS (Figure 6), and the multiscale bias outperforms a single-scale bias on all four metrics on mixture GPs (Appendix F.8, Table 15(a)). Its FlashAttention compatibility substantially reduces peak memory, with savings that grow with the input-set size (Table 6), and FaMA applies directly to higher-dimensional inputs in UCI regression and BO.

Q2: Standard Time Conditioning with Multiscale Bias

We thank the reviewer for this question. Figure 6 compares input–time concatenation, temporal modulation, and temporal modulation with multiscale bias, but it does not include standard time conditioning combined with the multiscale bias. To address the reviewer's concern, we complete the factorial design below, and also include adaLN conditioning (Peebles & Xie, 2023) as an alternative baseline.

Why time-dependent attention is needed. For a GP prior y1∼N(0,K)y_1 \sim \mathcal{N}(0, K) under the linear OT path, E[y1∣yt]=tK((1−t)2I+t2K)−1yt\mathbb{E}[y_1 \mid y_t] = tK\left((1-t)^2 I + t^2 K\right)^{-1} y_t. The optimal velocity therefore mixes function values across points with a matrix that changes along the flow, from broad kernel smoothing near t=0t = 0 to near-identity as t→1t \to 1. Standard conditioning modulates the token features shared by queries, keys, and values, whereas FaMA applies separate time-dependent modulations to each projection, making pairwise attention logits explicitly flow-dependent.

Table R2. Ablation of time conditioning with and without the multiscale spatial bias on 1D GP. MMD is the mean ± std over 10 sampling seeds. NCRPS is the mean ± std over 500 test functions.

Time conditioningMultiscale biasMMD ↓NCRPS ↓
Concatenation (basic)✗0.055 ± 0.0040.058 ± 0.012
Concatenation✓0.042 ± 0.0030.051 ± 0.008
adaLN✓0.039 ± 0.0040.050 ± 0.005
Temporal modulation✗0.042 ± 0.0060.054 ± 0.017
FaMA✓0.035 ± 0.0050.049 ± 0.004
AuthorsOpenReview ↗

Response to Reviewer 6TL1 (Continued 2)

Results. The two components of FaMA are complementary. The multiscale bias consistently improves prior quality, but combining it with standard time conditioning (concatenation or adaLN) does not match FaMA. The gain of FaMA therefore does not come from the spatial bias alone: flow-dependent attention logits provide a further improvement. We have added these results to Appendix F.8 (Table 15(b)).


References

Chung, Hyungjin, et al. "Diffusion posterior sampling for general noisy inverse problems." ICLR. 2023.

Dutordoir, Vincent, et al. "Neural diffusion processes." ICML. 2023.

Garnelo, Marta, et al. "Neural processes." arXiv preprint arXiv:1807.01622. 2018.

Hamad, Hussen Abu, and Dan Rosenbaum. "Flow matching neural processes." NeurIPS. 2025.

Mathieu, Emile, et al. "Geometric neural diffusion processes." NeurIPS. 2023.

Peebles, William, and Saining Xie. "Scalable diffusion models with transformers." ICCV. 2023.

Pokle, Ashwini, et al. "Training-free linear image inverses via flows." TMLR. 2024.

Press, Ofir, Noah A. Smith, and Mike Lewis. "Train short, test long: Attention with linear biases enables input length extrapolation." ICLR. 2022.

Raffel, Colin, et al. "Exploring the limits of transfer learning with a unified text-to-text transformer." JMLR. 2020.

Shaw, Peter, Jakob Uszkoreit, and Ashish Vaswani. "Self-attention with relative position representations." NAACL. 2018.

Su, Jianlin, et al. "RoFormer: Enhanced transformer with rotary position embedding." Neurocomputing. 2024.

We hope that our responses and additional experiments address the reviewer's concerns. We are happy to respond promptly if the reviewer has any further questions. If the reviewer finds that the main concerns have been addressed, we would be grateful if the reviewer could reconsider the evaluation and score accordingly.

Reviewer_ynWLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes the Structured Flow Process (SFP), a flow-matching-based Neural Process (NP) that learns a stochastic-process prior over finite-dimensional marginals of irregularly sampled functions, and performs posterior inference via context-guided sampling. The main contribution is architectural: a Flow-adaptive Multihead Attention (FaMA) block with two components: (1) a temporal modulation that conditions the queries, keys, and values on the flow time, and (2) a multiscale spatial bias that encodes relative distances between input locations, using log-spaced length scales across attention heads. In addition, the authors reformulate the spatial bias as an augmented query–key inner product, making it compatible with FlashAttention without materializing the full pairwise bias matrix. On the theoretical side, the paper proves that the velocity model is permutation equivariant and that the induced distributions are therefore permutation invariant; marginal consistency is not guaranteed and is only evaluated empirically. Experiments compare SFP against conditional NPs (AttCNP, ConvCNP, LNP, TNP) and generative function-space models (NDP, SPL-OFM). SFP is most closely related to NDP and SPL-OFM, which also use diffusion or flow-based generative models to learn function priors; based on my understanding, the key difference lies in the velocity-model architecture. The evaluation covers prior modeling (MMD, SWD) and posterior inference (NRMSE, NCRPS) on mixture-of-GPs, Navier–Stokes, and global wind-field tasks, as well as downstream UCI regression (ASN, CCS, AutoMPG, EE) and Bayesian optimization. SFP achieves the best or competitive results on most metrics, though not uniformly (e.g., SPL-OFM obtains a lower NRMSE on Navier–Stokes, and SFP ties TNP on ASN NRMSE).

Critical strengths
  1. SFP is evaluated on a diverse set of tasks spanning synthetic stochastic processes (mixture of GPs), physical dynamical systems (Navier–Stokes), and real geophysical data (ERA5 wind fields), as well as downstream UCI regression and Bayesian optimization. The authors also release their code, which supports reproducibility and future research.
  2. The authors reformulate the multiscale spatial bias as an augmented query–key inner product. This makes the biased attention compatible with FlashAttention, yielding memory savings. Practical efficiency considerations like this are often underexplored in the NP literature, so this is a useful and reusable contribution.
  3. The figures (e.g., Figs. 2-4) effectively illustrate the architecture, the efficient attention design, and the qualitative behavior on each task, making the contributions and experimental settings easy to follow.
Critical weaknesses
  1. The contributions are not clearly delineated from prior work. Using flow-based generative models for NPs, for both prior modeling and posterior inference, has already been explored (e.g., NDP, SPL-OFM). The guidance-based posterior sampling likewise follows existing zeroshot imputation methods. Section 3.1 largely describes standard conditional flow matching plus guided imputation, yet it is presented as part of the proposed methodology. This makes it hard to tell which components are new. I suggest moving this content to the background section and stating explicitly that the main contribution is the FaMA backbone.
  2. The two components of FaMA closely follow existing designs. The temporal modulation applies gated multiplicative conditioning (Perez et al., 2018; Miyato et al., 2024) to the queries, keys, and values. The multiscale spatial bias is similar to head-specific geometric scales in ALiBi [2].
  3. The theoretical results are redundant and should be moved to Appendix. Lemma 3.1 follows directly from the absence of index-based positional encodings: coordinates enter only through pointwise features and pairwise distances, so equivariance follows from the standard set-transformer argument. Proposition 3.2 is a known property of continuous normalizing flows and is also shared by NDP. These results hold for any equivariant velocity network, so they neither motivate the specific FaMA design nor distinguish SFP from the baselines. In addition, Section 3.3 discusses marginal consistency but provides no formal result.
  4. The scalability claim is not substantiated. According to Table 1, scalability is the only property distinguishing SFP from NDP. However, the paper does not define what scalability means here or evaluate it systematically. No comparison of training or inference cost against NDP is provided (see Q1).

Minor

  • The labels of Fig 6 are missing.
  • dd is not properly defined.
  • Typo: “Eq. equation 1” in Line 141.
Questions for authors
  1. Table 1 lists scalability as the only property distinguishing SFP from NDP. Could the authors clarify:
  • (a) What does “scalability” refer to here? input-set size, input dimension, training cost, or inference cost?
  • (b) Can you provide a direct comparison of scalability in FLOPs between SFP, NDP, and SPL-OFM under different model sizes and dataset sizes?
  • (c) How does SFP perform and scale on substantially larger input sets (e.g., ≥104\geq 10^4 points), where FlashAttention might matter most?
  1. Could the authors state which components of Section 3.1 differ from existing work such as NDP, SPL-OFM?

  2. How does the multiscale spatial bias differ from existing distance-based attention biases, such as Gaussian distance biases in self-attention [1], and head-specific scales as in ALiBi [2]?

  3. Since Lemma 3.1 and Proposition 3.2 hold for models without positional encodings, what insight do they provide beyond the standard set-transformer arguments?

  4. (Minor) Could the authors:

  • (a) add a legend to Fig. 6 indicating which curve corresponds to each model variant?
  • (b) clarify the meaning of dd in Section 2, Section 2.2, and Eq. (3)?
  • (c) explain why some of the baselines in Fig. 4 (a) do not have gray lines?
  • (d) explain why different color spectum are used ubiquitous Fig. 4 (c)? I think it is harder to interpret the results using different colors from the ground truth target.
  • (e) explain whether the conclusions drawn from Fig. 5 and 6 remain meaningful given that many of the curves fall within a standard deviation?

References
[1] Yang et al. Modeling Localness for Self-Attention Networks. EMNLP 2018.

[2] Press et al. Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation. 2021.

8 comments
AuthorsOpenReview ↗

Response to Reviewer ynWL

We sincerely thank the reviewer for the detailed review and for the time invested in evaluating our work. We are glad that the reviewer finds our evaluation comprehensive, considers the FlashAttention-compatible reformulation of the spatial bias a useful and reusable contribution, and finds our figures helpful in following the method.

We appreciate the reviewer's constructive comments, especially the concrete suggestions on positioning and presentation. We address each point below. Since several weaknesses and questions concern the same issue, we answer them together: W1 with Q2, W2 with Q3, W3 with Q4, and W4 with Q1.

Weakness

W1 and Q2: Delineation from Prior Work

We thank the reviewer for this constructive suggestion. We would like to clarify that conditional flow matching and guided posterior sampling are not listed among our contributions (Section 1). We do not claim likelihood guidance or the Tweedie estimator as contributions. Their role in SFP is to turn an unconditional finite-marginal prior, learned without context–target splits, into a posterior sampler for arbitrary context–query partitions and observation models without retraining.

We present them in Section 3.1 to frame flow matching within the NP framework, so that readers can easily follow the full method in one place: how SFP learns a prior over finite-dimensional marginals of irregular input sets, and how this prior is used for posterior inference. Our main contributions are SFP as a flow-based generative NP over irregular inputs and the FaMA block (Section 3.2). In the revision, we have clarified this attribution in Section 3.1 by rephrasing "we propose efficient sampling" as "we adopt the training-free guided sampling of Pokle et al. (2024)" (Line 157).

To answer Q2 directly, Section 3.1 differs from NDP (Dutordoir et al., 2023) and SPL-OFM (Shi et al., 2025) in the following respects:

NDPSPL-OFMSFP
Generative modelDDPM, 1,000 stepsOperator flow matching with a GP sourceConditional OT flow matching with an i.i.d. Gaussian source
Inputs at training timeIrregular setsRegular grids (FNO backbone)Irregular, unordered sets of varying size
Posterior inferenceRePaint-style resamplingSGLD on the flow log-density (22–4×1044 \times 10^4 iterations)Training-free likelihood guidance (adapted from Chung et al., 2023; Pokle et al., 2024)
BackboneBi-dimensional attentionTime-conditioned FNOFaMA

W2 and Q3: Relation of FaMA to Existing Designs

We thank the reviewer for these references. We would like to clarify that we have already cited them and discussed the relationship of our approach with existing methods (Perez et al., 2018; Miyato et al., 2024 in Section 3.2; Press et al., 2022 in Section 2), and that Section 3.2 discusses why and how we adapt them to NPs with flow matching. Each follows directly from a structural property of the target in our setting, that correlations among the finite marginals evolve along the flow and depend on the relative distance between inputs. Our contribution lies not in the individual mechanisms, but in identifying which structure a flow-based NP over irregular inputs requires, and in realizing it while preserving permutation equivariance and exact FlashAttention compatibility.

The spatial bias is particularly important for scaling input dimensions. As dd grows, a fixed number of observations covers the input space increasingly sparsely, so purely content-based attention must infer which points are relevant from few scattered samples, and attention to unrelated distant points acts as noise. The spatial bias supplies a distance-based prior that makes training more sample-efficient. Normalizing the squared distance by dd (Eq. 3) keeps this prior on a comparable scale across input dimensions, and the multiscale heads adapt the effective neighborhood size to the data density. This underlies SFP's applicability up to d=8d = 8 in UCI regression and BO, where NDP becomes unstable and SPL-OFM is restricted to regular grids by its FNO backbone. We detail the differences below.

  • Temporal modulation. FiLM-style conditioning modulates token features before a shared projection. FaMA instead applies separate time-dependent modulations to the query, key, and value projections, so that the pairwise attention logits change with flow time. This is motivated by the structure of the target velocity: for a GP prior, the optimal denoiser E[y1∣yt]=tK((1−t)2I+t2K)−1yt\mathbb{E}[y_1 \mid y_t] = tK\left((1-t)^2 I + t^2 K\right)^{-1} y_t mixes function values across points with a matrix that shifts from broad kernel smoothing near t=0t = 0 to near-identity as t→1t \to 1. We kindly refer the reviewer to our response to Reviewer i8PY (W2) for the detailed derivation and motivation for both temporal modulation and the multiscale spatial bias.
AuthorsOpenReview ↗

Response to Reviewer ynWL (Continued)

  • Multiscale spatial bias versus ALiBi and Gaussian localness bias (Q3).
    • ALiBi (Press et al., 2022) adds a linear penalty on the token-index distance ∣i−j∣|i - j| in 1D causal sequences, with head-specific slopes chosen for length extrapolation. FaMA penalizes the squared Euclidean distance between continuous, multi-dimensional coordinates of unordered sets. Since exp⁡(−∥xi−xj∥2/dℓh2)\exp(-\|x_i - x_j\|^2 / d\ell_h^2) is an RBF kernel, each head reweights content attention by an RBF kernel with length scale ℓh\ell_h, matching the distance-decaying correlations of stationary processes; the log-spaced heads cover several length scales.
    • Gaussian localness bias (Yang et al., 2018) centers a Gaussian window at a predicted position in a 1D token sequence to strengthen local attention in machine translation. FaMA's bias is centered at each query's own coordinates in continuous space and uses fixed, log-spaced length scales across heads without additional parameters.
    • Efficiency. A linear distance bias on continuous multi-dimensional coordinates is not separable and requires materializing the N×NN \times N bias matrix. The squared-distance bias admits an exact low-rank decomposition (Eq. 4), which enables standard FlashAttention.

Finally, we appreciate the reviewer's comments on the relation to existing designs. We have added Yang et al. (2018) and this comparison to the related work. A qualitative comparison with ALiBi-style and RoPE-style mechanisms is provided in our response to Reviewer 6TL1 (Q1).

W3 and Q4: Theoretical Results

We thank the reviewer for these comments. We do not claim the permutation results (Lemma C.2 and Proposition C.3 in the revision) as a distinguishing contribution. Their purpose is to verify that the two components FaMA adds, time-modulated projections and the distance bias, do not break equivariance. Mechanisms commonly used for spatial data, such as index-based positional encodings or grid-based FNO backbones, do not preserve it on unordered sets. We would like to further clarify that prior generative NPs such as NDP (Dutordoir et al., 2023) mainly establish permutation equivariance at the network level. Proposition C.3 complements this by explicitly connecting an equivariant velocity field to permutation-invariant distributions along the entire flow, which is the property required for an exchangeable stochastic process. In the revision, we have condensed Section 3.3 into a short description of these properties, moved the formal statements and proofs to Appendix C, and revised contribution (3) accordingly.

We also thank the reviewer for noting that Section 3.3 gives no formal result on marginal consistency. We have added the following result to Section 3.3, with the formal statement and proof in Appendix C.2 (Proposition C.5); it clarifies what is and is not guaranteed:

  • Exact target. The source N(0,I)\mathcal{N}(0, I) is i.i.d. across input points, and the conditional OT path interpolates each point independently. Hence, for nested sets XS⊂XLX_S \subset X_L, restricting the target path on XLX_L to XSX_S gives exactly the target path on XSX_S. Under Assumption C.1, the exact flow-matching velocity therefore induces a marginally consistent model.
  • Learned model. The learned velocity only approximates this target, so SFP is approximately consistent, as for NDP and FMNP (Hamad & Rosenbaum, 2025). Appendix F.4 (Table 11) quantifies this: SFP is comparable to SPL-OFM and close to the finite-sample exact-GP floor. We further evaluated marginal consistency on wind fields and on larger input sets (up to the full 64×3264 \times 32 grid with 2,048 points). SFP remains close to the reference in all settings, and the gap does not grow with the input-set size. We kindly refer the reviewer to our response to Reviewer ZWEN (Q1, Table R3) for the full results.
AuthorsOpenReview ↗

Response to Reviewer ynWL (Continued 2)

W4 and Q1: Scalability

We thank the reviewer for pointing out that "scalability" in Table 1 is undefined. We have split this column into two explicit properties in the revised Table 1.

  • (a) Meaning of scalability. We meant two things: scalability to higher input dimensions and to larger input sets.
  • Input dimension. SFP remains applicable up to d=8d = 8 in UCI regression and BO. SPL-OFM requires regular grids whose size grows exponentially with dd, and NDP was numerically unstable beyond 1D in our experiments despite following the official implementation.
  • Input-set size. The separable spatial bias enables FlashAttention with memory linear in NN, and the memory savings grow with the input-set size (Appendix D.3, Table 6). Our new stress test in Table R2 extends this comparison to larger input sets.

We would like to further clarify that this is not a claim of lower FLOPs than grid-based operators: on a regular grid, FNO evaluates in O(Nlog⁡N)O(N \log N), whereas attention is O(N2)O(N^2) in compute. SFP's advantage lies instead in handling irregular inputs at scale, where FNO does not directly apply, and whose regular grids would also grow exponentially with the input dimension.

  • (b) FLOPs comparison. We report FLOPs, parameters, and training cost under different model and dataset sizes:

    Table R1. Computational cost of NDP, SPL-OFM, and SFP. GFLOPs are measured per network function evaluation (NFE) on input sets of NN points; NFE per sample is the number of evaluations used to draw one prior sample.

    MethodModel sizeParamsGFLOPs / NFE (NN = 256)GFLOPs / NFE (NN = 4,096)NFE per sampleTraining GPU-hours
    NDP(default) Bi-dim Attention, hidden 128, 5 layers2.77M1.67174.631,0002.65
    SPL-OFM(default) FNO, 4 layers, 16 modes, hidden 1289.25M0.2312.493740.47
    SFPhidden 128, 6 layers, 4 heads1.33M0.8042.71002.07
    SFPhidden 256, 8 layers, 8 heads4.91M1.44122.91004.18
  • (c) Larger input sets (N≥104N \geq 10^4). We profile the peak memory and runtime of a single velocity evaluation on FaMA block with standard attention versus FlashAttention for NN from 4,096 to 65,536 on a 40 GB A100 GPU. Standard attention runs out of memory beyond 16,384 points, whereas FaMA with FlashAttention scales to 65,536 points.

    Table R2. Memory and runtime per velocity evaluation for large input sets.

    NNStandard mem. (MiB)Flash mem. (MiB)Standard time (ms)Flash time (ms)
    4,096263245417.7912.48
    8,192824862865.5218.62
    16,38432856760252.4826.56
    32,768OOM1048OOM46.01
    65,536OOM1216OOM174.75

We sincerely thank the reviewer for these comments and questions, which helped us further analyze and present the scalability advantages of SFP and the FaMA block. We have included these results in Appendix D.3.

Minor Weaknesses and Q5

We greatly appreciate the reviewer's careful reading of our manuscript.

  • (a) Figure 6 legend. We have added legends. The three curves correspond to the basic version (flow time appended to each token), the version with temporal modulation, and full FaMA with temporal modulation and multiscale spatial bias.

  • (b) Definition of dd. Throughout the paper, dd denotes the input dimension, X∈RN×dX \in \mathbb{R}^{N \times d}. In Section 2.2, we unintentionally reused dd for the dimension of the flow state; we have replaced it with NN. In Eq. (3), dividing the squared distance by dd keeps the bias on a comparable scale across input dimensions, since the expected squared distance between uniformly sampled inputs grows linearly with dd.

  • (c) Missing gray lines in Figure 4(a). Gray lines denote posterior function samples. For TNP (Nguyen & Grover, 2022), which outputs a Gaussian predictive distribution, Figure 4(a) shows the predictive mean and uncertainty band. LNP and AttCNP do sample functions via a global Gaussian latent variable, which makes their samples have limited variability and largely overlap with the mean. We have clarified this in the caption.

  • (d) Color maps in Figure 4(c). The model panels show the absolute error of the posterior mean, not the predicted field, as indicated by their color bars. Since errors are much smaller than field values, a shared scale would make the differences between models hard to see. All model panels therefore share a separate error scale and are directly comparable with each other. We have further clarified this in the caption.

Reviewer_i8PYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

In this work, the authors proposed Structured Flow Process (SFP), which is a new flow-based generative model for stochastic processes. they introduce several modeling simplifications that reduces the training algorithm almost identical to regular flow matching, and design a sampling algorithm to accompany the simple training algorithm. For the network structure, they propose Flow-adaptive Multihead Attention (FaMA), which combines a temporal modulation of attention with a multiscale spacial bias in the attention mechanism. Their experiment show superior performance against baselines.

Critical strengths

I think this paper genuinely proposes a meaningful direction in simplifying the formulation and modeling assumption of stochastic processes. In comparison to its prior works, SPF uses many assumptions that seem too simple to be good (e.g. directly using Gaussian noise instead of GP, the Gaussian surrogate via Tweedie's formula, etc), yet it still proves its practical advantages through experiments. Overall I think this shows a surprisingly simple direction to explore in the future.

Critical weaknesses
  1. This paper is unfortunately not very well written. As a matter of fact, the entire paper, including its illustrative figures, reads like something that AI generated. Sentences are packed like word salad and words are simply there for filling the space as opposed to explaining the concepts. I am not against using AI to improve papers, but this is just simply not very readable.
  2. Following up on the writing, most of its design choices are not very well motivated and most are just stated as some fact rather than justified novelties. This problem is especially severe in Section 3.2, where the introduction of the temporal modulation and the specific formulation of spatial bias are basically just based on intuitions at best and with no real prior evidence of necessity.
  3. Figure 6 has no legends or labels, so I don't really know what I am looking at, despite this is an important figure to separate the contribution of FaMA (which the authors claim to be a key novelty).
Questions for authors
  1. Has the authors also conducted ablation studies on the NFE (number of flow steps) budget? How does these methods compare under different budgets?
  2. Has the authors tried FaMA on other NP settings?
4 comments
AuthorsOpenReview ↗

Response to Reviewer i8PY

We sincerely appreciate the reviewer for the thoughtful review and for the time invested in evaluating our work. We are glad that the reviewer sees SFP as a meaningful and surprisingly simple direction for modeling stochastic processes, whose simplifying assumptions still prove effective in practice.

We appreciate the reviewer's candid comments on presentation and motivation. We address each point below and describe the corresponding revisions.

Weakness

W1: Writing Quality

We thank the reviewer for this candid feedback and take it seriously. We used AI tools only to polish the grammar of text written by the authors, and all figures were created by the authors. We agree that the current text is too dense and that several sentences state claims without explaining them. To address the reviewer's concerns, we have made the following changes in our revision:

  • rewrote lengthy sentences in Sections 1 and 3 into shorter and clearer ones;
  • fixed notation and presentation issues, including the overloaded symbol dd (input dimension in Sections 2 and 3.2), inconsistent lemma and proposition numbering in Appendix C, the duplicated paragraph in Appendix D.2, and the missing legends in Figure 6.

We would also be grateful if the reviewer could point to specific passages that were especially hard to follow, so that we can address them directly.

W2: Motivation of the Design Choices in Section 3.2

We thank the reviewer for this important comment. We clarify below that both components follow from how the target velocity behaves along the flow.

Motivating example. Consider a GP prior y1∼N(0,K)y_1 \sim \mathcal{N}(0, K) on an input set XX, with source y0∼N(0,I)y_0 \sim \mathcal{N}(0, I) and yt=(1−t) y0+t y1y_t = (1-t)\, y_0 + t\, y_1. The optimal velocity is determined by the denoiser E[y1∣yt]=Mt yt,Mt=tK((1−t)2I+t2K)−1.\mathbb{E}[y_1 \mid y_t] = M_t\, y_t, \qquad M_t = tK\left((1-t)^2 I + t^2 K\right)^{-1}. The matrix MtM_t specifies how the value at each point should aggregate information from the other points. Writing λ\lambda for an eigenvalue of KK, its eigenvalues are tλ/((1−t)2+t2λ)t\lambda / ((1-t)^2 + t^2\lambda). Two observations follow:

  1. The interaction pattern changes with flow time. Near t=0t = 0, Mt≈tKM_t \approx tK performs broad kernel smoothing; as t→1t \to 1, Mt→IM_t \to I becomes local. An attention layer approximating this map therefore needs attention weights that depend on tt. Concatenating tt to each token (Figure 6, basic version) leaves this dependence to be learned implicitly; temporal modulation applies time-dependent projections to queries, keys, and values, so the attention logits themselves change with tt.
  2. Different spatial scales dominate at different times. Large-scale components (large λ\lambda) are resolved early in the flow, and fine-scale components (small λ\lambda) only near t=1t = 1. Moreover, our datasets mix length scales: the mixture-GP task samples length scales from a log-normal distribution and combines RBF and Matérn kernels, and NS and wind fields contain both local and global structure. This motivates assigning different length scales to different heads.

Why a squared-distance bias. Since exp⁡(Bij(h))=exp⁡(−∥xi−xj∥2/dℓh2)\exp(B^{(h)}_{ij}) = \exp(-\|x_i - x_j\|^2 / d\ell_h^2) is an RBF kernel, each head computes content attention reweighted by an RBF kernel with length scale ℓh\ell_h. This encodes the distance-decaying correlations of stationary processes (Williams & Rasmussen, 2006) as a soft prior, while the learned query–key term can still capture non-stationary or anisotropic structure. The squared form also admits an exact low-rank decomposition, which makes the bias compatible with FlashAttention (Eq. 4).

Empirical evidence. Each component is supported by controlled ablations: temporal modulation and multiscale bias each improve MMD and NCRPS on mixture GPs and NS (Figure 6); multiscale bias outperforms a single-scale bias on all four metrics (Appendix F.8, Table 15(a)); and performance is stable over a practical range of the scale parameter cc (Appendix F.7, Table 14).

W3: Figure 6 Legend

We greatly thank the reviewer for pointing this out and apologize for the omission. We have added legends and axis labels to Figure 6. The three curves correspond to (1) the basic version, which appends flow time to each token; (2) the version with temporal modulation; and (3) full FaMA with temporal modulation and multiscale spatial bias.

AuthorsOpenReview ↗

Response to Reviewer i8PY (Continued)

Questions

Q1: Ablation on the Number of Flow Steps

We thank the reviewer for this question. Appendix F.6 (Table 13) varies the number of Euler steps over {5,10,20,50,100,200}\{5, 10, 20, 50, 100, 200\} on the GP and wind-field (WF) tasks:

Table R1. Sensitivity to the number of sampling steps (batch size 32 for GP, 8 for WF).

TaskStepsTime (ms/sample)NRMSE ↓NCRPS ↓Calibration error ↓
GP52.350.44 ± 0.870.34 ± 0.550.19 ± 0.09
GP209.170.13 ± 0.040.05 ± 0.050.08 ± 0.03
GP5022.750.12 ± 0.050.05 ± 0.040.07 ± 0.03
GP10045.530.05 ± 0.040.03 ± 0.020.05 ± 0.02
WF526.890.53 ± 0.210.28 ± 0.130.22 ± 0.10
WF20106.540.33 ± 0.020.19 ± 0.020.14 ± 0.02
WF50265.570.30 ± 0.010.17 ± 0.020.14 ± 0.02
WF100530.550.28 ± 0.010.17 ± 0.010.09 ± 0.01

Results. Performance improves rapidly up to 20–50 steps, with smaller gains afterwards, consistent with Hamad & Rosenbaum (2025). Sampling time grows linearly with the number of steps. With 20 steps, SFP already attains lower NCRPS than TNP, the strongest single-pass baseline (GP: 0.05 vs. 0.07; WF: 0.19 vs. 0.25).

Comparison under equal budgets. By default, NDP uses 1,000 diffusion steps, SPL-OFM uses an adaptive ODE solver with about 374 function evaluations (NFE) for prior sampling, and SFP uses 100 Euler steps. To further address the reviewer's concern, we conducted an additional experiment that compares the prior quality of all methods under equal sampling budgets, i.e., at matched NFE:

Table R2. Prior quality (102×10^2\timesMMD ↓) at matched NFE (mean ± std). "–" indicates that NDP training is numerically unstable on WF (Table 2a).

TaskNFENDPSPL-OFMSFP
GP2017.20 ± 0.307.70 ± 0.606.80 ± 0.30
GP5012.10 ± 0.304.73 ± 0.503.80 ± 0.40
GP1008.80 ± 0.503.41 ± 0.703.47 ± 0.50
WF20–34.63 ± 0.1028.41 ± 0.10
WF50–27.90 ± 0.1024.40 ± 0.30
WF100–23.43 ± 0.1022.20 ± 0.20

Results. At matched NFE, SFP achieves the lowest MMD in five of the six settings. SFP outperforms NDP at every budget on GP, and SPL-OFM catches up only at 100 NFE. Against SPL-OFM, the advantage of SFP is largest at small budgets: up to 20% on GP (50 NFE) and 18% on WF (20 NFE), where SFP with 20 NFE is already close to SPL-OFM with 50 NFE. This shows that SFP's advantage is not due to a larger sampling budget and is most pronounced when only few steps are affordable.

Q2: FaMA in Other NP Settings

We thank the reviewer for this suggestion. FaMA has two components with different scopes. The multiscale spatial bias applies to any attention-based NP over continuous inputs, whereas temporal modulation applies to NPs with a generative time variable. Flow Matching Neural Processes (FMNP; Hamad & Rosenbaum, 2025) meet both conditions: FMNP is a context-conditioned flow-matching NP built on standard self-attention. We therefore replace its attention blocks with full FaMA, keep all other components and training settings unchanged, and evaluate both models on the 1D GP and wind-field (WF) benchmarks:

Table R3. FaMA as a plug-in attention block in FMNP (mean ± std over training seeds).

TaskModelParamsNRMSE ↓NCRPS ↓
GPFMNP0.77M0.06 ± <0.010.03 ± <0.01
GPFMNP + FaMA0.73M0.04 ± <0.010.03 ± <0.01
WFFMNP1.48M0.32 ± 0.030.24 ± 0.01
WFFMNP + FaMA1.42M0.30 ± 0.020.21 ± 0.01

Results. With slightly fewer parameters, FaMA matches or improves FMNP on both tasks: on GP it reduces NRMSE with unchanged NCRPS, and on the real-world WF task it reduces both NCRPS and NRMSE. These results show that the benefit of FaMA is not specific to SFP: FaMA serves as a plug-in attention block that improves other attention-based generative NPs. We have added these results to Appendix F.10 (Table 17(b)).


References

Hamad, Hussen Abu, and Dan Rosenbaum. "Flow matching neural processes." NeurIPS. 2025.

Williams, Christopher K. I., and Carl Edward Rasmussen. Gaussian Processes for Machine Learning. MIT Press, 2006.

We hope that our responses and the revisions address the reviewer's concerns. We are happy to respond promptly if the reviewer has any further questions. If the reviewer finds that the main concerns have been addressed, we would be grateful if the reviewer could reconsider the evaluation and score accordingly.

Dear Reviewer i8PY,

We appreciate the time you have spent on our paper. As the discussion period ends soon, we would like to kindly ask whether our responses have addressed your concerns. To summarize, we revised lengthy sentences and fixed notation and presentation issues (W1, W3), motivated both FaMA components through how the optimal denoiser changes along the flow (W2), showed that SFP performs well with few sampling steps and outperforms baselines at matched budgets (Q1), and demonstrated that FaMA also improves FMNP as a plug-in attention block (Q2).

Any further feedback would be very welcome, and we are happy to answer additional questions before the discussion closes.

Best,

Authors of Submission

#24747

Thank you for your rebuttal! I really appreciate all the efforts the authors made during this period. However, I do believe that this paper can be benefit from another iteration and therefore I would keep my score.

Reviewer_ZWENOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes using flow matching to learn function distributions from irregular observations, adding flow time modulation and multi-scale spatial biasing into the attention mechanism. It handles changing observation locations, counts, and orderings, and performs well on synthetic processes, physical fields, wind fields, and downstream tasks. The theory guarantees permutation invariance, but marginal consistency across different-sized input sets is only approximate. Plus, posterior sampling relies on approximate guidance and costs more computation than a single forward pass.

Critical strengths

1.Handles functional data with irregular observation locations, counts, and orderings.

2.Both flow time modulation and multi-scale spatial biasing are backed by ablation studies.

3.Proves permutation invariance of finite-dimensional distributions.

4.Covers synthetic processes, physical fields, real wind fields, and downstream tasks.

5.Spatial biasing is compatible with FlashAttention, reducing memory usage.

Critical weaknesses

1.Doesn't guarantee exact marginal consistency across input sets of different sizes.

2.Posterior guidance relies on a Gaussian surrogate likelihood and tuning parameters.

3.Iterative sampling is slower than single-pass forward prediction.

4.Different input dimensions still need separate configuration.

5.Some baselines were trained under different input conditions; Navier–Stokes NRMSE isn't state-of-the-art.

Questions for authors

1.How severe is the marginal inconsistency on wind fields and larger input sets?

2.Is posterior guidance still reliable under non-Gaussian noise?

3.How would spatial biasing extend to anisotropic or non-Euclidean inputs?

4.Do the comparisons hold under equal training compute and sampling budgets?

5.Can generated function samples stay consistent as query points are continuously added?

7 comments
AuthorsOpenReview ↗

Response to Reviewer ZWEN

We sincerely thank the reviewer for the thoughtful review and for the time invested in evaluating our work. We are glad that the reviewer recognizes the effectiveness of SFP, our broad evaluation, and the FlashAttention-compatible spatial bias.

We appreciate the reviewer's constructive comments. We address each point below with additional experiments on marginal consistency, non-Gaussian likelihoods, and computational cost.

Weakness

W1: Exact Marginal Consistency

We appreciate the reviewer’s valuable comment on marginal consistency. The lack of an exact guarantee reflects an inherent trade-off rather than an oversight. We further explain from the three points below.

  • (1) Exact consistency requires restricting the model class. It holds by construction only when all finite marginals derive from a single explicit object, as in Gaussian Processes (GPs), latent-variable NPs with a conditionally independent decoder (Garnelo et al., 2018), and Markov NPs (Xu et al., 2023). Such guarantees require Gaussianity or conditional independence, which limit the dependencies the model can represent (Foong et al., 2020; Dutordoir et al., 2023). Expressive models that parameterize the joint over a finite input set with a set-based network, such as NDP, autoregressive CNPs, and SFP, do not guarantee exact consistency but deliver better empirical performance in general.

  • (2) In SFP, the learning target is exactly consistent; only the learned velocity is approximate. The source N(0,I)\mathcal{N}(0, I) is i.i.d. across input points, and the conditional OT path interpolates each point independently. Hence, for nested sets XS⊂XLX_S \subset X_L, the target path on XSX_S is exactly the marginal of the target path on XLX_L, and any inconsistency comes only from the approximation error of vθv_\theta. We have added this result to Section 3.3, with the formal statement and proof in Appendix C.2 (Proposition C.5). Removing the remaining inconsistency by design would require the velocity on XSX_S to match the conditional expectation of the velocity on XLX_L. This is an integral constraint that couples all set sizes. Training on randomly sized subsets encourages SFP to satisfy it approximately.

  • (3) Empirically, the remaining inconsistency is small. We would like to refer the reviewer to Appendix F.4 (Table 11), which compares samples drawn directly on XSX_S with samples drawn on XLX_L and then restricted to XSX_S. SFP (102×10^2\timesMMD 4.15, SWD 0.11) is comparable to SPL-OFM (4.02, 0.13) and close to the finite-sample floor of the exact GP (2.87, 0.08). It also reduces MMD and SWD by about 65% and 45% relative to NDP (11.92, 0.20). Our new results on wind fields and larger input sets (See Q1, Table R3) further show that this gap remains small and does not grow with the input-set size.

W2: Posterior guidance relies on a Gaussian surrogate likelihood and tuning parameters.

We thank the reviewer for this thoughtful comment. We clarify that the surrogate likelihood itself has no free parameters. The guidance only tunes a single scalar and the resulting posterior is not restricted to be Gaussian. We further address the reviewer's concerns from the following three perspectives.

  • (1) The surrogate variance is fixed by the flow path. The surrogate approximates only the intermediate distribution p(y1∣yt)p(y_1 \mid y_t). Its variance rt2=(1−t)2/((1−t)2+t2)r_t^2 = (1-t)^2/((1-t)^2+t^2) is the posterior variance of y1y_1 given yty_t when the normalized data are unit-variance Gaussian, and it vanishes as t→1t \to 1. The surrogate becomes exact at the end of the flow. The observation variance is a fixed nugget (σy=10−3\sigma_y = 10^{-3}), following standard NP benchmarks (Appendix F.5). This keeps posterior inference training-free: one unconditional prior serves all context sets, yet it performs on par with FMNP, which is trained specifically for conditional prediction (Table 17(a)).

  • (2) Only the guidance scale λ\lambda is tuned, and it is empirically robust. We select λ\lambda from a six-value grid on validation NCRPS and fix it at test time. Performance is stable for λ∈[1,3]\lambda \in [1, 3] (Table 12), and only two values are used across all five tasks: λ=1\lambda = 1 for GP, UCI, and BO, and λ=3\lambda = 3 for NS and WF (Table 18). In comparison, SPL-OFM posterior sampling relies on SGLD with tuned iteration counts, burn-in, learning-rate schedules, Hutchinson samples, and noise levels (Appendix D.2; Shi et al., 2025, Appendices O–P).

AuthorsOpenReview ↗

Response to Reviewer ZWEN (Continued)

  • (3) The Gaussian likelihood is a problem-specific choice, not a requirement of SFP. We use a Gaussian model p(yc∣y1)p(y^c \mid y_1) because all our tasks are regression problems. Guidance only requires a differentiable log-likelihood, so the likelihood model can be replaced for the specific problem without retraining the prior: for example, a Bernoulli likelihood for classification problems, or a Poisson likelihood Poisson(yc∣exp⁡(Pc y^1∣t))\mathrm{Poisson}(y^c \mid \exp(P_c\, \hat y_{1|t})) for count data. In these cases, the guidance term can instead use the plug-in estimate ∇ytlog⁡p(yc∣y^1∣t)\nabla_{y_t} \log p(y^c \mid \hat y_{1|t}), as in diffusion posterior sampling (Chung et al., 2023). To further address the reviewer's concern, we apply the SFP prior trained on 2D GP samples to binary classification on three standard 2D benchmarks from scikit-learn (Pedregosa et al., 2011): two moons, concentric circles, and a linearly separable dataset. SFP models a latent function ff, and the labels follow p(yic∣f)=Bernoulli(yic∣σ(f(xi)))p(y^c_i \mid f) = \mathrm{Bernoulli}(y^c_i \mid \sigma(f(x_i))) with the logistic link σ\sigma. Predictive class probabilities at held-out points are obtained by averaging σ(f)\sigma(f) over posterior samples. We compare against GP classification (GPC) with the Laplace approximation and an RBF kernel fitted by maximizing the marginal likelihood on each dataset (Williams & Rasmussen, 2006). We report accuracy and negative log-likelihood (NLL) on 500 test points, averaged over 10 random datasets for each context size.

Table R1. Binary classification on 2D benchmarks (mean ± std over 10 random datasets, 500 test points each).

DatasetContextAcc. GPC ↑Acc. SFP ↑NLL GPC ↓NLL SFP ↓
Moons250.85 ± 0.030.86 ± 0.020.41 ± 0.050.35 ± 0.04
Moons500.87 ± 0.030.87 ± 0.020.36 ± 0.030.35 ± 0.06
Moons1000.88 ± 0.020.90 ± 0.020.30 ± 0.020.30 ± 0.04
Circles250.81 ± 0.040.81 ± 0.030.45 ± 0.070.43 ± 0.05
Circles500.86 ± 0.020.86 ± 0.030.36 ± 0.040.34 ± 0.05
Circles1000.88 ± 0.010.88 ± 0.010.30 ± 0.030.29 ± 0.03
Linear250.86 ± 0.070.86 ± 0.070.38 ± 0.090.33 ± 0.11
Linear500.86 ± 0.080.87 ± 0.060.33 ± 0.110.30 ± 0.10
Linear1000.88 ± 0.050.88 ± 0.050.29 ± 0.080.28 ± 0.09

Results. SFP achieves lower NLL than GPC in all nine settings while attaining comparable accuracy. The gain is largest with few context points (e.g., 25 points), where the learned prior matters most, and it narrows as more labels become available. Notably, GPC must be fitted on each dataset, whereas SFP reuses the same prior and only changes the likelihood in the guidance, showing the flexibility of decoupling the prior from the observation model.

We thank the reviewer for this comment, which motivated us to demonstrate SFP with non-Gaussian likelihoods. We have added these results to Appendix F.

W3: Iterative sampling is slower than single-pass forward prediction.

We appreciate the reviewer's comment on inference cost, which we also acknowledged in our Limitations. Our contribution is not to reduce sampling cost, but to improve the expressiveness–efficiency trade-off within continuous generative NPs (see our discussions in Section 1). We address the reviewer's concern from the three perspectives below.

  • (1) Iterative sampling enables a more expressive posterior. Iterative generation is the standard inference mechanism of continuous generative NPs, including NDP, FMNP, and SPL-OFM. It replaces the single-pass decoder of conditional NPs, which typically outputs factorized or Gaussian predictives. This is reflected in the consistent posterior gains over conditional NPs in Table 2b and other continuous NPs works (Dutordoir et al., 2023).

  • (2) The number of steps offers a controllable trade-off. Appendix F.6 (Table 13) varies the Euler steps from 5 to 200. Sampling time grows linearly, while most of the gain is reached within 20–50 steps. Even with 20 steps, SFP attains lower NCRPS than TNP, the strongest single-pass baseline:

    Table R2. Sampling cost and posterior quality.

    TaskMethodStepsTime (ms/sample)NRMSE ↓NCRPS ↓
    GPTNP1–0.12 ± 0.090.07 ± 0.04
    GPSFP209.170.13 ± 0.040.05 ± 0.05
    GPSFP10045.530.05 ± 0.040.03 ± 0.02
    WFTNP1–0.42 ± 0.030.25 ± 0.01
    WFSFP20106.540.33 ± 0.020.19 ± 0.02
    WFSFP100530.550.28 ± 0.010.17 ± 0.01
AuthorsOpenReview ↗

Response to Reviewer ZWEN (Continued 2)

  • (3) SFP is cheaper than other generative NPs. The FlashAttention-compatible spatial bias reduces peak memory by 82.8% and runtime by 30% on NS, and by 97.7% at 16,384 points (Table 6). Posterior guidance is computed as a vector–Jacobian product (VJP) without materializing the Jacobian, and the VJP accounts for only 31–39% of each guided step (Table 7). In comparison, NDP uses 1,000 diffusion steps and SPL-OFM posterior sampling runs 22–4×1044 \times 10^4 SGLD iterations, each requiring a flow log-density estimate (Appendix D.2), whereas SFP uses 100 Euler steps by default. As SFP is trained with flow matching, few-step samplers and distillation approaches for flow models are directly applicable, which we leave for future work.

W4: Different input dimensions still need separate configuration.

We thank the reviewer for this comment, which we also acknowledge in our Limitations. We clarify its scope below.

We would like to clarify first that such dimension-dependent form is standard for NPs, and SFP scales further in dd than its closest baselines. Conditional NPs such as AttCNP and TNP, as well as NDP, likewise fix the input dimension at training time; dimension-agnostic NPs (Lee et al., 2025) are a dedicated line of work. Among generative NPs, SFP remains applicable up to d=8d = 8, whereas SPL-OFM requires grids that grow exponentially with dd and NDP becomes unstable beyond 1D in our experiments (Section 4.2, finding 4).

FaMA is compatible with dimension-agnostic extensions. The spatial bias depends on inputs only through scalar pairwise distances, so it is already dimension-agnostic in form. Replacing the pointwise embedding with a dimension-agnostic tokenizer would therefore yield a single model across input dimensions, which we leave for future work.

W5: Some baselines were trained under different input conditions; Navier–Stokes NRMSE isn't state-of-the-art.

We thank the reviewer for raising this point. We would like to clarify that the different input structure reflects a limitation of SPL-OFM, not an unfair comparison. SPL-OFM relies on an FNO backbone, which only processes data on regular grids. This is precisely the limitation SFP is designed to remove (Table 1), and it also makes SPL-OFM inapplicable to UCI regression and BO. We therefore trained SPL-OFM on the grids it requires, while all other methods follow the same irregular-input protocol as SFP (Appendix D.2). Actually, this protocol makes learning harder for SFP: FNO exploits the known grid layout, whereas SFP must infer spatial correlations from unordered coordinate–value pairs without access to the grid layout. SFP nevertheless matches or outperforms SPL-OFM on most metrics in Tables 2a and 2b.

On NS, the setting most favorable to SPL-OFM, the fields are simulated with a pseudo-spectral solver on a periodic 64×6464 \times 64 grid, which matches the Fourier basis of FNO. SPL-OFM therefore attains a lower NRMSE, which measures only the posterior mean, whereas SFP achieves better prior quality and better NCRPS, which evaluates the full predictive distribution. This pattern holds across five training seeds (Table 16).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.