- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces Structured Flow Process (SFP), a flow-matching framework for learning function distributions from irregular observations. Its Flow-adaptive Multihead Attention combines time-conditioned interactions with multiscale spatial biases, enabling flexible prior generation, guided posterior inference, and permutation invariance.
Critical strengths
- The paper addresses probabilistic function modeling under irregular observations, an important challenge for scientific machine learning.
- FaMA combines flow-time modulation and multiscale spatial biases to capture evolving spatial dependencies.
- SFP achieves competitive prior modeling and posterior inference results across synthetic, physical, and real-world datasets.
Critical weaknesses
- The posterior sampling mechanism largely follows existing guidance-based methods, with limited methodological innovation.
- Theoretical results establish permutation invariance but do not guarantee marginal consistency across different input sets.
- Iterative flow-based inference is expensive, which makes it hard for practical deployment.
Questions for authors
See Weaknesses and below:
- How does FaMA compare against established relative positional attention mechanisms under matched model capacity and computational budgets?
- Would standard time conditioning combined with multiscale spatial bias achieve comparable performance without the proposed time-dependent QKV modulation?
5 comments
Response to Reviewer 6TL1 (Continued)
- SFP is cheaper than other generative NPs. SFP uses 100 Euler steps by default, compared with 1,000 diffusion steps for NDP and – SGLD iterations for SPL-OFM posterior sampling. We kindly refer the reviewer to our responses to Reviewer ZWEN (Q4, Table R4) for the full cost table and Reviewer ynWL (W4, Table R2) for large-scale memory profiling. Since SFP is trained with standard flow matching, few-step samplers and distillation for flow models are directly applicable, which we leave for future work.
We thank the reviewer for raising this point, which motivated us to profile SFP at larger scales and further demonstrate its scalability and cost advantages over other generative NPs. We have added these results to Appendix D.3.
Questions
Q1: Comparison with Relative Positional Attention
We thank the reviewer for this question. FaMA's spatial bias differs from relative positional attention in three respects:
-
Input type. Relative position encodings such as Shaw et al. (2018), T5 biases (Raffel et al., 2020), and ALiBi (Press et al., 2022) are designed for discrete token indices in 1D sequences. FaMA operates on continuous, multi-dimensional coordinates of irregular sets.
-
Inductive bias. Since is an RBF kernel, each FaMA head computes content attention reweighted by an RBF kernel with length scale : . This matches the distance-decaying correlations of stationary processes, and the log-spaced heads cover several length scales. RoPE (Su et al., 2024) instead rotates queries and keys by position, which modulates content similarity but does not impose a distance decay.
-
Efficiency. FaMA's squared-distance bias is decomposable: expands exactly into inner products of augmented queries and keys (See Eq. 4 in manuscript). FaMA therefore runs on standard FlashAttention without materializing the bias, which keeps memory linear in and scales to 65,536 points (see our response to Reviewer ynWL, W4, Table R2). By contrast, a linear distance bias on continuous multi-dimensional coordinates is not decomposable and requires the full bias matrix.
-
Existing evidence. The ablations in our paper isolate the effect of the spatial bias: adding the multiscale bias on top of temporal modulation improves MMD and NCRPS on mixture GPs and NS (Figure 6), and the multiscale bias outperforms a single-scale bias on all four metrics on mixture GPs (Appendix F.8, Table 15(a)). Its FlashAttention compatibility substantially reduces peak memory, with savings that grow with the input-set size (Table 6), and FaMA applies directly to higher-dimensional inputs in UCI regression and BO.
Q2: Standard Time Conditioning with Multiscale Bias
We thank the reviewer for this question. Figure 6 compares input–time concatenation, temporal modulation, and temporal modulation with multiscale bias, but it does not include standard time conditioning combined with the multiscale bias. To address the reviewer's concern, we complete the factorial design below, and also include adaLN conditioning (Peebles & Xie, 2023) as an alternative baseline.
Why time-dependent attention is needed. For a GP prior under the linear OT path, . The optimal velocity therefore mixes function values across points with a matrix that changes along the flow, from broad kernel smoothing near to near-identity as . Standard conditioning modulates the token features shared by queries, keys, and values, whereas FaMA applies separate time-dependent modulations to each projection, making pairwise attention logits explicitly flow-dependent.
Table R2. Ablation of time conditioning with and without the multiscale spatial bias on 1D GP. MMD is the mean ± std over 10 sampling seeds. NCRPS is the mean ± std over 500 test functions.
| Time conditioning | Multiscale bias | MMD ↓ | NCRPS ↓ |
|---|---|---|---|
| Concatenation (basic) | ✗ | 0.055 ± 0.004 | 0.058 ± 0.012 |
| Concatenation | ✓ | 0.042 ± 0.003 | 0.051 ± 0.008 |
| adaLN | ✓ | 0.039 ± 0.004 | 0.050 ± 0.005 |
| Temporal modulation | ✗ | 0.042 ± 0.006 | 0.054 ± 0.017 |
| FaMA | ✓ | 0.035 ± 0.005 | 0.049 ± 0.004 |
Response to Reviewer 6TL1 (Continued 2)
Results. The two components of FaMA are complementary. The multiscale bias consistently improves prior quality, but combining it with standard time conditioning (concatenation or adaLN) does not match FaMA. The gain of FaMA therefore does not come from the spatial bias alone: flow-dependent attention logits provide a further improvement. We have added these results to Appendix F.8 (Table 15(b)).
References
Chung, Hyungjin, et al. "Diffusion posterior sampling for general noisy inverse problems." ICLR. 2023.
Dutordoir, Vincent, et al. "Neural diffusion processes." ICML. 2023.
Garnelo, Marta, et al. "Neural processes." arXiv preprint arXiv:1807.01622. 2018.
Hamad, Hussen Abu, and Dan Rosenbaum. "Flow matching neural processes." NeurIPS. 2025.
Mathieu, Emile, et al. "Geometric neural diffusion processes." NeurIPS. 2023.
Peebles, William, and Saining Xie. "Scalable diffusion models with transformers." ICCV. 2023.
Pokle, Ashwini, et al. "Training-free linear image inverses via flows." TMLR. 2024.
Press, Ofir, Noah A. Smith, and Mike Lewis. "Train short, test long: Attention with linear biases enables input length extrapolation." ICLR. 2022.
Raffel, Colin, et al. "Exploring the limits of transfer learning with a unified text-to-text transformer." JMLR. 2020.
Shaw, Peter, Jakob Uszkoreit, and Ashish Vaswani. "Self-attention with relative position representations." NAACL. 2018.
Su, Jianlin, et al. "RoFormer: Enhanced transformer with rotary position embedding." Neurocomputing. 2024.
We hope that our responses and additional experiments address the reviewer's concerns. We are happy to respond promptly if the reviewer has any further questions. If the reviewer finds that the main concerns have been addressed, we would be grateful if the reviewer could reconsider the evaluation and score accordingly.
Response to Reviewer 6TL1
We sincerely thank the reviewer for the thoughtful review and for the time invested in evaluating our work. We are glad that the reviewer finds the problem important, the combination of flow-time modulation and multiscale spatial bias well suited to evolving spatial dependencies, and our results competitive across synthetic, physical, and real-world datasets.
We appreciate the reviewer's constructive comments. We address each point below.
Weakness
W1: Novelty of Posterior Sampling
We thank the reviewer for this comment. We clarify that posterior sampling is not claimed as a main contribution (Section 1); our contributions are SFP as a flow-based generative NP over irregular inputs and the FaMA block. Building on training-free guidance (Chung et al., 2023; Pokle et al., 2024) is a deliberate choice that decouples the prior from the observation model:
We have clarified the attribution of the introduced guidance in Section 3.1.
W2: Marginal Consistency
We appreciate the reviewer’s valuable comment on MC. We clarify the scope of our theoretical results: SFP guarantees permutation invariance of its finite-dimensional distributions but not exact MC; MC is encouraged approximately by training on randomly sampled subsets. Many expressive generative NPs, such as NDP (Dutordoir et al., 2023), GeoNDP (Mathieu et al., 2023), and FMNP (Hamad & Rosenbaum, 2025), share this limitation. Exact MC by construction is available only for restricted model classes such as GPs and latent-variable NPs with conditionally independent decoders (Garnelo et al., 2018), whose structural assumptions limit the dependencies they can represent.
We would like to further clarify two points:
(1) The learning target of SFP is exactly consistent. The source N(0,I) is i.i.d. across input points, and the conditional OT path interpolates each point independently. Hence, for nested sets XS⊂XL, the target path on XS is exactly the marginal of the target path on XL. Under the regularity conditions of Assumption C.1, the exact flow-matching velocity therefore induces a consistent model, and any inconsistency of SFP comes only from the approximation error of vθ. We have added this result to Section 3.3, with the formal statement and proof in Appendix C.2 (Proposition C.5).
(2) Empirically, the remaining inconsistency is small. Appendix F.4 (Table 11) compares samples drawn directly on XS with samples drawn on XL and then restricted to XS. SFP (102×MMD 4.15, SWD 0.11) is comparable to SPL-OFM (4.02, 0.13), close to the exact GP (2.87, 0.08), and well below NDP (11.92, 0.20). To further assess this gap, we conducted new MC tests on the real-world wind-field task and on larger GP input sets. SFP stays close to the finite-sample reference in all settings, and the gap does not grow with the input-set size. We kindly refer the reviewer to our response to Reviewer ZWEN (Q1, Table R3) for details.
We thank the reviewer for this important comment, which helped us clarify the scope of our theoretical results and strengthen the empirical evidence for marginal consistency.
W3: Inference Cost
We thank the reviewer for this comment, which we also acknowledge in our Limitations. Iterative sampling is the standard inference mechanism of continuous generative NPs such as NDP, FMNP, and SPL-OFM. It is the price of posteriors that capture correlated and non-Gaussian structure (e.g., the bimodal posterior in Appendix F.12), which single-pass NPs with Gaussian predictives cannot represent. Within this class, SFP improves the cost–quality trade-off: