- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes a diagnostic for flow matching posterior estimation (FMPE), a widely used method for simulation-based inference (SBI), based on the variability of generated endpoints across independently trained flows. The central idea is to consider how the endpoint variability changes with the number of simulated training samples. The authors establish a central limit theorem for the learned endpoint, and show that endpoint fluctuations around the population regularised estimator exhibit a scaling law. These results are established under a neural tangent kernel (NTK) limit for a regularised two-layer neural network. Based on this result, they construct a statistical test that assesses whether the observed endpoint variability follows the theoretical scaling. The proposed diagnostic is evaluated through several experiments, including a synthetic bimodal posterior and two more complex epidemiological simulators.
Critical strengths
- The authors provide a careful theoretical analysis of endpoint variability in flow matching posterior estimation, providing reasonable justification for the proposed diagnostic.
- The diagnostic offers a complementary approach to more conventional posterior-accuracy metrics (e.g., W1, MMD, etc).
Critical weaknesses
- The theoretical results apply to an idealised infinite-width, regularised two-layer neural network. Meanwhile, the experiments use substantially different architectures and optimisation procedures. As a result, it is not clear whether the predicted scaling law applies directly for the models considered in the paper.
- The diagnostic has a limited capacity to distinguish between different sources of error. In particular, the estimator could exhibit the predicted scaling, whilst still same retaining substantial approximation or optimisation bias. In my view, this limits the ability of the diagnostic to distinguish insufficient training data from "suboptimal pipelines".
- The pseudo-datasets used to perform the test are generated from an initially fitted approximation, rather than the true simulator. There is no (theoretical) guarantee that the diagnostic behaviour transfers between these distributions. The empirical justification is limited to a single synthetic example.
Questions for authors
- What justifies the assumption that diagnostic behaviour under pseudo-data generated from a fitted FMPE model is representative of behaviour under the true simulator? Does the empirical validation shown in Figure 2 hold more widely (e.g., when the fitted FMPE model has significant approximation error)?
- The predicted scaling law appears compatible with estimators that retain substantial approximation or optimisation bias. Is there any reason (theoretical or otherwise) to expect the scaling law to hold specifically for well-tuned FMPE configurations? Could the authors clarify which types of sub-optimal configuration the diagnostic is expected to identify, and how one should interpret failure to reject the null?
- In Figure 1 (Section 5.1), left panel, the average plateaus over sample sizes, so it is not clear to me what additional information the additional diagnostic contributes. The authors make the point that this plateau is obscured if one doesn't average over repeated fits, but the proposed diagnostic is itself based on repeated model fits, so I do not find this argument entirely convincing.
- In Figure 1, central panel, the average continues to decrease as the training-set size increases, suggesting that additional data does improve posterior approximation quality in this example. Is there any evidence that the observed endpoint floor is attributable to insufficient optimisation? What conclusion should be drawn about whether to use more training data, or to modify the optimisation procedure?
- In Figure 3 (Section 5.4), how were the reference posterior distributions obtained (and validated)? Further, what distinguishes the well-tuned and problematic FMPE setups?
- In Figures 1-2 and Table 1 (Sections 5.1-5.3), the velocity is parameterised as a sample mean "buffer" plus a residual field (Appendix F.2). Since the buffer already has fluctuations, could the authors clarify how much of the endpoint scaling observed in practice is attributable to this term? Do the empirical results still look the same if this is removed or altered (e.g., fixed equal to the population mean)?
7 comments
Official Comment by Authors [2]
W2. Thank you for raising this important point. We agree that observing the predicted endpoint-variability scaling does not imply that the fitted posterior is unbiased, correctly specified, or free of approximation, optimization, regularization, or numerical error. In particular, an estimator may retain substantial persistent bias while its sampling variability across training datasets still decreases at the predicted rate.
Our intended interpretation of the diagnostic is therefore deliberately asymmetric. A systematic departure from the predicted scaling provides evidence that the fitted FMPE estimator is not behaving according to the reference sampling regime and therefore warrants further scrutiny. Conversely, failure to reject should not be interpreted as certifying posterior correctness, ruling out persistent bias, or establishing that limited training data are the only source of poor posterior accuracy. We have revised Sections 1, 3 and 5 to make this scope explicit and to avoid language suggesting that the diagnostic can uniquely identify the source of posterior error.
We also added controlled experiments with deliberately biased estimators in Appendix F.6 (Figure 19). In these experiments, strong regularization induces substantial persistent bias while the reference configuration still exhibits scaling close to the predicted rate, directly confirming the reviewer’s point. When an additional suboptimal network configuration is imposed, however, the curves remain similar while the endpoint-variability scaling departs markedly from the theoretical rate. Thus, the diagnostic does not detect persistent bias itself, but can still provide additional information beyond by revealing configuration-related scaling departures in the presence of substantial bias.
Similarly, our poor-training experiments are not intended to imply that all insufficiently optimized FMPE models must violate the predicted scaling. Rather, they show that certain practically relevant optimization failures can produce detectable departures from it. We now frame the diagnostic as complementary to posterior-accuracy and posterior-checking methods, rather than as a tool for uniquely distinguishing insufficient training data from all possible forms of suboptimality.
W3. We address this concern in our response to Q1 below.
Official Comment by Authors [3]
Q1. We agree that a general theoretical guarantee of transfer from the original simulator distribution to the fitted pseudo-data distribution is difficult to establish. We therefore distinguish what follows from the theory from what we validate empirically.
One direction does follow from our theoretical analysis. The endpoint CLT does not require the training data to be generated by the original simulator; it applies to a fixed data-generating distribution satisfying the stated moment conditions. Once the fitted model used to generate pseudo-data is fixed, the pseudo-datasets are i.i.d. draws from such a distribution, so the scaling is likewise predicted for an estimator operating in the theoretical regime. What the theory does not guarantee is transfer of diagnostic failures between distributions: a configuration that departs from the predicted scaling under the original simulator is not theoretically guaranteed to exhibit the same departure under pseudo-data.
We have therefore substantially expanded the empirical validation of this transfer. The former Figure 2 has been replaced by Table 3, which reports pseudo-dataset results for all configurations over 10 replications, alongside the corresponding results obtained from the original simulator-generated datasets in Table 2 (see also Tables 11 and 17 in Appendix F.5). For example, for the inaccurate ODE solver at high severity, we obtain with on simulator-generated datasets and with on pseudo-datasets. More broadly, the two procedures give similar qualitative conclusions: the reference configuration is not rejected in either case, all high-severity configurations are rejected in both, and at the 0.05 level the decisions differ only for the inaccurate ODE solver at middle severity, which is rejected on simulator-generated datasets () but not on pseudo-datasets (); the shallow-network configuration at low severity is borderline under pseudo-data (). We also evaluate the pseudo-dataset procedure in the CRKP and BDEISS-CT case studies, where the reference configuration is not rejected and poor training is rejected under both the original simulator-generated datasets and pseudo-datasets (Tables 21--24).
These comparisons already include pseudo-datasets generated by models trained under suboptimal configurations and therefore with non-negligible approximation error. We additionally introduce a more extreme stress test in Appendix F.6 in which one posterior mode is deliberately removed from the pseudo-data distribution. Even under this severe generator misspecification, the diagnostic outcome continues to reflect the configuration being retrained: the reference configuration gives and is not rejected, whereas poor training at high severity gives and is rejected.
This stress test also illustrates an important limitation of the pseudo-dataset procedure. It evaluates whether a configuration exhibits the predicted scaling under data generated from the fitted model, but it does not expose errors of that generating model itself, such as a missing mode or underestimated posterior spread.
Q2. We address these points in our response to Weakness 2 above. In brief, we do not expect the predicted scaling law to hold exclusively for well-tuned FMPE configurations. As our added biased-estimator experiment demonstrates, a substantially biased estimator may still exhibit the predicted scaling. Our intended interpretation is therefore asymmetric: departures from the predicted scaling can identify particular forms of configuration-related failure that alter endpoint-variability behavior, whereas failure to reject the null indicates only consistency with the reference sampling regime and should not be interpreted as evidence that the configuration is optimal or that the posterior is unbiased. The revised manuscript now makes this scope explicit.
Official Comment by Authors [1]
W1. We thank the reviewer for the constructive feedback. We agree that our formal theoretical guarantee applies only under the stated infinite-width, regularized two-layer setting, and we do not claim that the theorem directly establishes the N−1/2 scaling law for the finite, deeper architectures used in our experiments. We have clarified this limitation in the revised manuscript and now state explicitly in Section 6 that extending the theory to finite-width and deeper FMPE architectures remains an important direction for future work.
At the same time, there is precedent in the NTK literature for using predictions derived under idealized infinite-width settings as hypotheses to be evaluated empirically in finite neural networks. For example, Lee et al. [1] derive an NTK-based characterization of training dynamics in the infinite-width limit and subsequently examine how closely finite neural networks follow the corresponding linearized dynamics. Similarly, McNamara et al. [2] establish their formal convergence results under an idealized two-layer setting and empirically investigate whether the predicted behavior extends to finite and more complex architectures. These studies do not provide a theoretical guarantee for our setting, but motivate examining whether the predicted asymptotic behavior remains informative outside the exact regime covered by the theorem.
Our experiments serve this empirical role. We use a finite, multilayer residual velocity network trained by stochastic optimization, rather than the two-layer infinite-width model analyzed theoretically. Under the reference configuration, the fitted endpoint variability nevertheless follows the predicted N−1/2 scaling across the controlled benchmark and both epidemiological simulation studies. In contrast, systematic departures emerge as network capacity, optimization budget, or numerical accuracy are degraded. We have also added replications across training seeds and additional robustness analyses in Appendix F.5 to show that this behavior is not explained simply by initialization or optimization randomness.
We therefore view the theoretical result as providing the reference scaling regime underlying the diagnostic, while the experiments test empirically whether this prediction remains useful for the practical finite-network FMPE architectures considered here. We have revised the manuscript to make this distinction between theoretical guarantee and empirical generalization explicit.
[1] Lee, J., Xiao, L., Schoenholz, S. S., Bahri, Y., Novak, R., Sohl-Dickstein, J., & Pennington, J. (2019). Wide neural networks of any depth evolve as linear models under gradient descent. Advances in Neural Information Processing Systems, 32.
[2] McNamara, D., Loper, J., & Regier, J. (2024). Globally convergent variational inference. Advances in Neural Information Processing Systems, 37.