acceptodds
Under review as a conference paper at ICLR 2027

Beyond Accuracy: Diagnosing Flow Matching Posterior Estimation through Endpoint Variability

Abstract

Flow matching posterior estimation (FMPE) offers a flexible and increasingly important approach to simulation-based Bayesian inference, enabling expressive posterior approximation through learned continuous-time transport. Yet its performance can be difficult to diagnose through conventional posterior-accuracy metrics, as they cannot reliably distinguish expected sample-size-driven variability from systematic errors induced by suboptimal configurations. In this work, we develop a theoretically grounded diagnostic based on the scaling of endpoint variability across training-set sizes. Building on the predicted scaling law, we construct a statistical test for assessing whether the sample-size-driven behavior of FMPE is consistent with theoretical expectations, and develop a pseudo-dataset strategy that reduces the need for extensive repeated simulator runs. Experiments, including domain-specific evaluations with complex epidemiological simulators, support the validity of our proposed diagnostic and show that it can identify suboptimal FMPE configurations.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Simulation-based Inference, Bayesian Inference, Flow Matching, Machine Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_yR1LOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a diagnostic for flow matching posterior estimation (FMPE), a widely used method for simulation-based inference (SBI), based on the variability of generated endpoints across independently trained flows. The central idea is to consider how the endpoint variability changes with the number of simulated training samples. The authors establish a central limit theorem for the learned endpoint, and show that endpoint fluctuations around the population regularised estimator exhibit a scaling law. These results are established under a neural tangent kernel (NTK) limit for a regularised two-layer neural network. Based on this result, they construct a statistical test that assesses whether the observed endpoint variability follows the theoretical scaling. The proposed diagnostic is evaluated through several experiments, including a synthetic bimodal posterior and two more complex epidemiological simulators.

Critical strengths
  • The authors provide a careful theoretical analysis of endpoint variability in flow matching posterior estimation, providing reasonable justification for the proposed diagnostic.
  • The diagnostic offers a complementary approach to more conventional posterior-accuracy metrics (e.g., W1, MMD, etc).
Critical weaknesses
  • The theoretical results apply to an idealised infinite-width, regularised two-layer neural network. Meanwhile, the experiments use substantially different architectures and optimisation procedures. As a result, it is not clear whether the predicted scaling law applies directly for the models considered in the paper.
  • The diagnostic has a limited capacity to distinguish between different sources of error. In particular, the estimator could exhibit the predicted scaling, whilst still same retaining substantial approximation or optimisation bias. In my view, this limits the ability of the diagnostic to distinguish insufficient training data from "suboptimal pipelines".
  • The pseudo-datasets used to perform the test are generated from an initially fitted approximation, rather than the true simulator. There is no (theoretical) guarantee that the diagnostic behaviour transfers between these distributions. The empirical justification is limited to a single synthetic example.
Questions for authors
  • What justifies the assumption that diagnostic behaviour under pseudo-data generated from a fitted FMPE model is representative of behaviour under the true simulator? Does the empirical validation shown in Figure 2 hold more widely (e.g., when the fitted FMPE model has significant approximation error)?
  • The predicted scaling law appears compatible with estimators that retain substantial approximation or optimisation bias. Is there any reason (theoretical or otherwise) to expect the scaling law to hold specifically for well-tuned FMPE configurations? Could the authors clarify which types of sub-optimal configuration the diagnostic is expected to identify, and how one should interpret failure to reject the null?
  • In Figure 1 (Section 5.1), left panel, the average W1W_1 plateaus over sample sizes, so it is not clear to me what additional information the additional diagnostic contributes. The authors make the point that this W1W_1 plateau is obscured if one doesn't average over repeated fits, but the proposed diagnostic is itself based on repeated model fits, so I do not find this argument entirely convincing.
  • In Figure 1, central panel, the average W1W_1 continues to decrease as the training-set size increases, suggesting that additional data does improve posterior approximation quality in this example. Is there any evidence that the observed endpoint floor is attributable to insufficient optimisation? What conclusion should be drawn about whether to use more training data, or to modify the optimisation procedure?
  • In Figure 3 (Section 5.4), how were the reference posterior distributions obtained (and validated)? Further, what distinguishes the well-tuned and problematic FMPE setups?
  • In Figures 1-2 and Table 1 (Sections 5.1-5.3), the velocity is parameterised as a sample mean "buffer" plus a residual field (Appendix F.2). Since the buffer already has N−12\smash{N^{-\frac{1}{2}}} fluctuations, could the authors clarify how much of the endpoint scaling observed in practice is attributable to this term? Do the empirical results still look the same if this is removed or altered (e.g., fixed equal to the population mean)?
7 comments
AuthorsOpenReview ↗

Official Comment by Authors [1]

W1. We thank the reviewer for the constructive feedback. We agree that our formal theoretical guarantee applies only under the stated infinite-width, regularized two-layer setting, and we do not claim that the theorem directly establishes the N−1/2N^{-1/2} scaling law for the finite, deeper architectures used in our experiments. We have clarified this limitation in the revised manuscript and now state explicitly in Section 6 that extending the theory to finite-width and deeper FMPE architectures remains an important direction for future work.

At the same time, there is precedent in the NTK literature for using predictions derived under idealized infinite-width settings as hypotheses to be evaluated empirically in finite neural networks. For example, Lee et al. [1] derive an NTK-based characterization of training dynamics in the infinite-width limit and subsequently examine how closely finite neural networks follow the corresponding linearized dynamics. Similarly, McNamara et al. [2] establish their formal convergence results under an idealized two-layer setting and empirically investigate whether the predicted behavior extends to finite and more complex architectures. These studies do not provide a theoretical guarantee for our setting, but motivate examining whether the predicted asymptotic behavior remains informative outside the exact regime covered by the theorem.

Our experiments serve this empirical role. We use a finite, multilayer residual velocity network trained by stochastic optimization, rather than the two-layer infinite-width model analyzed theoretically. Under the reference configuration, the fitted endpoint variability nevertheless follows the predicted N−1/2N^{-1/2} scaling across the controlled benchmark and both epidemiological simulation studies. In contrast, systematic departures emerge as network capacity, optimization budget, or numerical accuracy are degraded. We have also added replications across training seeds and additional robustness analyses in Appendix F.5 to show that this behavior is not explained simply by initialization or optimization randomness.

We therefore view the theoretical result as providing the reference scaling regime underlying the diagnostic, while the experiments test empirically whether this prediction remains useful for the practical finite-network FMPE architectures considered here. We have revised the manuscript to make this distinction between theoretical guarantee and empirical generalization explicit.

[1] Lee, J., Xiao, L., Schoenholz, S. S., Bahri, Y., Novak, R., Sohl-Dickstein, J., & Pennington, J. (2019). Wide neural networks of any depth evolve as linear models under gradient descent. Advances in Neural Information Processing Systems, 32.

[2] McNamara, D., Loper, J., & Regier, J. (2024). Globally convergent variational inference. Advances in Neural Information Processing Systems, 37.

AuthorsOpenReview ↗

Official Comment by Authors [2]

W2. Thank you for raising this important point. We agree that observing the predicted N−1/2N^{-1/2} endpoint-variability scaling does not imply that the fitted posterior is unbiased, correctly specified, or free of approximation, optimization, regularization, or numerical error. In particular, an estimator may retain substantial persistent bias while its sampling variability across training datasets still decreases at the predicted N−1/2N^{-1/2} rate.

Our intended interpretation of the diagnostic is therefore deliberately asymmetric. A systematic departure from the predicted scaling provides evidence that the fitted FMPE estimator is not behaving according to the reference sampling regime and therefore warrants further scrutiny. Conversely, failure to reject H0:α=0.5H_0:\alpha=0.5 should not be interpreted as certifying posterior correctness, ruling out persistent bias, or establishing that limited training data are the only source of poor posterior accuracy. We have revised Sections 1, 3 and 5 to make this scope explicit and to avoid language suggesting that the diagnostic can uniquely identify the source of posterior error.

We also added controlled experiments with deliberately biased estimators in Appendix F.6 (Figure 19). In these experiments, strong regularization induces substantial persistent bias while the reference configuration still exhibits scaling close to the predicted N−1/2N^{-1/2} rate, directly confirming the reviewer’s point. When an additional suboptimal network configuration is imposed, however, the W1W_1 curves remain similar while the endpoint-variability scaling departs markedly from the theoretical rate. Thus, the diagnostic does not detect persistent bias itself, but can still provide additional information beyond W1W_1 by revealing configuration-related scaling departures in the presence of substantial bias.

Similarly, our poor-training experiments are not intended to imply that all insufficiently optimized FMPE models must violate the predicted scaling. Rather, they show that certain practically relevant optimization failures can produce detectable departures from it. We now frame the diagnostic as complementary to posterior-accuracy and posterior-checking methods, rather than as a tool for uniquely distinguishing insufficient training data from all possible forms of suboptimality.

W3. We address this concern in our response to Q1 below.

AuthorsOpenReview ↗

Official Comment by Authors [3]

Q1. We agree that a general theoretical guarantee of transfer from the original simulator distribution to the fitted pseudo-data distribution is difficult to establish. We therefore distinguish what follows from the theory from what we validate empirically.

One direction does follow from our theoretical analysis. The endpoint CLT does not require the training data to be generated by the original simulator; it applies to a fixed data-generating distribution satisfying the stated moment conditions. Once the fitted model used to generate pseudo-data is fixed, the pseudo-datasets are i.i.d. draws from such a distribution, so the N−1/2N^{-1/2} scaling is likewise predicted for an estimator operating in the theoretical regime. What the theory does not guarantee is transfer of diagnostic failures between distributions: a configuration that departs from the predicted scaling under the original simulator is not theoretically guaranteed to exhibit the same departure under pseudo-data.

We have therefore substantially expanded the empirical validation of this transfer. The former Figure 2 has been replaced by Table 3, which reports pseudo-dataset results for all configurations over 10 replications, alongside the corresponding results obtained from the original simulator-generated datasets in Table 2 (see also Tables 11 and 17 in Appendix F.5). For example, for the inaccurate ODE solver at high severity, we obtain α^=0.02±0.01\widehat{\alpha}=0.02\pm0.01 with p<10−3p<10^{-3} on simulator-generated datasets and α^=0.01±0.01\widehat{\alpha}=0.01\pm0.01 with p<10−3p<10^{-3} on pseudo-datasets. More broadly, the two procedures give similar qualitative conclusions: the reference configuration is not rejected in either case, all high-severity configurations are rejected in both, and at the 0.05 level the decisions differ only for the inaccurate ODE solver at middle severity, which is rejected on simulator-generated datasets (p=0.01p=0.01) but not on pseudo-datasets (p=0.07p=0.07); the shallow-network configuration at low severity is borderline under pseudo-data (p=0.05p=0.05). We also evaluate the pseudo-dataset procedure in the CRKP and BDEISS-CT case studies, where the reference configuration is not rejected and poor training is rejected under both the original simulator-generated datasets and pseudo-datasets (Tables 21--24).

These comparisons already include pseudo-datasets generated by models trained under suboptimal configurations and therefore with non-negligible approximation error. We additionally introduce a more extreme stress test in Appendix F.6 in which one posterior mode is deliberately removed from the pseudo-data distribution. Even under this severe generator misspecification, the diagnostic outcome continues to reflect the configuration being retrained: the reference configuration gives α^=0.49±0.02\widehat{\alpha}=0.49\pm0.02 and is not rejected, whereas poor training at high severity gives α^=0.03±0.03\widehat{\alpha}=0.03\pm0.03 and is rejected.

This stress test also illustrates an important limitation of the pseudo-dataset procedure. It evaluates whether a configuration exhibits the predicted scaling under data generated from the fitted model, but it does not expose errors of that generating model itself, such as a missing mode or underestimated posterior spread.

Q2. We address these points in our response to Weakness 2 above. In brief, we do not expect the predicted scaling law to hold exclusively for well-tuned FMPE configurations. As our added biased-estimator experiment demonstrates, a substantially biased estimator may still exhibit the predicted N−1/2N^{-1/2} scaling. Our intended interpretation is therefore asymmetric: departures from the predicted scaling can identify particular forms of configuration-related failure that alter endpoint-variability behavior, whereas failure to reject the null indicates only consistency with the reference sampling regime and should not be interpreted as evidence that the configuration is optimal or that the posterior is unbiased. The revised manuscript now makes this scope explicit.

Reviewer_Hja4OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents a method to diagnose the quality of approximated posteriors obtained through flow matching posterior estimation. They derive a central-limit theorem for their neural flow matching estimator, derive asymptotic bounds and present experimental results comparing their diagnostic method against Wasserstein metrics and test whether it is a good metric to distinguish raw variance from poor posterior reconstruction. The theoretical results provided help frame their proposed solution, and the authors validate these results empirically too.

Critical strengths
  • The authors do a thorough job at justifying their proposed method with a theoretical analysis. This helps framing the work and understanding the contributions.
  • The problem is well positioned and the arguments in the paper are clear.
Critical weaknesses
  • The empirical evaluation and evidence presented is slightly weak, the particular examples and datasets used are not really well justified or described, and it makes it difficult to evaluate the overall validity and relevance of the method.
  • Some of the sections and paragraphs in the text are not clear and obfuscate the paper's arguments.
Questions for authors
  • Fundamentally, while I appreciate the theoretical justification for the method, I find the empirical results shallow, under-explained and under-justified. First, section 5.1 does not really include a detailed explanation on what datasets or benchmarks are used for the plots, or how the results have been collected.
  • Second, the 'shallow network' and 'poor training' regime characteristics seem picked arbitrarily (why 20,000 steps and not more/less for poor training?).
  • Third, the way the results are interpreted feels hand-wavy. Authors state that 'the variability of W1W_1 can obscure the underlying lack of improvement'. I am not sure how can one conclude this decisively from the presented results. Furthermore, I understand the point that the authors try to make with the poor training plot is that the fact that W1W_1 decreases with NN in this regime is misleading because with interrupted training runs one cannot be sure the obtained posterior is accurate. This does not seem obvious from the results alone, and I believe requires further justification.
  • Fourth, it is not obvious to me what the purpose of section 5.4 is. If it is to show that the author's diagnostic method + a well tuned FMPE performs well in the benchmarks, that seems reasonable, but given it does not really improve over NPE one wonders what is the conclusion to be drawn from this.
2 comments
AuthorsOpenReview ↗

Official Comment by Authors [1]

W1 and W2. We thank the reviewer for these thoughtful comments. We have substantially expanded the empirical evaluation and revised the presentation to improve clarity. In particular, we now provide additional experiments across multiple types and severities of suboptimal FMPE configurations, expanded pseudo-dataset evaluations, and two domain-specific epidemiological case studies, together with additional robustness analyses. We have also revised several sections of the manuscript to clarify the motivation, interpretation, and intended scope of the diagnostic. Detailed responses to each point are provided below.

Q1. We have revised both the main text and appendix so that the experimental tasks, datasets, reference posteriors, and result-collection procedures are stated explicitly.

In the main text, the opening paragraph of Section 5 now explicitly identifies and describes the benchmark used in Sections 5.1--5.3, namely a one-dimensional bimodal posterior conditioned on a one-dimensional observation, as well as the two epidemiological simulation studies used in Section 5.4. All training data are generated from the corresponding simulators. We also added Table 1, which summarizes the reference and suboptimal FMPE configurations used in Sections 5.1--5.3.

We also expanded Appendix F.1 to provide additional details on the simulators, generated datasets, and construction and validation of the reference posteriors. Specifically, the reference posterior is available analytically for the bimodal benchmark, is obtained using MCMC for CRKP, and using sequential ABC for BDEISS-CT.

Appendix F.4 now gives a detailed description of how the reported results are collected and how the regression diagnostic is performed. For each configuration and each of five training-set sizes, we fit 20 FMPE models on independently simulated training datasets, compute the endpoint of each fitted model at a fixed probe (z,x)(z,x), and regress the log endpoint variance on log⁡N\log N. The training-set sizes are 15,000--240,000 for the bimodal benchmark and 0.15M--2.4M for the epidemiological case studies. The W1W_1 values in Figure 1 are computed for each fitted model relative to samples from the corresponding reference posterior. Full neural-network architectures, training specifications, and configuration details are provided in Appendices F.2 and F.3.

Q2. We agree that the rationale for these choices was not sufficiently explained in the original version. In the revised experiments, we consider three severity levels for each type of suboptimal configuration, with each level perturbing only one factor of the reference configuration (Table 1 in the main text; Table 9 in Appendix F.3). For example, for poor training we use 20K optimization steps at high severity, 30K at middle severity, and 50K at low severity, compared with 60K steps in the reference configuration. Section 5.2 now reports how the diagnostic changes across these severity levels.

The severity levels were selected to span the range over which the fitted α^\hat{\alpha} changes meaningfully rather than to represent uniquely optimal thresholds. We now explain this rationale and provide a broader scan of each factor in Appendix F.3 (Figure 13). For poor training, reducing the optimization budget further from 20K to 10K steps produces little additional change in the fitted scaling behavior: α^\hat{\alpha} is already close to zero at 20K (α^=0.04\hat{\alpha}=0.04 in the scan). In contrast, increasing the budget to 30K produces a clearly different fitted value, α^=0.22\hat{\alpha}=0.22. The replicated results show the same pattern, with α^=0.01±0.01\hat{\alpha}=0.01\pm0.01 at 20K and 0.20±0.030.20\pm0.03 at 30K (Table 2). We therefore use 20K as the high-severity setting because further reducing the training budget produces little additional change in the diagnostic. At the other end, 70K optimization steps yield essentially the same fitted α^\hat{\alpha} as the 60K-step reference configuration.

The shallow-network severity levels were selected using the same principle. Reducing the high-severity architecture from (width,depth)=(32,2)(\mathrm{width},\mathrm{depth})=(32,2) to (16,2)(16,2) leaves α^\hat{\alpha} close to zero, whereas the middle-severity architecture (64,3)(64,3) produces a clearly different value, α^=0.11\hat{\alpha}=0.11. Thus, the selected levels are intended to represent mild, intermediate, and strong perturbations spanning the informative transition in scaling behavior, rather than arbitrary cutoffs.

AuthorsOpenReview ↗

Official Comment by Authors [2]

Q3. We agree that our original interpretation was stated too strongly, and we have revised Section 5.1 to distinguish the two examples more carefully.

For the shallow-network configuration, we now directly quantify the effect that motivated our original statement. Although the average W1W_1 remains approximately constant across training-set sizes, if one independently selects a single fitted model at each NN and constructs a W1W_1 trajectory, more than 50% of such trajectories are decreasing in our experiment. Thus, when only one fitted model is observed at each training-set size, as would often be the case in practice, variability in W1W_1 across independently fitted models can create the appearance of improving posterior accuracy even when the average W1W_1 has plateaued. We now report this result explicitly in Section F.5 and Figure 15.

For the poor-training configuration, we agree that a decreasing W1W_1 curve alone does not establish that the posterior is inaccurate. We have revised the corresponding interpretation in Section 5.1 to avoid this implication. Our intended point is more focused: the poor-training and reference configurations both show decreasing average W1W_1 as NN increases, but their endpoint-variability behavior is markedly different. The poor-training configuration strongly departs from the predicted N−1/2N^{-1/2} scaling, whereas the reference configuration remains consistent with it. Thus, the scaling diagnostic provides additional information about the behavior of the fitted estimator that is not apparent from the W1W_1 trend alone. We now frame endpoint-variability scaling and posterior-accuracy metrics as providing complementary information, rather than interpreting a decreasing W1W_1 curve itself as evidence that the posterior is accurate or inaccurate.

Q4. We agree that the purpose of Section 5.4 was not sufficiently clear in the original version. The goal of this section is not to show that a well-tuned FMPE necessarily outperforms NPE in posterior accuracy. Rather, it is to evaluate whether the endpoint-variability diagnostic remains informative in more realistic, domain-specific inference problems where posterior accuracy alone may be difficult to interpret.

In both epidemiological case studies, the reference-configuration FMPE has higher W1W_1 than NPE at smaller training-set sizes and becomes competitive only after substantially more simulated training data are available. From the W1W_1 curves alone, however, it is not clear whether this reflects a fundamentally problematic FMPE configuration or simply the data requirements of the fitted estimator. The endpoint-variability diagnostic provides additional information: the reference configuration exhibits scaling close to the theoretically predicted N−1/2N^{-1/2} rate in both simulators, whereas the poor-training configuration shows clear departures from this rate.

We have revised Section 5.4 to make this purpose explicit. Our conclusion is therefore not that FMPE is superior to NPE, but that the scaling diagnostic can provide information about the behavior of an FMPE estimator that is not available from posterior-accuracy curves alone. As emphasized elsewhere in the revised manuscript, agreement with the predicted scaling does not establish that the posterior is accurate or that FMPE is preferable to another inference method; the diagnostic is intended to complement, rather than replace, posterior-accuracy comparisons.

Reviewer_hfmjOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a diagnostic for flow matching posterior estimation based on how fitted transport endpoints vary across independently simulated training sets. For a fixed observation and base-noise sample, the authors establish an endpoint central limit theorem for a regularized kernel estimator, connected to a two-layer neural network through an infinite-width, long-training-time limit. This motivates testing whether endpoint standard deviation decreases as (N^{-1/2}). Experiments on a scalar bimodal problem and two epidemiological simulators show departures from the predicted scaling under several deliberately degraded configurations.

Critical strengths

The theoretical analysis gives the paper a well reference regime. The progression from the empirical ridge solution to an influence-function expansion and then to the endpoint CLT is well organized. The experiments investigate several concrete sources of poor performance, including limited network capacity, insufficient optimization, and inaccurate numerical integration. The epidemiological examples also illustrate a relevant practical situation: a more expressive estimator can initially have worse posterior accuracy than a simpler alternative. These examples give a reasonable motivation for studying information beyond an accuracy curve alone.

Critical weaknesses

The evidence does not yet establish how reliably the test distinguishes limited data from a suboptimal configuration. The CLT describes fluctuations around (R_\lambda), rather than around the true posterior transport. An estimator with persistent approximation or regularization bias can still have (N^{-1/2}) fluctuations. Likewise, a fixed numerical solver can preserve this rate while introducing endpoint bias, provided its output remains a smooth function of the fitted field. The caveat in Section 3 is appropriate, but the interpretations in Sections 5.4 and 6 go further by suggesting that the test can help attribute poor accuracy to insufficient data. The experiments show selected failures that violate the rate; they do not establish how often materially inaccurate configurations retain it. This also matters for the pseudo-dataset procedure: retraining on samples from an inaccurate fitted model may test reproducibility of that model’s errors rather than expose them.

The experiments do not sufficiently separate the sources of endpoint variability or validate the resulting test. The theory isolates training-data fluctuations after taking the width and optimization-time limits, whereas Appendix F.2 uses finite residual networks, stochastic optimization, and a fixed training budget. The treatment of initialization and optimization seeds across replicates is not clear. A variance floor from these sources could explain a flattened scaling curve without identifying the underlying source of posterior error. The architecture also adds the empirical parameter mean directly to the velocity field; this term has (1/N) variance under the toy prior, making an ablation particularly relevant. Appendix F.3 specifies normality checks, size exclusions, and additional diagnostics, but their outcomes are not presented. Individual fitted slopes and p-values therefore provide limited evidence about the procedure’s false-positive rate and detection power.

Questions for authors
  1. How are initialization, minibatch, and base-noise seeds handled across independent fits? Can repeated fits on the same dataset separate training randomness from between-dataset variability? Does the main result persist without the empirical-mean offset in Appendix F.2?

  2. Have you evaluated configurations with substantial persistent posterior bias that nevertheless retain (N^{-1/2}) endpoint scaling? For the pseudo-dataset procedure, what happens when the generating estimator misses a mode or substantially underestimates posterior spread?

  3. Can you report the regime-check results described in Appendix F.3, including any excluded training sizes, and empirical rejection frequencies over repeated complete diagnostic runs? These would help assess calibration after applying the exclusion rule.

4 comments
AuthorsOpenReview ↗

Official Comment by Authors [1]

W1. Thank you for the careful review and raising this point. We agree that observing the predicted N−1/2N^{-1/2} endpoint-variability scaling does not imply that the fitted posterior is unbiased, correctly specified, or free of approximation, optimization, regularization, or numerical error. Indeed, as noted in Section 3, an estimator may retain systematic bias while still exhibiting the predicted sampling-rate behavior.

Our intended interpretation of the diagnostic is therefore deliberately asymmetric. A systematic departure from the predicted scaling provides evidence that the fitted FMPE estimator is not behaving according to the reference sampling regime and therefore warrants further scrutiny. Conversely, failure to reject H0:α=0.5H_0:\alpha=0.5 should not be interpreted as certifying that the FMPE configuration is optimal, that the fitted posterior is unbiased, or that limited training data are the only source of poor posterior accuracy.

We agree that some wording in Sections 1 and 5 could be read more strongly than intended, particularly phrases such as “distinguish limited-data effects from suboptimal configurations.” We have revised these statements to make clear that the diagnostic provides additional information about estimator scaling behavior rather than uniquely identifying the source of posterior error. Agreement with the predicted scaling indicates only that the observed endpoint variability is consistent with the theoretical sampling regime and does not rule out persistent bias or other forms of model error.

To examine this point empirically, we also added controlled experiments with deliberately biased estimators in Appendix F.6 (Figure 19). We induce substantial persistent bias through strong regularization and find that the biased reference configuration still exhibits scaling close to the predicted N−1/2N^{-1/2} rate, as expected. When an additional suboptimal network configuration is imposed, however, the W1W_1 curves remain similar while the endpoint-variability scaling departs markedly from the theoretical rate. This experiment illustrates both the limitation and the intended role of the diagnostic: it does not detect persistent bias itself, but can still provide additional information beyond W1W_1 by revealing configuration-related scaling departures in the presence of substantial bias.

W2. Thank you for these important points. We address the concerns regarding training randomness, the empirical-mean buffer, and validation of the diagnostic in our responses to Questions 1 and 3 below. In particular, the revised experiments include replications across training seeds, an ablation removing the empirical-mean buffer, fitted slopes after separating training randomness from between-dataset variability, and the previously unreported regime-check, Shapiro--Wilk, and Brown--Forsythe results.

AuthorsOpenReview ↗

Official Comment by Authors [2]

Q1. Each fit involves two seeds: a dataset seed, which generates the simulated training set, and a training seed, which controls the network initialization, minibatch order, and base-noise sampling. Within one run of the diagnostic, all fits share a single training seed, while only the dataset seed varies across fits. The first version of the paper reported one such run. Thus, training randomness does not directly contribute to the between-dataset endpoint variance within a run, although interactions between the training procedure and individual datasets may remain. We now state this explicitly in the “Seed setting” paragraph of Appendix F.2.

Following the reviewer’s suggestion, we repeated the entire diagnostic 10 times on the same set of datasets, using a different training seed for each replication (Table 11; also summarized in Table 2 of the main text). These replications allow us to estimate the contribution associated with training randomness and its interaction with the datasets. We then re-estimated α^\hat{\alpha} after separating this component from the between-dataset variability, yielding an adjusted α^\hat{\alpha} (Table 12). The conclusions remain unchanged. The largest adjustments occur for poor training (middle and high severity), large target variance (middle and high severity), and shallow network (middle severity), where α^\hat{\alpha} increases by approximately 0.05--0.07. For example, under poor training at high severity, the fitted α^\hat{\alpha} is 0.01±0.010.01 \pm 0.01, while the adjusted α^\hat{\alpha} is 0.08. For the other configurations, the adjustment is at most 0.04. After adjustment, all affected estimates remain below 0.5, with the closest being 0.39 for large target variance at middle severity, and all high-severity configurations remain at or below 0.12. These results indicate that training randomness and its interaction with the datasets are not the primary explanation for the observed flattening of endpoint-variance scaling. Additional results are provided in Appendix F.5 for the bimodal task (Tables 11--12), the pseudo-dataset experiments (Tables 17--18), and the epidemiological case studies (Tables 21--24).

The main result also persists after removing the empirical-mean offset. All replication experiments described above were conducted without this offset, and the fitted α^\hat{\alpha} values remain close to those reported in the first version, which used the offset. On the bimodal task, the difference is at most 0.05 (Table 10 versus Table 11), and under the pseudo-dataset strategy it is at most 0.07 (Table 16 versus Table 17). In particular, the reference configuration gives α^=0.49\hat{\alpha}=0.49 with the offset and α^=0.50±0.04\hat{\alpha}=0.50\pm0.04 without it. The main text now reports the results without the empirical-mean offset (Tables 2 and 3). These results directly address the concern that the offset itself might induce the observed N−1/2N^{-1/2} scaling.

AuthorsOpenReview ↗

Official Comment by Authors [3]

Q2. To create substantial persistent bias, we take the reference configuration and increase the regularization from λ=0.01\lambda=0.01 to λ=4.0\lambda=4.0. The resulting estimator has a clearly larger average W1W_1 than the reference configuration at every training-set size (Figure 19), while retaining the predicted scaling: across 10 replications, α^=0.48±0.04\hat{\alpha}=0.48\pm0.04 (p=0.79p=0.79 for the median α^\hat{\alpha}), and the adjusted α^\hat{\alpha} is 0.51. This is consistent with the theory, in which the CLT is centered at RλR_\lambda for any fixed λ\lambda, and confirms the reviewer’s point that the diagnostic does not identify persistent bias itself.

The diagnostic can nevertheless remain informative about additional configuration-related failures in the presence of such bias. When we further replace the network with the shallow-network configuration at middle severity, while keeping the same λ=4.0\lambda=4.0, the average W1W_1 remains within approximately 0.015 of that of the biased reference estimator, whereas the scaling behavior changes substantially: α^=0.13±0.03\hat{\alpha}=0.13\pm0.03 (p<10−3p<10^{-3}), with adjusted α^=0.16\hat{\alpha}=0.16. Thus, two estimators with similarly poor W1W_1 can exhibit markedly different endpoint-variability scaling.

For the pseudo-dataset procedure, we did not observe a naturally occurring case in which the generating estimator completely missed a mode in our original experiments (e.g., Figures 16 and 17). We therefore constructed an extreme stress test by retaining only pseudo-samples satisfying x>0.5128x>0.5128 and θ>0\theta>0 (Figure 18). This removes one of the two posterior modes and substantially reduces posterior spread. The diagnostic outcome then reflects the configuration being retrained rather than the error of the pseudo-data generator: the reference configuration retrained on the unimodal pseudo-dataset is not rejected (α^=0.49±0.02\hat{\alpha}=0.49\pm0.02, p=0.90p=0.90 for the median α^\hat{\alpha}), whereas poor training at high severity is rejected (α^=0.03±0.03\hat{\alpha}=0.03\pm0.03, p<10−3p<10^{-3}). The Brown--Forsythe test gives the same qualitative conclusions.

We therefore agree with the reviewer’s interpretation. The pseudo-dataset procedure evaluates whether a retrained configuration exhibits the predicted scaling on data generated from the fitted model; it does not expose misspecification of that generating model itself, such as a missing mode or underestimated posterior spread. Those errors must be assessed with posterior-accuracy or posterior-predictive checks, which our diagnostic is intended to complement rather than replace. We have added this limitation explicitly to Section 6, where we now state that the pseudo-dataset strategy does not expose errors of the model used to generate the pseudo-datasets.

Q3. We now report the outcomes of the regime check, the Fisher-combined Shapiro–-Wilk test and the Brown–-Forsythe test to Appendix F.5 (the regime check itself is now described in Appendix F.4). The regime check fails only rarely, and only at the smallest size N=15,000N = 15{,}000 (Table 13). No size is excluded for the reference configuration or for any low-severity configuration. In the worst case, large target variance (high severity), it excludes 6% of the sizes (i.e., 3 sizes in 5 * 10 = 50 sizes; 5 sizes for one test and there are 10 replications). For all the configurations, we get p>0.05p > 0.05 in the Fisher-combined Shapiro–-Wilk test in every replication (Tables 14 and 19). There is no guaranteed ordering between the Brown-–Forsythe p-values and our regression p-values, but the two tests lead to similar statistical conclusions (Tables 15 and 20): for instance, in the first replication, the Brown-–Forsythe test gives p<10−3p < 10^{-3} for all high-severity configurations and p<0.10p < 0.10 for all middle-severity ones, and it does not reject the low-severity and reference configurations. Corresponding diagnostics for the epidemiological case studies are in Tables 21-–24.

AuthorsOpenReview ↗

Official Comment by Authors [4]

An update of our experiment. We conducted one further experiment to verify our diagnostic can provide additional information compared to average W1W_1.

In addition to the poor training configuration, we use shallow-network FMPE configurations in the CRKP and BDEISS-CT epidemiological case studies (Appendix F.7). In both case studies, reducing network capacity leads to only modest changes in the average W1W_1 curves relative to the reference FMPE configuration, making the difference difficult to assess from posterior accuracy alone. In contrast, the endpoint-variability diagnostic shows departures from the predicted scaling. Across 10 replications, the shallow-network configuration yields α^=0.34±0.02\hat{\alpha}=0.34\pm0.02 for CRKP and α^=0.36±0.03\hat{\alpha}=0.36\pm0.03 for BDEISS-CT, compared with values close to 0.50.5 under the reference configuration. These additional results further support that endpoint-variability scaling can provide information beyond average W1W_1 when assessing FMPE behavior.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary Paragraph for Area Chairs and Reviewers

We thank the reviewers for their thoughtful and constructive feedback. We believe the main concerns raised fall into two categories, both of which we address through revisions to the manuscript and substantial additional experiments.

First, several comments highlighted the importance of clarifying the intended interpretation of the diagnostic. Our proposed test is deliberately asymmetric: a significant departure from the theoretically predicted N−1/2N^{-1/2} endpoint-variability scaling indicates that the fitted FMPE is not exhibiting the expected sample-size dependence and therefore warrants further scrutiny. Conversely, agreement with the predicted scaling indicates that the estimator's variability behaves as expected with increasing training-set size, but does not by itself establish posterior accuracy or rule out persistent bias. We have clarified this interpretation in Section 3 and revised the corresponding language throughout Sections 1 and 5. We also added controlled experiments with deliberately biased estimators in Appendix F.6 (Figure 18 and 19), showing that bias and endpoint-variability scaling capture different aspects of estimator behavior: substantial bias can coexist with the predicted scaling, while the diagnostic can still reveal additional configuration-related scaling departures that are not apparent from W1W_1 alone.

Second, reviewers requested stronger empirical validation of the diagnostic. In response, we substantially expanded the experiments. Section 5.2 and Table 2 now evaluate multiple types and severities of suboptimal FMPE configurations with replications across training seeds. We additionally removed the empirical-mean buffer used in the original implementation and found that this had negligible influence on the fitted scaling behavior (Appendix F.5, Table 10 vs. Table 11), directly addressing the concern that the buffer itself might induce the observed N−1/2N^{-1/2} scaling. Section 5.3 and Table 3 provide expanded pseudo-dataset results, while Section 5.4 and Figure 2 extend the evaluation to two complex epidemiological simulators. Appendix F.5 provides further robustness analyses, including fitted slopes after separating training randomness from between-dataset variability (Table 12), regime-check outcomes (Table 13), Fisher-combined Shapiro--Wilk tests, Brown--Forsythe tests (Tables 14--15), and additional pseudo-dataset and epidemiological results.

Detailed point-by-point response to each reviewer are provided below.

AuthorsOpenReview ↗

Official Comment by Authors

We thank the reviewers again for their careful reviews and constructive discussion. As a final revision, we added one further experiment to the first revision, using shallow-network FMPE configurations in the CRKP and BDEISS-CT epidemiological case studies (Appendix F.7).

In both case studies, reducing network capacity leads to only modest changes in the average W1W_1 curves relative to the reference FMPE configuration, making the difference difficult to assess from posterior accuracy alone. In contrast, the endpoint-variability diagnostic shows departures from the predicted scaling. Across 10 replications, the shallow-network configuration yields α^=0.34±0.02\hat{\alpha}=0.34\pm0.02 for CRKP and α^=0.36±0.03\hat{\alpha}=0.36\pm0.03 for BDEISS-CT, compared with values close to 0.50.5 under the reference configuration. These additional results further support that endpoint-variability scaling can provide information beyond average W1W_1 when assessing FMPE behavior.

We appreciate the reviewers’ feedback, which has helped us clarify the interpretation of the diagnostic and substantially strengthen the empirical evaluation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.