- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies whether approximate simulations help train a neural posterior estimator when only a limited number of exact simulations are available. It compares exact-only training, mixed training, and surrogate pretraining followed by exact-data fine-tuning. The authors find that mixed training introduces bias, while two-stage training provides little improvement over exact-only training in their experiments. The paper also examines Fano-based allocation of exact simulations and proposes a classifier-based diagnostic for assessing surrogate quality before training an NPE. Experiments primarily focus on stochastic gene-expression models, with additional evaluations on SIR and single-cell data.
Critical strengths
-
The matched simulator controls help identify the sources of inference error. The comparison between LNA and NB isolates the effect of distributional shape, while the comparison between CLE and the hybrid highlights the importance of preserving discrete promoter dynamics. These experiments provide useful insights into why certain approximations lead to poor posterior estimates.
-
The paper compares exact-only, mixed, and two-stage training under the same exact-simulation budget. This provides a controlled evaluation of whether surrogate simulations offer benefits beyond using the available exact simulations alone. The use of multiple evaluation metrics, including coverage, bias, and SBC, further strengthens the empirical analysis.
Critical weaknesses
-
The main experiments do not sufficiently test the regime where surrogate simulations would be most valuable. In Section 4.2, exact-only training already achieves near-nominal coverage 0.91 for with only 160 simulations. Therefore, the finding that surrogate pretraining provides no improvement may reflect the relative simplicity of the benchmark rather than a general limitation of multifidelity NPE. This concern is further compounded by Table 4, where several exact simulators are faster than the CLE surrogate.
-
Incorrect interpretation of classifier AUC as total variation distance. The diagnostic in Section 4.4 relies on the approximation $\mathrm{TV}\approx 2\mathrm{AUC}-1$, but this identity generally holds for optimal classification accuracy, not ROC AUC. While classifier AUC may still provide a useful ranking of surrogates, the reported discrepancy values cannot be interpreted as estimates of total variation distance without further justification. This undermines the quantitative interpretation of one of the paper's main contributions.
Questions for authors
-
Can the authors evaluate whether two-stage training provides benefits in settings where the exact-only NPE is substantially undertrained or poorly calibrated due to a limited simulation budget? Such experiments would help clarify whether the negative results in Section 4.2 generalize beyond the relatively simple benchmark considered.
-
Can the authors justify the approximation in Section 4.4? Since this relationship generally holds for optimal classification accuracy rather than ROC AUC, could the authors provide a theoretically justified TV estimator and verify whether the surrogate ranking remains consistent?