- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2(+1)
Read review
Paper summary
The authors propose BISIP, a method that determines how to generate NPE training samples that would be informative for reconstructing the observations , which can reduce computational cost. To do so, they parameterize the NPE as a unimodal Gaussian with diagonal covariance and define a surrogate metric that diagnoses where the NPE has the highest bias for generating the next training samples. This metric is dependent on differentiating the forward model, either using a neural surrogate or if the simulator is simple enough to be differentiated in the first place. Their method is validated on both toy experiments and a realistic cosmological reconstruction inverse problem, where it outperforms baselines.
Critical strengths
Generating training samples in SBI is costly, while lacking training coverage can cause bias in the resulting posterior. This problem is important for many real-world applications, and this paper aims to address both of these.
Critical weaknesses
The posterior has a highly limited parameterization; namely, a unimodal Gaussian with diagonal covariance. It is also unclear if the bias metric extends to more complex or multimodal posteriors where the true posterior mean error should actually be high.
The metric for computing is dependent on the learned forward model surrogate . Since we already use (which may contain errors) to decide which extra data to simulate, it would be useful to compare to generating much more data with the learned and training on that, to see if this "middle ground" of using for decision but not simulation is useful.
Questions for authors
How does BISIP perform on posterior distributions that have multimodality, i.e. from symmetries in the simulation operator?
In the case where can be differentiated without a surrogate, how does the learned compare to the true Jacobian? And in the same scenario, how does the actual bias compare to the metric, particularly for multimodal posteriors?
3 comments
Response to Authors
I thank the authors for their response, which answers some of my concerns, and I'll raise the soundness score. However, I still think that even though the goal is a single reconstruction of a multimodal posterior, the bias metric still depends on the posterior mean which is usually not close to any true solution in a multimodal case. It is also still unclear for the ionization field problem whether the bias-selected data that uses the surrogate as a metric has any improvement from just generating more training pairs with .
Response
We thank the reviewer for the acknowledging the response. We address further points below
I still think that even though the goal is a single reconstruction of a multimodal posterior, the bias metric S still depends on the posterior mean which is usually not close to any true solution in a multimodal case. We acknowledge that the proposed algorithm may not be able to handle multimodal simulation-based inverse problems, and is not tested on such problems. We claim that the proposed algorithm works well for problems with unimodal inverse solutions and support that empirically. Such problems abound in practice and are challenging to solve.
It is also still unclear for the ionization field problem whether the bias-selected data that uses the surrogate F as a metric has any improvement from just generating more training pairs with F.
We understand this important point of the reviwer To clarify, in the settings we target, the training set cannot simply be scaled up. Simulator calls are expensive, so the labelled budget is small . Real observations are also limited, and the model must work on observations that may lie outside the support of the training data. With so few labelled pairs, which pairs we add matters more than how many. BISIP uses the bias score to direct the limited simulator budget toward the observations where the current inverse model is least reliable, rather than spending it on regions it already handles well.
The question is therefore whether, at an equal simulator budget, targeted acquisition beats untargeted data generation. We test this with the two comparisons below.
Random (known as Rand) baseline (Table 3). Observations are selected uniformly at random from the unlabelled pool, independently of S(φ, y*). For each selected observation , we sample an inverse candidate from the standard posterior predictive,
$u_j=\mu_\phi(y_j)+\sigma_\phi(y_j)\odot\epsilon, \qquad \epsilon\sim\mathcal N(0,I).$
Each candidate is evaluated with the forward simulator F, without the learned surrogate, and the resulting pair is added to the training set.
Ablation of acquisition components (Section A.5). Additionaly, , we ran a controlled ablation on the ionization-field reconstruction benchmark with four configurations: Random, Random + BISIP Inflation, BISIP Selection Only, and full BISIP to highlight the strength of components of BISIP . All use the same evaluation procedure. Results are provided in the table below
| Configuration | ↓ | ↓ | NLL ↓ |
|---|---|---|---|
| Random | |||
| Random + BISIP Inflation | |||
| BISIP Selection Only | |||
| BISIP |
Bias-based selection, alone or in the full method, improves on Random on all three metrics. Inflation alone gives no clear gain over Random ( 4.01 vs 3.94, within error bars), so most of the improvement comes from selection. Selection Only and full BISIP are comparable within error bars.
Comparison with training with additional data. To test directly whether targeted acquisition helps beyond simply enlarging the training set, we added a passive baseline. Starting from the same 150 initial examples, we add 48 independently generated examples from the original 21cmFAST–SiMPLE-GEN–DataAugment pipeline, with no inverse-model-based selection and no surrogate-guided acquisition. We train on the resulting 198 examples and evaluate it with the same held-out data and training protocol as BISIP.Results are below
| Configuration | ↓ | ↓ | NLL ↓ |
|---|---|---|---|
| Random Passive ( only) |
This comparison allows us to assess whether BISIP provides benefits beyond those obtained by simply acquiring additional samples using and clearly if we compare it to Table 3 in manuscript and are provided in beginning of the response, all acquisition variants outperform the Random passive on every metric.
Response (1/1)
We sincerely thank you for the valuable feedback and for your acknowledgment of our work. We address each comment below.
W1. Unimodal diagonal Gaussian posterior; unclear whether S extends to complex or multimodal posteriors.
Q1. How does BISIP perform on multimodal posteriors, e.g. from symmetries in the simulation operator?
In the present work, our primary focus is to improve inverse reconstruction (i.e., a single solution), rather than faithful recovery of arbitrary posterior distributions (i.e., all possible solutions). The major challenges we tackle are scalability to high dimensions and unknown priors. We employ a Gaussian posterior with diagonal covariance primarily for computational scalability, since it allows us to work efficiently in high dimensions and facilitates rapid adaptation through sequential simulation acquisition.
We acknowledge that this choice limits the ability of the approach to represent multimodal posteriors. In such cases, BISIP may converge to one of the solutions. Moreover, when the symmetries are known, equivalent solutions can be obtained by applying the symmetry transformations to the recovered solution. Also, simulation-based inverse problems with "unimodal" solutions are very common in natural sciences.
W2. S depends on the learned forward surrogate F^, which may contain errors. Compare against generating much more data with F^ and training on that.
Q2. Where F can be differentiated without a surrogate, how does the Jacobian of the learned surrogate compare to the true Jacobian? How does the actual bias compare to S, particularly for multimodal posteriors?
Role of the learned forward surrogate. We thank the reviewer for raising these important questions. The learned surrogate F^ is not a central component of BISIP and is introduced only for the high-dimensional Ionization Field Reconstruction benchmark. We have therefore moved its description to Section 5.3 to clarify its benchmark-specific role. For the other two benchmarks, we use the known forward operators and their exact Jacobians, so no learned surrogate is required. In the ionization-field problem, F^ approximates the forward operator and its Jacobian for estimating the bias score S(ϕ,y⋆). It is not used to generate additional training pairs.
Assessment of the surrogate-guided bias score. To examine whether the estimated score is a useful indicator of the actual reconstruction bias B, we compare S(ϕ,yi) with the ground-truth bias B(yi)=∥μϕ(yi)−ui⋆∥22 on held-out observations. We have added this analysis to Appendix A.6 of the revised manuscript. The results show a strong association between the estimated score and the actual bias throughout acquisition. In particular, at the final acquisition round, the Pearson correlation between S(ϕ,y⋆) and B is approximately 0.77 for the Inverse Laplace benchmark, 0.85 for ionization-field reconstruction, and 0.78 for the linear inverse problem with κ=100.