acceptodds
Under review as a conference paper at ICLR 2027

BISIP: Bias-Guided Stochastic Search for Simulation-Based Inverse Problems

Abstract

Complex simulations, common in science and engineering, lead to challenging inverse problems. Bayesian methods, in particular deep learning-based ones, are widely used but are limited by data scarcity, computational cost of simulations, and distribution shift during testing. To address these challenges, we develop a stochastic search-based method to iteratively refine the inverse estimate. We use forward-consistency to estimate model bias, which, together with the model variance, guides the active acquisition of training samples to improve inverse estimation during testing. We evaluate the proposed approach on inverse problems of increasing complexity, including linear inverse with heteroscedastic noise, inverse Laplace transform, and non-linear reconstruction of neutral hydrogen distribution in the early universe from sparse cosmological observations. Across the evaluated settings, our method consistently improves inverse estimation compared with the considered active-learning and sequential inference baselines. In the cosmological reconstruction task, BISIP achieves the lowest estimation error among the evaluated methods while requiring substantially fewer simulator evaluations.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Inverse Problems, Simulation based inference, Stochastic Search, Uncertainty Quantification, Active Learning, Physical Sciences

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Ymy1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2(+1)
Read review
Paper summary

The authors propose BISIP, a method that determines how to generate NPE training samples that would be informative for reconstructing the observations y∗y^*, which can reduce computational cost. To do so, they parameterize the NPE as a unimodal Gaussian with diagonal covariance and define a surrogate metric that diagnoses where the NPE has the highest bias for generating the next training samples. This metric is dependent on differentiating the forward model, either using a neural surrogate or if the simulator is simple enough to be differentiated in the first place. Their method is validated on both toy experiments and a realistic cosmological reconstruction inverse problem, where it outperforms baselines.

Critical strengths

Generating training samples in SBI is costly, while lacking training coverage can cause bias in the resulting posterior. This problem is important for many real-world applications, and this paper aims to address both of these.

Critical weaknesses

The posterior has a highly limited parameterization; namely, a unimodal Gaussian with diagonal covariance. It is also unclear if the bias metric SS extends to more complex or multimodal posteriors where the true posterior mean error should actually be high.

The metric for computing SS is dependent on the learned forward model surrogate FF. Since we already use FF (which may contain errors) to decide which extra data to simulate, it would be useful to compare to generating much more data with the learned FF and training on that, to see if this "middle ground" of using FF for decision but not simulation is useful.

Questions for authors

How does BISIP perform on posterior distributions that have multimodality, i.e. from symmetries in the simulation operator?

In the case where FF can be differentiated without a surrogate, how does the learned ∇F\nabla F compare to the true Jacobian? And in the same scenario, how does the actual bias compare to the SS metric, particularly for multimodal posteriors?

3 comments
AuthorsOpenReview ↗

Response (1/1)

We sincerely thank you for the valuable feedback and for your acknowledgment of our work. We address each comment below.

W1. Unimodal diagonal Gaussian posterior; unclear whether S extends to complex or multimodal posteriors.

Q1. How does BISIP perform on multimodal posteriors, e.g. from symmetries in the simulation operator?

In the present work, our primary focus is to improve inverse reconstruction (i.e., a single solution), rather than faithful recovery of arbitrary posterior distributions (i.e., all possible solutions). The major challenges we tackle are scalability to high dimensions and unknown priors. We employ a Gaussian posterior with diagonal covariance primarily for computational scalability, since it allows us to work efficiently in high dimensions and facilitates rapid adaptation through sequential simulation acquisition.

We acknowledge that this choice limits the ability of the approach to represent multimodal posteriors. In such cases, BISIP may converge to one of the solutions. Moreover, when the symmetries are known, equivalent solutions can be obtained by applying the symmetry transformations to the recovered solution. Also, simulation-based inverse problems with "unimodal" solutions are very common in natural sciences.

W2. SS depends on the learned forward surrogate F^\hat{F}, which may contain errors. Compare against generating much more data with F^\hat{F} and training on that.

Q2. Where FF can be differentiated without a surrogate, how does the Jacobian of the learned surrogate compare to the true Jacobian? How does the actual bias compare to SS, particularly for multimodal posteriors?

Role of the learned forward surrogate. We thank the reviewer for raising these important questions. The learned surrogate F^\hat{F} is not a central component of BISIP and is introduced only for the high-dimensional Ionization Field Reconstruction benchmark. We have therefore moved its description to Section 5.3 to clarify its benchmark-specific role. For the other two benchmarks, we use the known forward operators and their exact Jacobians, so no learned surrogate is required. In the ionization-field problem, F^\hat{F} approximates the forward operator and its Jacobian for estimating the bias score S(ϕ,y⋆)S(\phi, y^\star). It is not used to generate additional training pairs.

Assessment of the surrogate-guided bias score. To examine whether the estimated score is a useful indicator of the actual reconstruction bias BB, we compare S(ϕ,yi)S(\phi, y_i) with the ground-truth bias B(yi)=∥μϕ(yi)−ui⋆∥22B(y_i) = \lVert \mu_\phi(y_i) - u_i^\star \rVert_2^2 on held-out observations. We have added this analysis to Appendix A.6 of the revised manuscript. The results show a strong association between the estimated score and the actual bias throughout acquisition. In particular, at the final acquisition round, the Pearson correlation between S(ϕ,y⋆)S(\phi, y^\star) and BB is approximately 0.77 for the Inverse Laplace benchmark, 0.85 for ionization-field reconstruction, and 0.78 for the linear inverse problem with κ=100\kappa=100.

Reviewer_Ymy1OpenReview ↗

Response to Authors

I thank the authors for their response, which answers some of my concerns, and I'll raise the soundness score. However, I still think that even though the goal is a single reconstruction of a multimodal posterior, the bias metric SS still depends on the posterior mean which is usually not close to any true solution in a multimodal case. It is also still unclear for the ionization field problem whether the bias-selected data that uses the surrogate FF as a metric has any improvement from just generating more training pairs with FF.

We thank the reviewer for the acknowledging the response. We address further points below

I still think that even though the goal is a single reconstruction of a multimodal posterior, the bias metric S still depends on the posterior mean which is usually not close to any true solution in a multimodal case. We acknowledge that the proposed algorithm may not be able to handle multimodal simulation-based inverse problems, and is not tested on such problems. We claim that the proposed algorithm works well for problems with unimodal inverse solutions and support that empirically. Such problems abound in practice and are challenging to solve.

It is also still unclear for the ionization field problem whether the bias-selected data that uses the surrogate F as a metric has any improvement from just generating more training pairs with F.

We understand this important point of the reviwer To clarify, in the settings we target, the training set cannot simply be scaled up. Simulator calls are expensive, so the labelled budget is small . Real observations are also limited, and the model must work on observations that may lie outside the support of the training data. With so few labelled pairs, which pairs we add matters more than how many. BISIP uses the bias score S(ϕ,y∗)S(\phi, y^*) to direct the limited simulator budget toward the observations where the current inverse model is least reliable, rather than spending it on regions it already handles well.

The question is therefore whether, at an equal simulator budget, targeted acquisition beats untargeted data generation. We test this with the two comparisons below.

Random (known as Rand) baseline (Table 3). Observations are selected uniformly at random from the unlabelled pool, independently of S(φ, y*). For each selected observation yj∗y_j^*, we sample an inverse candidate from the standard posterior predictive,

$u_j=\mu_\phi(y_j)+\sigma_\phi(y_j)\odot\epsilon, \qquad \epsilon\sim\mathcal N(0,I).$

Each candidate is evaluated with the forward simulator F, without the learned surrogate, and the resulting pair is added to the training set.

Ablation of acquisition components (Section A.5). Additionaly, , we ran a controlled ablation on the ionization-field reconstruction benchmark with four configurations: Random, Random + BISIP Inflation, BISIP Selection Only, and full BISIP to highlight the strength of components of BISIP . All use the same evaluation procedure. Results are provided in the table below

ConfigurationEestE_\mathrm{est} ↓ErecE_\mathrm{rec} ↓NLL ↓
Random3.94±0.113.94 \pm 0.116.15±0.636.15 \pm 0.63−1.2489±0.1589-1.2489 \pm 0.1589
Random + BISIP Inflation4.011±0.144.011 \pm 0.145.95±0.345.95 \pm 0.34−1.2595±0.1566-1.2595 \pm 0.1566
BISIP Selection Only3.392±0.273.392 \pm 0.275.77±0.215.77 \pm 0.21−1.3638±0.1256-1.3638 \pm 0.1256
BISIP3.57±0.083.57 \pm 0.085.60±0.165.60 \pm 0.16−1.3133±0.1693-1.3133 \pm 0.1693

Bias-based selection, alone or in the full method, improves on Random on all three metrics. Inflation alone gives no clear gain over Random (EestE_{est} 4.01 vs 3.94, within error bars), so most of the improvement comes from selection. Selection Only and full BISIP are comparable within error bars.

Comparison with training with additional data. To test directly whether targeted acquisition helps beyond simply enlarging the training set, we added a passive baseline. Starting from the same 150 initial examples, we add 48 independently generated examples from the original 21cmFAST–SiMPLE-GEN–DataAugment pipeline, with no inverse-model-based selection and no surrogate-guided acquisition. We train on the resulting 198 examples and evaluate it with the same held-out data and training protocol as BISIP.Results are below

ConfigurationEestE_\mathrm{est} ↓ErecE_\mathrm{rec} ↓NLL ↓
Random Passive (FF only)4.10±0.074.10 \pm 0.077.24±0.277.24 \pm 0.27−1.116±0.1732-1.116 \pm 0.1732

This comparison allows us to assess whether BISIP provides benefits beyond those obtained by simply acquiring additional samples using FF and clearly if we compare it to Table 3 in manuscript and are provided in beginning of the response, all acquisition variants outperform the Random passive on every metric.

Reviewer_Hd27OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes using an (amortized) forward model to estimate the bias in posterior estimates in simulation based inference. This is done by using the difference in reconstructed outputs from the true output, and mapping this information back to the parameter space using an approximate inverse Jacobian of the amortized forward model. This is used to inform an active learning loop where we pick the observations with the largest bias, sample from a widened posterior for those observations, and simulate the real forward model to gain more data. The model is retrained, and the loop starts again.

Critical strengths
  • Estimating the bias in the posterior distribution and using that to inform new sample acquisition is a good idea.
  • The method provides a clear improvement over previous methods, especially with large condition numbers
Critical weaknesses
  • The derivation for posterior bias seems to assume that the forward model does not have noise. This should be stated as a caveat, and discussion should be included for how this affects the central claims of the paper.
  • It seems unclear under what assumptions is Equation 10. a good approximation to Eq.9. The Pearson correlation reported in Section 5.2 does not seem particularly good (even though the method does seem to work in practice). Could the authors provide a more thorough motivation for Eq.10., and a more detailed experimental characterization? It seems that it would be important to have a better understanding on when should we expect to trust the estimator in practice. E.g., is there any simple case where the estimator is exact? If so, how close is that to the experimental setting used in the paper in practice?
  • It seems that the paper does not have a proof of convergence to the true posterior. This can make it more difficult to reason about how the method works. This should be either stated as a clear caveat, or additional work should be done to establish such a convergence argument.
Questions for authors
  • I may be misunderstanding, but seems that the expectation in the definition of S(ϕ,y∗)S(\phi, y*) is redundant; E[∣μϕ−u∗∣2]=∣μϕ−u∗∣2E[|\mu_\phi - u*|^2] = |\mu_\phi - u*|^2. Following that, Theorem 3.1. sneaks in the result in the assumption: If the model is unbiased as in E[u]=μϕ=u∗E[u] = \mu_\phi = u*, then S(ϕ,y∗)=0S(\phi, y*) = 0 trivially.
  • In the ionization field estimation problem, it seems that both an ionization field and a corresponding halo catalogue is given as input to the forward model. If the posterior estimator only outputs the ionization field, how is the corresponding halo catalogue obtained for the forward simulator?
4 comments
AuthorsOpenReview ↗

Response (1/2)

We sincerely thank the reviewer for the valuable feedback and for your acknowledgment of our work. We address the comment below.

W1. The derivation of posterior bias seems to assume a noise-free forward model. This should be stated as a caveat, with a discussion of how it affects the central claims.

We acknowledge that the derivation of the posterior-bias estimate in Section 3.1 (Equation 5) considers a deterministic forward mapping, y∗=F(u∗)y^*=F(u^*), and does not explicitly model measurement noise. However the experiments include measurement noise and the results are reported over those noisy settings. So, this way of modeling does not affect our central claims. We have added this to section 3.1 L130 in revised manuscript.

Another interpretation of this question could be that we assume a perfect forward model is known. We acknowledge that our methods assume that the forward model FF is known perfectly. In many scientific problems, such as the cosmological problem, access to real observations with known ground-truth latent causes is unavailable. Hence, FF guides the epistemic investigation into the latent causes. Our paper addresses the problem of estimating those latent causes, given a particular FF.

W2. It is unclear when Eq. 10 is a good approximation of Eq. 9. The Pearson correlation in Section 5.2 is not particularly good. Please give a more thorough motivation for Eq. 10 and a more detailed experimental characterization, including when the estimator can be trusted. Is there a simple case where it is exact, and how close is that to the experimental setting?

Thanks for pointing this out. We acknowledge that the original presentation did not clearly distinguish between the general bias-score formulation and the computational approximation used in one of the testbeds.

Eq. 9 is the general result, while Eq. 10 (in former pdf) is used only for the high-dimensional ionization-field reconstruction benchmark (Sec 5.3) for computational simplification. For the other two benchmarks we use the full forward-operator formulation (Eq. 9).

To make this distinction explicit, we have moved the approximate formulation (Eq. 10) from the general methodological discussion to Sec 5.3, where the Ionization-field reconstruction experiment is described.

Second, the motivation for the approximation in Eq. 10 ( Eq. 14 in revised pdf) and when it should be trusted: In the ionization-field reconstruction problem, directly evaluating the full inverse-Jacobian expression becomes computationally expensive. We observe that the interactions are localised in space, leading to small values of entries in the Jacobian matrix as we go away from the main diagonal. We therefore employ the approximation in Eq 10; the inverse-Jacobian structure can be captured through its diagonal entries. This makes bias-score evaluation computationally tractable, improves scalability, and accelerates the overall iterative search procedure.

Additional experiments were conducted to test whether off-diagonal interactions improve the bias approximation. For this we also retain Jacobian interactions within spatial neighbourhoods of radius r=1r=1 and r=2r=2, instead of using only the diagonal.

Forming the full Jacobian is too expensive in high dimensions, so we build each local approximation JrJ_r matrix-free, using Rademacher probes (entries i.i.d. in {−1,+1}\{-1,+1\}) and Jacobian-vector products (JVPs). The latent correction vrv_r solves (Jr⊤Jr+λI) vr=Jr⊤(y⋆−F^ψ(u)),(J_r^\top J_r + \lambda I)\, v_r = J_r^\top\big(y^\star - \hat{F}_\psi(u)\big), where λ\lambda is a regularization parameter. We solve this with conjugate gradient, which needs only matrix-vector products. The resulting vrv_r gives the acquisition score for each variant.

We compare r=1r=1 and r=2r=2 to see how wider local interactions affect the bias estimate and acquisition performance. Results are below.

MetricRadius r=1r=1Radius r=2r=2Radiusr=0r=0
EestE_{est}3.59 ±\pm 0.123.46 ±\pm 0.333.57 ±\pm 0.08
NLL-1.107 ±\pm 0.122-0.771 ±\pm 0.211-1.313 ±\pm 0.169
ErecE_{rec}5.63 ±\pm 0.185.51 ±\pm 0.215.60 ±\pm 0.16

Here $r=0 $ is the BISIP method. We observe that increasing the radius does not provide any substantial gains in the performance.

AuthorsOpenReview ↗

Response (2/2)

The Pearson correlation in Sec 5.2 is not particularly good. Thank you for pointing this out. We acknowledge that in former pdf , the Inverse Laplace results in Sec 5.2 under "Empirical Validation of S(ϕ,y⋆)S(\phi, y^\star)" were inadvertently reported for a single random seed. We have corrected this in Section 5.2 (L343) of the revised manuscript, which now reports results averaged over 10 independent seeds. Additionally , we have added an thorough analysis of the relationship between the proposed score SS and the actual reconstruction bias B(yi)=∥∣μϕ−ui∗∥∣22B(y_i)=\left\||\mu_\phi-u_i^*\right\||_2^2 in Appendix A.6 of the revised manuscript where we observe that across all three benchmarks, SS maintains a consistently strong rank correlation with BB throughout acquisition.

At the final acquisition round, averaged over multiple runs, the Pearson correlation between S(ϕ,y⋆)S(\phi, y^\star) and BB is approximately 0.77 for Inverse Laplace, 0.85 for ionization-field reconstruction, and 0.78 for the linear inverse problem with κ=100\kappa =100

For the severely ill-posed Inverse Laplace problem, we observe that runs with larger reconstruction error ErecE_{\mathrm{rec}} tend to exhibit weaker correlation between SS and BB, whereas runs with lower reconstruction error show a stronger association shown in the table. Since ErecE_{\mathrm{rec}} is easy to compute at test time, it can serve as a diagnostic of run reliability: a run with large ErecE_{\mathrm{rec}} can be repeated, whereas runs with low ErecE_{\mathrm{rec}} show good correlation and are more trustworthy. In our experiments, roughly 60% of Inverse Laplace runs are good in this sense. This behaviour is shown in the table below-

SeedErec(×10−2)E_{\mathrm{rec}}(\times 10^{-2})Correlation (SS-BB)
Seed 129.890.36
Seed 224.280.44
Seed 319.870.62
Seed 415.870.78
Seed 514.730.89

W3: It seems that the paper does not have a proof of convergence to the true posterior. This can make it more difficult to reason about how the method works. This should be either stated as a clear caveat, or additional work should be done to establish such a convergence argument.

We have added a proof of convergence in appendix A.1 of the revised paper. It is as below:

"BISIP acquires samples in regions where the estimated bias StS_t is the largest (Eq. (10) in revised pdf) (tt indexes the acquisition steps). Furthermore, St>0S_t>0 widens the search region ensuring that the probability of acquiring new data in the neighbourhood of true u∗u^{*} is non-zero. Under the standard assumptions that additional training data does not increase the estimation error, the squared bias Bt=∥∣μt−u∗∥∣2B_t=\||\mu_t-u^*\||^2 forms a non-negative supermartingale. Consequently, BtB_t converges to zero, asymptotically, and then, by Theorem 3.1, StS_t also converges to zero."

Moreover, we show that the probability of acquiring a sample in the neighbourhood of true u∗u^* is proportional to sd−1/2s_d^{-1/2}, where sds_d is the estimated bias along dimension dd, for a large bias. Thus, the probability of acquiring good data does not decay drastically even when the bias is large, thereby, helping convergence.

Q1. The expectation in the definition of S(ϕ,y⋆)S(\phi, y^\star) seems redundant, since E∥uϕ−u⋆∥2=∥uϕ−u⋆∥2\mathbb{E}\lVert u_\phi - u^\star\rVert^2 = \lVert u_\phi - u^\star\rVert^2. Theorem 3.1 then appears to assume its own result: if E[u]=uϕ=u⋆\mathbb{E}[u] = u_\phi = u^\star, then S=0S = 0 trivially.

i. Thank for pointing out. The definition of bias does not need E. Now we define the bias as B(ϕ,y⋆)=∥μϕ−u⋆∥B(\phi, y^\star) = \lVert \mu_\phi - u^\star \rVert, and S(ϕ,y⋆)S(\phi, y^\star) is its estimate. The revised paper is updated accordingly.

ii. BB is the bias and SS is its estimate. When the model is unbiased, B=0B=0, but it is not obvious that the estimate SS is also 0. Hence, the theorem 3.1.

Q2. In the ionization field problem, the forward model takes both an ionization field and a halo catalogue. If the posterior estimator only outputs the ionization field, how is the halo catalogue obtained for the forward simulator?

In the Cosmological Reionization Field Reconstruction experiment, in the y=F(u)y = F(u) forward model, uu represents the ionization field and y represents the Lyman-alpha emitter galaxies.  Haloes are simply locations in the simulation box where the matter density is greater than a threshold value. They are obtained from domain knowledge (cosmology). It can be treated as a part of FF. We have now included a description of how the halo catalogue is generated by 21cmFAST using the excursion-set formalism in the Appendix section A.3.4 under "Generation of mock galaxy observations" paragraph.

I thank the authors for the additional experiments, they address many of my concerns. However, I still have some issues with the questions raised:

The motivation for Eq. 10 ( Eq. 14 in revised pdf)

As I understand it, the motivation is 1) you assume that the Jacobian has small off-diagonal terms. It is well approximated by a diagonal matrix. 2) this results in being able to approximate the inverse Jacobian with a diagonal matrix as well. 3) The provided equation is a valid approximation of the diagonal of the inverse diagonal. I'm happy to grant 1), but I don't think that 2) holds in general. E.g., the inverse of a discrete Laplacian matrix for is typically dense. For 3), I'm not sure if the paper provided a clear justification for why is the equation a valid estimate of the diagonal of the inverse Jacobian. The additional experiments are very helpful, but is the radius=0 result there equivalent with using the original approximation? Furthermore, if at all possible, it would be helpful to have direct empirical analysis on the bias of the estimate.

Proof of convergence

I thank the authors for the additional clarification, but the provided proof seems to miss intermediate steps that clearly show that it is true. It also seems incorrect: The supermartingale theorem states that supermartingales converge to some finite value, not zero. And even if the difference of the posterior mean compared to u* goes to zero, that does not directly imply that the posterior approximation becomes correct. I think the convergene behaviour should be presented as an empirical observation or with a stronger proof.

Q2

This partially resolves the concern, but I'm still not sure about the central concern of where exactly does the halo catalogue come from for the forward model after we have estimated the ionization field? Is the halo catalogue fixed? Or is there a specific method to generate the halo catalogue based on the ionization field somehow, or are the halo catalogues sampled independently?

We thank the reviewer for acknowledging the response. We address further points below.

i. As I understand it, the motivation is 1) you assume that the Jacobian has small off-diagonal terms. It is well approximated by a diagonal matrix. 2) this results in being able to approximate the inverse Jacobian with a diagonal matrix as well. 3) The provided equation is a valid approximation of the diagonal of the inverse diagonal. I'm happy to grant 1), but I don't think that 2) holds in general. E.g., the inverse of a discrete Laplacian matrix for is typically dense. For 3), I'm not sure if the paper provided a clear justification for why is the equation a valid estimate of the diagonal of the inverse Jacobian. Furthermore, if at all possible, it would be helpful to have direct empirical analysis on the bias of the estimate.

We agree that (2) does not hold in general; we do not claim that too. The paper proposes different ways to approximate S for the cosmology testbed as follows (in the increasing order of relaxation and decreasing computational complexity):

  • use the full Jacobian (eq 9)
  • use a band matrix approximation with r>0.
  • use a diagonal approximation, i.e., r=0 (eq 14)

We find that the overall empirical performance is not significantly different across these methods (see Table shared in the previously response). Also to clarify, r=0r=0 is equivalent to using the original approximation that is Eq. 14 in the revised manuscript.

I thank the authors for the additional clarification, but the provided proof seems to miss intermediate steps that clearly show that it is true. It also seems incorrect: The supermartingale theorem states that supermartingales converge to some finite value, not zero. And even if the difference of the posterior mean compared to u∗u^* goes to zero, that does not directly imply that the posterior approximation becomes correct. I think the convergene behaviour should be presented as an empirical observation or with a stronger proof.

We acknowledge that this is not a formal proof, but intuitive arguments to support our hypothesis. Nevertheless, if the bias is non-zero, the probability of acquiring samples in the vicinity of u∗u^* is >0>0. Under additional assumption that such samples lead to a positive expected reduction of the bias, the limiting value cannot remain >0>0. This is not a rigorous proof, but the empirical results support that the bias rapidly decreases with acquisition steps. ii. We acknowledge that the algorithm does not guarantee that qϕq_\phi converges to the true posterior, but to an unbiased posterior; for how much does the estimated posterior represent the true posterior depends on the β−\beta-NLL objective (a kind of maximum likelihood estimator) and the expressiveness of the model (ϕ\phi). A principled calibration of variance (confidence intervals) or the complete posterior could be taken up as future studies.

iii. This partially resolves the concern, but I'm still not sure about the central concern of where exactly does the halo catalogue come from for the forward model after we have estimated the ionization field? Is the halo catalogue fixed? Or is there a specific method to generate the halo catalogue based on the ionization field somehow, or are the halo catalogues sampled independently?

In our experiments, we fix the halo field (catalogue). Generalising our results to stochastic variations in halo catalogues is straightforward and routinely done in the field but was deemed unnecessary for the present work. Hence, to focus on the problem at hand and to avoid computational overhead, we chose not to vary halo fields in our testbench. A fair comparison across baselines substantiates the effectiveness of BISIP. However, we also note that our testbench has a very large simulation box (sizes of uu and yy), which, because of ergodicity, captures the effect of stochastic variations in halo field to some extent.

Reviewer_mMqzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors introduce BISIP as a novel approach for sample-efficient posterior inference in the context of simulation-based inverse problems, particularly where the forward model F\mathcal{F} is non-linear, high dimensional, expensive, without a tractable likelihood, and with the presence of prior shift. Given a forward model with heteroskedastic noise (y=F(u)+ϵy = \mathcal{F}(u) + \epsilon ), they introduce a neural estimator qϕ(u∣y)q_\phi(u| y), with the neural estimator parameterized by a diagonal Gaussian qϕ(u∣y)=N(u;μϕ(y),diag(σϕ2(y)))q_\phi(u | y) = \mathcal{N}(u; \mu_\phi(y), \text{diag}(\sigma^2_\phi(y))). It is first trained on a fixed set of paired samples Dt=0=(ui,yi)i=1ND_{t=0}=(u_i, y_i)_{i=1}^N with a β\beta-NLL objective. A local-linear approximation for posterior bias, combined with a (differentiable) surrogate, is used within an active learning loop to improve the posterior estimator efficiently through collecting additional samples. The authors cover related work and validate the approach on three increasingly challenging datasets, including a case where ill-posedness can be directly controlled. The authors conclude that BISIP outperforms several approaches and sets a new benchmark for high-dimensional field reconstruction in cosmology. The main methodological contribution is the combination of the local-linear bias estimate as part of the acquisition strategy within an active learning loop.

Critical strengths
  • S1: The paper demonstrated consistent, slight performance improvements on the task relative to other baseline models. For the synthetic data this is shown in Table 1, 2 and Figure 2, where BISIP typically has the best metric scores (with some exceptions), although for the cosmological data advantage is less clear for NLL and ErecE_\text{rec}.
  • S2: The paper’s methodology is very clear and well-articulated which makes it easy to follow.
Critical weaknesses
  • W1: missing ablations to demonstrate performance of different components.
    • The overall algorithm has multiple moving parts (approximations for posterior bias, the surrogate, composite loss, hyperparameters β\beta, γ\gamma, kk). These are all combined together and it is unclear from the presented results which part is doing the work of providing improved performance since not all components are ablated. This is partly mitigated where the composite loss is mentioned for the training across all the active learning models for the linear problem, although it could still impact the relative performance.
    • Relatedly the Rand method, whereby a central aspect of the bias estimator is no longer used, has EestE_\text{est} similar to BISIP in some cases (e.g. κ=10\kappa=10 in Table 1) and is better or equal to the alternative active learning baselines in some cases (e.g. EestE_\text{est} in Table 2 and Table 3). Additionally, there are results where BISIP does not perform better than comparison methods within error illustrated (e.g. ErecE_\text{rec} in Table 1 and 2, and differences with respect to the active learning baselines are within error bars for NLL and ErecE_\text{rec} in Table 3).
    • While the performance of the bias approximation is validated for the Laplace dataset, the results illustrate only partial ability to acquire the most biased samples (e.g. 40% of top 5%). It's also unclear how this changes during the active learning acquisition, and it is not validated in the other two datasets. Validating this further would be beneficial since this is central to BISIP and its use in the acquisition context is particularly stress testing the approximation as values are being selected farthest from the true value.
  • W2: performance benefit relating to prior shift.
    • Unknown/static priors are introduced as a key motivation over other Bayesian frameworks. BISIP's performance advantage in this respect is not clearly shown. For example, for the linear and Laplace datasets, posteriors could have been estimated with approaches making use of their exact likelihoods and combined with different prior assumptions to validate this aspect of the purported benefit.
Questions for authors

Relating to W1:

  • Q1: Several components of the algorithm are not ablated and so it's not easy to determine their relative benefit:
    • Hyperparameters γ\gamma and kk are set for individual datasets without justification for the values used. Ablations for these, in particular γ=0\gamma=0 which removes the reconstruction part of the composite loss, would be useful for identifying which components lead to performance advantages.
    • β\beta is stated as tuned per problem but the details of this tuning (i.e. the data and measures used) are not provided - could further detail regarding this be added and potentially the sensitivity of performance to its value?
    • A clarification, for the Rand baseline, how is uju_j sampled, does it still use sds_d for sampled uj∼N(μϕ(yj∗),diag(sd+σd2))u_j \sim \mathcal{N}(\mu_\phi(y_j^*), \text{diag}(s_d + \sigma_d^2)) or is it uj∼N(μϕ(yj∗),diag(σd2))u_j \sim \mathcal{N}(\mu_\phi(y_j^*), \text{diag}(\sigma_d^2))?
  • Q2: The quality of the approximation of the posterior bias SS is only validated in the Laplace dataset. A broader examination of the approximation (across the 3 datasets, at different points in the acquisition) would be beneficial to validate the method.

Relating to W2:

  • Q3: For the synthetic cases, would it be possible to generate posteriors making use of the exact likelihood under different prior assumptions to provide a further baseline and demonstrate the effect of prior choice/shift?
  • Q4: Is it possible for the prior choices for each model to be articulated and contrasted across models?

Minor comments:

  • Q5: It's not stated the overall run times for the different approaches - can these be reported?
  • Q6: Reproducibility is limited since no code is available - can this be made available?
  • Q7: The captions in the Figures 2 and 3 should provide additional detail about what is plotted as well as the meaning of the shaded regions.
  • Q8: In Eq. 12, the composite loss relating to reconstruction is only over the selected observations Bt\mathcal{B}_t - could the rationale for this over including all observations be provided?
  • Q9: L302 should be γ\gamma instead of λ\lambda for consistency with elsewhere.
4 comments
AuthorsOpenReview ↗

Response (1/3)

We sincerely thank the reviewer for the valuable feedback and for your acknowledgment of our work. We address the comment below.

W1. The algorithm has many components (posterior bias approximation, surrogate, composite loss, β, γ, κ). Not all are ablated, so it is unclear which part drives the improvement. This is partly mitigated where the composite loss is mentioned for the training across all the active learning models for the linear problem, although it could still impact the relative performance.

We thank the reviewer for pointing out that the original submission did not sufficiently disentangle the contributions of the individual components of BISIP. BISIP has two distinct components in its active-acquisition mechanism,

  1. Bias-guided observation selection. For each observation in the unlabelled pool, BISIP computes an estimated bias score and selects observations with high estimated posterior bias.
  2. Bias-inflated candidate sampling. For each selected observation, BISIP expands the proposal distribution using the estimated bias and the predictive uncertainty of qϕq_\phi using the covariance matrix diag(σϕ2(y⋆)+s(ϕ,y⋆))\mathrm{diag}\big(\sigma_\phi^2(y^\star) + s(\phi, y^\star)\big)

To investigate the individual contributions of the acquisition mechanisms, we conducted a controlled ablation study on the Inverse Laplace Transform and Ionization Field Reconstruction benchmark. We compared four configurations namely Random baseline, Random selection with bias-inflated sampling (Random + BISIP Inflation) , bias-guided selection with ordinary posterior sampling (BISIP Selection Only), and the complete BISIP method. All configurations use the same evaluation procedure.

Results on Ionization Field Reconstruction

ConfigurationEestE_{\mathrm{est}} (×10−2\times 10^{-2}) ↓ErecE_{\mathrm{rec}} (×10−2\times 10^{-2}) ↓NLL ↓
Random3.94 ±\pm 0.116.15 ±\pm 0.63-1.2489 ±\pm 0.1589
Random + BISIP Inflation3.87 ±\pm 0.145.95 ±\pm 0.34-1.2595 ±\pm 0.1566
BISIP Selection Only3.431 ±\pm 0.275.77 ±\pm 0.21-1.3638 ±\pm 0.1256
BISIP3.57 ±\pm 0.085.6 ±\pm 0.16-1.3133 ±\pm 0.1693

Results on Inverse Laplace Transform Benchmark

ConfigurationEestE_{\mathrm{est}} (×10−2\times 10^{-2}) ↓ErecE_{\mathrm{rec}} (×10−2\times 10^{-2}) ↓NLL ↓
Random10.9 ±\pm 0.84.3 ±\pm 2.0-2.42 ±\pm 0.08
Random + BISIP Inflation8.42 ±\pm 0.382.34 ±\pm 0.2-2.67 ±\pm 0.07
BISIP Selection Only8.63 ±\pm 0.282.66 ±\pm 0.64-2.69 ±\pm 0.05
BISIP9.1 ±\pm 0.42.5 ±\pm 0.7-2.6 ±\pm 0.05

Analysis The contribution of selection and proposal inflation depends on the benchmark. For ionization-field reconstruction, bias-guided selection alone gives the best mean EestE_{\mathrm{est}} and NLL, while full BISIP gives the lowest mean ErecE_{\mathrm{rec}}. The Inverse Laplace Transform problem is highly ill-posed, and the differences among the three modified configurations are small. Selection alone and inflation alone both improve clearly over Random, so either component can account for the gain. Combining them gives no consistent additional improvement, so the benefit of BISIP on this problem come from either of the components.

Q1a Hyperparameters γ\gamma and kk are set per dataset without justification. Please add ablations, in particular γ=0\gamma = 0 (removes the reconstruction part of the composite loss).

The composite loss is applied consistently across all acquisition strategies and benchmarks, including the baselines. Additionally, we evaluate the contribution of forward-consistency regularization by comparing the complete objective with a variant that removes the forward-consistency term γ=0\gamma=0, while keeping the acquisition procedure fixed on the Inverse Laplace Transform and Reionization Field Reconstruction Benchmark. It is hown in table below where values are (×10−2\times 10^{-2}) for EestE_{\mathrm{est}} and ErecE_{\mathrm{rec}}.

Inverse Laplace Transform

ConfigurationEestE_{\mathrm{est}} ↓ErecE_{\mathrm{rec}} ↓NLL ↓
BISIP, γ=0\gamma=08.86 ±\pm 0.32.87 ±\pm 0.3-2.63 ±\pm 0.076
BISIP9.1 ±\pm 0.42.5 ±\pm 0.7-2.6 ±\pm 0.05

Ionization Field Reconstruction

ConfigurationEestE_{\mathrm{est}} ↓ErecE_{\mathrm{rec}} ↓NLL ↓
BISIP, γ=0\gamma=03.79 ±\pm 0.125.98 ±\pm 0.2-1.264 ±\pm 0.087
BISIP3.57 ±\pm 0.085.6 ±\pm 0.16-1.313 ±\pm 0.1693

On Inverse Laplace, the difference between the two configurations is not statistically significant. On high-dimensional ionization-field reconstruction, the benefit of forward-consistency regularization is more pronounced.

We have added these experiments and the discussion to Appendix A.5 of the revised manuscript.

AuthorsOpenReview ↗

Response (2/3)

Choice of acquisition batch size kk. We set kk separately for each benchmark, according to problem complexity, and use the same kk for all acquisition baselines within a benchmark.

  • Ionization-field reconstruction: kk is set by the cost of simulator queries and the need to balance frequent model updates against acquisition cost. Following the intuition from batch gradient methods, smaller batches allow more frequent updates and can help the model avoid poor local minima. The price is slower overall convergence, but it can lead to a better solution.
  • Linear inverse and Inverse Laplace: kk is chosen using the change in EestE_{est} between consecutive acquisition rounds, with a threshold of less than 0.001%0.001\%.

Q1b. β\beta is tuned per problem, but the tuning details (data, measures) are not given. Please add them, and ideally a sensitivity analysis.

We follow Seitzer et al. (2022) and use β=0.5\beta=0.5 as our starting point. This is also the value used by TORRCH, a baseline in our Ionization-field Reconstruction experiments. We adjusted it only where needed, within the range [0.4,0.6][0.4, 0.6], and use the same β\beta for all acquisition strategies within a benchmark. There is a typo in Sec 2.1 L95 of reporting it to be in range [0.4,0.8][0.4, 0.8].

  • Ionization-field reconstruction: β=0.5\beta=0.5.
  • Inverse Laplace: β=0.4\beta=0.4. With 0.50.5, the reconstruction became increasingly noisy and fluctuated across acquisition cycles for all baselines. β=0.4\beta=0.4 gave stable behaviour.
  • Linear inverse: β=0.5\beta=0.5 for k=1000k=1000 and k=100k=100, consistent with the other benchmarks. For k=10k=10, β=0.5\beta=0.5 and β=0.6\beta=0.6 give similar performance.

We have clarified this in Sec 2.1 of the revised manuscript.

W1.2. The Rand baseline (which does not use the central bias estimator) is similar to BISIP in some cases (e.g. κ=10\kappa=10, Table 1) and better than or equal to the other AL baselines in others (EestE_{\mathrm{est}}, Tables 2 and 3). BISIP is also not better within error bars in several cases (ErecE_{\mathrm{rec}} in Tables 1 and 2; NLL and ErecE_{\mathrm{rec}} in Table 3).

On the Linear Inverse benchmark, Rand is competitive with BISIP at κ=10\kappa=10 because the problem is only mildly ill-posed there, and all methods perform similarly. As the problem complexity increases , BISIP's gains over Rand and the other acquisition baselines become larger and statistically significant.

Q1c. For the Rand baseline, how is uju_j sampled? Does it use sds_d, i.e. uj∼N(μϕ(yj⋆), diag(sd+σd2))u_j\sim\mathcal{N}\big(\mu_\phi(y_j^\star),\ \mathrm{diag}(s_d + \sigma_d^2)\big), or uj∼N(μϕ(yj⋆), diag(σd2))u_j \sim \mathcal{N}\big(\mu_\phi(y_j^\star),\ \mathrm{diag}(\sigma_d^2)\big)?

In the Rand baseline, observations are selected uniformly at random from the unlabelled pool, independently of the bias score. For each selected yj⋆y_j^\star, the candidate is sampled from the standard posterior predictive, with no bias correction: uj=μϕ(yj⋆)+σϕ(yj⋆)⊙ϵ,ϵ∼N(0,I),u_j = \mu_\phi(y_j^\star) + \sigma_\phi(y_j^\star) \odot \epsilon, \qquad \epsilon \sim \mathcal{N}(0, I), that is, uj∼N(μϕ(yj⋆), diag(σϕ2(yj⋆)))u_j \sim \mathcal{N}\big(\mu_\phi(y_j^\star),\ \mathrm{diag}(\sigma_\phi^2(y_j^\star))\big). The score S(ϕ,y⋆)S(\phi, y^\star) is not used. The candidate is then passed through FF, and the resulting pair is added to the training set.

W1.3 and Q2. The quality of the approximation to S is validated only on the Laplace dataset. Please examine it across all 3 datasets and at different points of the acquisition.

We thank the reviewer for this suggestion. Appendix A.6 of the revised manuscript now validates the bias approximation on all three benchmarks and at multiple stages of acquisition. On a fixed held-out set with known ground truth, we compare S(ϕ,yi)S(\phi, y_i) with the actual bias B(yi)=∥μϕ(yi)−ui⋆∥22B(y_i)=\lVert \mu_\phi(y_i)-u_i^\star\rVert_2^2, and show how SS tracks BB across the complete acquisition cycle through Fig 9, 10, 11 in the same appendix, together with the evolution of the posterior variance.

In the original manuscript, the Inverse Laplace results in "Empirical Validation of S(ϕ,y⋆)S(\phi, y^\star)" were inadvertently reported for a single seed. Section 5.2 (L343) now reports results averaged over 10 seeds.

Across the acquisition cycle, the estimated score S shows a consistently strong correlation with the actual reconstruction bias B. For Inverse Laplace benchmark, the Pearson correlation varies between 0.63 and 0.77 over the entire acquisition cycle. At the final acquisition round, the Pearson correlations computed on seed-averaged scores are approximately 0.77 (Inverse Laplace), 0.85 (ionization-field reconstruction) and 0.78 (linear inverse, κ=100\kappa=100). Despite the severe ill-conditioning of the forward operator in the Inverse Laplace benchmark (κ(A)≈1016\kappa(A)\approx 10^{16}), the score identifies a substantial fraction of the highly biased observations, with Top-5% and Top-20% overlap reaching 0.57 and 0.70, respectively.

AuthorsOpenReview ↗

Response (3/3)

W2. and Q3. For the synthetic cases, would it be possible to generate posteriors making use of the exact likelihood under different prior assumptions to provide a further baseline and demonstrate the effect of prior choice/shift?

We thank the reviewer for this suggestion. Our experiments are set in the simulation-based inference (SBI) setting, where the likelihood is unavailable to the inference methods, which rely only on simulator-generated samples. For the linear inverse and Inverse Laplace benchmarks, the analytical forward models and likelihoods are known to us but are not given to the methods. An exact-likelihood posterior under different priors would therefore use information beyond what the evaluated methods have, making it an oracle reference rather than an SBI baseline. This matters most for ionization-field reconstruction, where the likelihood is not explicitly available.

Q4. Is it possible for the prior choices for each model to be articulated and contrasted across models?

Our methods does not use any information regarding test prior. For a fair comparison, all methods use the same training prior and are evaluated under the same, distinct test prior, thereby ensuring a consistent prior-shift setting across all methods.

The choices of training and test priors are explicitly described for each benchmark in the Appendix: Section A.3.1 (Latent Field Generation) for the linear inverse benchmark, Section A.3.2 (Equations 21–26) for the inverse Laplace transform benchmark, and Section A.3.5 for the ionization-field reconstruction benchmark.

Q5. Report the overall run times of the different approaches.

We thank the reviewer for pointing this out. We have now update the run times for diffrent approaches in Appendix in L844, L997 and L1132 for all three benchmarks respectively.

Q6. Reproducibility is limited since no code is available - can this be made available?

We are providing a ZIP archive containing the code accompanying this work. Upon acceptance, the code will be made publicly available along with a comprehensive README detailing the setup and execution instructions. The implementation details, and hyperparameter configurations is there in the Appendix for reproducibility.

Q7. Captions of Figures 2 and 3 should state what is plotted and what the shaded regions mean.

Thanks for pointing this out. We have made the changes in the updated manuscript.

Q8. In Eq. 12, the composite loss relating to reconstruction is only over the selected observations. Bt\mathcal{B}_t. Could the rationale for this, rather than including all observations, be provided?

We thank the reviewer for pointing this out. The restriction of the reconstruction loss to only the selected observations in Equation 12 was an inadvertent typo in the manuscript. We have corrected the equation in the revised version to ensure that the reconstruction loss accounts for the full dataset of observations, Dt+1\mathcal{D}_{t+1}, rather than only the selected observations.

Q9. L302: notation should be consistent with elsewhere in the paper.

Thank you for pointing this out. We have corrected this now.

I thank the authors for their efforts in responding to my questions and their updates to the paper. The revisions and additional ablations help provide further evidence of the effect of the different components, clarify the method significantly and the paper is improved.

However, I will retain the same score as initially provided. My main remaining considerations are that:

  • (relating to W1/Q1 and the new results provided): The ablations do not show a consistent advantage for the combined components within the uncertainty reported, making it unclear which combination to use in practice.
  • (relating to W2/Q3): The benefit of the method under varying extent of prior shift remains under-explored in the synthetic problems.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.