- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors tackle Bayesian Experimental Design (BED) under model misspecification using distributionally robust optimization (DRO), proposing the Distributionally Robust Bayesian Experimental Design (DR-BED) framework. Their approach constructs frequentist confidence bands for the conditional outcome distribution and uses these to select robust experimental designs. The method is clearly motivated and supported by theoretical derivations and experiments on polynomial regression, pharmacokinetics, and source-location benchmarks. DR-BED is a promising approach toward more robust and potentially better-calibrated BED under model misspecification, although its reliance on smoothness and shape assumptions, restriction to scalar outcomes, and computational complexity may limit its applicability.
Critical strengths
The authors approach the problem of model misspecification in BED/BOED using an operations research perspective that is novel in this setting and well-motivated. Their formulation is clearly explained, combining nonparametric confidence-band estimation with distributionally robust optimization. The theoretical coverage guarantees and computationally tractable dual reformulation are particularly valuable contributions. Their evaluations use established BED benchmarks and demonstrate promising improvements in predictive performance under model misspecification.
Critical weaknesses
-
Justification of the surrogate information-gain objective. The proposed DR-EIG replaces standard EIG with a surrogate based on the midpoint of the confidence bands. However, it is unclear whether this surrogate adequately captures information gain. In particular, the claim in Appendix A.6 that the objective reduces to standard EIG under correct model specification appears mathematically questionable, as the expectations over model parameters and outcomes are taken independently rather than jointly. This is consequential because the motivation of the proposed acquisition function relies on its connection to information gain. Recent work by Callahan & Catanach (2026), On the Misinformation in a Statistical Experiment, establishes an axiomatic framework for evaluating information-gain objectives under model misspecification. The authors should clarify whether their proposed surrogate satisfies appropriate information-gain properties or provide an alternative theoretical justification.
-
Limited empirical validation and insufficient baseline specification. While DR-BED demonstrates promising performance on standard benchmarks, its improvements are not consistent across all metrics, particularly in the higher-dimensional comparison with GBOED. Furthermore, the experiments do not directly evaluate information gain or posterior calibration, despite these being central motivations of the framework. The implementation of Bayesian Adaptive Design (BAD) is also underspecified. Although the authors cite Foster et al. (2019), Variational Bayesian Optimal Experimental Design, this work introduces multiple EIG estimation approaches, and it is unclear which estimator and optimization procedure are used in the baseline. The posterior inference method and comparative computational budgets should also be clarified to establish the reproducibility and fairness of the experimental comparisons.
Overall, I find this to be a promising and interesting contribution that brings a valuable distributionally robust optimization perspective to Bayesian experimental design. My current recommendation reflects concerns about the mathematical justification of the proposed surrogate information-gain objective and some important implementation details. I would be open to revising my assessment upward if the authors can adequately address these concerns during the rebuttal.
Questions for authors
-
Validity of the surrogate EIG objective: Could the authors clarify the derivation in Appendix A.6, particularly the claim in Equations 110–111 that DR-EIG reduces to standard EIG under correct model specification? Standard EIG takes expectations over the joint distribution of model parameters and outcomes, whereas the proposed surrogate appears to take these expectations independently. Under what assumptions are these expressions equivalent?
-
Reproducibility and fairness of baseline comparisons: Could the authors specify which EIG estimator and optimization algorithm from Foster et al. (2019) are used for the BAD baseline? Additionally, how are posterior distributions estimated or sampled from across the methods, and are the computational budgets comparable? These details would help establish whether the reported improvements arise from the proposed acquisition function rather than differences in implementation or computational effort.
-
Evidence for improved posterior calibration: Theorem 3.1 guarantees coverage of the true DGP by the ambiguity set, but this does not necessarily imply improved calibration of the Bayesian posterior. Could the authors clarify what theoretical or empirical evidence supports their claims about posterior calibration? In particular, does the reported improvement in predictive NLL reflect better calibration, or primarily improved predictive fit?
-
Degenerate bins in the ambiguity-set construction: In Equation 4, the set defining may be empty when an individual observation carries a large kernel weight, and concentrated weights may also produce repeated bin boundaries. How does the proposed algorithm handle these cases? Are additional assumptions on the kernel weights or effective sample size required to ensure the validity of the ambiguity-set construction and the coverage guarantee in Theorem 3.1?
5 comments
Rebuttal-Part 2
W1, Q1
The reviewer is right that the reduction claimed in the submitted Appendix A.6 did not hold as stated, and we are grateful for the suggested reference Callahan & Catanach (2026) and the opportunity to revise the integrand function (please refer to W1 of the response to the similar concern of Reviewer ssyv). We revised the analysis, the integrand, and the theoretical justification, and we now provide a full verification against the axioms of Callahan & Catanach (2026). The ambiguity set, the coverage guarantee, the conic dual, and Algorithm 2 hold for any integrand satisfying Assumption 3 and are unchanged.
(i) Joint expectations and an exact decomposition. The parameter expectation in the integrand is now taken under the posterior given , so the utility is a joint expectation under , as in the EIG. By Bayes' rule, the integrand with reference decomposes exactly as
This decomposition makes the reviewer's concern precise: under correct specification, the band concentrates around the true outcome distribution rather than the predictive , and, averaged over the true parameter, the second term cancels the first, so that full weight on does not preserve the information-gain ranking of designs. Based on the reference suggested by the reviewer, we revised our integrand as follows.
(ii) Revised integrand. Under misspecification, need not necessarily correspond to any quantity of the true DGP, so the EIG's reference is computed under a model that may not describe the real system, whereas is built from the observed data. We therefore combine both references:
$f(\boldsymbol d,y)=\mathbb E_{\boldsymbol\theta\sim p(\boldsymbol\theta\mid y,\boldsymbol d)}\big[\log p(y\mid\boldsymbol\theta,\boldsymbol d)\big]-(1-\nu_t)\log p(y\mid\boldsymbol d)-\nu_t\log\hat q(y\mid\boldsymbol d), \qquad \nu_t=(1-\alpha)\frac{t}{t+t_0},$
so that trust in the data-driven reference grows as the bands tighten and approaches the coverage level . Its properties (see revised Appendix A.6) are:
- (a) It recovers the standard EIG exactly when , for every .
- (b) Under correct specification and without ambiguity, averaged over the true parameter, it equals for every design (Proposition A.2), which answers the reviewer's question on the conditions for the reduction. The criterion ranks designs exactly as the EIG does.
(iii) Verification against the axioms of Callahan & Catanach (2026). We are especially grateful for this reference. The new Appendix A.6.1 recalls the classical axioms (G1)--(G3) and the robust axioms (R1)--(R4), and checks each of them for DR-EIG. In their setting, the averaged criterion admits the exact information--penalty decomposition (Proposition A.3)
where the first term is , the second term is , and inside the second term is the misspecification gap and the KL term is the marginal mismatch that separates EGIG from in their Lemma 4.4. The penalty vanishes under correct specification.
With , DR-EIG satisfies (R1); has the information--penalty structure of (R3), with satisfying (G1)--(G3); satisfies (R2) and the non-negativity of the penalty in (R3) under explicit conditions; and satisfies (R4) for the true information and the marginal penalty (Table 4). We state the remaining differences, and their consequences, explicitly. They concern only the posterior gap , which requires the true posterior, i.e., a simulator of the true process.
The analysis also explains the design choice: without the reference (), (R2) fails whenever the working model learns spuriously, and with full weight (), the information part vanishes. We position DR-BED as the simulator-free complement to EGIG, which attains the information--penalty structure from observed data, together with finite-sample coverage and regret guarantees; when a simulator is available, a distributionally robust EGIG is an interesting future research direction.
Rebuttal-Part 3
W2, Q2, Q3
Below we outline how we implement these suggestions in the revision.
Robust baselines and consistency across metrics. The revised Section 4.2 compares DR-BED (using the new integrand) with the robust BED method. Our new results show that the performance of DR-BED improves with the revised utility function. The revised Table 2 summarises the performance, averaged over rounds 1--15 of the experiments, on the source-location finding problem in different dimensions.
In the four-dimensional setting, DR-BED improves on GBOED across all metrics, with lower MMD, W1, MSE, and NLL, where the largest improvement is a nearly % reduction in MMD. These results demonstrate both a closer match to the true DGP and better predictive distributions under model misspecification. In the eight-dimensional setting, DR-BED again leads on all metrics, although the differences between the methods are smaller than in the four-dimensional case. This shows that DR-BED remains effective in terms of the distributional discrepancy metrics MMD and W1 as the dimensionality increases, consistent with its distributionally robust formulation.
Calibration. We thank the reviewer for this suggestion. We agree that this distinction matters, and we have conducted new experiments. Please see the new experimental results in the paragraph Calibration and Predictive Interval Coverage in Appendix A.7.6 of the revised paper, starting at line 2916.
Our experimental results show that, across all three benchmarks (polynomial regression, pharmacokinetic (PK), and source-location finding), DR-BED achieves the lowest PIT calibration error across the rounds of experiments. In the source-location finding problem, its error stays at about or below, while GBOED's rises from about to , and DR-BED's predictive interval coverage stays close to the nominal level, whereas the other baselines fall below it. Similar results hold for the pharmacokinetic and polynomial regression problems, where DR-BED again achieves the lowest PIT calibration error and the best predictive interval coverage among the methods.
In Appendix A.7.6, we also measure the discrepancy between the resulting posterior predictive distribution and the true DGP (MMD, W1), the point accuracy (MSE), and the predictive likelihood of outcomes from the true DGP (NLL). The calibration of the ambiguity set is evaluated directly in Figure 3, which shows that the empirical coverage of the true DGP by the bands is at least the nominal level. Following the reviewer's Question 4, we have removed the statements that attribute the NLL improvement to posterior calibration.
BAD and the other baselines. The full specification of every baseline, including the estimator, the design optimiser, the sample sizes, and the belief update, is now given in Appendix A.7.5.
BAD. BAD is a greedy sequential design over a finite candidate set. At each round , it evaluates candidates (with seed ) and selects the candidate with the largest EIG under the current posterior. EIG is estimated with the prior contrastive estimation (PCE) lower bound, using outer posterior samples and contrastive samples. We have corrected the citation for the EIG estimator and optimisation algorithm to Foster et al. (2019).
Posterior inference is shared across baselines. Within each case study, baselines use the same inference procedure, given in Appendix A.7.5: the exact conjugate Gaussian posterior (polynomial), full-covariance Gaussian VI on (PK), and importance sampling with a mixture of 8 Gaussian VI fits as proposal followed by independence Metropolis--Hastings resampling (source location). GBOED uses the belief updates prescribed by their paper.
For a given seed, every method receives the same assumed parametric likelihood , the same prior , the same number of observations, and the same observation noise. Differences between methods therefore arise from their design-selection criteria.
Computational budgets. Following the suggestion from Reviewer C8ho22, in the new Appendix A.9 we report per-round design times for all methods. DR-BED is more expensive per round because it solves the band LPs inside the subgradient loop. We note that the baselines estimate the EIG under the assumed model, whose bias under misspecification does not vanish with a larger Monte Carlo budget, whereas DR-BED evaluates the design against all outcome distributions consistent with the data. The improvements therefore arise from the design criterion rather than from computational effort.
Note that the band LPs for each outcome sample do not depend on each other, so they could be solved in parallel. Our current implementation solves them one by one, which is one of the reasons for its higher computational time.
Rebuttal
Summary
Many thanks for a thorough review, which we believe helped us substantially improve the paper. We first address the three limitations mentioned in the summary, and then the detailed points.
Smoothness and shape assumptions. These are structural conditions, weaker than specifying a parametric likelihood, and they follow the shape-restricted confidence-band literature (Jiang, 2017; Chen et al., 2022). In our implementation, the mode is estimated from the posterior predictive (up to two modes are allowed), the number of bins M is chosen adaptively, and the density bound U is enlarged automatically if a band LP is infeasible. The Hölder constant and the bandwidth are user-specified. Appendix A.5.1 and Table 3 now document every input for each case study.
Scalar outcomes. We agree that this is a limitation, and we state it in the Conclusion. Scalar outcomes are, however, the standard setting of the BED benchmarks we use (polynomial regression, pharmacokinetics, and source location). Extending the bands to multivariate outcomes, for example via conditional copulas or bands on one-dimensional projections, is an interesting direction for future work.
Computational cost. The cost is dominated by the band LPs within the stochastic subgradient loop. These LPs are independent across outcome samples. The new Appendix A.9 provides a cost breakdown and an equal-budget comparison with all baselines.
Jiang, H. (2017). Uniform convergence rates for kernel density estimation. Proceedings of the 34th International Conference on Machine Learning, PMLR, pp. 1694–1703.
Chen. X., Lin, Q., and Xu, G. (2022) Distributionally robust optimization with confidence bands for probability density functions. INFORMS Journal on Optimization, 4(1):65–89, 2022.