acceptodds
Under review as a conference paper at ICLR 2027

Distributionally Robust Bayesian Experimental Design via Confidence-Band Ambiguity Sets

Abstract

Model misspecification in Bayesian Experimental Design (BED) can lead to overconfident posterior inference and unreliable design selection. This limitation hinders the use of the approach in safety-critical applications, such as healthcare, where experimental outcomes must be accompanied by guarantees at a specified confidence level. To address this challenge, we propose a Distributionally Robust BED (DR-BED) framework that introduces a robust optimisation problem with a novel ambiguity set, constructed from a pair of lower-bound and upper-bound functions as confidence bands for the true Data Generating Process (DGP). We establish theoretical guarantees showing that, under standard regularity conditions on the experimental setting, namely Hölder continuity and compactness of the design space and bounded support of the experimental outcome, the constructed ambiguity set contains the true DGP with the experimenter-specified confidence level . Notably, the confidence level can be updated dynamically throughout the sequential experimentation process, allowing experts to explore different robustness-efficiency trade-offs without repeating the experimental workflow. Building on the theoretical guarantees, we reformulate the DR-BED optimisation problem in its dual form and develop a stochastic subgradient algorithm for its practical implementation. Experiments on controlled and benchmark problems demonstrate that, under model misspecification, DR-BED consistently produces improved posterior beliefs and achieves lower distances to the true DGP compared to the baseline across multiple distributional metrics.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Distributionally Robust Optimisation, Bayesian Experimental Design, Model Misspecification, Ambiguity Sets

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_j19KOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors tackle Bayesian Experimental Design (BED) under model misspecification using distributionally robust optimization (DRO), proposing the Distributionally Robust Bayesian Experimental Design (DR-BED) framework. Their approach constructs frequentist confidence bands for the conditional outcome distribution and uses these to select robust experimental designs. The method is clearly motivated and supported by theoretical derivations and experiments on polynomial regression, pharmacokinetics, and source-location benchmarks. DR-BED is a promising approach toward more robust and potentially better-calibrated BED under model misspecification, although its reliance on smoothness and shape assumptions, restriction to scalar outcomes, and computational complexity may limit its applicability.

Critical strengths

The authors approach the problem of model misspecification in BED/BOED using an operations research perspective that is novel in this setting and well-motivated. Their formulation is clearly explained, combining nonparametric confidence-band estimation with distributionally robust optimization. The theoretical coverage guarantees and computationally tractable dual reformulation are particularly valuable contributions. Their evaluations use established BED benchmarks and demonstrate promising improvements in predictive performance under model misspecification.

Critical weaknesses
  1. Justification of the surrogate information-gain objective. The proposed DR-EIG replaces standard EIG with a surrogate based on the midpoint of the confidence bands. However, it is unclear whether this surrogate adequately captures information gain. In particular, the claim in Appendix A.6 that the objective reduces to standard EIG under correct model specification appears mathematically questionable, as the expectations over model parameters and outcomes are taken independently rather than jointly. This is consequential because the motivation of the proposed acquisition function relies on its connection to information gain. Recent work by Callahan & Catanach (2026), On the Misinformation in a Statistical Experiment, establishes an axiomatic framework for evaluating information-gain objectives under model misspecification. The authors should clarify whether their proposed surrogate satisfies appropriate information-gain properties or provide an alternative theoretical justification.

  2. Limited empirical validation and insufficient baseline specification. While DR-BED demonstrates promising performance on standard benchmarks, its improvements are not consistent across all metrics, particularly in the higher-dimensional comparison with GBOED. Furthermore, the experiments do not directly evaluate information gain or posterior calibration, despite these being central motivations of the framework. The implementation of Bayesian Adaptive Design (BAD) is also underspecified. Although the authors cite Foster et al. (2019), Variational Bayesian Optimal Experimental Design, this work introduces multiple EIG estimation approaches, and it is unclear which estimator and optimization procedure are used in the baseline. The posterior inference method and comparative computational budgets should also be clarified to establish the reproducibility and fairness of the experimental comparisons.

Overall, I find this to be a promising and interesting contribution that brings a valuable distributionally robust optimization perspective to Bayesian experimental design. My current recommendation reflects concerns about the mathematical justification of the proposed surrogate information-gain objective and some important implementation details. I would be open to revising my assessment upward if the authors can adequately address these concerns during the rebuttal.

Questions for authors
  1. Validity of the surrogate EIG objective: Could the authors clarify the derivation in Appendix A.6, particularly the claim in Equations 110–111 that DR-EIG reduces to standard EIG under correct model specification? Standard EIG takes expectations over the joint distribution of model parameters and outcomes, whereas the proposed surrogate appears to take these expectations independently. Under what assumptions are these expressions equivalent?

  2. Reproducibility and fairness of baseline comparisons: Could the authors specify which EIG estimator and optimization algorithm from Foster et al. (2019) are used for the BAD baseline? Additionally, how are posterior distributions estimated or sampled from across the methods, and are the computational budgets comparable? These details would help establish whether the reported improvements arise from the proposed acquisition function rather than differences in implementation or computational effort.

  3. Evidence for improved posterior calibration: Theorem 3.1 guarantees coverage of the true DGP by the ambiguity set, but this does not necessarily imply improved calibration of the Bayesian posterior. Could the authors clarify what theoretical or empirical evidence supports their claims about posterior calibration? In particular, does the reported improvement in predictive NLL reflect better calibration, or primarily improved predictive fit?

  4. Degenerate bins in the ambiguity-set construction: In Equation 4, the set defining km(d)k_m(d) may be empty when an individual observation carries a large kernel weight, and concentrated weights may also produce repeated bin boundaries. How does the proposed algorithm handle these cases? Are additional assumptions on the kernel weights or effective sample size required to ensure the validity of the ambiguity-set construction and the coverage guarantee in Theorem 3.1?

5 comments

Summary

Many thanks for a thorough review, which we believe helped us substantially improve the paper. We first address the three limitations mentioned in the summary, and then the detailed points.

  • Smoothness and shape assumptions. These are structural conditions, weaker than specifying a parametric likelihood, and they follow the shape-restricted confidence-band literature (Jiang, 2017; Chen et al., 2022). In our implementation, the mode is estimated from the posterior predictive (up to two modes are allowed), the number of bins MM is chosen adaptively, and the density bound U\mathcal U is enlarged automatically if a band LP is infeasible. The Hölder constant and the bandwidth are user-specified. Appendix A.5.1 and Table 3 now document every input for each case study.

  • Scalar outcomes. We agree that this is a limitation, and we state it in the Conclusion. Scalar outcomes are, however, the standard setting of the BED benchmarks we use (polynomial regression, pharmacokinetics, and source location). Extending the bands to multivariate outcomes, for example via conditional copulas or bands on one-dimensional projections, is an interesting direction for future work.

  • Computational cost. The cost is dominated by the band LPs within the stochastic subgradient loop. These LPs are independent across outcome samples. The new Appendix A.9 provides a cost breakdown and an equal-budget comparison with all baselines.

Jiang, H. (2017). Uniform convergence rates for kernel density estimation. Proceedings of the 34th International Conference on Machine Learning, PMLR, pp. 1694–1703.

Chen. X., Lin, Q., and Xu, G. (2022) Distributionally robust optimization with confidence bands for probability density functions. INFORMS Journal on Optimization, 4(1):65–89, 2022.

AuthorsOpenReview ↗

Rebuttal-Part 2

W1, Q1

The reviewer is right that the reduction claimed in the submitted Appendix A.6 did not hold as stated, and we are grateful for the suggested reference Callahan & Catanach (2026) and the opportunity to revise the integrand function (please refer to W1 of the response to the similar concern of Reviewer ssyv). We revised the analysis, the integrand, and the theoretical justification, and we now provide a full verification against the axioms of Callahan & Catanach (2026). The ambiguity set, the coverage guarantee, the conic dual, and Algorithm 2 hold for any integrand satisfying Assumption 3 and are unchanged.

(i) Joint expectations and an exact decomposition. The parameter expectation in the integrand is now taken under the posterior given yy, so the utility is a joint expectation under q(y∣d) p(θ∣y,d)q(y\mid\boldsymbol d)\,p(\boldsymbol\theta\mid y,\boldsymbol d), as in the EIG. By Bayes' rule, the integrand with reference q^\hat q decomposes exactly as

Eθ∼p(θ∣y,d)[log⁡p(y∣θ,d)]−log⁡q^(y∣d)=KL(p(θ∣y,d)∥p(θ))+log⁡p(y∣d)−log⁡q^(y∣d).\mathbb E_{\boldsymbol\theta\sim p(\boldsymbol\theta\mid y,\boldsymbol d)}\big[\log p(y\mid\boldsymbol\theta,\boldsymbol d)\big]-\log\hat q(y\mid\boldsymbol d)=\mathrm{KL}\big(p(\boldsymbol\theta\mid y,\boldsymbol d)\|p(\boldsymbol\theta)\big)+\log p(y\mid\boldsymbol d)-\log\hat q(y\mid\boldsymbol d).

This decomposition makes the reviewer's concern precise: under correct specification, the band concentrates around the true outcome distribution p(⋅∣θ~,d)p(\cdot\mid\tilde{\boldsymbol\theta},\boldsymbol d) rather than the predictive p(⋅∣d)p(\cdot\mid\boldsymbol d), and, averaged over the true parameter, the second term cancels the first, so that full weight on q^\hat q does not preserve the information-gain ranking of designs. Based on the reference suggested by the reviewer, we revised our integrand as follows.

(ii) Revised integrand. Under misspecification, θ\boldsymbol\theta need not necessarily correspond to any quantity of the true DGP, so the EIG's reference log⁡p(y∣d)\log p(y\mid\boldsymbol d) is computed under a model that may not describe the real system, whereas q^\hat q is built from the observed data. We therefore combine both references:

$f(\boldsymbol d,y)=\mathbb E_{\boldsymbol\theta\sim p(\boldsymbol\theta\mid y,\boldsymbol d)}\big[\log p(y\mid\boldsymbol\theta,\boldsymbol d)\big]-(1-\nu_t)\log p(y\mid\boldsymbol d)-\nu_t\log\hat q(y\mid\boldsymbol d), \qquad \nu_t=(1-\alpha)\frac{t}{t+t_0},$

so that trust in the data-driven reference grows as the bands tighten and approaches the coverage level 1−α1-\alpha. Its properties (see revised Appendix A.6) are:

  • (a) It recovers the standard EIG exactly when q=q^=p(⋅∣d)q=\hat q=p(\cdot\mid\boldsymbol d), for every νt\nu_t.
  • (b) Under correct specification and without ambiguity, averaged over the true parameter, it equals (1−νt)EIG(d)(1-\nu_t)\mathrm{EIG}(\boldsymbol d) for every design (Proposition A.2), which answers the reviewer's question on the conditions for the reduction. The criterion ranks designs exactly as the EIG does.

(iii) Verification against the axioms of Callahan & Catanach (2026). We are especially grateful for this reference. The new Appendix A.6.1 recalls the classical axioms (G1)--(G3) and the robust axioms (R1)--(R4), and checks each of them for DR-EIG. In their setting, the averaged criterion admits the exact information--penalty decomposition (Proposition A.3)

Uˉ(d)=(1−νt)EIG⋆(d)−[ΔKL(d)+νtKL(pM⋆(y∣d)∥∥pM(y∣d))],\bar U(\boldsymbol d)={(1-\nu_t)\mathrm{EIG}^\star(\boldsymbol d)}-{\Big[\Delta_{\mathrm{KL}}(\boldsymbol d)+\nu_t \mathrm{KL}\big(p_{\mathcal M^\star}(y\mid\boldsymbol d)\|\|p_{\mathcal M}(y\mid\boldsymbol d)\big)\Big]},

where the first term is UtrueU_\text{true}, the second term is ΔU\Delta_U, and inside the second term ΔKL\Delta_{\mathrm{KL}} is the misspecification gap and the KL term is the marginal mismatch that separates EGIG from IKL⋆I^\star_{\mathrm{KL}} in their Lemma 4.4. The penalty vanishes under correct specification.

With 0<νt<10<\nu_t<1, DR-EIG satisfies (R1); has the information--penalty structure of (R3), with UtrueU^{\mathrm{true}} satisfying (G1)--(G3); satisfies (R2) and the non-negativity of the penalty in (R3) under explicit conditions; and satisfies (R4) for the true information and the marginal penalty (Table 4). We state the remaining differences, and their consequences, explicitly. They concern only the posterior gap ΔKL\Delta_{\mathrm{KL}}, which requires the true posterior, i.e., a simulator of the true process.

The analysis also explains the design choice: without the reference (νt=0\nu_t=0), (R2) fails whenever the working model learns spuriously, and with full weight (νt=1\nu_t=1), the information part vanishes. We position DR-BED as the simulator-free complement to EGIG, which attains the information--penalty structure from observed data, together with finite-sample coverage and regret guarantees; when a simulator is available, a distributionally robust EGIG is an interesting future research direction.

AuthorsOpenReview ↗

Rebuttal-Part 3

W2, Q2, Q3

Below we outline how we implement these suggestions in the revision.

Robust baselines and consistency across metrics. The revised Section 4.2 compares DR-BED (using the new integrand) with the robust BED method. Our new results show that the performance of DR-BED improves with the revised utility function. The revised Table 2 summarises the performance, averaged over rounds 1--15 of the experiments, on the source-location finding problem in different dimensions.

In the four-dimensional setting, DR-BED improves on GBOED across all metrics, with lower MMD, W1, MSE, and NLL, where the largest improvement is a nearly 8080% reduction in MMD. These results demonstrate both a closer match to the true DGP and better predictive distributions under model misspecification. In the eight-dimensional setting, DR-BED again leads on all metrics, although the differences between the methods are smaller than in the four-dimensional case. This shows that DR-BED remains effective in terms of the distributional discrepancy metrics MMD and W1 as the dimensionality increases, consistent with its distributionally robust formulation.

Calibration. We thank the reviewer for this suggestion. We agree that this distinction matters, and we have conducted new experiments. Please see the new experimental results in the paragraph Calibration and Predictive Interval Coverage in Appendix A.7.6 of the revised paper, starting at line 2916.

Our experimental results show that, across all three benchmarks (polynomial regression, pharmacokinetic (PK), and source-location finding), DR-BED achieves the lowest PIT calibration error across the rounds of experiments. In the source-location finding problem, its error stays at about 0.010.01 or below, while GBOED's rises from about 0.040.04 to 0.090.09, and DR-BED's predictive interval coverage stays close to the nominal level, whereas the other baselines fall below it. Similar results hold for the pharmacokinetic and polynomial regression problems, where DR-BED again achieves the lowest PIT calibration error and the best predictive interval coverage among the methods.

In Appendix A.7.6, we also measure the discrepancy between the resulting posterior predictive distribution and the true DGP (MMD, W1), the point accuracy (MSE), and the predictive likelihood of outcomes from the true DGP (NLL). The calibration of the ambiguity set is evaluated directly in Figure 3, which shows that the empirical coverage of the true DGP by the bands is at least the nominal level. Following the reviewer's Question 4, we have removed the statements that attribute the NLL improvement to posterior calibration.

BAD and the other baselines. The full specification of every baseline, including the estimator, the design optimiser, the sample sizes, and the belief update, is now given in Appendix A.7.5.

BAD. BAD is a greedy sequential design over a finite candidate set. At each round rr, it evaluates C=80C=80 candidates (with seed 777+r777+r) and selects the candidate with the largest EIG under the current posterior. EIG is estimated with the prior contrastive estimation (PCE) lower bound, using N=100N=100 outer posterior samples and M=20M=20 contrastive samples. We have corrected the citation for the EIG estimator and optimisation algorithm to Foster et al. (2019).

Posterior inference is shared across baselines. Within each case study, baselines use the same inference procedure, given in Appendix A.7.5: the exact conjugate Gaussian posterior (polynomial), full-covariance Gaussian VI on log⁡θ\log\boldsymbol\theta (PK), and importance sampling with a mixture of 8 Gaussian VI fits as proposal followed by independence Metropolis--Hastings resampling (source location). GBOED uses the belief updates prescribed by their paper.

For a given seed, every method receives the same assumed parametric likelihood p(y∣d,θ)p(y\mid\boldsymbol d,\boldsymbol\theta), the same prior p(θ∣H0)p(\boldsymbol\theta\mid\mathcal H_0), the same number of observations, and the same observation noise. Differences between methods therefore arise from their design-selection criteria.

Computational budgets. Following the suggestion from Reviewer C8ho22, in the new Appendix A.9 we report per-round design times for all methods. DR-BED is more expensive per round because it solves the band LPs inside the subgradient loop. We note that the baselines estimate the EIG under the assumed model, whose bias under misspecification does not vanish with a larger Monte Carlo budget, whereas DR-BED evaluates the design against all outcome distributions consistent with the data. The improvements therefore arise from the design criterion rather than from computational effort.

Note that the band LPs for each outcome sample do not depend on each other, so they could be solved in parallel. Our current implementation solves them one by one, which is one of the reasons for its higher computational time.

Reviewer_ssywOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies Bayesian experimental design under likelihood misspecification. It constructs conditional density bands from historical observations using kernel weighting, bootstrap calibration, and shape restrictions. These bands define an ambiguity set of plausible outcome distributions. The proposed experimental choice maximizes the worst-case value of a proposed utility over this set. The paper develops a scalar dual reformulation and a stochastic optimization procedure, and presents coverage and regret guarantees. Experiments consider polynomial regression, pharmacokinetics, and source localization.

Critical strengths

The paper considers a practically important problem: experimental designs can be sensitive to inaccurate likelihood models, and this is relevant especially when experiments are expensive. Using historical observations to construct explicit distributional uncertainty sets and guide next experiment choices is a reasonable direction. The scalar dual formulation also offers a useful computational approach to optimizing the proposed utility. Comparisons with random design, BAD, and GBOED provide encouraging evidence on several predictive metrics.

Critical weaknesses
  1. The proposed utility (U(d,y)) does not appear to recover expected information gain. My understanding is that the utility function defined by Eq. (13) and (14) balances the negative of the KL divergence between (p(\cdot\mid\theta,d)) and a reference distribution (\widehat q(\cdot\mid d)) (the more similar, the better), and the scope of the ambiguity set (the less ambiguous, the better). If there is no ambiguity, an experiment having the same outcome distribution under all (\theta) receives utility zero, while an experiment that distinguishes (\theta) receives a negative utility. This raises a concern about whether the utility function, or even the entire workflow, requires redesign, which would materially affect the central contribution.

  2. The connection between the theoretical guarantees, the implemented procedure, and the reported experimental results requires clarification. The coverage proof assumes i.i.d. design points, but the algorithm selects designs adaptively. In addition, the regret proof relies on simultaneous coverage across designs, while Theorem 3.1 states a fixed-design guarantee. Consequently, the coverage and regret guarantees do not seem to apply directly to the implemented adaptive procedure. The experimental setup also requires clarification regarding the conditional mode function, density upper bound, and smoothness constants required by Algorithm 1. Furthermore, the negative polynomial-regression NLL values in Table 1 appear inconsistent with the unit Gaussian noise variance in Eq. (156) and (158) and the NLL definition in Eq. (153), as detailed in the questions below. These issues affect the interpretation of the theoretical guarantees and experimental evidence.

I currently lean towards rejection due to the above theoretical and experimental weaknesses. They seem to be key issues affecting the central claims of the paper, rather than minor mistakes. It would be difficult to raise the assessment to an accepting score if any of these issues is not properly and convincingly addressed.

Questions for authors
  1. Clarification of the utility. Is my interpretation of the utility in Eq. (13) and (14) correct? In particular, how should its preference for an experiment with identical outcome distributions under all (\theta), over an experiment that distinguishes (\theta), be interpreted when there is no ambiguity? Does this imply a necessary redesign of the utility function or the workflow?

  2. Clarification of the guarantees. How does the coverage guarantee apply to the implemented adaptive procedure when the proof assumes i.i.d. design points? How is the simultaneous coverage required by the regret proof justified by the fixed-design guarantee in Theorem 3.1? Please explain clearly how the guarantees and the algorithm relate to each other.

  3. Clarification of the experimental inputs. How are the conditional mode function, density upper bound, and smoothness constants required by Algorithm 1 obtained or specified in the experiments?

  4. Clarification of the NLL results. The polynomial likelihoods in Eq. (156) and (158) have unit Gaussian noise variance. Eq. (153) defines NLL using a posterior mixture of that assumed likelihood. Each component density is at most (1/\sqrt{2\pi}), so the mixture has the same upper bound. Consequently, the per-observation NLL must satisfy [ \mathrm{NLL}\ge\frac12\log(2\pi)\approx0.919, ] but Table 1 reports negative values. How can these results be reconciled with the stated likelihoods and evaluation formula? Please provide the experimental and evaluation details needed to explain this discrepancy.

6 comments

We thank the reviewer for the technically detailed review. We agree that, on several points, the original manuscript required a sharper characterisation; the resulting changes lead to a revision of the manuscript addressing these points. Importantly, we suggest that these changes have strengthened the existing claims, without substantial redesign of the method. The revision addresses every issue at its root.

  • (1) Under correct specification, the revised criterion induces exactly the same design ranking as EIG, while under misspecification, it equals information gain minus an explicit misspecification penalty.
  • (2) The coverage guarantee now accommodates adaptive design selection.
  • (3) The regret guarantee is supported by a proved simultaneous-coverage result rather than an implicit assumption.
  • (4) We have addressed the experimental ambiguities identified by the reviewer by specifying all algorithmic inputs and the evaluation scale.

We believe these revisions resolve the technical issues raised and strengthen the manuscript accordingly, and we do not hesitate to answer further questions.

AuthorsOpenReview ↗

Rebuttal-part 2

W1, Q1.

We thank the reviewer for this observation, which we also address in the response to Reviewer j19K (W1).

The reviewer's interpretation of the submitted utility is correct. Without ambiguity it equals −Ep(θ)[KL(q~∥p(⋅∣θ,d))]≤0-\mathbb E_{p(\boldsymbol\theta)}[\mathrm{KL}(\tilde q\|p(\cdot\mid\boldsymbol\theta,\boldsymbol d))]\leq0, and even with the posterior expectation it averages to zero over the true parameter for every design. This calls for a change of the integrand only, not of the workflow. The ambiguity set (constructed by Algorithm 1), the coverage guarantee, the conic dual F(d,λ)F(\boldsymbol d,\lambda), and the stochastic subgradient method (Algorithm 2) hold for any integrand satisfying Assumption 3 and are therefore unchanged. Therefore, we believe that we addressed the issue without the need to redesign the workflow.

(i) Previous version of utility. Taking the parameter expectation under the posterior, as in the EIG, is necessary but not sufficient. Write p(θ)p(\boldsymbol\theta) for the current belief and p(y∣d)=Ep(θ)[p(y∣θ,d)]p(y\mid\boldsymbol d)=\mathbb E_{p(\boldsymbol\theta)}[p(y\mid\boldsymbol\theta,\boldsymbol d)]. By Bayes' rule, $\log p(y\mid\boldsymbol\theta,\boldsymbol d)-\log p(y\mid\boldsymbol d) =\log p(\boldsymbol\theta\mid y,\boldsymbol d)-\log p(\boldsymbol\theta)$,

hence

Eθ∼p(θ∣y,d)[log⁡p(y∣θ,d)]−log⁡q^(y∣d)=KL(p(θ∣y,d)∥p(θ))+log⁡p(y∣d)−log⁡q^(y∣d),\mathbb E_{\boldsymbol\theta\sim p(\boldsymbol\theta\mid y,\boldsymbol d)}\big[\log p(y\mid\boldsymbol\theta,\boldsymbol d)\big]-\log\hat q(y\mid\boldsymbol d)={\mathrm{KL}\big(p(\boldsymbol\theta\mid y,\boldsymbol d)\|p(\boldsymbol\theta)\big)}+{\log p(y\mid\boldsymbol d)-\log\hat q(y\mid\boldsymbol d)},

where the first term represents information gained from yy, and the second and third terms capture model-reference consistency. Under correct specification and without ambiguity, the second term averages to −EIG(d)-\mathrm{EIG}(\boldsymbol d) over the true parameter, so that the utility averages to zero for every design. Therefore, following the reviewer's suggestion, we revised the integrand as follows.

(ii) Revised integrand. Under misspecification, θ\boldsymbol\theta need not correspond to any quantity of the true DGP, so the reference log⁡p(y∣d)\log p(y\mid\boldsymbol d) of the EIG is computed under a model that may not describe the real system, whereas q^\hat q is built from the observed data. We therefore keep both references and weight them:

f(\boldsymbol d,y) &= \mathbb E_{\boldsymbol\theta\sim p(\boldsymbol\theta\mid y,\boldsymbol d)} \big[\log p(y\mid\boldsymbol\theta,\boldsymbol d)\big] -(1-\nu_t)\log p(y\mid\boldsymbol d) -\nu_t\log\hat q(y\mid\boldsymbol d) \\ &= \mathrm{KL}\big( p(\boldsymbol\theta\mid y,\boldsymbol d) \|p(\boldsymbol\theta) \big) + \nu_t \Big[ \log p(y\mid\boldsymbol d) -\log\hat q(y\mid\boldsymbol d) \Big]. \end{aligned}$$
AuthorsOpenReview ↗

Rebuttal-part 3

That is, the surprisal of yy is measured against the geometric combination p(y∣d)1−νtq^(y∣d)νtp(y\mid\boldsymbol d)^{1-\nu_t}\hat q(y\mid\boldsymbol d)^{\nu_t}. The weight increases over rounds as

νt=(1−α)tt+t0,\nu_t=(1-\alpha)\frac{t}{t+t_0},

so trust in the data-driven reference grows as the bands tighten and approaches the coverage level 1−α1-\alpha. Since νt\nu_t is fixed within a round, all results of Section 3 hold at each round. The revised criterion has the following properties; see Appendix A.6:

  • (a) Information-gain ranking under correct specification. Without ambiguity and with a correct model, the utility averaged over the true parameter equals (1−νt)EIG(d)(1-\nu_t)\mathrm{EIG}(\boldsymbol d) for every design (Proposition A.2), so designs are ranked exactly as by the EIG. This also shows why νt<1\nu_t<1 is required.

  • (b) Reduction to the EIG. For every νt\nu_t, U(d,q)=EIG(d)U(\boldsymbol d,q)=\mathrm{EIG}(\boldsymbol d) when q=q^=p(⋅∣d)q=\hat q=p(\cdot\mid\boldsymbol d); more generally, for q=p(⋅∣d)q=p(\cdot\mid\boldsymbol d), U=EIG(d)+νtKL(p(⋅∣d)∥q^(⋅∣d)).U = \mathrm{EIG}(\boldsymbol d) + \nu_t\mathrm{KL}\big(p(\cdot\mid\boldsymbol d)\|\hat q(\cdot\mid\boldsymbol d)\big).

  • (c) Information minus a misspecification penalty. Under misspecification, in the setting of Callahan & Catanach (2026), the averaged utility decomposes as (1−νt)EIG⋆(d)−[ΔKL(d)+νtKL(p⋆(y∣d)∥p(y∣d))],(1-\nu_t)\mathrm{EIG}^\star(\boldsymbol d) - \Big[\Delta_{\mathrm{KL}}(\boldsymbol d)+\nu_t\mathrm{KL}\big(p^\star(y\mid\boldsymbol d)\|p(y\mid\boldsymbol d)\big)\Big], where EIG⋆\mathrm{EIG}^\star is the information under the true process, ΔKL\Delta_{\mathrm{KL}} is their posterior misspecification gap, and the last term is the mismatch between the true and model marginals. A negative value therefore indicates an experiment that is misleading under the assumed model, rather than an informative one.

  • (d) Role of q^\hat q. Dropping the reference (νt=0\nu_t=0) gives a non-negative criterion that also rewards spurious learning on experiments that are uninformative about the real system; the data-driven reference is what penalises such experiments without requiring a simulator of the true process.

The dual (Eq. 18) and Algorithm 2 change only through ff and its derivative, which gains the term

\mathbb E_{\boldsymbol\theta\mid y,\boldsymbol d} \left[ \partial_{\boldsymbol d} \log p(y\mid\boldsymbol\theta,\boldsymbol d) \right],$$ estimated with the same self-normalised importance weights and requiring no additional likelihood evaluations. All experiments in the revision use the revised integrand. **We emphasise that the central contribution, namely the data-driven confidence-band ambiguity set with its coverage guarantee and the tractable dual reformulation, is independent of the choice of integrand and is not affected by this change.**
Reviewer_C8hoOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a distributionally robust Bayesian experimental design (DR-BED) framework to address model misspecification. The method constructs confidence-band ambiguity sets that are updated during sequential experiments and optimises experimental designs against worst-case distributions. The authors provide theoretical coverage guarantees and evaluate the approach on several benchmark problems.

Critical strengths
  1. The paper addresses a relevant problem in Bayesian experimental design and provides theoretical guarantees under certain assumptions.

  2. The experiments cover multiple settings and show promising improvements over the considered baselines.

Critical weaknesses
  1. It is unclear whether more expressive probabilistic models could avoid or alleviate model misspecification. Comparisons with such alternatives would help establish the necessity of the approach.

  2. In Table 5, DR-BED requires approximately 24 seconds per design selection, compared with 1 second for BAD and 6 seconds for GBOED. It is unclear whether the performance improvements due to the additional computational cost. A more balanced comparison under similar computational budgets would be helpful.

  3. The proposed method relies on many assumptions about the true DGP, including a known mode, an upper bound on its density, unimodality, and Hölder continuity. How these assumptions can be satisfied in practice? Besides, given this prior knowledge, it would also be reasonable to reduce model misspecification directly.

Questions for authors

Please refer to the weaknessess section.

3 comments

W1. We thank the reviewer for this question. We agree that a richer likelihood model is often the first option an experimenter would consider to address model misspecification. However, we view richer models and DR-BED as two complementary, rather than competing, approaches to addressing misspecification in BED.

Following the reviewer's suggestion, in the new Appendix A.10 we discuss and experimentally compare all the baselines under more expressive likelihood models, showing that DR-BED can outperform the baselines in two scenarios with expressive likelihoods. This further illustrates the capability of DR-BED to wrap around any likelihood, including expressive probabilistic models, since the likelihood enters only through the integrand f(d,y)f(\boldsymbol d,y).

Based on the new results presented in Appendix A.10:

(i) Expressiveness reduces, but does not eliminate, misspecification. Whether a model class contains the true DGP cannot generally be verified before design decisions are made. Standard BED treats the assumed model as correct, so residual misspecification can affect design selection. DR-BED instead constructs its ambiguity set non-parametrically from observed data and structural assumptions, and Theorem 3.1 does not require a well-specified likelihood.

(ii) Expressive models are less reliable when data are scarce. This is precisely the BED regime, where experiments are expensive and data are limited. More expressive likelihoods can have broad, prior-dominated posteriors, while nested EIG estimation can be high-variance and computationally demanding.

AuthorsOpenReview ↗

Rebuttal-part 2

W2. We thank the reviewer for this suggestion, and we have implemented the comparison under similar computational budgets, presented in Appendix A.9 through the new Figures 13 and 14. Our new results show the following:

(i) Why additional computation alone does not close the gap. The baselines spend their computation on estimating the EIG under the assumed model more accurately. Under misspecification, this criterion is itself biased because it is evaluated under a likelihood that does not describe the real system; a larger Monte Carlo budget reduces the variance of the estimate but not this bias. DR-BED instead spends its computation on evaluating the design against all outcome distributions that are consistent with the data. Appendix A.9 reports the per-round cost of all methods.

(ii) Where the cost comes from and how it can be reduced. The per-round cost of DR-BED is dominated by the K×BK\times B linear programs that construct the confidence bands (Algorithm 1) inside the stochastic subgradient loop (Algorithm 2), with K=15K=15 and B=40B=40 in our experiments. The cost therefore scales linearly in KK and BB, which can be traded off directly against the accuracy of the subgradients. Note that the band LPs for each outcome sample do not depend on each other, so they could be solved in parallel. Our current implementation solves them one by one, which is one of the reasons for its higher computational time.

(iii) Cost relative to the experiment. BED is primarily intended for settings in which each experiment is expensive and/or slow, such as laboratory assays, pharmacokinetic blood sampling, or sensor deployment, where obtaining a single observation can take minutes to days. In such settings, one or two minutes of computation per design decision is negligible relative to the cost of the experiment, and a more robust design can save far more than it costs.

W3. We would like to clarify that these are structural conditions, substantially weaker than specifying a parametric likelihood, and closely follow assumptions in the shape-restricted confidence-band literature (Jiang, 2017; Chen et al., 2022).

In particular, Chen et al. (2022) assume bounded support, unimodality with a known mode, and a known upper density bound, together with Hölder continuity to establish contraction of the confidence bands and convergence of the robust objective. Their construction also allows the mode to be specified by an interval rather than exactly.

(i) Why these conditions do not remove misspecification. These conditions constrain, but do not identify, the true DGP. Infinitely many conditional densities can satisfy the same mode, density bound, shape, and smoothness constraints while differing in their tails, skewness, and variance. Turning this partial knowledge into a single likelihood would therefore require additional modelling assumptions, reintroducing precisely the misspecification we seek to guard against. DR-BED instead retains this partial knowledge as structural constraints and optimises against all densities consistent with both these constraints and the observed data.

(ii) How these conditions are handled in practice. Following this question, Appendix A.5.1 explains the selection of all hyperparameters in Algorithm 1.

  • Mode. The mode need not be known: at each target design, we estimate it from the current posterior predictive distribution under the assumed model.

  • Unimodality. We relax this condition in practice: the LP in Appendix A.3.1 imposes piecewise monotonicity around each detected mode and allows up to two modes. Unimodality naturally arises when observations consist of a signal plus measurement noise, as in our pharmacokinetic and polynomial benchmarks.

  • Density upper bound. We set U=Δy/(2πσmin⁡)\mathcal U=\Delta_y/(\sqrt{2\pi}\sigma_{\min}) on the normalised outcome scale, where σmin⁡\sigma_{\min} is a lower bound on measurement noise (e.g., instrument precision). A loose bound simply enlarges the ambiguity set, making the resulting design more conservative; if the LP is infeasible, U\mathcal U is enlarged by a fixed factor.

  • Hölder continuity. This is a standard smoothness condition in non-parametric estimation. Since a worst-case Hölder constant can be overly conservative, we treat LL as a calibration constant and report its values in Table 3.

References:

Jiang, H. (2017). Uniform convergence rates for kernel density estimation. Proceedings of the 34th International Conference on Machine Learning, PMLR, pp. 1694–1703.

Chen. X., Lin, Q., and Xu, G. (2022) Distributionally robust optimization with confidence bands for probability density functions. INFORMS Journal on Optimization, 4(1):65–89, 2022.

AuthorsOpenReview ↗

Rebuttal-Part3

Summary

We thank the reviewer for the thoughtful review and constructive suggestions. We agree that the original manuscript would benefit from a clearer positioning of DR-BED relative to more expressive models. The resulting changes add new experiments and clarifications, and we believe they strengthen the existing claims.

  • (1) DR-BED complements expressive likelihood models. The new comparison in Appendix A.10 shows that DR-BED still outperforms the baselines.
  • (2) DR-BED’s performance gains are now assessed under similar computational budgets (Appendix A.9, Figures 13–14). This shows that giving the baselines more computation does not close the gap under misspecification.
  • (3) The structural assumptions on the true DGP now come with a practical procedure for setting each of them. We also clarify that these conditions constrain the true DGP without identifying it, so they cannot be used to remove misspecification directly.

We believe that these revisions adequately address the concerns raised by the reviewer and further strengthen the manuscript. We would be pleased to address any additional questions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

We thank the three reviewers for their constructive and careful reviews. We are encouraged that the reviewers found the problem practically important (R1, R2, R3), the operations-research perspective novel and well-motivated (R3), the combination of non-parametric confidence bands with distributionally robust optimisation clearly explained (R3), and the coverage guarantee and tractable scalar dual reformulation particularly valuable contributions (R2, R3).

The reviews raised four main themes. We have addressed each of them in the revised manuscript.

1. Comparison to expressive likelihood models and equal computational budget (R1).

In the new Appendix A.10, we now discuss and experimentally compare all the baselines under more expressive likelihood models, showing that DR-BED can outperform the baselines in two scenarios with expressive likelihoods. The new Appendix A.9 provides a cost breakdown and an equal-budget comparison with all baselines.

2. The design criterion and its connection to information gain (R2, R3).

We now give an exact decomposition of the integrand into an information-gain term, KL(p(θ∣y,d)∥p(θ))\mathrm{KL}(p(\boldsymbol\theta\mid y,\boldsymbol d)\|p(\boldsymbol\theta)), and a model–reference consistency term, log⁡p(y∣d)−log⁡q^(y∣d)\log p(y\mid\boldsymbol d)-\log\hat q(y\mid\boldsymbol d).

The resulting criterion (Appendix A.6):

  • recovers the EIG exactly when q=q^=p(⋅∣d)q=\hat q=p(\cdot\mid\boldsymbol d);
  • under correct specification and without ambiguity, equals (1−νt)EIG(d)(1-\nu_t)\mathrm{EIG}(\boldsymbol d) on average over the true parameter, and therefore ranks designs exactly as the EIG does;
  • under misspecification, decomposes as true information minus an explicit misspecification penalty, (1−νt)EIG⋆−[ΔKL+νtKL(p⋆(y∣d)∥p(y∣d))](1-\nu_t)\mathrm{EIG}^\star-[\Delta_{\mathrm{KL}}+\nu_t\mathrm{KL}(p^\star(y\mid\boldsymbol d)\|p(y\mid\boldsymbol d))], in the framework of Callahan & Catanach (2026), against whose robust axioms we verify it in the new Appendix A.6.1.

Note that the ambiguity set, the coverage theorem, the conic dual and the stochastic subgradient algorithm are unchanged; only the integrand and its derivative change, and the derivative requires no additional likelihood evaluations. All experiments use the revised criterion.

3. Theory for the implemented sequential procedure (R2).

(i) The coverage guarantee now applies to adaptively selected designs.

(ii) We resolved the degenerate-bin issue raised by R3: the condition max⁡iwi(d)≤1/M\max_i w_i(\boldsymbol d)\le 1/M guarantees well-defined bins and holds automatically for the observations (Section 3.1).

4. Experimental evaluation and reproducibility (R1, R2, R3).

  • A full specification of every baseline, inference procedure, and common random streams for all methods (Appendix A.7.5).
  • Comparisons with a robust BED method in addition to Random and BAD (Section 4.2).
  • A comparison in which all methods use a more expressive likelihood (Appendix A.10).
  • A per-case specification of all inputs of Algorithm 1 (Appendix A.5.1, Table 3).
  • Correction of the noise level stated in the appendix, and an explicit statement of the normalised outcome scale on which NLL is computed (Appendix A.7.3).
  • A clarified interpretation of the results, which distinguishes predictive fit from calibration.

We believe these revisions address the concerns raised and strengthen both the theoretical and the empirical support of the paper.

References

Callahan, J., & Catanach, T. (2026). On the Misinformation in a Statistical Experiment. The 29th International Conference on Artificial Intelligence and Statistics.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.