acceptodds
Under review as a conference paper at ICLR 2027

Beyond Terminal Objectives: Bayesian Experimental Design with Multistep Actions and Losses

Abstract

Existing loss-driven Bayesian experimental design (BED) methods learn policies that target a terminal loss evaluated at a fixed end point. We argue that this terminal formulation is ill-suited to cases where we can take intermediary actions or our experiment endpoint is unknown. To address this, we introduce MA-BED (Multistep Action-BED), which extends the loss-driven BED formulation to include intermediary actions and losses throughout the acquisition trajectory. This assignment of value to incomplete trajectories allows us to encourage the data to be as informative as possible at every stage of the experiment, such that we are encouraged to learn quickly. In turn, this allows us to support problems where separate actions must be taken during acquisition, and problems with unknown, stochastic, or adaptively chosen experiment endpoints. The intermediate signal additionally shortens credit assignment, stabilizing optimization over long rollouts. Indeed, because of the practical training benefits this provides, we find MA-BED not only improves intermediate decisions compared with terminal-loss baselines on target tracking, adaptive stopping, and classical design problems, it can also improve the terminal decisions for longer rollouts.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Bayesian experimental design, experimental design, data acquisition, loss-driven design, Bayesian decision theory

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_umVzOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper extends loss-driven, policy-based Bayesian experimental design (Action-BED) from a single terminal decision to decisions and losses at every acquisition step. MA-BED trains design and action policies end-to-end on a weighted sum of per-step losses, which supports online tasks, stochastic horizons and adaptive stopping. Experiments on tracking, CES with dropout, and source location show gains over Action-BED, DAD and random design.

Critical strengths

The framework is well motivated and seems broadly applicable. Many experimental design problems involve intermediate decisions or unknown endpoints (tracking, adaptive trials, monitoring), and MA-BED handles them with a simple objective at negligible extra cost.

A careful evaluation protocol. A bit surprisingly, it also helps with long-horizon terminal problems.

Critical weaknesses

The main gains are expected by construction. The dynamic tracking, CES-dropout and random-horizon experiments evaluate exactly MA-BED’s training objective, while baselines are trained on terminal objectives. Natural baselines, such as Action-BED with randomly sampled horizons, are missing.

Questions for authors

When should the gains carry over to terminal-only problems? This is perhaps the most interesting finding. The long-horizon results (Tables 12–13) suggest that MA-BED helps as the horizon grows, but they are preliminary. Could the authors strengthen this conclusion, for example with more settings, and an analysis of when and why the gains appear?

3 comments

We thank the reviewer for their constructive suggestions. In particular, we have added a new random-endpoint ablation to disentangle the benefits of objective matching from those of dense intermediate supervision. We also provide additional empirical and theoretical evidence clarifying when MA-BED can improve terminal performance. We address each concern separately below.


Weakness: Objective matching versus dense supervision

The main gains are expected by construction. The dynamic tracking, CES-dropout and random-horizon experiments evaluate exactly MA-BED’s training objective, while baselines are trained on terminal objectives. Natural baselines, such as Action-BED with randomly sampled horizons, are missing.

We thank the reviewer for this suggestion. We added exactly this baseline and found that while it does improve over Action-BED, it does not do as well as MA-BED and our conclusions still hold.

Specifically, we trained Action-BED on one endpoint per trajectory, sampled from the deployment-horizon distribution, with architectures, horizon distribution and training budget matched to MA-BED (Table G in our response to Reviewer ekp7; full details there, Weakness 3). Since both methods optimize the same objective in expectation, this removes the objective-mismatch advantage and isolates the effect of dense intermediate supervision, which we find remains beneficial. In particular, MA-BED achieves lower mean random-horizon error across all evaluated settings, with the clearest additional gains for d=2d=2 and longer horizons, although differences are small for d=5d=5, T∈30,50T\in{30,50}.

We note also that MA-BED frequently achieves better terminal performance than random-endpoint training. For instance, at d=2d=2, T=30T=30, random-endpoint training recovers most of the RH-MSE improvement but substantially worsens terminal MSE (12.7212.72 vs. 7.137.13 for MA-BED). These findings are consistent with the lower gradient variance guaranteed by MA-BED's Rao–Blackwellised estimator. We will incorporate this ablation into the revised manuscript.

Finally, we reiterate that MA-BED often outperforms Action-BED on the terminal loss as well, suggesting that multistep supervision can improve terminal performance even when the training objective differs from the evaluation objective.

Question: When multistep supervision improves terminal performance?

When should the gains carry over to terminal-only problems? This is perhaps the most interesting finding. The long-horizon results (Tables 12–13) suggest that MA-BED helps as the horizon grows, but they are preliminary. Could the authors strengthen this conclusion, for example with more settings, and an analysis of when and why the gains appear?

We thank the reviewer for this question. Intermediate losses help terminal training when (i) long-horizon credit assignment makes terminal-only training optimization-limited, which becomes more likely as TT grows, and (ii) intermediate losses provide sufficiently aligned, low-noise gradients. We highlight that Appendix C already provides significant theoretical investigation of these mechanisms, characterizing how intermediate losses introduce shorter credit-assignment paths, when their gradients remain aligned with the terminal objective and thus provide descent directions (Proposition 7), and under which conditions they improve the gradient signal-to-noise ratio (Proposition 9). We also provide further empirical and theoretical insights below that go beyond the original submission, that we have incorporated in the revised manuscript.

Explaining the gains through gradient diagnostics. We monitor these diagnostics on location finding (T=30T=30) to illustrate the theoretical conditions established in Appendix C (the gradient-diagnostics table below). The terminal descent factor MM measures whether the multistep gradient remains a descent direction for the terminal objective (if M>0M>0), while the relative SNR measures the improvement in signal-to-noise ratio along the terminal-gradient direction (if relative SNR >1>1). Both conditions hold throughout the first 75% of training, during which MA-BED converges faster to better terminal performance than Action-BED. At the final checkpoint, when MA-BED has already converged, M=−0.24M=-0.24 and the SNR ratio falls to 0.580.58, while MA-BED’s MSE slightly increases from 7.017.01 to 7.137.13. Beyond illustrating our theoretical analysis, these diagnostics offer a practical way to assess intermediate loss choices and tune auxiliary supervision during training to improve terminal performance.

Empirical evidence. The horizon-dependent gains observed in Tables 12–13 are further strengthened by our new five-seed experiments (see Table C in our response to Reviewer GDjv), which confirm the robustness of MA-BED’s terminal improvements and suggest greater training stability, with less variability across runs and fewer poor optimization outcomes. Complementing this, our new double-pendulum experiments (Table B, Reviewer GDjv) show that MA-BED also improves terminal MSE on a distinct dynamical task (1.1421.142 vs. 1.2411.241). Finally, our new weighting ablation (Table A, Reviewer GDjv) further shows that terminal performance depends on balancing intermediate and late-horizon supervision: γ=0.6\gamma=0.6 achieves the best terminal MSE (6.646.64), compared with 8.328.32 for Action-BED and 8.008.00 for uniform weighting. Smaller γ\gamma provides insufficient auxiliary supervision, whereas larger γ\gamma places insufficient emphasis on later horizons. This identifies the weighting schedule as a tunable hyperparameter for improving terminal performance.

Further theoretical characterization. Assuming the terminal objective is LL-smooth, we derive a sufficient condition under which an exact multistep gradient update yields strictly lower terminal loss than a terminal-only update, starting from the same parameters with learning rate η\eta and auxiliary weight β>0\beta>0:

L\eta\cdot ||g_T|| \cdot ||g_A|| +\frac{L\eta\beta}{2}||g_A||^2.$$ Here, $g_T$ and $g_A$ denote the terminal and auxiliary gradients with respect to the design-policy parameters, holding the downstream policy fixed. This extends Proposition 7's first-order descent condition to a finite-step comparison of exact gradient updates. The formal statement and proof will be added to Appendix C.4. --- ### Table H - Gradient diagnostics (Location finding $T=30$). Terminal MSE in $10^{-3}$. $M>0$ indicates a terminal descent direction; relative SNR $>1$ indicates increased terminal-direction SNR. |Training progress|Action-BED MSE|MA-BED MSE|Descent factor M|Relative SNR| |---|---|---|---|---| |10%|17.23|12.26|8.18|3.44| |25%|12.71|8.81|1.82|6.55| |50%|9.59|7.44|2.46|2.17| |75%|9.44|7.01|1.05|5.30| |100%|8.32|7.13|-0.24|0.58| $M=1+\beta\langle g_T,g_A\rangle/\|g_T\|^2$; relative SNR $=\mathrm{SNR}_\beta/\mathrm{SNR}_T$, measured along the terminal-gradient direction.

Thank you for the additional experiments that resolves my weakness concern. I keep my rating.

Reviewer_ekp7OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MA-BED extends Action-BED to settings where decisions are needed during data collection or the experiment may end at an uncertain time. It jointly trains design and intermediate-action policies using weighted losses over acquisition prefixes, and examines how ideal losses preserve terminal optima and how auxiliary gradients affect optimization. Experiments on dynamic tracking, CES preference recommendation, and source localization show improvements in intermediate decisions and stopping efficiency, with some gains in terminal performance over longer horizons.

Critical strengths

Valuing decisions throughout data collection gives a common objective for online prediction and tasks with uncertain stopping times. In two-dimensional localization, improvements in both intermediate and terminal error persist across five training seeds, with training times similar to Action-BED (Tables 14–15).

Critical weaknesses

a. Sequential experimental design already allows cumulative stage utilities [1], but sec 5 does not clearly explain what MA-BED adds at the objective level. b JADAI already uses intermediate posterior losses to jointly train design and inference networks [2], but sec 5 leaves MA-BED’s contribution beyond this training approach unclear. c Action-BED is trained for a fixed endpoint in the random-horizon experiments, so the comparison does not separate the benefits of matching the task objective from those of supervision at every step (Appendices E.2–E.3).

[1] Xun Huan and Youssef M. Marzouk. Sequential Bayesian optimal experimental design via approximate dynamic programming. arXiv:1604.08320, 2016. [2] Niels Bracher, Lars Kühmichel, Desi R. Ivanova, Xavier Intes, Paul-Christian Bürkner, and Stefan T. Radev. JADAI: Jointly Amortizing Adaptive Design and Bayesian Inference. arXiv:2512.22999, 2025.

Questions for authors
  1. If JADAI’s Eq. (18) uses the task losses in this paper with matched backpropagation settings, how do its training gradients differ from Eq. (6)?
  2. Under the same horizon distribution and compute budget, how does Action-BED trained with randomly sampled endpoints compare with supervision at every prefix?
  3. Correct Proposition 6’s conditions for histories with changing dimensions, and explain wheter this affects the optimization interpretation in sec 4.3 (Appendix C.1 and C.3).
4 comments

We thank the reviewer for their detailed comments. In particular, their suggestion to compare against Action-BED trained with randomly sampled endpoints led to a new ablation that separates the effect of matching the task objective from that of per-prefix supervision, which we believe makes the paper’s empirical case considerably stronger. We address each point below.


Question 1 & Weakness 2: Relation to JADAI and gradient comparison

JADAI already uses intermediate posterior losses to jointly train design and inference networks [2], but sec 5 leaves MA-BED’s contribution beyond this training approach unclear.

If JADAI’s Eq. (18) uses the task losses in this paper with matched backpropagation settings, how do its training gradients differ from Eq. (6)?

We thank the reviewer for highlighting this closely related work. While JADAI (Bracher et al., 2026) shares aspects of MA-BED’s training mechanism, the two methods differ fundamentally in their objectives and contributions. JADAI introduces intermediate losses derived from its posterior estimator to densify the training signal for terminal EIG through a telescoping construction. In contrast, MA-BED introduces a general loss-driven formulation in which intermediate downstream decisions and their associated losses define the design objective itself. We will revise Section 5 to explicitly discuss JADAI and better articulate our distinct contributions.

  • At the gradient level, JADAI’s Eq. (18) amounts to a sum of intermediate-loss gradients induced by its stop-gradient telescoping construction. With identical losses, parameterizations, and backpropagation settings, this aggregation can indeed recover our Eq. (6) for λt≡1\lambda_t\equiv1. However, the ability to optimize arbitrary intermediate task losses is precisely what MA-BED establishes through Proposition 1, which lifts the multistep Bayesian decision problem to joint optimization over design and intermediate action policies. This removes the need for posterior inference and extends policy-based BED to genuinely intermediate decision problems.

  • Moreover, JADAI’s practical gradient estimator differs from MA-BED’s full pathwise differentiation. By detaching the history at each policy input, JADAI retains direct supervision from future losses to past designs but omits gradient paths capturing how earlier observations influence subsequent adaptive design decisions. Thus, its gradients do not fully account for the effect of early experiments on future policy adaptation, unlike MA-BED’s end-to-end optimization of the adaptive trajectory.

  • Finally, JADAI's posterior-inference setting can be recovered within MA-BED by taking downstream actions as posterior distributions and using logarithmic scoring losses.

Additional empirical comparison. To further substantiate that the choice of intermediate losses is central to MA-BED, rather than merely an implementation detail, we compare against JADAI on its two-source location finding benchmark (D=K=2D=K=2). MA-BED uses the task-specific, permutation-invariant QLL loss

=\min_{\sigma\in\mathcal S_K}\frac{1}{KD} \sum_{k,d}\left[ \Delta_{\sigma,k,d}^2+ \log(\Delta_{\sigma,k,d}^2+\epsilon) \right],$$ where $\Delta_{\sigma,k,d}=\theta_{\sigma(k),d}-\hat\theta_{k,d}$. This loss is known to provide an effective supervision signal for learning informative designs (Action-BED [3]), even though maximizing EIG is not its explicit objective. We match the generative model and evaluation settings, and training budgets where feasible. Using a single policy trained with maximum horizon $T=30$ for each method, we compare sPCE, directly targeted by JADAI. As shown in the JADAI comparison table below, MA-BED outperforms JADAI at every horizon, nearly saturating the sPCE estimator’s ceiling at $t=30$, denoting a much higher effective EIG. Thus, despite similarities in their training mechanisms, the differences are significant and MA-BED can learn highly informative policies without explicitly targeting information gain precisely thanks to its ability to choose the relevant task losses. --- ### Table F - JADAI comparison (Location finding, $d=K=2$). sPCE lower bounds, following JADAI’s protocol: $5\times10^5$ contrastive samples at $t=10,20$; $10^6$ at $t=30$. Policies are trained to $T=30$. JADAI numbers come from its paper. |Method|t=10|t=20|t=30| |---|---|---|---| |JADAI|6.74±0.04|10.90±0.03|12.82±0.03| |MA-BED (QLL)|**7.21±0.06**|**12.42±0.04**|**13.63±0.02**|

Question 2 & Weakness 3: Random-endpoint ablation

Action-BED is trained for a fixed endpoint in the random-horizon experiments, so the comparison does not separate the benefits of matching the task objective from those of supervision at every step (Appendices E.2–E.3). Under the same horizon distribution and compute budget, how does Action-BED trained with randomly sampled endpoints compare with supervision at every prefix?

We agree with the reviewer that the original random-horizon comparison does not fully disentangle the benefits of dense intermediate supervision from those of optimizing the appropriate deployment objective. We have added this ablation for the locating finding task: Action-BED trained on one endpoint per trajectory, sampled from the deployment-horizon distribution, with the same architectures, horizon distribution and training budget as MA-BED (Table G below). It optimizes the same objective as MA-BED, so it separates objective matching from per-prefix supervision. With λt=P(τ=t)\lambda_t = P(\tau = t), MA-BED’s gradient is the conditional expectation of the random-endpoint gradient over τ\tau (i.e., its Rao–Blackwellisation), and therefore has no higher variance. Given the limited rebuttal period, we report the configurations completed so far and will extend this ablation to the remaining settings in the revised manuscript.

Experimental results. Random-endpoint training substantially improves RH-MSE over terminal-only Action-BED, confirming the importance of objective matching. Nevertheless, MA-BED achieves the lowest mean RH-MSE across all evaluated settings, with particularly pronounced improvements for d=2d=2 and longer horizons. Moreover, MA-BED frequently achieves better terminal performance than both baselines, despite not directly optimizing the terminal objective.

Overall, this random-horizon baseline is not an existing method but a purpose-built ablation designed to match MA-BED’s objective. It demonstrates that MA-BED’s gains cannot be attributed solely to objective matching, but also reflect the optimization benefits of dense intermediate supervision, although their magnitude depends on the setting. We will incorporate this ablation into the revised manuscript.


Table G — Random-endpoint ablation (Location Finding, K=2)

All methods use p(τ=t)∝0.95T−tp(\tau=t)\propto0.95^{T-t}. Results are mean ± standard error over 50,000 rollouts. sPCE and sNMC are computed with 10710^7 contrastive samples. ABED = Action-BED; ABED-τ = Action-BED (random endpoint).

d = 2 — RH-MSE (10−210^{-2}), terminal MSE (10−310^{-3}).

TMethodRH-MSE ↓MSE ↓sPCE ↑sNMC ↑
30ABED12.69±0.068.32±0.2411.60±0.0511.72±0.06
ABED-τ9.01±0.0612.72±0.3311.17±0.0511.23±0.05
MA-BED8.34±0.067.13±0.2212.78±0.0513.13±0.06
50ABED6.30±0.022.95±0.0713.24±0.0413.68±0.06
ABED-τ3.31±0.024.50±0.1313.08±0.0413.35±0.05
MA-BED2.70±0.021.37±0.0315.09±0.0317.99±0.12

d = 5 — RH-MSE and terminal MSE (10−110^{-1}).

TMethodRH-MSE ↓MSE ↓sPCE ↑sNMC ↑
30ABED5.82±0.014.23±0.026.47±0.076.50±0.08
ABED-τ5.78±0.024.41±0.026.41±0.086.43±0.08
MA-BED5.75±0.024.30±0.026.62±0.086.67±0.08
50ABED4.21±0.022.66±0.0110.34±0.0911.06±0.12
ABED-τ4.07±0.012.72±0.0210.59±0.0911.52±0.13
MA-BED4.05±0.012.68±0.0110.51±0.1011.51±0.14
75ABED2.84±0.011.70±0.0113.12±0.0815.26±0.14
ABED-τ2.60±0.011.58±0.0113.90±0.0719.27±0.22
MA-BED2.47±0.011.41±0.0114.23±0.0721.17±0.23

Weakness 1: Relation to sOED

Sequential experimental design already allows cumulative stage utilities [1], but sec 5 does not clearly explain what MA-BED adds at the objective level.

The central contribution of MA-BED is not merely allowing cumulative stage utilities, but providing a practical, end-to-end method for optimizing general intermediate decision losses. Indeed, the sOED formulation in Eq. 1 of Huan and Marzouk (2016) [1] is sufficiently general to encompass our multistep EFL by taking gt(ht)=−λtRt(ht)g_t(h_t)=-\lambda_t R_t(h_t), where Rt(ht)=inf⁡atEp(θ∣ht)[ℓt(at,θ,ht)]R_t(h_t)=\inf_{a_t}\mathbb E_{p(\theta\mid h_t)}[\ell_t(a_t,\theta,h_t)] is the posterior Bayes risk. However, the approach developed in [1] focuses on terminal information gain, potentially combined with intermediate experimental costs, rather than task-specific downstream decisions throughout acquisition.

More importantly, applying their approximate dynamic programming approach to our objective would require repeated posterior inference, evaluation of intermediate Bayes risks, and approximation of Bellman value functions over belief states, making general nonlinear losses and long horizons computationally challenging. MA-BED instead makes these objectives directly tractable: Proposition 1 lifts the intermediate Bayesian decisions into jointly optimized action policies, enabling end-to-end training from simulated trajectories without explicit posterior inference, nested decision optimization, or Bellman value approximation. This provides a practical, scalable approach to a broader class of intermediate decision problems. We will clarify this distinction and correct our characterization of [1] in Section 5.


Question 3: Correction of Proposition 6

Correct Proposition 6’s conditions for histories with changing dimensions, and explain wheter this affects the optimization interpretation in sec 4.3 (Appendix C.1 and C.3).

We thank the reviewer for pointing out this technical issue. We agree that Proposition 6 requires additional care when the rollout state represents a growing history with changing dimensions.

When sts_t represents the growing history, Jk∈Rnk×nk−1J_k\in\mathbb{R}^{n_k\times n_{k-1}} with nk≥nk−1n_k\geq n_{k-1}. Defining σmin⁡(A)=min⁡∥v∥=1∥Av∥\sigma_{\min}(A)=\min_{\|v\|=1}\|Av\| and assuming full-column-rank Jacobians, the singular-value product bound and the upper (contractive) bound of Proposition 6 remain valid. However, the expansive bound must be restricted to reachable directions:

\geq \left(\prod_{k=i+1}^{t}m_k\right) \|\Pi_{\operatorname{Im}(P_{i\to t})}q_t\|.$$ Indeed, $P_{i\to t}^{\top}$ annihilates components orthogonal to its reachable subspace, even when the forward Jacobians have positive minimum singular values. We will explicitly state the dimensionality and rank assumptions and correct Proposition 6 and its proof accordingly. **Crucially, this correction does not affect the optimization conclusions of Section 4.3.** Proposition 6 serves only to illustrate, under simplified sufficient conditions, how gradient sensitivity can depend multiplicatively on temporal path length. It was not intended to provide a universal guarantee against vanishing or exploding gradients. The exact gradient decomposition (Proposition 5), and hence the presence of shorter credit-assignment paths under multistep supervision, remain fully unchanged for variable-dimensional histories. Likewise, the terminal-descent and gradient-SNR conditions (Propositions 7 and 9) are independent of Proposition 6 and remain unaffected. --- ### References [1] Xun Huan and Youssef M. Marzouk. **Sequential Bayesian Optimal Experimental Design via Approximate Dynamic Programming.** arXiv:1604.08320, 2016. [2] Niels Leif Bracher, Lars Kühmichel, Desi R. Ivanova, Xavier Intes, Paul-Christian Bürkner, and Stefan T. Radev. **JADAI: Jointly Amortizing Adaptive Design and Bayesian Inference.** Proceedings of the 43rd International Conference on Machine Learning (ICML), 2026. [3] Tom Rossa, Angus Phillips, and Tom Rainforth. **Action-BED: Task-Driven Bayesian Experimental Design with Singly Intractable Objectives.** arXiv:2606.23662, 2026. [4] Daolang Huang, Xinyi Wen, Ayush Bharti, Samuel Kaski, and Luigi Acerbi. **ALINE: Joint Amortization for Bayesian Inference and Active Data Acquisition.** Advances in Neural Information Processing Systems (NeurIPS), 2025. [5] Tom Blau, Edwin V. Bonilla, Iadine Chades, and Amir Dezfouli. **Optimizing Sequential Experimental Design with Deep Reinforcement Learning.** Proceedings of the 39th International Conference on Machine Learning (ICML), PMLR 162:2107–2128, 2022.

I have updated my evaluation.

Reviewer_GDjvOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper extends Action-BED, a loss-driven Bayesian experimental design method, to multi-step losses, enabling learning from intermediate endpoints rather than relying on a fixed terminal endpoint. The proposed method demonstrates improvements in trajectory loss, long-horizon prediction, and terminal performance on small-scale, controlled synthetic tasks.

Critical strengths
  • The paper is easy to read and follow, which is great.
  • The paper provides a clear motivation for the multi-step loss.
  • The controlled experiments provide insights into how the algorithm actually works.
Critical weaknesses
  • Lack of novelty. The paper builds primarily on Action-BED. If we set the horizon to 1 in Action-BED, it reduces the objective to a per-step loss. By summing losses over multiple steps, the same effect could potentially be achieved. Moreover, similar analyses have already been conducted in finite-horizon episodic MDPs, so the theoretical analysis does not appear to be novel.

  • Unclear design of the loss function. One could naively decompose the terminal loss into multiple per-step losses or design a more principled decomposition that helps minimize the terminal loss. If the main contribution is the use of a multi-step loss, the paper should explain how to choose the loss function at each step, especially given that the loss depends not only on the action taken at that step but also on the posterior sample. Moreover, there are per-step loss coefficients λt\lambda_t to be specified. An experimental study over different coefficients is worth exploring.

  • Limited experiments and unclear practical advantages. The experiments are limited to toy examples. Although these examples provide useful insights, they do not serve as realistic benchmarks for evaluating the algorithm's effectiveness when the underlying model is unknown. This problem is closely related to POMDPs and Bayes-adaptive MDPs: the former infer latent states that generate observations, while the latter learn the underlying MDP model or its parameters. There is also a rich body of work on approximate solutions to these problems that require minimal assumptions about the environment. Since these problems can generally be framed in terms of actively inferring latent variables from observable histories, it is unclear how the proposed method improves upon existing approaches in addressing the underlying challenges, such as posterior learning, non-stationary policies, and long-horizon dependencies.

  • Insufficient statistical evidence. Most experiments use only a single random seed. Although Appendix Table 14 reports results on robustness across multiple seeds, the evaluation is limited to a small-scale setting with d=2. If the system dynamics are known exactly, why not vary the initial-state parameters to evaluate the method's robustness more thoroughly? Furthermore, Action-BED already includes a MNIST example and a double-pendulum task, both of which would be worthwhile benchmarks for comparison if you want to demonstrate advantages.

Questions for authors

What is the difference between a design and an action? Isn't a design by definition an action? As I read the details of the dynamic tracking task, I find that the action is actually the reconstruction of the target position. However, this seems to conflate the action with what it truly means. Furthermore, in the analysis, the design and action policies are indeed compiled together. Therefore, a more detailed explanation of their respective roles is needed.

6 comments

We thank the reviewer for their comments. We proceed to address each of their concerns separately below. We highlight in particular the numerous new experiments we have performed that we believe address some of the key concerns raised.

Weakness 1: Lack of novelty

The paper builds primarily on Action-BED. If we set the horizon to 1 in Action-BED, it reduces the objective to a per-step loss. By summing losses over multiple steps, the same effect could potentially be achieved...

We believe there has been a misunderstanding here: MA-BED is not Action-BED with T=1T=1. Training Action-BED with horizon 11 at every step gives a myopic policy: each design is chosen only for its immediate loss. In MA-BED, the design at step tt is trained against all subsequent losses ℓt,…,ℓT\ell_t,\ldots,\ell_T, so it remains non-myopic. Summing per-step losses over a single shared rollout, while jointly optimizing designs and actions under this objective, is precisely our approach. In particular, formulating this within loss-driven BED, establishing its Bayes-optimality (Proposition 1), and making the joint optimization of design and action policies tractable is a novel contribution to the experimental design literature that extend loss-driven BED beyond terminal objectives. This enables problems that are difficult to address within existing policy-based BED frameworks, including online decisions (tracking), random horizons (CES, location finding), and adaptive stopping.

...similar analyses have already been conducted in finite-horizon episodic MDPs, so the theoretical analysis does not appear to be novel.

We agree that Bellman recursions and advantage-based constructions are standard in finite-horizon MDPs, and we do not claim novelty in these principles themselves. Our contribution is to establish their implications for loss-driven BED by constructing Bellman-consistent intermediate losses from posterior decision risks. Unlike a generic decomposition of terminal loss, these intermediate losses preserve terminal-optimal design policies under arbitrary non-negative weights with λT>0\lambda_T>0, without requiring a telescoping objective. We will clarify this connection and the scope of our theoretical contribution.


Weakness 2: Unclear design of the loss function

One could naively decompose the terminal loss into multiple per-step losses or design a more principled decomposition that helps minimize the terminal loss. If the main contribution is the use of a multi-step loss, the paper should explain how to choose the loss function at each step, especially given that the loss depends not only on the action taken at that step but also on the posterior sample. Moreover, there are per-step loss coefficients to be specified.

We should clarify that in MA-BED, intermediate losses are not a decomposition of a terminal loss, but relate to additional actions that are taken at each step: in standard BED we only have a single terminal action, but MA-BED is extending this to allow intermediate actions as well. The intermediate losses are thus part of the problem specification itself, rather than something the user needs to “design”. For example, in dynamic tracking, ℓt\ell_t is the tracking error at step tt, while in CES, λt\lambda_t is the probability that the participant leaves at step tt. In both cases the problem defines ℓt\ell_t and λt\lambda_t, and they need not be designed.

Only in the special case where the task is purely terminal, and we are using MA-BED to try and improve training, does the question of choosing ℓt\ell_t arise. Here we already do provide explicit guidance on how the ℓt\ell_t should be chosen: Theorem 1 shows that they should try to approximate the Bellman-consistent losses, as these preserve the terminal optimum for any λ\lambda.

... An experimental study over different coefficients is worth exploring.

We agree that this would be helpful in the terminal-only setting, thank you for the suggestion. To this end, we ran an ablation over the weights λt∝γT−t\lambda_t \propto \gamma^{T-t} for location finding (the weighting-ablation table below). Every γ\gamma improves terminal MSE, sPCE and sNMC over Action-BED, so the gains are robust to this choice.

Table A - Weighting ablation (Location finding d=2d=2, T=30T=30).

MA-BED uses normalized weights λt∝γT−t\lambda_t\propto\gamma^{T-t}; terminal MSE in 10−310^{-3}.

MethodγMSE ↓sPCE ↑sNMC ↑
Action-BED—8.32±0.2411.60±0.0511.72±0.06
MA-BED0.407.59±0.1612.03±0.0512.18±0.06
MA-BED0.507.92±0.2212.00±0.0512.13±0.05
MA-BED0.606.64±0.2712.58±0.0512.90±0.06
MA-BED0.706.83±0.1812.29±0.0412.47±0.06
MA-BED0.806.99±0.2312.76±0.0513.10±0.07
MA-BED0.907.02±0.2713.20±0.0513.67±0.07
MA-BED0.957.13±0.2212.78±0.0513.13±0.06
MA-BED0.997.59±0.2612.83±0.0513.19±0.07
MA-BED1.008.00±0.2612.75±0.0513.07±0.06

Weakness 3. Limited experiments and unclear practical advantages

The experiments are limited to toy examples. Although these examples provide useful insights, they do not serve as realistic benchmarks for evaluating the algorithm's effectiveness when the underlying model is unknown...

In BED, the generative model is assumed given, and uncertainty about its parameters can be incorporated into the prior. Since designs are chosen before any data exist, there is nothing from which an unknown model could be learned, and simulation-based benchmarks are the norm. Our evaluation goes beyond that norm: besides the standard location finding and CES benchmarks, we study tracking, random horizons and, following your suggestion, a double pendulum (Table B), with gains over Action-BED on the objectives they target. We have also added five-seed evaluations (Tables C-D), a weighting ablation (Table A), a random-endpoint baseline (Table G, Reviewer ekp7) and a comparison with JADAI (Table F, Reviewer ekp7).

...This problem is closely related to POMDPs and Bayes-adaptive MDPs... There is also a rich body of work on approximate solutions to these problems that require minimal assumptions about the environment. Since these problems can generally be framed in terms of actively inferring latent variables from observable histories, it is unclear how the proposed method improves upon existing approaches in addressing the underlying challenges, such as posterior learning, non-stationary policies, and long-horizon dependencies.

Thank you for this comment. Our work is primarily a Bayesian experimental design contribution: standard BED methods, including Action-BED, only have a terminal loss, whereas we write BED objectives with intermediate actions and losses and show (Proposition 1) that the resulting nested objective can be lifted to a single expectation over jointly optimized design and action policies. This gives a practical end-to-end algorithm that avoids explicit belief or posterior computation altogether, since the action policy directly outputs the decision. Our starting point and baselines are therefore those of the BED literature, with the standard BED assumption of a known likelihood with unknown parameters θ\theta.

That said, the resulting problem does have the structure of a POMDP with information rewards (Spaan et al., 2015; Satsangi et al., 2018), and following your comment we have made this connection explicit in a new paragraph in the related-work section (reproduced in our response to your final comment below). We see the POMDP literature as complementary, since it offers other ways of optimising objectives of this form. The challenges you list are handled as in policy-based BED: the policy conditions on the full history and so is non-stationary, and long horizons are addressed by the dense supervision analysed in Section 4.3.


Question: What is the difference between a design and an action?

Isn't a design by definition an action? ...a more detailed explanation of their respective roles is needed.

Thank you for this question. Answering it helped us clarify how MA-BED relates to the POMDP literature, and we have added a paragraph to the related work on this. The two terms play distinct roles in decision-theoretic BED, whereas in RL and optimal control both would be called “actions’’:

  • A design ξt\xi_t determines which data are observed. It affects future observations but does not (typically) enter the loss.
  • An action ata_t is the downstream decision made from the data collected so far, for example an estimate of a statistic of interest. It enters the loss but does not affect future observations.

In dynamic tracking, the design is where to measure, and the action is the reconstructed target position. Our policies are trained jointly, but they play different roles.

This separation is exactly what links MA-BED to POMDPs. The added paragraph reads:

“The MA-BED formulation is closely related to partially observable Markov decision processes (POMDPs; Åström, 1965; Aoki, 1967; Littman et al., 1995). In particular, its sequential decision structure can be represented through composite actions (at,ξt+1)(a_t, \xi_{t+1}), where ata_t determines the current decision loss and ξt+1\xi_{t+1} governs future observations. This mirrors POMDPs with information rewards (POMDP-IR; Spaan et al., 2015; Satsangi et al., 2018), which introduce prediction actions alongside sensing actions, and is closely related to belief-dependent reward formulations in ρ\rhoPOMDPs (Araya et al., 2010). While these frameworks establish related decision-theoretic formulations, their solutions primarily rely on POMDP planning algorithms. MA-BED instead casts these objectives within loss-driven BED and jointly learns amortized acquisition and decision policies through end-to-end optimization, avoiding explicit posterior inference and planning at deployment.”

Weakness 4: Insufficient statistical evidence

Thank you for pointing this out. We are extending our multi-seed evaluation across all experiments where it was previously missing. Given the substantial computational cost, some runs are still ongoing, but all results will be included in the next version of the paper. In the meantime, preliminary results from additional seeds confirm all previously reported trends, with even stronger gains on CES, suggesting that the seed originally reported may have been particularly favorable to Action-BED; see Tables C-D below. These findings further support the robustness of our conclusions. We have also added an ablation varying the initial-state parameters in the tracking task, as requested, to assess robustness beyond random-seed variability; see Table E below.

As per your suggestion, we have evaluated Action-BED and MA-BED on the double pendulum task, using the exact experimental setting from Action-BED [7], with uniform weighting for RH-MSE and per-step losses defined as the MSE of the predicted physical parameters governing the dynamics. MA-BED improves both RH-MSE and terminal MSE, providing further evidence that multistep supervision can also enhance terminal performance on a distinct task.

Table B - Double pendulum.

RH-MSE and terminal MSE in 10−410^{-4}, mean ± s.e. over 3 seeds and 50,000 shared evaluation trajectories.

MethodRH-MSE ↓Terminal MSE ↓
Action-BED4.277±0.0431.241±0.010
MA-BED4.035±0.0041.142±0.004

Table C — Training-seed variability (Location finding)

Mean ± s.e. over five seeds. sPCE/sNMC evaluated with 10⁷ contrastive samples for T=30 and 10⁸ otherwise to reduce approximation bias and variability in the EIG estimates.

d = 2 — MSE in 10⁻³, RH-MSE in 10⁻².

TMethodMSE ↓RH-MSE ↓sPCE ↑sNMC ↑
30Action-BED8.19±0.3412.72±0.1911.65±0.1411.77±0.15
MA-BED6.73±0.028.21±0.0613.03±0.0713.47±0.10
50Action-BED3.20±0.126.13±0.1113.29±0.0713.44±0.09
MA-BED1.46±0.082.82±0.0315.86±0.0716.37±0.11

d = 5 — MSE and RH-MSE in 10⁻¹.

TMethodMSE ↓RH-MSE ↓sPCE ↑sNMC ↑
30Action-BED4.26±0.015.84±0.016.52±0.046.55±0.05
MA-BED4.32±0.015.76±0.016.57±0.036.61±0.03
50Action-BED2.74±0.024.22±0.0610.26±0.1610.44±0.19
MA-BED2.65±0.014.03±0.0111.16±0.0911.60±0.13
75Action-BED1.61±0.082.76±0.0514.69±0.5017.02±1.47
MA-BED1.45±0.042.48±0.0215.34±0.0618.92±0.42

Table D - Training-seed variability (CES)

Expected normalized recommendation regret (10−210^{-2}), mean ± s.e. over five seeds; qq is the dropout rate.

Methodq=0.05q=0.1q=0.2q=0.3q=0.4
Action-BED2.61±0.404.76±0.618.72±0.9310.96±0.8514.26±0.91
MA-BED1.28±0.052.34±0.024.48±0.026.48±0.038.29±0.03

Table E - Tracking initialization robustness (T=40T=40)

Uniformly weighted trajectory MSE (10−310^{-3}) under different initial-position and initial-velocity distributions: x0∼N(μx,σx2I2)x_0\sim\mathcal N(\mu_x,\sigma_x^2I_2) and v0∼N(μv,σv2I2)v_0\sim\mathcal N(\mu_v,\sigma_v^2I_2). Columns give (σx,σv)(\sigma_x,\sigma_v): Low (0.075,0.04)(0.075,0.04), Original (0.15,0.08)(0.15,0.08), High (0.30,0.16)(0.30,0.16). Methods are retrained for each setting with matched architectures and budgets. Mean ± s.e. over 10,000 simulated trajectories.

MethodLowOriginalHigh
Action-BED24.26±0.3745.24±0.3663.77±0.41
MA-BED1.87±0.143.22±0.058.27±0.11

References

[1] Satsangi, Y., Whiteson, S., Oliehoek, F. A., and Spaan, M. T. J. (2018). Exploiting Submodular Value Functions for Scaling Up Active Perception. Autonomous Robots, 42(2), 209–233.

[2] Spaan, M. T. J., Veiga, T. S., and Lima, P. U. (2015). Decision-Theoretic Planning Under Uncertainty with Information Rewards for Active Cooperative Perception. Autonomous Agents and Multi-Agent Systems, 29(6), 1157–1185.

[3] Åström, K. J. (1965). Optimal Control of Markov Processes with Incomplete State Information. Journal of Mathematical Analysis and Applications, 10(1), 174–205.

[4] Aoki, M. (1967). Optimization of Stochastic Systems: Topics in Discrete-Time Systems. Mathematics in Science and Engineering, Academic Press.

[5] Littman, M. L., Cassandra, A. R., and Kaelbling, L. P. (1995). Learning Policies for Partially Observable Environments: Scaling Up. International Conference on Machine Learning (ICML).

[6] Araya, M., Buffet, O., Thomas, V., and Charpillet, F. (2010). A POMDP Extension with Belief-Dependent Rewards. Advances in Neural Information Processing Systems (NeurIPS).

[7] Tom Rossa, Angus Phillips, and Tom Rainforth. Action-BED: Task-Driven Bayesian Experimental Design with Singly Intractable Objectives. arXiv:2606.23662, 2026.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal

We sincerely thank all reviewers for their constructive feedback, which has helped us substantially strengthen the paper. We have uploaded a revised manuscript, with modifications highlighted in red.

First, isolating the sources of improvement. Following Reviewers ekp7 and umVz, we introduced an Action-BED ablation trained on randomly sampled endpoints, matching MA-BED's expected objective. This confirms that objective matching accounts for an important part of the gains, while dense per-prefix supervision provides additional benefits, particularly at longer horizons, and consistently improves terminal performance over random-endpoint training in the evaluated settings.

Second, strengthening the theoretical and empirical evidence. We have substantially expanded our empirical analysis, adding five-seed evaluations on CES and location finding, tracking robustness experiments, an ablation of temporal weights, and practical gradient diagnostics. We also established a stronger theoretical condition for improved terminal optimization. Furthermore, we report new comparisons with JADAI and experiments on a stochastic double pendulum in our responses, providing additional evidence of the practical benefits of MA-BED.

Ongoing experiments. Some additional evaluations are still running, including further robustness seeds at (T,d)=(75,2)(T,d)=(75,2) and (100,5)(100,5), the random-endpoint ablation on CES, and expanded comparisons with JADAI and on the double-pendulum task across additional settings and baselines. We intend to incorporate these results into the manuscript once the corresponding evaluations are complete.

We hope these new results, clarifications, and manuscript revisions address the reviewers' concerns and strengthen the evidence supporting our conclusions. We would be very grateful if the reviewers would consider revisiting their assessments and increasing their scores in light of these improvements.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.