- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper extends loss-driven, policy-based Bayesian experimental design (Action-BED) from a single terminal decision to decisions and losses at every acquisition step. MA-BED trains design and action policies end-to-end on a weighted sum of per-step losses, which supports online tasks, stochastic horizons and adaptive stopping. Experiments on tracking, CES with dropout, and source location show gains over Action-BED, DAD and random design.
Critical strengths
The framework is well motivated and seems broadly applicable. Many experimental design problems involve intermediate decisions or unknown endpoints (tracking, adaptive trials, monitoring), and MA-BED handles them with a simple objective at negligible extra cost.
A careful evaluation protocol. A bit surprisingly, it also helps with long-horizon terminal problems.
Critical weaknesses
The main gains are expected by construction. The dynamic tracking, CES-dropout and random-horizon experiments evaluate exactly MA-BED’s training objective, while baselines are trained on terminal objectives. Natural baselines, such as Action-BED with randomly sampled horizons, are missing.
Questions for authors
When should the gains carry over to terminal-only problems? This is perhaps the most interesting finding. The long-horizon results (Tables 12–13) suggest that MA-BED helps as the horizon grows, but they are preliminary. Could the authors strengthen this conclusion, for example with more settings, and an analysis of when and why the gains appear?
3 comments
Question: When multistep supervision improves terminal performance?
When should the gains carry over to terminal-only problems? This is perhaps the most interesting finding. The long-horizon results (Tables 12–13) suggest that MA-BED helps as the horizon grows, but they are preliminary. Could the authors strengthen this conclusion, for example with more settings, and an analysis of when and why the gains appear?
We thank the reviewer for this question. Intermediate losses help terminal training when (i) long-horizon credit assignment makes terminal-only training optimization-limited, which becomes more likely as grows, and (ii) intermediate losses provide sufficiently aligned, low-noise gradients. We highlight that Appendix C already provides significant theoretical investigation of these mechanisms, characterizing how intermediate losses introduce shorter credit-assignment paths, when their gradients remain aligned with the terminal objective and thus provide descent directions (Proposition 7), and under which conditions they improve the gradient signal-to-noise ratio (Proposition 9). We also provide further empirical and theoretical insights below that go beyond the original submission, that we have incorporated in the revised manuscript.
Explaining the gains through gradient diagnostics. We monitor these diagnostics on location finding () to illustrate the theoretical conditions established in Appendix C (the gradient-diagnostics table below). The terminal descent factor measures whether the multistep gradient remains a descent direction for the terminal objective (if ), while the relative SNR measures the improvement in signal-to-noise ratio along the terminal-gradient direction (if relative SNR ). Both conditions hold throughout the first 75% of training, during which MA-BED converges faster to better terminal performance than Action-BED. At the final checkpoint, when MA-BED has already converged, and the SNR ratio falls to , while MA-BED’s MSE slightly increases from to . Beyond illustrating our theoretical analysis, these diagnostics offer a practical way to assess intermediate loss choices and tune auxiliary supervision during training to improve terminal performance.
Empirical evidence. The horizon-dependent gains observed in Tables 12–13 are further strengthened by our new five-seed experiments (see Table C in our response to Reviewer GDjv), which confirm the robustness of MA-BED’s terminal improvements and suggest greater training stability, with less variability across runs and fewer poor optimization outcomes. Complementing this, our new double-pendulum experiments (Table B, Reviewer GDjv) show that MA-BED also improves terminal MSE on a distinct dynamical task ( vs. ). Finally, our new weighting ablation (Table A, Reviewer GDjv) further shows that terminal performance depends on balancing intermediate and late-horizon supervision: achieves the best terminal MSE (), compared with for Action-BED and for uniform weighting. Smaller provides insufficient auxiliary supervision, whereas larger places insufficient emphasis on later horizons. This identifies the weighting schedule as a tunable hyperparameter for improving terminal performance.
Further theoretical characterization. Assuming the terminal objective is -smooth, we derive a sufficient condition under which an exact multistep gradient update yields strictly lower terminal loss than a terminal-only update, starting from the same parameters with learning rate and auxiliary weight :
L\eta\cdot ||g_T|| \cdot ||g_A|| +\frac{L\eta\beta}{2}||g_A||^2.$$ Here, $g_T$ and $g_A$ denote the terminal and auxiliary gradients with respect to the design-policy parameters, holding the downstream policy fixed. This extends Proposition 7's first-order descent condition to a finite-step comparison of exact gradient updates. The formal statement and proof will be added to Appendix C.4. --- ### Table H - Gradient diagnostics (Location finding $T=30$). Terminal MSE in $10^{-3}$. $M>0$ indicates a terminal descent direction; relative SNR $>1$ indicates increased terminal-direction SNR. |Training progress|Action-BED MSE|MA-BED MSE|Descent factor M|Relative SNR| |---|---|---|---|---| |10%|17.23|12.26|8.18|3.44| |25%|12.71|8.81|1.82|6.55| |50%|9.59|7.44|2.46|2.17| |75%|9.44|7.01|1.05|5.30| |100%|8.32|7.13|-0.24|0.58| $M=1+\beta\langle g_T,g_A\rangle/\|g_T\|^2$; relative SNR $=\mathrm{SNR}_\beta/\mathrm{SNR}_T$, measured along the terminal-gradient direction.Thank you for the additional experiments that resolves my weakness concern. I keep my rating.
We thank the reviewer for their constructive suggestions. In particular, we have added a new random-endpoint ablation to disentangle the benefits of objective matching from those of dense intermediate supervision. We also provide additional empirical and theoretical evidence clarifying when MA-BED can improve terminal performance. We address each concern separately below.
Weakness: Objective matching versus dense supervision
We thank the reviewer for this suggestion. We added exactly this baseline and found that while it does improve over Action-BED, it does not do as well as MA-BED and our conclusions still hold.
Specifically, we trained Action-BED on one endpoint per trajectory, sampled from the deployment-horizon distribution, with architectures, horizon distribution and training budget matched to MA-BED (Table G in our response to Reviewer ekp7; full details there, Weakness 3). Since both methods optimize the same objective in expectation, this removes the objective-mismatch advantage and isolates the effect of dense intermediate supervision, which we find remains beneficial. In particular, MA-BED achieves lower mean random-horizon error across all evaluated settings, with the clearest additional gains for d=2 and longer horizons, although differences are small for d=5, T∈30,50.
We note also that MA-BED frequently achieves better terminal performance than random-endpoint training. For instance, at d=2, T=30, random-endpoint training recovers most of the RH-MSE improvement but substantially worsens terminal MSE (12.72 vs. 7.13 for MA-BED). These findings are consistent with the lower gradient variance guaranteed by MA-BED's Rao–Blackwellised estimator. We will incorporate this ablation into the revised manuscript.
Finally, we reiterate that MA-BED often outperforms Action-BED on the terminal loss as well, suggesting that multistep supervision can improve terminal performance even when the training objective differs from the evaluation objective.