Beyond Terminal Objectives: Bayesian Experimental Design with Multistep Actions and Losses

Existing loss-driven Bayesian experimental design (BED) methods learn policies that target a terminal loss evaluated at a fixed end point. We argue that this terminal formulation is ill-suited to cases where we can take intermediary actions or our experiment endpoint is unknown. To address this, we introduce MA-BED (Multistep Action-BED), which extends the loss-driven BED formulation to include intermediary actions and losses throughout the acquisition trajectory. This assignment of value to incomplete trajectories allows us to encourage the data to be as informative as possible at every stage of the experiment, such that we are encouraged to learn quickly. In turn, this allows us to support problems where separate actions must be taken during acquisition, and problems with unknown, stochastic, or adaptively chosen experiment endpoints. The intermediate signal additionally shortens credit assignment, stabilizing optimization over long rollouts. Indeed, because of the practical training benefits this provides, we find MA-BED not only improves intermediate decisions compared with terminal-loss baselines on target tracking, adaptive stopping, and classical design problems, it can also improve the terminal decisions for longer rollouts.