acceptodds
Under review as a conference paper at ICLR 2027

The Value of Mechanistic Priors in Sequential Decision Making

Abstract

Hybrid mechanistic models, physical priors with learned residuals, promise to reduce the data required for good decisions, but have no computable criterion to test this. We characterize the value of mechanistic priors in sequential decision-making within both asymptotic and burn-in regimes. To formalize this, we introduce the mechanistic information of a model: the mutual information between the model's recommended policy and the true optimal policy , bounded via a centered, occupancy-weighted bias. In the asymptotic regime (large ), matched bounds reveal that Bayesian regret scales with the residual entropy , delivering a theoretical sample complexity reduction of compared to an uninformed baseline. We further provide a computable pre-trial model certificate. Complementarily, in the clinically relevant burn-in regime (small ), we establish a lower bound on the penalty incurred by confidently wrong priors. We demonstrate both the asymptotic and burn-in bounds on an illustrative in-silico 5-fluorouracil (5-FU) chemotherapy plant whose structure follows published FOLFOX pharmacokinetics. The hybrid prior reduces cumulative regret by relative to standard body-surface-area dosing and relative to an uninformed learner, and remains below both under every calibration bias tested. Finally, we show that priors sensitive to distribution shift can lose half of their mechanistic information under a small distribution shift, motivating physically grounded priors for safety-critical applications.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Multi armed bandits, Mechanistic priors, Bayesian regret, Thompson sampling

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_anVROpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper analyses when mechanistic priors in multi armed bandits can improve decision making. The authors expose the fact that there is no offline criterion which can certify incorporating such prior models will help decision making, and therefore propose a criterion which they call the mechanistic information (the mutual information between the optimal and recommended policy). The paper proceeds by proposing a (supposedly) offline-computable pre-trial model certificate. Finally, the paper analyzes low sample regimes and derives bounds, as well as the asymptotic regime. Their approach is validated on synthetic data.

Critical strengths
  • The paper aims to fill a gap that is highly relevant. I truly believe that the proposed research gap is highly valuable.
  • The paper is self-contained, introduces the necessary notation, assumptions, and provides all the proofs (I did not check them in detail, but they appear to be sound).
Critical weaknesses
  • The title and motivation are somewhat misleading. The paper is called "the value of mechanistic priors in sequential decision making", and the motivation and introduction mirror this motivation. However, the work the proceeds by only covering the static bandit setting. There is nothing wrong with proposing a new theory for the easier bandit setting, but the paper should be titled and framed accordingly.

  • The proposed certificate assumes oracle information, and is therefore generally not usable in practice. This is my main concern: Deriving an RmechR_{mech} as the mutual information between optimal and proposed policy is fine, but building on this as the fundation for everything that follows (e.g., HmechH_{mech}) is problematic. The main motivation is to propose a certificate that is computable offline and can guide the deployment of mechanistic priors. However, this leads to a large mismatch between theory and practice. There is no way to obtain these quantities from a mechanistic prior in general, which makes the proposed approach, generally, inapplicable in practice.

  • This connects directly to the second weakness: the proposed method is only validated on purely synthetic data (likely because the optimal policy, and therefore the quantities RmechR_{mech} and HmechH_{mech} can be obtained in closed form. This is, unfortunately, insufficient for validation, and strengthens my second concern above.

  • The proposed theory is a basic adaptation of existing work on quantifying regret and information tradeoff (Shufaro et al., 2025) and Bayesian regret upper bounds for Thompson sampling (Russo & Van Roy, 2016). To be fair with the authors, the second one is acknowledged by the authors. Still, this limits the novelty of the proposed theory.


Shufaro et al., 2025. On Bits and Bandits: Quantifying the Regret-Information Trade-off. In ICLR, 2025.

Russo & Van Roy. An Information-Theoretic Analysis of Thompson Sampling In JMLR, 2016.

Questions for authors
  • Can the authors explain how the relevant quantities can be computed when there is no oracle information available?

  • Can the authors explain how their approach translates to sequential decision making?

Reviewer_khfiOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies how to evaluate a mechanistic model (known dynamics plus learned residual) as a prior for sequential decision making, before any run trial. The authors argue that held-out prediction error is not the criterion, since a shared offset or an error in a region no policy visits does not change the recommendation. They reduce continuous control to a K-armed policy bandit, define mechanistic information Rmech=I(π∗;π^)R_{\text{mech}} = I(\pi^*; \hat\pi), and bound it by a Gaussian-channel capacity C(Bμ)C(B_\mu) through a centered, occupancy-weighted bias BμB_\mu. Bayesian regret is bounded above and below in terms of the residual entropy HmechH_{\text{mech}} (tight up to log⁡K\sqrt{\log K}), giving a sample-complexity ratio Θ(H(μ)/Hmech)\Theta(H(\mu)/H_{\text{mech}}), and a rule-out certificate Bμcrit(s)B_\mu^{\text{crit}}(s) is derived that any model must satisfy to save a factor ss. A burn-in bound prices confidently wrong priors, and a shift result shows mechanistic information can halve under a small KL shift. On an in-silico 5-FU dosing plant, the hybrid prior roughly halves cumulative regret relative to standard dosing and to an uninformed learner, and stays below both under every calibration bias; injecting the model once as a prior degrades gracefully, while re-solving it every cycle re-asserts the drift and ends worse than standard care.

Critical strengths
  1. The problem is well motivated, treating the hybrid prior as constructed rather than assumed is a reasonable step beyond previous work like Russo & Van Roy (2016) and Shufaro et al. (2025).

  2. The experiments are transparent: constants are labeled by provenance, cohort separation is checkable, and the authors report that their theoretical floor is exceeded everywhere rather than hiding the gap.

  3. The empirical finding that a prior injected once degrades gracefully while a per-cycle controller does not is useful for system design.

Critical weaknesses
  1. The hybrid's empirical gain is partly confounded with offline structural knowledge. It receives a prior table P(π∗∣π^)P(\pi^* \mid \hat\pi) estimated against known optimal arms, and a shape matrix QQ built from counterfactual outcomes at all arms. Neither is available to a practitioner, and neither is given to the baselines. An ablation (e.g. TS with a covariate-based prior from the same cohort, or a scrambled π^→μhyb\hat\pi \to \mu_{\text{hyb}} mapping) would clarify how much comes from the ODE itself.

  2. Proposition 1 assumes M^=h−1(Z)\hat{\mathcal{M}} = h^{-1}(Z), which attributes all model error to measurement noise and excludes structural misspecification. The main experiment, however, uses a misspecified model class. A relaxed version with an explicit misspecification term would make the theory match its use.

  3. The theoretical ratio H(μ)/Hmech\sqrt{H(\mu)/H_{\text{mech}}} is loose and insensitive at the measured operating points: predicted floor about 1.12 against observed roughly 1.85. The theory therefore does not quantitatively predict the empirical gain.

  4. Narrow empirical scope: While the paper frames its theory broadly for scientific machine learning and continuous control, the empirical evaluation is restricted to a single in-silico 5-FU dosing task. Moreover, this setting is mathematically degenerate: it reduces to a 1D scalar parameter identification problem (p=1p=1 clearance parameter, dZ=1d_Z=1 blood draw) with static, cycle-constant scalar actions. Evaluating a second, structurally distinct physical domain—especially one with multivariate parameters (p>dZp > d_Z) or time-varying control trajectories—is needed to support the broad generality claimed in the introduction.

Questions for authors
  1. How much of the reported gain survives if the ODE is removed and only cohort-derived information is used?

  2. Can Proposition 1 be relaxed to M^∉supp(ρ)\hat{\mathcal{M}} \notin \mathrm{supp}(\rho), and can BμB_\mu be decomposed into measurement error and structural error?

  3. The certificate is one-sided and needs the occupancy μ\mu, so outside simulation it requires a reference model. The "computable today" examples in the paper assume uniform occupancy, and Remark 1 concedes the one-cycle reading falls below the resolution of the bounds. Does the rule-out verdict change under a concentrated rather than uniform occupancy?

  4. Can you construct a model that fails the one-cycle screen and confirm that the hybrid then fails to save a cycle?

Reviewer_t6R1OpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper asks when a hybrid mechanistic model (physical prior combined with learned residuals) reduces the number of interactions needed to make good sequential decisions. It argues that commonly used held-out predictive accuracy is the wrong measure and instead treats the mechanistic model as a prior for a bandit learner. The control problem is reduced to a KK-armed policy bandit, and the model's value is quantified by its mechanistic information RmechR_{\mathrm{mech}}, the mutual information between the optimal policy and the policy the calibrated model recommends before any interaction. Building on Russo & Van Roy (2016) and Shufaro et al. (2025), the authors derive Bayesian regret bounds. From these they obtain a critical bias BμcritB_\mu^{\mathrm{crit}}: a threshold on a decision-relevant, occupancy-weighted model. For the finite-horizon regime, they lower-bound the burn-in cost of a confident, but wrong prior and show that population-fitted priors can lose information under distribution shift. The framework is illustrated throughout with adaptive 5-FU chemotherapy dosing from Tan et al (2026). In a simulation study, Thompson sampling with a one-compartment surrogate prior reduces regret significantly to both body-surface-area dosing and an uninformed learner and remains robust across calibration biases.

Critical strengths

S1 Well-written, with a strong running example. The 5-FU dosing example grounds the framework in a clinically meaningful problem and is used consistently throughout. The deployed surrogate (one-compartment) is a simplification of a two-compartment Michaelis–Menten population model. Throughput the authors discuss the Theorems or Assumptions in light of the running example, which is helpful. This makes the contribution legible and its relevance to medical applications convincing. 

S2 The paper makes a clear and important point: held-out predictive error is neither necessary nor sufficient for a model to be useful in sequential decision-making. It connects hybrid modeling to information-theoretic regret analysis by measuring the mutual information between the optimal policy and the recommended policy and by restricting the model errors that count to decision-relevant ones rather than general predictive performance.

Critical weaknesses

W1 Too few figures (for a paper with a concrete application). The empirical support is mostly reported as single numbers in the running examples. Figures showing the regret bounds against empirical regret over time for the 5-FU example (whether, and how closely, they are obeyed or attained) or for the theoretical results showing how regret scales with the residual entropy HmechH_{\mathrm{mech}} , would make the paper more readable and allows to judge how informative the bounds are at least in the simulated setting.

W2 "pre-trial" seems like an overclaim

  • The abstract and introduction present the critical bias as a computable certificate that can be evaluated before any interaction. The Limitations section then partly retract this (L473): BμB_\mu "needs per-arm rewards no clinical dataset contains, so outside simulation it needs a reference model to supply them." This seems to contradict the pre-trial claim.
  • Instead, it seems, a "trusted" simulator of the true dynamics is needed. Thus the papers should make the assumptions/conditions for the pre-trial certificate to be computable more clear.
  • Ensure claims about pre-trial computability are aligned with the Limitations throughout.

W3 Insufficient positioning in the literature. The methodological contribution rests on essentially one line of prior work (Shufaro et al., 2025), while most remaining citations concern the running example. The related-work section is very brief and largely repeats references already used in the main text. Relevant areas might include Thompson sampling under misspecified priors, work discussing how to transfer a population-level prior to an individual, and model-informed precision dosing in pharmacometrics. A short search (without knowing too much about the topic) lead to these seemlingly relevant papers:

  • Bayesian decision-making under misspecified priors with applications to meta-learning (Simchowitz et al, 2021)
  • On the Prior Sensitivity of Thompson Sampling (Liu & Liu, 2016)
  • Meta-Thompson Sampling (Kveton et al, 2021)
Questions for authors

Clarifications

  • Theorem 1 holds for some prior μ\mu. Does a lower bound of the same form hold for the specific prior induced by ρ\rho? If not, how should Theorem 3 and Running Examples 3–4 be interpreted? How is μ\mu chosen while RmechR_{\mathrm{mech}}, which depends on μ\mu, is held fixed?
  • Assumption 3 seems pretty strong. Could you explain to me to what extend this is reasonable?
  • Reweard model in the setup is explained to have Gaussian noise, but then the example
  • L187: The reward model in the setup is introduced with Gaussian noise. In this light, why is the reward ∈{0,1}\in\{0,1\} (Bernoulli noise) and not e.g. distance to the therapeutic window with Gaussian noise in order to closely match the setup in L107
  • L212: What is PK?
  • L323: In Running Example 5, the burn-in bound corresponds to ≈30\approx 30 suboptimal plays, exceeding the N=12N=12 cycles. Does that mean, a confident, but wrong prior cannot be overturned within the dosage cycles?
  • L348: Why do the LLM priors come in here?
  • Running Example 7 is unclear to me, what is a simulated plant?

Minor Comments

  • Figure 1 is placed too far from its first reference and is too small to read. Its parameters and symbols should be explained in the caption, so the figure can be understood on its own.
  • L19: μ\mu is not defined, what does it mean?
  • in the Introduction: "action," "decision" and "interaction" should be related explicitly; "interaction" is never defined. 
  • Consider moving the introduction of the 5-FU example further up, give some context and intuitive explanation why it is a suitable example. (Lot's of information becomes much clearer while reading the next sections, but people might stop reading here, when not yet intrigued by the work)
    • L168: the 5-FU model is mentioned before it is introduced. - L41–47: unclear how actions relate to predictive accuracy; a small toy example carried throughout would help. The last sentence is hard to parse.
  • L103: l\mathcal{l} is not defined
  • Running Example 1: add context for non-specialists (5-FU is a chemotherapy drug given as 46-hour infusion for colorectal cancer), justify why the two- vs. one-compartment pairing is realistic, and show the sequential structure (one round per cycle).
  • The link of mechanistic models and LLM priors is not clear, but get quite some appearance. Can you elaborate why?
  • I appreciate the honest discussion of limitations.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.