acceptodds
Under review as a conference paper at ICLR 2027

Bellman Recursion Diffusion for Off-Policy Evaluation

Abstract

Off-policy evaluation (OPE) estimates the value of an evaluation policy from data collected by a different behavior policy. A generative model of the policy's discounted future answers value queries without sequential rollouts, but training one by bootstrapping on its own predictions amplifies fitting error by up to the effective horizon . We introduce Bellman Recursion Diffusion (\method), which trains a conditional diffusion model on the discounted distribution of future transitions of a frozen one-step dynamics model. Each training target lets the frozen model simulate the first transitions and bootstraps only the remaining mass , which reduces the amplification to . We prove that every probability-valued Bellman target buys contraction at the same rate per model call, so depth decides only where the horizon is paid for; that the Bellman structure holds at every noise level, so denoising score matching fits it; and that the estimator obeys an OPE bound separating generative from dynamics error. On five control benchmarks, deeper blocks bring the generator closer to the frozen model, and on Walker2d one-step bootstrapping remains far from it even after fifty times more sweeps. A released one-step occupancy model, TD-flow, has higher log-RMSE than \method on all five tasks and lower rank correlation on four. \method is more accurate on both metrics than model-based rollout on all four horizon-matched tasks, and than STITCH-OPE on three of them.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, off policy evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_6gE5OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes a type of model-based off policy evaluation that learns a model of future rollouts, including both the policy prediction and the state-dynamics, using a diffusion model to directly learn the distribution of (s_t, a_t, s_t+1) at some future t.

Critical strengths

None

Critical weaknesses
  • Many pieces appear to be AI-generated in writing (e.g., mentioning specific seeds and their results).
  • Writing is quite difficult to follow, and clearly below the level of a reasonable paper that merits acceptance.
Questions for authors

None

Reviewer_XrEgOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes Bellman Recursion Diffusion (BRD), a method for off-policy evaluation (OPE) that learns a policy's discounted transition occupancy under a frozen transition dynamics model fitted to offline data. The method iteratively trains a diffusion model, whose targets combine simulations of the frozen transition model with a bootstrapped tail from the diffusion model of the previous iteration. The authors derive bounds separating generator and dynamics errors and report better OPE performance than external baselines on different control tasks.

Critical strengths
  1. The paper identifies how fitting errors propagate through generative bootstrapping and distinguishes them from errors inherited from the dynamics model.
  2. The multi-step Bellman construction is interesting and resembles the principle behind other multistep methods in RL, e.g. n-step returns. A longer simulated prefix reduces reliance on the bootstrapped estimate. It offers an interpretable way to control error propagation during generator training.
Critical weaknesses

The benefit over directly using the fitted dynamics. BRD introduces another approximation while retaining the model's dynamics bias. In Section 4 of the paper, the MBR baseline uses a separately fitted model. Its poorer results therefore do not establish an advantage over rolling out the model already available to BRD. The paper misses a comparison of OPE accuracy of BRD directly against an MBR baseline, where both methods share the same transition model. This comparison is central to deciding whether the additional generator is worthwhile. It remains also unclear when the extra training pays off. BRD requires a separate generator for each evaluation policy, but the paper does not show why this extra training is worthwhile. A concrete example of reusing the generator for repeated evaluations of the same policy would help, together with an estimate of how many evaluations are needed for the savings to offset the training cost.

Sensitivity to hyperparameters.
While Table 1 compares two block depths across all tasks, a more extensive OPE sweep over mm is limited to Walker2d (Table 16). Also, the paper does not examine sensitivity to the target-pool size NN or how mm, QQ and NN interact with each other and with dynamics-model error in terms of OPE performance.

Clarity (i) Experimental reporting: Section 4 is difficult to follow because important details are scattered across the main text and appendix. Each experiment should first state its question, setup, reference values and metrics, then present the results and explain what they establish. Specific examples include:

  • Tables 1–2 do not consistently report the number of training seeds.
  • Figure 2(a) needs a precise definition of the distributions being compared and how the sliced Wasserstein distance is computed.
  • The main text does not explain how P^\hat P and r^\hat r are fitted or how these choices relate to existing model-based methods.
  • “Changing only the Bellman target” combines results from Table 1 with additional experiments discussed in the appendix. The numbers in lines 373–376 are difficult to trace. Direct table references and explicit identification of the configuration behind each number would help.

(ii): Connections between the method's components. The paper would benefit from a more coherent explanation of the training procedure. A short overview should introduce the training roots and the frozen predecessor model before presenting the target construction. It should then explain how the sampling rule in Equation (9) implements the block Bellman target in Equation (6), and how each training sweep approximates one fixed-point update.

(iii) Attribution of theoretical results. Although related work is cited, the distinction between established results and new contributions remains unclear. For example, Equation (4) closely resembles Equation (2) of [1]. The paper should explicitly identify which results are standard and which are new, particularly among the lemmas and propositions in the appendix.

[1] Michael Janner, Igor Mordatch, and Sergey Levine. γ-Models: Generative Temporal Difference Learning for Infinite-Horizon Prediction. NeurIPS, 2020.

I recommend rejection as the practical benefit for OPE remains insufficiently established and the presentation makes important comparisons difficult to assess.

Questions for authors
  • How are P^\hat P and r^\hat r fitted? Which modeling and training choices differ from the MBR baseline and established model-based OPE implementations?
  • Using identical dynamics and reward models, how does BRD compare with direct rollouts of its frozen model in log-RMSE, Spearman’s rank correlation, and total computational cost?
  • What is the intuition behind sampling training roots from a mixture of βm⋆\beta_m^\star and d0πed_0^{\pi_e}? Appendix D.11 compares the mixture with sampling from d0πed_0^{\pi_e} alone. What advantage does the mixture offer over sampling only from βm⋆\beta_m^\star?
  • Have you tested your method within existing policy optimization algorithms, for example for advantage estimation in PPO? If not, how might BRD be used in this setting? Could restricting policy updates to a trust region allow the generator to be reused or updated efficiently as the policy changes?
Reviewer_X5aBOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper addresses off-policy evaluation: estimating the expected return of a target policy using data collected under a different behavior policy. The authors propose Bellman Recursion Diffusion (BRD), which trains a conditional diffusion model to represent the discounted distribution of future transitions under the evaluation policy. The main idea is a multi-step Bellman recursion, which the authors prove theoretical results. Experiments on five control benchmarks demonstrate that increasing the recursion depth generally improves the diffusion model's approximation to the frozen dynamics model's future occupancy distribution, and BRD performs competitively against existing OPE estimators.

Critical strengths

The paper identifies an important problem with generative temporal-difference learning: approximation errors accumulate through repeated bootstrapping.

Theorem 3.2 establishes a contraction–work relationship:

1−κ(v)=(1−γ)cost⁡(v).1-\kappa(v)=(1-\gamma)\operatorname{cost}(v).

Within the paper's class of probability-valued combinations of Bellman powers and its model-call cost definition, this characterizes the trade-off between computational work and contraction.

The authors isolate the effects of recursion depth while holding the learned dynamics model fixed. They also examine occupancy-distribution fidelity, value prediction error, and policy-ranking accuracy, rather than relying exclusively on a single OPE metric.

Critical weaknesses

The theoretical improvement addresses generative approximation error rather than the fundamental statistical error of OPE. It remains unclear how important the former is relative to the latter in realistic offline RL settings.

The theoretical advantage comes at an increased training cost. The paper even acknowledges that it does not establish an end-to-end computational speedup.

Experimental improvements are modest.

Questions for authors

See weakness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.