The benefit over directly using the fitted dynamics.
BRD introduces another approximation while retaining the model's dynamics bias. In Section 4 of the paper, the MBR baseline uses a separately fitted model. Its poorer results therefore do not establish an advantage over rolling out the model already available to BRD. The paper misses a comparison of OPE accuracy of BRD directly against an MBR baseline, where both methods share the same transition model. This comparison is central to deciding whether the additional generator is worthwhile. It remains also unclear when the extra training pays off. BRD requires a separate generator for each evaluation policy, but the paper does not show why this extra training is worthwhile. A concrete example of reusing the generator for repeated evaluations of the same policy would help, together with an estimate of how many evaluations are needed for the savings to offset the training cost.
Sensitivity to hyperparameters.
While Table 1 compares two block depths across all tasks, a more extensive OPE sweep over m is limited to Walker2d (Table 16). Also, the paper does not examine sensitivity to the target-pool size N or how m, Q and N interact with each other and with dynamics-model error in terms of OPE performance.
Clarity
(i) Experimental reporting: Section 4 is difficult to follow because important details are scattered across the main text and appendix. Each experiment should first state its question, setup, reference values and metrics, then present the results and explain what they establish. Specific examples include:
- Tables 1–2 do not consistently report the number of training seeds.
- Figure 2(a) needs a precise definition of the distributions being compared and how the sliced Wasserstein distance is computed.
- The main text does not explain how P^ and r^ are fitted or how these choices relate to existing model-based methods.
- “Changing only the Bellman target” combines results from Table 1 with additional experiments discussed in the appendix. The numbers in lines 373–376 are difficult to trace. Direct table references and explicit identification of the configuration behind each number would help.
(ii): Connections between the method's components. The paper would benefit from a more coherent explanation of the training procedure. A short overview should introduce the training roots and the frozen predecessor model before presenting the target construction. It should then explain how the sampling rule in Equation (9) implements the block Bellman target in Equation (6), and how each training sweep approximates one fixed-point update.
(iii) Attribution of theoretical results. Although related work is cited, the distinction between established results and new contributions remains unclear. For example, Equation (4) closely resembles Equation (2) of [1]. The paper should explicitly identify which results are standard and which are new, particularly among the lemmas and propositions in the appendix.
[1] Michael Janner, Igor Mordatch, and Sergey Levine. γ-Models: Generative Temporal Difference Learning for Infinite-Horizon Prediction. NeurIPS, 2020.
I recommend rejection as the practical benefit for OPE remains insufficiently established and the presentation makes important comparisons difficult to assess.