- Rating
- 2
- Confidence
- 2(−1)
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies offline-to-online reinforcement learning with a fixed generative reference policy. A critic-induced Gibbs tilt is represented through a density martingale whose logarithm satisfies a quadratic BSDE. Its initial value supplies the regularized Bellman continuation, while its martingale integrand determines the policy drift. VBCDP estimates these quantities using scalar critic evaluations. Validation includes two controlled one-dimensional problems and six DeepMind Control Suite tasks.
Critical strengths
1.The formulation provides a coherent connection between generative policy improvement and critic continuation. Although the probabilistic ingredients are classical, their integration into a fixed-reference actor–critic construction is useful. Theorem 1 and Proposition 2 clearly identify the underlying regularized control objective. 2.The controlled experiments check values, drifts, and endpoint distributions, including a bimodal target. The five-seed RL results also show consistent improvement over matched frozen references across all six tasks. Together, these provide meaningful evidence that scalar critic evaluations can guide the proposed policy adaptation.
Critical weaknesses
1.The baseline comparisons do not isolate the algorithmic contribution. VBCDP benefits from a pretrained reference and 50% offline data in replay, whereas Appendix C.4 identifies the comparison as a from-scratch benchmark. The offline dataset’s source, size, and collection budget are not specified, and offline preparation is excluded from the cost accounting. Equal online interaction budgets therefore do not establish comparable resources. Improvement over a frozen reference alone does not establish an advantage over alternative adaptation methods. 2.The practical solver’s approximation quality remains insufficiently validated. Exact identification holds at the continuous-time population level, while the RL implementation uses 8–16 paths per state and 5–40 Euler steps. Appendix B.5 acknowledges downward bias in the log-moment estimate, which directly affects critic targets, but does not quantify its magnitude. The one-dimensional checks and temperature ablation do not establish how sensitive the RL results are to path count or discretization.
Questions for authors
1.Could you clarify the offline dataset’s provenance, size, and collection budget, and provide a competitive adaptation baseline with comparable offline data, pretraining opportunities, and online interactions? 2.How do soft-value estimation error and final return change as the path count and Euler resolution increase? A controlled comparison against a high-sample value estimate would help assess whether these approximations materially affect the proposed mechanism.