acceptodds
Under review as a conference paper at ICLR 2027

Reinforcement Learning by Change of Measure: A BSDE Theory for Generative Policies

Abstract

Online reinforcement learning with generative policies requires translating scalar critic values into changes of the multi-step stochastic dynamics that generate actions. Existing online generative-policy methods typically focus on actor-side objectives based on value reweighting, importance sampling, or critic action gradients. We seek a unified formulation that also identifies the soft continuation value used in the Bellman update. We introduce a unified change-of-measure formulation of actor-critic updates for online RL with generative policies, in which an offline generative policy defines a reference process and critic-induced improvement is recast as a controlled change of its path law toward the desired Gibbs endpoint distribution. We prove that this change of measure induces a positive martingale whose logarithm satisfies a quadratic backward stochastic differential equation (QBSDE), identifying the soft value and the Doob-Girsanov drift correction as two components of the same probabilistic transformation. This equivalence converts Boltzmann policy improvement into a QBSDE estimation problem under the reference process. Building on this principle, we develop Value-Only BSDE-Controlled Diffusion Policies (VBCDP), which learns the value and policy drift directly from scalar critic evaluations without requiring critic action gradients. Controlled experiments recover analytic Gibbs–Doob solutions, while results on six DeepMind Control Suite tasks show consistent improvement over matched frozen references, with mean gains of -, and competitive performance against modern generative-policy baselines.

Primary area:
learning theory
Keywords:
Diffusion Policies, BSDE Diffusion

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_PLWaOpenReview ↗
Rating
2
Confidence
2(−1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies offline-to-online reinforcement learning with a fixed generative reference policy. A critic-induced Gibbs tilt is represented through a density martingale whose logarithm satisfies a quadratic BSDE. Its initial value supplies the regularized Bellman continuation, while its martingale integrand determines the policy drift. VBCDP estimates these quantities using scalar critic evaluations. Validation includes two controlled one-dimensional problems and six DeepMind Control Suite tasks.

Critical strengths

1.The formulation provides a coherent connection between generative policy improvement and critic continuation. Although the probabilistic ingredients are classical, their integration into a fixed-reference actor–critic construction is useful. Theorem 1 and Proposition 2 clearly identify the underlying regularized control objective. 2.The controlled experiments check values, drifts, and endpoint distributions, including a bimodal target. The five-seed RL results also show consistent improvement over matched frozen references across all six tasks. Together, these provide meaningful evidence that scalar critic evaluations can guide the proposed policy adaptation.

Critical weaknesses

1.The baseline comparisons do not isolate the algorithmic contribution. VBCDP benefits from a pretrained reference and 50% offline data in replay, whereas Appendix C.4 identifies the comparison as a from-scratch benchmark. The offline dataset’s source, size, and collection budget are not specified, and offline preparation is excluded from the cost accounting. Equal online interaction budgets therefore do not establish comparable resources. Improvement over a frozen reference alone does not establish an advantage over alternative adaptation methods. 2.The practical solver’s approximation quality remains insufficiently validated. Exact identification holds at the continuous-time population level, while the RL implementation uses 8–16 paths per state and 5–40 Euler steps. Appendix B.5 acknowledges downward bias in the log-moment estimate, which directly affects critic targets, but does not quantify its magnitude. The one-dimensional checks and temperature ablation do not establish how sensitive the RL results are to path count or discretization.

Questions for authors

1.Could you clarify the offline dataset’s provenance, size, and collection budget, and provide a competitive adaptation baseline with comparable offline data, pretraining opportunities, and online interactions? 2.How do soft-value estimation error and final return change as the path count and Euler resolution increase? A controlled comparison against a high-sample value estimate would help assess whether these approximations materially affect the proposed mechanism.

Reviewer_N2K3OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper tries to provide a framework that connects traditional concepts and methods in stochastic control to generative policy optimization in RL (reinforcement learning), focusing on connections between endpoint distributions, path-space measures, and controlled trajectories.

Critical strengths

Not sure whether there's any. The central theoretical results, in particular Theorem 1 and Corollary 1, appear to be direct specializations of established results in stochastic control. For instance, Nüsken and Richter (2021) has already established closely related equivalences.

Critical weaknesses

The paper's writing appears to rely heavily on AI tools, fully of excessive jargons and inflated technical terms. Just a couple of examples from the first two pages:

"... A reference-relative Gibbs endpoint tilt is lifted to the reference path space. The positive conditional-exponential martingale propagates the terminal likelihood ratio backward in time ..."

"Our contributions progress from a new mechanism, through theory and computation, to empirical validation. First, ... a Gibbs endpoint tilt induces a positive martingale whose logarithm satisfies a QBSDE .... Second, we establish the endpoint–path–QBSDE–control equivalence, prove exact Gibbs–Doob identification by the terminal-residual objective, and develop a decoupled value-only solver under the simulable reference law. Third, controlled experiments recover ..., while online experiments show ...."

Questions for authors
  1. Algorithmic presentation (Lines 212–269, 783–809): What the paper lacks is some pseudocode that characterizes and illustrates its core algorithmic implementation.
  2. Table 1 (Lines 271–276) needs a clearer explanation of what each statistic aims to demonstrate and whether a larger or smaller value indicates better performance.
  3. Table 2 (Lines 325–331, 392–421): All other methods besides VBCDP need clearer explanations in the main text, particularly how they function and differ from one another. It would also be helpful to identify one widely accepted baseline that can serve as a reference point for subsequent comparisons.
  4. Additional experiments (Lines 318–323, 342–361): More variants of G(x) should be tested to provide more convincing evidence.
  5. Interpretation of numerical results (Lines 271–276, 325–331, 349–374): The experimental section contains too many numerical values reported with unnecessary precision, often juxtaposed without sufficient explanation.
  6. Figure 3 (Lines 392–405, 895–917): The authors should clarify how the offline initialization affects the fairness of the comparisons, as their method appears to start from significantly higher points than other methods.
Reviewer_6fEhOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to adapt a pretrained generative policy using scalar critic evaluations during online reinforcement learning. It fixes a reference action distribution, induced by a forward diffusion, and formulates policy improvement as maximizing expected critic value subject to a KL penalty relative to that reference. The resulting target is a reference-weighted Gibbs distribution.

The central construction lifts this endpoint reweighting to a change of measure on diffusion paths. The conditional exponential of the terminal critic value defines a positive martingale; its logarithm satisfies a quadratic BSDE. The initial scalar value gives the reference-relative soft Bellman continuation, while the martingale integrand gives the drift correction that realizes the improved policy. The paper also establishes exact identification through a terminal-residual objective and contraction of the fixed-reference regularized Bellman operator.

The proposed algorithm, Value-Only BSDE-Controlled Diffusion Policies (VBCDP), estimates the soft value using a Monte Carlo log-sum-exp target and learns the controller through a centered terminal-residual loss on reference trajectories. It requires scalar critic evaluations without critic action gradients.

Critical strengths
  1. Provides a coherent connection between the policy improvement objective, generative drift, and Bellman continuation. This paper gives an explicit identification of the scalar continuation value and the diffusion control associated with the same reference-relative optimization problem. The fixed reference makes the regularizer unambiguous across the MDP, and the derivation explains why the value used in the critic backup is compatible with the intended Gibbs actor.
  2. Experiments test the proposed mechanism and show consistent reference-relative improvement.
Critical weaknesses
  1. Insufficient empirical isolation and numerical validation. The online comparisons do not isolate the benefit of VBCDP under comparable offline information, initialization, and total resources. Also the small-sample, discretized online solver may not well approximate the common Gibbs target identified by the theory.
  2. Close connection to prior work that needs justification and clarification. The guidance term is closely related to what is presented in Contrastive Energy Prediction for Exact Energy-Guided Diffusion Sampling in Offline Reinforcement Learning.
Questions for authors
  1. How does the proposed construction relate to QGPO’s conditional log-exponential guidance potential after matching reference dynamics and temperature conventions? Please clarify which part is novel in the theory of this paper. (QGPO is the algorithm from paper Contrastive Energy Prediction for Exact Energy-Guided Diffusion Sampling in Offline Reinforcement Learning).
  2. Were the baselines given the same offline data and comparable pretrained initialization? If not, do VBCDP’s advantages persist when these are matched?
  3. How do you verify that the chosen numbers of reference paths and diffusion steps are sufficient to estimate the soft value accurately and produce a policy close to the target Gibbs distribution?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.