acceptodds
Under review as a conference paper at ICLR 2027

Variance-Reduced Off-Policy Preference Optimization via Regularizer Correction

Abstract

Direct Preference Optimization (DPO) regularizes a learned policy toward a reference policy using an implicit reverse KL divergence. Other convex regularizers are also well motivated for preference optimization but are known to perform worse in practice – especially under off-policy training where the preference data does not come from the policy currently being optimized. Moving beyond the bandit setting to multi-step MDPs, this effect gets exacerbated by the fact that constructing an implicit reward function for convex regularizers in multi-step MDPs requires sampling subsequent actions from the current policy, causing estimation errors to compound along trajectories. We formally characterize the regularizers that do not require subsequent action sampling and prove that the reverse KL is unique in providing a closed-form solution. To address error compounding in convex regularizers other than the reverse KL, we propose a regularity condition that minimizes the variance of the implicit reward estimator and derive a canonical form satisfying this condition by design. Our experimental results on tabular MDPs and for large language model post-training show that the proposed correction consistently improves on alternative approaches to estimating separable convex regularizers.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning from Human Feedback, Direct Preference Optimization

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_vpZaOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper addresses the high variance of implicit reward estimation under non-reverse-KL regularization in long-sequence preference optimization. Similar to GRPO and REINFORCE, it employs a baseline technique to reduce variance while attempting to preserve the expectation of the estimator. Experimental results suggest that the proposed method can improve preference optimization performance in certain settings.

Critical strengths
  1. The paper proposes using a baseline technique to reduce the variance of implicit reward estimation. This is a classical technique, the motivation is reasonable and worth investigating.
  2. The general intuition behind the theoretical analysis is sound: a better unbiased estimator should have lower variance.
  3. The experiments are relatively comprehensive, covering both mathematical reasoning and preference alignment tasks.
Critical weaknesses
  1. The theoretical analysis is potentially misleading. The paper targets off-policy settings, yet many important theoretical analyses are conducted under on-policy assumptions. Consequently, many of the proposed theorems and conclusions do not directly apply to the proposed method and cannot establish its advantages. For example, Theorem 2(ii) argues that the proposed implicit reward estimator has lower variance by analyzing the variance of $ \Psi_a^\mu $ under the distribution $ \pi $. However, the paper focuses on off-policy settings, where samples are not drawn from $ \pi $. Therefore, a variance reduction under $ \pi $ does not necessarily imply a variance reduction in the actual off-policy setting. Policy mismatch is a fundamental issue in off-policy settings and should not be ignored. Similar analyses under distributions that do not match the actual application appear repeatedly in the appendix (e.g., Appendix B.7), making it difficult to assess the validity and practical relevance of many theoretical conclusions.
  2. The characterization of off-policy estimation error is insufficient. The paper characterizes the variance using $ \delta = |\pi-\pi_{\text{ref}}|_1 $. However, $ \delta $ can be large in off-policy settings, and simply comparing asymptotic orders is insufficient to establish the effectiveness of the proposed method.
  3. The practical method is inconsistent with the theoretical analysis. Equation 21 uses the token-level reward $ \frac{f}{u} $. Under off-policy sampling, the expectations of $ \frac{f_c}{u} $ and $ \frac{f}{u} $ are no longer equal, which introduces bias into the loss.
Questions for authors
  1. Which of the theorems presented in the paper actually hold under off-policy settings?
  2. Which theoretical derivations have been manually checked, and which can the authors confidently guarantee are correct?
6 comments
AuthorsOpenReview ↗

Author's response (1/2)

Thank you for your detailed review. Your main concern about policy mismatch in off-policy settings is valid, and we have revised the theoretical analysis accordingly.

Summary

  • W1. Our paper targets off-policy settings, yet the submitted Theorem 2(ii) and Appendix B.7 were analyzed under the current policy. The revised PDF restates both for an explicit behavior policy πb\pi_b and bounds the MSE = variance + bias^2. The advantage of the canonical correction is preserved in off-policy setting, since its MSE is O((1+δb2)δ4)O((1+\delta_b^2)\delta^4), while otherwise is Θ(δ2)\Theta(\delta^2).
  • W2. The analysis should be local, because the correction itself can only modify the first-order term in u−1u - 1.
  • W3. The bias you describe exists and is now bounded in Theorem 2(iii). We intentionally do not use importance sampling, so that the effect of the canonical correction is not mixed with a second correction. Notably, we also added an experiment that measures how the bias, the variance, and the MSE change with the degree of policy mismatch (Figure 5 and Appendix C.6).
  • Q1 and Q2. Every theorem in the revised PDF holds under off-policy settings, and all derivations have been manually checked.

W1. The paper targets off-policy settings, yet many important theoretical analyses are conducted under on-policy assumptions. Theorem 2(ii) analyzes the variance under the current policy, from which samples are not drawn, and similar analyses appear in the appendix (e.g., Appendix B.7). Policy mismatch should not be ignored.

Your point is correct. A variance reduction under the current policy does not by itself imply one in the actual off-policy setting, and the submitted version did not address the mismatch.

In the revised PDF, the logged next action is drawn from a behavior policy πb\pi_b, and we analyze the MSE of the logged-sample estimator against the on-policy target, which decomposes into variance under πb\pi_b and bias^2. With δ=∥πθ−πref∥1\delta = \lVert \pi_\theta - \pi_{\mathrm{ref}} \rVert_1 and mismatch δb=∥πb−πθ∥1\delta_b = \lVert \pi_b - \pi_\theta \rVert_1, Theorem 2 now states the following.

  • (ii) The variance under πb\pi_b is O(δ4)O(\delta^4) for the canonical form and Θ(δ2)\Theta(\delta^2) otherwise, uniformly over πb\pi_b.
  • (iii) The bias is O(δbδ2)O(\delta_b\delta^2) for the canonical form and O(δbδ)O(\delta_b\delta) otherwise.
  • (iv) The MSE is O((1+δb2)δ4)O((1+\delta_b^2)\delta^4) for the canonical form and Θ(δ2)\Theta(\delta^2) otherwise.

In brief, the proof expands the integrand around ua=1u_a = 1 for each action separately, regardless of which policy the action is sampled from. The canonical correction removes the first-order term in u−1u - 1, and we track the effect of this term on the variance and on the bias. Importantly, within the family of objective-preserving corrections, the canonical correction is the unique choice that changes these asymptotic rates (Theorem 2(i)).

Appendix B.7 is revised in the same way, and it now bounds the gradient estimation error through the off-policy MSE instead of the on-policy variance (Proposition 1). We also clarify a notational point that may have added to the confusion. "a′∼μ(⋅∣s′)a' \sim \mu(\cdot \mid s')" at Line 1190 (Line 1334 in the revised PDF) was a typo for a′∼πθ(⋅∣s′)a' \sim \pi_\theta(\cdot \mid s'). In our paper, μ\mu is the correction map of Eq. 12 (Eq. 13 in the revised PDF), and it is not a behavior policy. The revised PDF writes the behavior policy as πb\pi_b throughout.

Again, thank you for your careful feedback. The revised theoretical results make the contribution of the paper clearer. We hope that our response resolves your concerns about the off-policy setting and leads to a positive reevaluation.

W2. The characterization of off-policy estimation error is insufficient. The variance is characterized by its order in the distance to the reference, which can be large in off-policy settings, so comparing asymptotic orders is insufficient to establish the effectiveness of the method.

The statement that the characterization through δ\delta is insufficient seems to come from a misunderstanding of our theoretical analysis. To clarify this point, the goal of our paper structurally requires the analysis around u≈1u \approx 1 (equivalently δ≈0\delta \approx 0), and it is not a simplification that we chose. Our relaxation looks for a correction that leaves the regularizer unchanged, and by Eq. 12 (Eq. 13 in the revised PDF), such a correction may modify only the first-order term in u−1u - 1. A Taylor expansion around u=1u = 1 is therefore unavoidable in order to characterize the effect of the correction. To correct this misunderstanding, the revised PDF restates this reason before Theorem 2 (Lines 315 in the revised PDF).

We hope that this clarifies the misunderstanding, and that the theoretical contribution of our method can be reevaluated. We would be glad to answer any point that remains unresolved.

AuthorsOpenReview ↗

Author's response (2/2)

W3. The practical method is inconsistent with the theoretical analysis. Equation 21 uses the token-level reward computed from the logged token. Under off-policy sampling, its expectation no longer equals the expectation under the current policy, which introduces bias into the loss.

You are right that the two expectations are no longer equal and that this introduces bias. The revised Theorem 2(iii) bounds this bias explicitly, and the canonical correction reduces it from O(δbδ)O(\delta_b \delta) to O(δbδ2)O(\delta_b \delta^2).

To avoid any concern, we note that the absence of importance sampling in Eq. 21 (Eq. 49 in the revised PDF) is an intentional control. The purpose of these experiments is to test whether the canonical correction remains effective in off-policy settings. Importance sampling is a second correction, and using both would make it impossible to distinguish which of the two an improvement comes from.

In addition, we ran an additional experiment that measures how the bias, the variance, and the MSE change as the data become more off-policy (Figure 5, Appendix C.6, and Figures 9 to 11 in the revised PDF). As Theorem 2(iii) predicts, the policy mismatch introduces bias under both forms. Nevertheless, the canonical form consistently achieves a lower MSE than the Amari form, and this advantage is driven primarily by the variance reduction.

Q1. Which of the theorems presented in the paper actually hold under off-policy settings?

In the revised PDF, all of the statements hold under off-policy settings.

  • Lemma 1 and Theorem 1 do not depend on how the next action is sampled. Lemma 1 is an identity for the implicit reward, and Theorem 1 characterizes the regularizers whose inner-term integrand is constant, for which both variance and bias vanish under every πb\pi_b (Line 283 in the revised PDF).
  • Theorem 2(i) (existence and uniqueness of the canonical form) follows directly from the definition of the regularizer and involves no sampling distribution.
  • Theorem 2(ii) to (iv) and Proposition 1 in the revised Appendix B.7 are now stated and proved for every behavior policy πb\pi_b with full support.

Before the revision, Section 3.3 (Theorem 2(ii)) and Appendix B.7 were stated for the on-policy variance, as you pointed out in W1. They are now stated for the off-policy MSE, and the earlier on-policy statements are recovered as the special case πb=πθ\pi_b = \pi_\theta.

Q2. Which theoretical derivations have been manually checked, and which can the authors confidently guarantee are correct?

All derivations in the paper, including the revised Theorem 2 and Appendix B.7, have been manually checked and we are highly confident that they are correct. We would like to clarify that the weaknesses raised above came from stating the analysis for the on-policy variance. They did not come from a flaw in the derivations, and the revision generalizes those statements to πb\pi_b, with the original on-policy results remaining valid as a special case.

Dear Reviewer vpZa,

We hope this message finds you well. Thank you again for your valuable time and careful review of our work.

We have posted our detailed response to your concerns. Following your comments, we revised both the theoretical and experimental parts of the paper. Specifically, we restated Theorem 2 and the related analysis for a behavior policy, so that every theorem now holds in the off-policy setting, and we bounded the bias from policy mismatch. We also added a new experiment that measures how the bias, the variance, and the MSE change with the degree of policy mismatch. As a result, your feedback helped us improve the contribution of the paper significantly, and we appreciate it.

We wanted to gently follow up to see whether our revision has adequately addressed your concerns, or if there are any remaining questions. We remain actively engaged in the discussion period and would be happy to provide any additional clarifications.

Best regards,

The Authors of Submission #8068

Reviewer_uBdaOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

In DPO, Reverse KL empirically surpasses other divergences. this paper identifies the root cause as high variance in implicit reward estimators. Theoretically, only reverse KL supports a zero-variance form free of next-action expectation sampling. The authors propose Canonical Correction, which adds an equivalent corrective term to reduce estimation variance without modifying the original optimization objective. The approach has negligible computational overhead and requires no extra networks or rollouts. Validated on toy MDPs and LLM preference alignment, corrected divergences produce stabler gradients and improved downstream performance.

Critical strengths
  1. The paper decouples regularizer design from implicit reward estimator design and asks whether performance differences between mathematically equivalent optimization objectives arise from estimation variance rather than the objectives themselves. This perspective revisits how divergences have been compared in DPO and suggests that prior experimental conclusions may have been confounded by estimation error.

  2. The two core theorems explain why reverse KL performs well and provide a general recipe for constructing low-variance estimators for other divergences. These derivations lead to Canonical Correction, a lightweight correction term that reduces estimation variance without requiring additional critic networks, reward models, or trajectory rollouts. The method introduces negligible computational overhead and can be readily integrated into existing LLM preference optimization pipelines.

Critical weaknesses

W1. Theorem 2 establishes a local variance guarantee under the current policy, while practical training uses logged actions from a behavior policy. Appendix B.7 assumes zero-mean residuals despite this mismatch, leaving the resulting bias and its impact on MSE insufficiently addressed.

W2. Although Figures 5 and 10 show improved training stability and policy recovery, they do not directly verify whether the variance reduction persists when the policy deviates substantially from the reference.

Questions for authors

Q1. In Appendix B.7, the next action is sampled from the behavior policy, but the residual is centered around an expectation under the current policy. Could the authors clarify the assumptions required for the residual to have zero conditional mean and for its second moment to equal the variance under the current policy? Do these assumptions hold in practical off-policy training with logged data?

Q2. When the behavior policy differs from the current policy, the estimation error may also include bias. Could the authors use the tabular MDP setting, where the target expectation can be computed exactly, to systematically vary the mismatch between the two policies and separately measure the bias, variance, and MSE of the Amari and canonical estimators? This would help establish whether the variance reduction translates into lower overall estimation error.

Q3. Theorem 2 provides a local variance guarantee near the reference policy. Figure 10 evaluates policy recovery across different target-policy peaks, while Figure 5 provides evidence of improved training stability. Could the authors directly measure estimator variance as a function of the actual policy deviation from the reference, to clarify whether the local variance advantage extends to larger policy shifts?

4 comments
AuthorsOpenReview ↗

Author's response (1/2)

Thank you for your careful reading and for concrete suggestions that improved the paper. We first summarize our response and then answer each point.

Summary

  • W1 and Q1. In the submitted version, Theorem 2 and Appendix B.7 were analyzed under the current policy, while practical training uses logged actions from a behavior policy. The revised PDF restates both for an explicit behavior policy πb\pi_b and bounds the MSE = variance + bias^2. Still, the advantage of the canonical correction is preserved, since its MSE is O((1+δb2)δ4)O((1+\delta_b^2)\delta^4), while otherwise is Θ(δ2)\Theta(\delta^2).
  • Q2. We ran the tabular experiment that you proposed, with three logging policies (uniform, unrelated, and adversarial). The mismatch does add bias, as Theorem 2(iii) predicts. Nevertheless, the canonical form consistently achieves a lower MSE than the Amari form. Under uniform logging, it remains approximately 10 times lower at every degree of mismatch (Figure 5 and Appendix C.6).
  • W2 and Q3. We measured the estimator error at the policies that DPO actually visits during training. The variance reduction of the canonical form persists when the policy deviates substantially from the reference.

W1. Theorem 2 gives a local variance guarantee under the current policy, while training uses logged actions from a behavior policy. Appendix B.7 assumes zero-mean residuals despite this mismatch, so the resulting bias and its impact on MSE are not addressed.

We agree, and thank you for identifying this precisely. The submitted Theorem 2(ii) and Appendix B.7 were on-policy statements, and neither accounted for the bias.

In the revised PDF, the logged next action is drawn from a behavior policy πb\pi_b with mismatch δb=∥πb−πθ∥1\delta_b = \lVert \pi_b - \pi_\theta \rVert_1, and Theorem 2 now has three parts in place of the former (ii), where δ=∥πθ−πref∥1\delta = \lVert \pi_\theta - \pi_{\mathrm{ref}} \rVert_1.

  • (ii) The variance under πb\pi_b is O(δ4)O(\delta^4) for the canonical form and Θ(δ2)\Theta(\delta^2) otherwise. The upper bound holds uniformly over πb\pi_b and does not involve δb\delta_b. In other words, the estimator based on a single logged next action incurs no additional noise in the off-policy setting compared to the on-policy setting, and the policy mismatch enters only through the bias (Lines 346 to 350 in the revised PDF).
  • (iii) The bias from the mismatch is O(δbδ2)O(\delta_b\delta^2) for the canonical form and O(δbδ)O(\delta_b\delta) otherwise.
  • (iv) The MSE is O((1+δb2)δ4)O((1+\delta_b^2)\delta^4) for the canonical form and Θ(δ2)\Theta(\delta^2) otherwise.

The canonical correction’s local MSE advantage is preserved under the mismatch for the following reason. The Taylor expansion of the integrand around ua=1u_a = 1 holds for each action separately, so it does not depend on which policy the action is sampled from. The canonical correction removes the first-order term of the expansion, thereby suppressing its contribution to both the variance and the bias caused by policy mismatch under off-policy sampling. Appendix B.7 is revised in the same way, as described under Q1.

W2. Figures 5 and 10 (which are Figures 6 and 14 in the revised PDF) show improved training stability and policy recovery, but do not directly verify whether the variance reduction persists when the policy deviates substantially from the reference.

In the revision, we additionally ran an experiment that measures this directly in the toy layered MDP, where every quantity can be computed in closed form (Section 4.1 and Appendix C.6). We trained DPO, and evaluated the variance of each estimator at the policies that it actually visits during training. With on-policy data, the canonical form reduces the variance to about 10% of that of the Amari form, and exactly to zero for reverse KL (Figures 5 and 9 in the revised PDF). Note that the canonical estimator reaches policies near the optimal policy, which deviate substantially from the reference, while the Amari estimator barely deviates from its initialization. Therefore, the variance reduction persists when the policy deviates substantially from the reference. We give more detail under Q3.

AuthorsOpenReview ↗

Author's response (2/2)

Q1. In Appendix B.7 the next action is sampled from the behavior policy, but the residual is centered around an expectation under the current policy. Which assumptions are required for the residual to have zero conditional mean and for its second moment to equal the variance under the current policy, and do they hold in practical off-policy training with logged data?

Before we begin, we apologize for a notational error that caused this confusion. The submitted Appendix B.7 was an on-policy analysis, and "a′∼μ(⋅∣s′)a' \sim \mu(\cdot \mid s')" at Line 1190 (which is Line 1334 in the revised PDF) was a typo for a′∼πθ(⋅∣s′).a' \sim \pi_\theta(\cdot \mid s'). In our paper, μ\mu denotes the correction map of Eq. 12 (which is Eq. 13 in the revised PDF).

The revised Appendix B.7 writes the behavior policy as πb\pi_b and drops both assumptions that you mentioned.

  • The conditional mean of the residual is no longer zero. It equals the bias of Theorem 2(iii).
  • Its conditional second moment is Vara′∼πb[Ψa′μ(ua′)]+Bias2\mathrm{Var}_{a' \sim \pi_b}[\Psi^{\mu}_{a'}(u_{a'})] + \mathrm{Bias}^2, not the variance under the current policy.
  • Along a trajectory the variance adds over transitions, while the bias can accumulate in the same direction, which costs a factor of the trajectory length TT. We also relaxed the assumption of independence across transitions. The revised proof only requires that the two trajectories of a preference pair are conditionally independent given their common initial state.

With the revised analysis, the conclusion of Appendix B.7 remains the same, and it is now stated as Proposition 1. The gradient estimation error of the canonical estimator is of a smaller order in δ\delta than that of any other estimator, and the behavior policy changes none of these orders.

Q2. When the behavior policy differs from the current policy, the estimation error may also include bias. Could the authors use the tabular MDP setting, where the target expectation can be computed exactly, to systematically vary the mismatch and separately measure the bias, variance, and MSE of the Amari and canonical estimators?

Thank you for this suggestion! We ran it exactly as proposed, and it turned out to be a more concrete argument for our method.

Under the toy layered MDP setting, we use the behavior policy πb=(1−λ)πθ+λπˉ\pi_b = (1-\lambda)\pi_\theta + \lambda\bar{\pi}, so that λ=0\lambda = 0 is exactly on-policy, and λ=1\lambda = 1 means that the data come entirely from a fixed logging policy πˉ\bar{\pi}. We consider three choices of πˉ\bar{\pi}, which are (i) the uniform policy used throughout the paper, (ii) an unrelated policy that is optimal for an independently drawn reward, and (iii) an adversarial policy that is optimal for the negated reward.

As λ\lambda grows, the mismatch introduces bias under both forms, as you expected and as Theorem 2(iii) predicts. Nevertheless, the variance reduction translates into lower total estimation error. Under uniform logging, the canonical MSE remains approximately 10×10\times lower than the Amari MSE at every λ\lambda (Figures 5 and 9 in the revised PDF). Under unrelated and adversarial logging, the canonical form also consistently achieves a lower MSE (Figures 10 and 11 in the revised PDF).

The experiments that you proposed are included in Section 4.1 (Figure 5) and Appendix C.6 of the revised PDF. Once again, we thank you for the comment, which allowed us to analyze our method more concretely.

Q3. Theorem 2 provides a local variance guarantee near the reference policy. Could the authors directly measure estimator variance as a function of the actual policy deviation from the reference, to clarify whether the local variance advantage extends to larger policy shifts?

Thank you for this suggestion. As described under W2, we measured the estimator variance at the policies that DPO actually visits during training, so that the measurement follows the actual policy deviation from the reference. To observe the training behavior as the current policy deviates farther from the reference, we additionally ran experiments under the three settings described in Q2. Since the reference policy is designed to be uniform in Appendix C.1, changing the logging policy from the unrelated policy to the adversarial policy represents moving progressively farther from the reference policy.

As shown in Figures 9 to 11 (Appendix C.6 in the revised PDF), the variance of the canonical form is lower than that of the Amari form in every scenario. Please note that the y axis is in log scale. In contrast, under adversarial logging, the bias shows almost no difference between the two estimators. In other words, our method generally lowers the MSE even when the policy shift is large, and this gain comes mainly from the variance. Therefore, the local variance advantage of Theorem 2 extends to larger policy shifts in our experiments.

Thank you for your thoughtful response. The clarifications have addressed my concerns, and I will maintain my positive rating.

We’re glad our rebuttal addressed your questions and concerns. Thank you for your feedback and for supporting the paper being presented at ICLR!

Reviewer_XVPgOpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies implicit reward estimation in off-policy preference optimization for multi-step MDPs. It characterizes the special role of reverse KL and proposes a correction to reduce estimation variance, with evaluations on tabular MDPs and LLM preference-learning tasks.

Critical strengths
  1. The theoretical analysis shows how the representation of a regularizer affects reward estimation and explains reverse KL's special role in multi-step preference optimization.
  2. The correction provides a design principle for improving estimation while preserving the underlying regularizer. Controlled MDP experiments and LLM evaluations support its usefulness.
Critical weaknesses

None.

Questions for authors

None.

3 comments
AuthorsOpenReview ↗

Author's response

Thank you for the positive assessment and for the clear summary of our contributions. For your information, the revised PDF extends Theorem 2 to the off-policy setting in response to the other reviews, and our conclusions remain unchanged. We would be glad to answer any further questions.

Thank you for the update on the off-policy extension of Theorem 2. I maintain my original rating with no further concerns.

Thank you for your continued support of our paper! We really appreciate your positive review.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.