- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
This paper addresses the high variance of implicit reward estimation under non-reverse-KL regularization in long-sequence preference optimization. Similar to GRPO and REINFORCE, it employs a baseline technique to reduce variance while attempting to preserve the expectation of the estimator. Experimental results suggest that the proposed method can improve preference optimization performance in certain settings.
Critical strengths
- The paper proposes using a baseline technique to reduce the variance of implicit reward estimation. This is a classical technique, the motivation is reasonable and worth investigating.
- The general intuition behind the theoretical analysis is sound: a better unbiased estimator should have lower variance.
- The experiments are relatively comprehensive, covering both mathematical reasoning and preference alignment tasks.
Critical weaknesses
- The theoretical analysis is potentially misleading. The paper targets off-policy settings, yet many important theoretical analyses are conducted under on-policy assumptions. Consequently, many of the proposed theorems and conclusions do not directly apply to the proposed method and cannot establish its advantages. For example, Theorem 2(ii) argues that the proposed implicit reward estimator has lower variance by analyzing the variance of $ \Psi_a^\mu $ under the distribution $ \pi $. However, the paper focuses on off-policy settings, where samples are not drawn from $ \pi $. Therefore, a variance reduction under $ \pi $ does not necessarily imply a variance reduction in the actual off-policy setting. Policy mismatch is a fundamental issue in off-policy settings and should not be ignored. Similar analyses under distributions that do not match the actual application appear repeatedly in the appendix (e.g., Appendix B.7), making it difficult to assess the validity and practical relevance of many theoretical conclusions.
- The characterization of off-policy estimation error is insufficient. The paper characterizes the variance using $ \delta = |\pi-\pi_{\text{ref}}|_1 $. However, $ \delta $ can be large in off-policy settings, and simply comparing asymptotic orders is insufficient to establish the effectiveness of the proposed method.
- The practical method is inconsistent with the theoretical analysis. Equation 21 uses the token-level reward $ \frac{f}{u} $. Under off-policy sampling, the expectations of $ \frac{f_c}{u} $ and $ \frac{f}{u} $ are no longer equal, which introduces bias into the loss.
Questions for authors
- Which of the theorems presented in the paper actually hold under off-policy settings?
- Which theoretical derivations have been manually checked, and which can the authors confidently guarantee are correct?
6 comments
Author's response (2/2)
W3. The practical method is inconsistent with the theoretical analysis. Equation 21 uses the token-level reward computed from the logged token. Under off-policy sampling, its expectation no longer equals the expectation under the current policy, which introduces bias into the loss.
You are right that the two expectations are no longer equal and that this introduces bias. The revised Theorem 2(iii) bounds this bias explicitly, and the canonical correction reduces it from to .
To avoid any concern, we note that the absence of importance sampling in Eq. 21 (Eq. 49 in the revised PDF) is an intentional control. The purpose of these experiments is to test whether the canonical correction remains effective in off-policy settings. Importance sampling is a second correction, and using both would make it impossible to distinguish which of the two an improvement comes from.
In addition, we ran an additional experiment that measures how the bias, the variance, and the MSE change as the data become more off-policy (Figure 5, Appendix C.6, and Figures 9 to 11 in the revised PDF). As Theorem 2(iii) predicts, the policy mismatch introduces bias under both forms. Nevertheless, the canonical form consistently achieves a lower MSE than the Amari form, and this advantage is driven primarily by the variance reduction.
Q1. Which of the theorems presented in the paper actually hold under off-policy settings?
In the revised PDF, all of the statements hold under off-policy settings.
- Lemma 1 and Theorem 1 do not depend on how the next action is sampled. Lemma 1 is an identity for the implicit reward, and Theorem 1 characterizes the regularizers whose inner-term integrand is constant, for which both variance and bias vanish under every
(Line 283 in the revised PDF). - Theorem 2(i) (existence and uniqueness of the canonical form) follows directly from the definition of the regularizer and involves no sampling distribution.
- Theorem 2(ii) to (iv) and Proposition 1 in the revised Appendix B.7 are now stated and proved for every behavior policy with full support.
Before the revision, Section 3.3 (Theorem 2(ii)) and Appendix B.7 were stated for the on-policy variance, as you pointed out in W1. They are now stated for the off-policy MSE, and the earlier on-policy statements are recovered as the special case .
Q2. Which theoretical derivations have been manually checked, and which can the authors confidently guarantee are correct?
All derivations in the paper, including the revised Theorem 2 and Appendix B.7, have been manually checked and we are highly confident that they are correct. We would like to clarify that the weaknesses raised above came from stating the analysis for the on-policy variance. They did not come from a flaw in the derivations, and the revision generalizes those statements to , with the original on-policy results remaining valid as a special case.
Dear Reviewer vpZa,
We hope this message finds you well. Thank you again for your valuable time and careful review of our work.
We have posted our detailed response to your concerns. Following your comments, we revised both the theoretical and experimental parts of the paper. Specifically, we restated Theorem 2 and the related analysis for a behavior policy, so that every theorem now holds in the off-policy setting, and we bounded the bias from policy mismatch. We also added a new experiment that measures how the bias, the variance, and the MSE change with the degree of policy mismatch. As a result, your feedback helped us improve the contribution of the paper significantly, and we appreciate it.
We wanted to gently follow up to see whether our revision has adequately addressed your concerns, or if there are any remaining questions. We remain actively engaged in the discussion period and would be happy to provide any additional clarifications.
Best regards,
The Authors of Submission #8068
Author's response (1/2)
Thank you for your detailed review. Your main concern about policy mismatch in off-policy settings is valid, and we have revised the theoretical analysis accordingly.
Summary
Your point is correct. A variance reduction under the current policy does not by itself imply one in the actual off-policy setting, and the submitted version did not address the mismatch.
In the revised PDF, the logged next action is drawn from a behavior policy πb, and we analyze the MSE of the logged-sample estimator against the on-policy target, which decomposes into variance under πb and bias^2. With δ=∥πθ−πref∥1 and mismatch δb=∥πb−πθ∥1, Theorem 2 now states the following.
In brief, the proof expands the integrand around ua=1 for each action separately, regardless of which policy the action is sampled from. The canonical correction removes the first-order term in u−1, and we track the effect of this term on the variance and on the bias. Importantly, within the family of objective-preserving corrections, the canonical correction is the unique choice that changes these asymptotic rates (Theorem 2(i)).
Appendix B.7 is revised in the same way, and it now bounds the gradient estimation error through the off-policy MSE instead of the on-policy variance (Proposition 1). We also clarify a notational point that may have added to the confusion. "a′∼μ(⋅∣s′)" at Line 1190
(Line 1334 in the revised PDF)was a typo for a′∼πθ(⋅∣s′). In our paper, μ is the correction map of Eq. 12(Eq. 13 in the revised PDF), and it is not a behavior policy. The revised PDF writes the behavior policy as πb throughout.Again, thank you for your careful feedback. The revised theoretical results make the contribution of the paper clearer. We hope that our response resolves your concerns about the off-policy setting and leads to a positive reevaluation.
The statement that the characterization through δ is insufficient seems to come from a misunderstanding of our theoretical analysis. To clarify this point, the goal of our paper structurally requires the analysis around u≈1 (equivalently δ≈0), and it is not a simplification that we chose. Our relaxation looks for a correction that leaves the regularizer unchanged, and by Eq. 12
(Eq. 13 in the revised PDF), such a correction may modify only the first-order term in u−1. A Taylor expansion around u=1 is therefore unavoidable in order to characterize the effect of the correction. To correct this misunderstanding, the revised PDF restates this reason before Theorem 2(Lines 315 in the revised PDF).We hope that this clarifies the misunderstanding, and that the theoretical contribution of our method can be reevaluated. We would be glad to answer any point that remains unresolved.