- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper proposes a single-trajectory offline policy optimization method for LLM alignment. The method leverages the closed-form optimal policy for the KL-regularized RL objective to construct a policy-fitting objective that utilizes pre-existing offline completions present in a training dataset to provide a single-trajectory learning signal. The method is supported by an empirical evaluation on the UltraFeedback dataset.
Critical strengths
None
Critical weaknesses
- Lack of novelty. The paper doesn't mention or acknowledge numerous papers that tackle exactly same issue and use very similar or sometimes an almost identical approach, e.g. A*-PO paper [1] that also approximates the value function from offline completions and uses the same squared loss, SPO [2], QRPO [3], AGRO [4], TBA [5], etc that also optimize the same squared loss while handling the normalization term in different ways. Therefore, the core regression objective and precomputed-normalizer approach are already present in prior work, including a very similar methods like A*-PO.
- Error in the theory. The proposed estimator for the partition function in eq. 13, namely , is not a correct monte-carlo estimate for the partition function in eq. 11. A simple argument demonstrating the error is that using that estimator the total probability mass of the completions from under the policy obtained by exactly fitting the proposed regression targets is 1, thereby assigning zero probability to all completions outside . A correct one would be , where is the probability distribution from which the completions in were sampled. The fact that is generally unknown in offline data makes the application of the proposed method very limited, therefore some other methods mentioned above set to a known policy, e.g. as in A*-PO [1] that uses reference model completions to estimate .
- Level of noise in . There is no analysis of the noise amplitude in the proposed estimate of , however, the QRPO paper [3] provides such a theoretical analysis and shows that in case the reward distribution has a right tail the noise level is catastrophic. The current paper evaluates the method in a setting with bounded rewards, which may not reveal the severity of this problem. However, the method presentation is not restricted to bounded rewards, and the absence of such analysis leaves an important limitation unaddressed.
[1] Kianté Brantley, Mingyu Chen, Zhaolin Gao, Jason D. Lee, Wen Sun, Wenhao Zhan, and Xuezhou Zhang. Accelerating RL for LLM Reasoning with Optimal Advantage Regression. NeurIPS, 2025.
[2] Taco Cohen, David W. Zhang, Kunhao Zheng, Yunhao Tang, Rémi Munos, and Gabriel Synnaeve. Soft Policy Optimization: Online Off-Policy RL for Sequence Models. arXiv preprint arXiv:2503.05453, 2025.
[3] Simon Matrenok, Skander Moalla, and Caglar Gulcehre. Quantile Reward Policy Optimization: Alignment with Pointwise Regression and Exact Partition Functions. NeurIPS, 2025.
[4] Yunhao Tang, Taco Cohen, David W. Zhang, Gabriel Synnaeve, and Rémi Munos. RL-finetuning LLMs from On- and Off-policy Data with a Single Algorithm. AISTATS, 2026.
[5] Brian R. Bartoldson, Siddarth Venkatraman, James Diffenderfer, Moksh Jain, Tal Ben-Nun, Seanie Lee, Minsu Kim, Johan Obando-Ceron, Yoshua Bengio, and Bhavya Kailkhura. Trajectory Balance with Asynchrony: Decoupling Exploration and Learning for Fast, Scalable LLM Post-Training. NeurIPS, 2025.
Questions for authors
- Why does the KL divergence in figure 2 start from a non-zero value given that the paper states that the policy starts from the reference checkpoint? Why do different methods have different starting points?
- Table 1 contains a column with GPT-4o and Claude-3.5 win-rates of RPO checkpoint against various methods, and table 5 contains a similar column with each RPO density variant’s win rates against SFT. Those columns allow us to compare the win-rates between SFT and RPO and observe that 75% density setting has higher reported judge win rates than 100% for every model, despite the claim that performance improves monotonically with density. Can the authors explain that? Likewise, why does the noise in rewards (with ) (table 6) improve those win rates for every model compared with the corresponding clean-reward RPO checkpoints?
- Which hyperparameter search ranges were used for each method?
2 comments
Thank you for the detailed response. At this points I would maintain my score.
We sincerely thank the reviewer for the careful and technically substantive assessment. The concerns about prior work, partition estimation, and empirical consistency are important. We agree that RPO's relationship to related approaches and the theoretical interpretation of its empirical partition require better qualification. We address each point below.
We agree that the manuscript should have discussed A*-PO, SPO, QRPO, AGRO, and TBA. A*-PO and QRPO are particularly related to RPO's core ingredients: regression toward reward-derived policy targets and normalization. We will substantially revise the related work and narrow our novelty claims.
Our contribution is not squared-error policy regression or offline normalization in general. Rather, RPO studies an empirical construction using same-prompt candidates from scalar-feedback datasets to define a prompt-dependent normalization, without training a separate value network. We acknowledge that this distinction alone does not establish broad methodological novelty. We will compare the assumptions, sampling requirements, normalization, and optimization settings of these methods, positioning RPO as a specific empirical variant rather than the first method to introduce this principle.
We appreciate the reviewer identifying this issue. The exact partition is
Z(x)=Ey∼πref[exp(τr(x,y))].
If completions are sampled from q(y∣x), an unbiased Monte-Carlo estimator generally requires
Z(x)=M1∑j=1Mq(yj∣x)πref(yj∣x)exp(rj/τ).
RPO's current estimator omits this correction. Unless candidates come from the reference policy or suitable sampling weights are available, it is therefore not an unbiased estimator of the full reference-policy partition. We agree this limits the current theoretical presentation.
Instead, the construction defines a normalization over the observed candidate set. Thus, the exact-fitting argument applies to this empirical support, not necessarily the full response space. We will correct the derivation and distinguish this finite-support objective from the full KL-regularized optimal policy. Recovering the latter requires knowledge of the sampling distribution and appropriate importance weighting, or another principled estimator. This qualification materially narrows our theoretical claim.
We agree that bounded rewards do not eliminate the variance caused by exponentiation. QRPO's analysis of this issue should have been discussed. Our bounded, standardized UltraFeedback rewards demonstrate behavior only in that setting, not robustness to arbitrary or right-tailed reward distributions. We will discuss how estimation noise depends on reward dispersion, temperature, and candidate count, and limit our robustness claims to the tested regime.
Q1. Non-zero initial KL. Identical initial policy and reference checkpoints must yield zero KL. We will clarify when the first measurement is taken and how KL is estimated. If the plotted values correspond to step zero, this is an inconsistency that must be corrected; identical initialization cannot explain non-zero values.
Q2. Density and reward-noise results. We agree that the reported judge win rates contradict the stated monotonic trend: 75% density outperforms 100%, while adding reward noise improves win rates across models. These results cannot support claims of monotonic improvement or straightforward robustness. We will audit checkpoint selection, evaluation, and table construction, report relevant variability, and revise the claims to match verified results. We will not attribute these differences to a mechanism without evidence.
Q3. Hyperparameter search. The manuscript reports a shared learning rate of 10−4, 150 warmup steps, and default τ=1.0, with an RPO temperature ablation over {0.1,1.0,5.0}. However, it does not provide complete method-specific search ranges. We will document the configurations actually tested for each baseline and distinguish fixed settings from searched hyperparameters, without implying a broader search.
We thank the reviewer again. These issues require substantive corrections, not merely clarification. We will narrow the theoretical claims to what the estimator supports, acknowledge prior work, and revise experimental conclusions wherever the evidence does not justify the current wording.