- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies distributionally robust RLHF under deployment-time shifts in contexts and human preferences. It proposes two PPO-based formulations: CORAL, which targets context distribution shift while keeping the preference model fixed, and JORAL, which addresses joint context and preference shifts. By exploiting KL-DRO duality, both robust objectives are reduced to tractable one-dimensional dual optimizations that induce adversarial sample weighting and can be integrated into PPO updates. The paper further connects the log-linear policy class commonly used in theoretical RLHF analyses to frozen-backbone LLMs with a trainable output head, and derives finite-sample robust policy-value regret bounds. Empirically, the methods are evaluated on Llama-3.2-1B-Instruct under preference-only, joint context–preference, and AlpacaEval-based shifted evaluation settings, where JORAL generally performs strongly and CORAL shows effectiveness under context shift.
Critical strengths
-
The paper provides a clear and meaningful formulation of distribution shift in RLHF. Rather than treating deployment shift as a single undifferentiated phenomenon, it distinguishes context shift from joint context–preference shift and develops CORAL and JORAL accordingly. In particular, CORAL robustifies the alignment value while retaining the KL-to-reference regularizer under the nominal context distribution, which gives the formulation a clear modeling interpretation rather than simply applying DRO mechanically to the entire RLHF objective. This decomposition is conceptually useful and directly motivates the two proposed methods.
-
The proposed robust objectives are translated into practical PPO-based algorithms in a relatively clean way. Strong duality reduces the inner KL-DRO problems to one-dimensional dual optimizations, yielding adversarial sample weights that can be incorporated into standard PPO updates. The paper also addresses the large action space of LLMs through Monte Carlo response sampling and provides concrete CORAL/JORAL training procedures. This makes the contribution substantially more complete than a purely abstract robust objective.
-
The theoretical and empirical components are both fairly substantial and generally well aligned with the claims. On the theoretical side, the paper carefully connects the commonly assumed log-linear policy class to frozen-backbone, last-layer LLM tuning at the next-token level, while explicitly stating the limits of this equivalence for full autoregressive responses. It also derives finite-sample robust regret bounds that account for reward-model estimation from pairwise preferences. On the empirical side, the evaluation distinguishes preference-only and joint shifts and includes a relatively clean context-shift experiment in which the prompt distribution changes while the correctness criterion is held fixed. JORAL also shows consistently strong performance across the tested shifted settings.
Critical weaknesses
-
The main theoretical guarantees do not directly cover the empirical training regime. The finite-sample analysis applies to log-linear policies and, in the LLM interpretation, to a frozen backbone with a trainable output head, whereas the experiments use full-network LLM fine-tuning. The authors explicitly acknowledge this gap, so I do not view it as invalidating either the theory or the experiments. However, it limits how directly the theoretical guarantees can be used to explain the behavior of the implemented full-model PPO training procedure.
-
The main experiments are conducted on a single 1B instruction-tuned backbone, and the reported uncertainty bars are computed across evaluation prompts rather than across independent training runs. This does not undermine the reported results, but it makes it difficult to assess how stable the observed gains are to stochastic optimization or how broadly they transfer across model families and scales. I hope if the authors can provide experiments on larger models or other model families such as Qwen to see if the results of the experiments still stand.
Questions for authors
- Clarification on robustness-parameter selection. The paper states that the robustness radius is swept over several values and that the best-performing value is reported for each DR method. Could the authors clarify whether is selected on a separate validation set or based on performance in the corresponding final evaluation environment? This matters because the empirically optimal radius varies substantially across the different shift settings, and test-time selection would give the reported best results an oracle-tuned interpretation.
3 comments
Thank you for the rebuttal. I think my problems are somewhat resolved and I decide to stick to my original score (3: Weak acceptance).
Thank you for your prompt response and positive feedback!
Rebuttal by authors
We would like to thank the reviewer for dedicating the time to review our paper and for providing the insightful comments. We have added experiments on aligning a 8B model, please refer to our updated draft.
Weakness 1
We thank the reviewer for this careful assessment, and we agree with the limitation identified here. Theorems 1-2 establish finite-sample guarantees for the log-linear policy class and, under our LLM interpretation, for a frozen backbone with a trainable output head. Our theoretical and empirical results thus provide complementary evidence. The theory establishes the statistical properties of the proposed robust objectives in a tractable LLM policy class, while the experiments demonstrate that the same objectives can be implemented through reweighted PPO updates and remain empirically effective under full-network fine-tuning.
Weakness 2
Thank you for your great suggestion. We have added experiments on the larger Llama-3.1-8B-Instruct, please check our updated manuscript. We note that this new experiment addresses the reviewer's two concerns: (i) it is a significantly larger model than the previous Llama-3.2-1B-Instruct; (ii) it comes from a different generation of the Llama family, and the two tested models also differ in architecture and training process, as summarized in the table below.
New results are now presented in Figures 2, 3(b) and Appendices C.4 and C.5 of the updated manuscript. In these experiments, JoRAL achieved again the best performance in most comparisons across settings (i) and (ii), demonstrating its superiority in distributional shift. Similar improvements are also observed in AlpacaEval setting (iii). The table below shows the AlpacaEval results for the 8B model with δ∈0.5,1,2,3,5.
Question 1
We thank the reviewer for identifying this important ambiguity. The latter interpretation is correct: for each distributionally robust method, we evaluated δ=0.01,0.05,0.1,0.5,1,2,3,5 and the summary Figure 1 and 3 report the best-performing value in the corresponding evaluation environment. The full panels of the algorithms performance under all tested δ's are deferred to Appendix C.4 and C.5.
We absolutely agree with the reviewer that the empirically optimal radius varies substantially across the different shift settings. When training Llama-3.2-1B-Instruct, in setting (i), where only preference shift is present, JoRAL performs best at δ≈1. In setting (ii), where the joint context-preference shift is large, both CORAL and JORAL benefit from a comparatively large radius, δ≥1; for JoRAL, the best performance in setting (ii) instead requires δ≈3 to account for the additional preference shift. The best δ also depends on the f-divergence underlying each method: W-DPO, which measures shift via the Wasserstein distance, achieves its best performance at a much smaller δ≈0.1, and at that same δ, it in fact outperforms JoRAL on some ArmoRM dimensions. This discussion is included in the experiment section.