acceptodds
Under review as a conference paper at ICLR 2027

Robust RLHF for LLM Alignment

Abstract

Reinforcement Learning with Human Feedback (RLHF) aligns large language models (LLMs) with human preferences, but its reliance on static, offline preference data leaves aligned models vulnerable to shifts in deployment contexts and user preferences. We introduce two distributionally robust (DR) alignment formulations under shifts measured by KL divergence, along with tractable PPO-based algorithms for each: JoRAL (joint context-preference shift robust alignment) and CoRAL, a simpler context-shift formulation in which the preference model remains unchanged. By directly connecting the log-linear policy class commonly used in theoretical analyses to modern LLMs, we prove that both CoRAL and JoRAL achieve sample complexity for the frozen-backbone, last-layer LLM policy class. Experiments on LLaMA-3.2-1B-Instruct show that JoRAL provides the most consistent and superior performance improvements across shifted evaluation environments, generally outperforming non-robust PPO and DPO and comparing favorably with existing robust-alignment methods, while CoRAL exhibits effective robustness to context shifts.

Primary area:
reinforcement learning
Keywords:
Large language model, post training, reinforcement learning with human feedback, distributionally robust optimization, sample complexity

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_GCfjOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies distributionally robust RLHF under deployment-time shifts in contexts and human preferences. It proposes two PPO-based formulations: CORAL, which targets context distribution shift while keeping the preference model fixed, and JORAL, which addresses joint context and preference shifts. By exploiting KL-DRO duality, both robust objectives are reduced to tractable one-dimensional dual optimizations that induce adversarial sample weighting and can be integrated into PPO updates. The paper further connects the log-linear policy class commonly used in theoretical RLHF analyses to frozen-backbone LLMs with a trainable output head, and derives finite-sample robust policy-value regret bounds. Empirically, the methods are evaluated on Llama-3.2-1B-Instruct under preference-only, joint context–preference, and AlpacaEval-based shifted evaluation settings, where JORAL generally performs strongly and CORAL shows effectiveness under context shift.

Critical strengths
  1. The paper provides a clear and meaningful formulation of distribution shift in RLHF. Rather than treating deployment shift as a single undifferentiated phenomenon, it distinguishes context shift from joint context–preference shift and develops CORAL and JORAL accordingly. In particular, CORAL robustifies the alignment value while retaining the KL-to-reference regularizer under the nominal context distribution, which gives the formulation a clear modeling interpretation rather than simply applying DRO mechanically to the entire RLHF objective. This decomposition is conceptually useful and directly motivates the two proposed methods.

  2. The proposed robust objectives are translated into practical PPO-based algorithms in a relatively clean way. Strong duality reduces the inner KL-DRO problems to one-dimensional dual optimizations, yielding adversarial sample weights that can be incorporated into standard PPO updates. The paper also addresses the large action space of LLMs through Monte Carlo response sampling and provides concrete CORAL/JORAL training procedures. This makes the contribution substantially more complete than a purely abstract robust objective.

  3. The theoretical and empirical components are both fairly substantial and generally well aligned with the claims. On the theoretical side, the paper carefully connects the commonly assumed log-linear policy class to frozen-backbone, last-layer LLM tuning at the next-token level, while explicitly stating the limits of this equivalence for full autoregressive responses. It also derives finite-sample robust regret bounds that account for reward-model estimation from pairwise preferences. On the empirical side, the evaluation distinguishes preference-only and joint shifts and includes a relatively clean context-shift experiment in which the prompt distribution changes while the correctness criterion is held fixed. JORAL also shows consistently strong performance across the tested shifted settings.

Critical weaknesses
  1. The main theoretical guarantees do not directly cover the empirical training regime. The finite-sample analysis applies to log-linear policies and, in the LLM interpretation, to a frozen backbone with a trainable output head, whereas the experiments use full-network LLM fine-tuning. The authors explicitly acknowledge this gap, so I do not view it as invalidating either the theory or the experiments. However, it limits how directly the theoretical guarantees can be used to explain the behavior of the implemented full-model PPO training procedure.

  2. The main experiments are conducted on a single 1B instruction-tuned backbone, and the reported uncertainty bars are computed across evaluation prompts rather than across independent training runs. This does not undermine the reported results, but it makes it difficult to assess how stable the observed gains are to stochastic optimization or how broadly they transfer across model families and scales. I hope if the authors can provide experiments on larger models or other model families such as Qwen to see if the results of the experiments still stand.

Questions for authors
  1. Clarification on robustness-parameter selection. The paper states that the robustness radius δ\delta is swept over several values and that the best-performing value is reported for each DR method. Could the authors clarify whether δ\delta is selected on a separate validation set or based on performance in the corresponding final evaluation environment? This matters because the empirically optimal radius varies substantially across the different shift settings, and test-time selection would give the reported best results an oracle-tuned interpretation.
3 comments
AuthorsOpenReview ↗

Rebuttal by authors

We would like to thank the reviewer for dedicating the time to review our paper and for providing the insightful comments. We have added experiments on aligning a 8B model, please refer to our updated draft.

Weakness 1

We thank the reviewer for this careful assessment, and we agree with the limitation identified here. Theorems 1-2 establish finite-sample guarantees for the log-linear policy class and, under our LLM interpretation, for a frozen backbone with a trainable output head. Our theoretical and empirical results thus provide complementary evidence. The theory establishes the statistical properties of the proposed robust objectives in a tractable LLM policy class, while the experiments demonstrate that the same objectives can be implemented through reweighted PPO updates and remain empirically effective under full-network fine-tuning.

Weakness 2

Thank you for your great suggestion. We have added experiments on the larger Llama-3.1-8B-Instruct, please check our updated manuscript. We note that this new experiment addresses the reviewer's two concerns: (i) it is a significantly larger model than the previous Llama-3.2-1B-Instruct; (ii) it comes from a different generation of the Llama family, and the two tested models also differ in architecture and training process, as summarized in the table below.

~Llama-3.2-1B-InstructLlama-3.1-8B-Instruct
Parameter sharingThe input token embeddings and output projection share a weight matrix.They use separate weight matrices.
Training recipeUses pruning followed by knowledge distillation.NA

New results are now presented in Figures 2, 3(b) and Appendices C.4 and C.5 of the updated manuscript. In these experiments, JoRAL achieved again the best performance in most comparisons across settings (i) and (ii), demonstrating its superiority in distributional shift. Similar improvements are also observed in AlpacaEval setting (iii). The table below shows the AlpacaEval results for the 8B model with δ∈0.5,1,2,3,5\delta \in\\{0.5, 1, 2, 3, 5\\}.

Policyδ\deltaLC Win RateOutput Length
CoRAL0.516.81 ± 0.312054 ± 45.55
CoRAL116.15 ± 0.322099 ± 46.33
CoRAL216.88 ± 0.362094 ± 45.52
CoRAL316.04 ± 0.292323 ± 45.18
CoRAL517.70 ± 0.322320 ± 45.62
JoRAL0.517.62 ± 0.331989 ± 45.12
JoRAL116.84 ± 0.391991 ± 43.81
JoRAL218.91 ± 0.362026 ± 43.27
JoRAL318.44 ± 0.322006 ± 42.20
JoRAL518.22 ± 0.352012 ± 43.07
WDPO0.518.55 ± 0.362107 ± 44.51
WDPO118.21 ± 0.392122 ± 44.71
WDPO218.36 ± 0.322111 ± 44.34
WDPO316.20 ± 0.322106 ± 44.81
WDPO517.30 ± 0.342116 ± 44.49
KLDPO0.518.05 ± 0.382117 ± 44.34
KLDPO116.92 ± 0.362113 ± 44.57
KLDPO218.55 ± 0.362101 ± 44.51
KLDPO317.12 ± 0.352115 ± 44.68
KLDPO517.31 ± 0.312113 ± 44.64
PPO—16.79 ± 0.322104 ± 45.08
DPO—16.97 ± 0.382140 ± 44.78

Question 1

We thank the reviewer for identifying this important ambiguity. The latter interpretation is correct: for each distributionally robust method, we evaluated δ=0.01,0.05,0.1,0.5,1,2,3,5\delta=\\{0.01,0.05,0.1,0.5,1,2,3,5\\} and the summary Figure 1 and 3 report the best-performing value in the corresponding evaluation environment. The full panels of the algorithms performance under all tested δ\delta's are deferred to Appendix C.4 and C.5.

We absolutely agree with the reviewer that the empirically optimal radius varies substantially across the different shift settings. When training Llama-3.2-1B-Instruct, in setting (i), where only preference shift is present, JoRAL performs best at δ≈1\delta\approx 1. In setting (ii), where the joint context-preference shift is large, both CORAL and JORAL benefit from a comparatively large radius, δ≥1\delta\geq1; for JoRAL, the best performance in setting (ii) instead requires δ≈3\delta\approx3 to account for the additional preference shift. The best δ\delta also depends on the f-divergence underlying each method: W-DPO, which measures shift via the Wasserstein distance, achieves its best performance at a much smaller δ≈0.1\delta\approx0.1, and at that same δ\delta, it in fact outperforms JoRAL on some ArmoRM dimensions. This discussion is included in the experiment section.

Thank you for the rebuttal. I think my problems are somewhat resolved and I decide to stick to my original score (3: Weak acceptance).

Thank you for your prompt response and positive feedback!

Reviewer_6XVaOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies the important problem of RLHF for LLM alignment. The idea behind this work is simple: to improve the robustness of RLHF (Section 2) by applying the established DRO framework to common reward models used in the RLHF literature (Eq. (6)). While the underlying idea may be simple, the theoretical contribution is relevant. Specifically, the author(s) propose two algorithms (CoRAL and JoRAL, Algorithms 1–2) and provide sample complexity bounds for these two algorithms (Theorems 1–2). The derivation of these theorems, which is deferred to the appendix, is non-trivial. The authors validate these algorithms experimentally, considering a single small LLM (LLAMA-3.2-1B-INSTRUCT) against four baselines.

Critical strengths

The paper addresses a very timely issue, namely LLM alignment with human feedback. I found the idea very interesting. The algorithms and guarantees are relevant and non-trivial from a theoretical perspective.

Critical weaknesses

The only major critical weakness lies in the experiments. Although the experimental framework is sound, the author(s) consider only a single small model for the experimental comparison, making the experimental evaluation insufficient. Another, less critical weakness is the lack of a "Conclusion" section in the main text and an "Ethics Statement", discussing the limitations of this work. Although the contribution of this work is mostly technical, I believe that these sections are necessary, given that the work focuses on AI safety.

Questions for authors

There is complementary work that looks at a similar question from a different perspective, e.g., [1, 2]. Have you considered benchmarking your work against these works? What are the advantages of your method over other techniques, e.g., [1, 2], that improve the robustness of RLHF?

Can you comment on the technical novelty of the proofs of Theorems 1–2?

Why did you consider only a single small model in your experiments? Can you comment on the scalability of your method to models that are significantly larger?

[1] Erhan Xu, Kai Ye, Hongyi Zhou, Luhan Zhu, Francesco Quinzan, Chengchun Shi: Doubly Robust Alignment for Large Language Models. NeurIPS 2025

[2] Rémi Munos, Michal Valko, Daniele Calandriello, Mohammad Gheshlaghi Azar, Mark Rowland, Daniel Guo, Yunhao Tang, Matthieu Geist, Thomas Mesnard, Côme Fiegel, Andrea Michi, Marco Selvi, Sertan Girgin, Nikola Momchev, Olivier Bachem, Daniel J. Mankowitz, Doina Precup, Bilal Piot: Nash Learning from Human Feedback. ICML 2024: 36743–36768

3 comments
AuthorsOpenReview ↗

Rebuttal by authors

We would like to thank the reviewer for dedicating the time to review our paper and for providing the insightful comments. We have added experiments on aligning a 8B model, please refer to our updated draft.

Weakness

A single small model

We appreciate your great suggestions. We have now added new experimental results on aligning the larger Llama-3.1-8B-Instruct; please see the updated draft for the full results. We also respond to this weakness in details in our response to your Question 3.

No conclusion & Ethics Statement

This is a valuable point. We agree that the conclusion belongs in the main text. Although we currently place it in Appendix B, due to space constraints, we plan to move it to the end of the main text in the final version. The current conclusion section discussed some potential limitation of CoRAL and JoRAL, as well as future directions. We have added an Ethics Statement before the References section (page 11).

Question 1

We thank the reviewer for pointing us to these complementary works. We have added both papers to the revised Related Work section.

Xu et al. [1] propose doubly robust preference optimization. Their estimator remains consistent when either the preference model or the reference policy is correctly specified. Thus, their method addresses preference-model and reference-policy misspecification under a nominal data-generating distribution. Munos et al. [2] instead replace scalar reward maximization with a general pairwise preference model and optimize a regularized Nash equilibrium. This formulation can represent richer and potentially non-transitive preferences. By contrast, our methods explicitly target deployment-time distribution shift.

An implementation-level comparison is possible in principle, but it would not be a like-for-like evaluation of the respective robustness guarantees. A faithful evaluation of [1] would require controlled misspecification of the preference model and reference policy, whereas evaluating [2] would require training a general pairwise preference model and introducing its self-play/Nash optimization pipeline. Our present experiments are instead designed to test robustness to prompt and preference distribution shifts.

Question 2

Thank you for raising this question. The technical novelty of Theorems 1-2 is two-layered. First, we carefully established the equivalence between the log-linear policy class and the frozen backbone last layer language model policy class, which suggests that Theorems 1-2 (and similar results in the literature [3-4]) apply to the frozen backbone last layer language model policy class.

Secondly, the core challenge in the regret analysis of Theorems 1-2 lies in taking a union bound over the stochastic policy class, which renders classic sample complexity techniques that rely on deterministic policy class complexity measures, such as the Hamming entropy integral [5], inapplicable. To address this, we leverage the Lipschitzness of the robust policy values under Assumption 1 and adopt the chaining method [6].

Question 3

We agree that the original evaluation on a 1B model was limited. At the initial submission deadline, the substantially greater computational cost of the 8B experiments prevented us from completing the full comparison in time. Now we have conducted the same experiments on aligning Llama-3.1-8B-Instruct and reported them in the updated manuscript. The results are now presented in Figures 2, 3(b) and Appendices C.4 and C.5 of the updated manuscript. In these experiments, JoRAL achieved again the best performance in most comparisons across settings (i), (ii), and (iii), demonstrating its superiority in distributional shift.

AuthorsOpenReview ↗

Rebuttal continued

Scalability on larger model

Both CoRAL and JoRAL scales well with the language model size. The table blow highlights selected performance improvements of Llama-3.1-8B-Instruct over Llama-3.2-1B-Instruct, which shows that both methods achieve higher instruction-following, truthfulness, and honesty scores with the 8B backbone at the reported matched δ\delta values.

Algorithmδ\deltaHelpSteer1 IFHelpSteer1 TruthfulnessHelpSteer1 HonestyHelpSteer2 IFHelpSteer2 TruthfulnessHelpSteer2 Honesty
CoRAL0.50.5↑30.2%\uparrow 30.2\%↑19.7%\uparrow 19.7\%↑20.6%\uparrow 20.6\%↑16.5%\uparrow 16.5\%↑14.4%\uparrow 14.4\%↑14.8%\uparrow 14.8\%
CoRAL11↑29.7%\uparrow 29.7\%↑18.8%\uparrow 18.8\%↑20.4%\uparrow 20.4\%↑13.9%\uparrow 13.9\%↑12.5%\uparrow 12.5\%↑13.4%\uparrow 13.4\%
CoRAL22↑22.3%\uparrow 22.3\%↑18.2%\uparrow 18.2\%↑18.6%\uparrow 18.6\%↑14.8%\uparrow 14.8\%↑13.2%\uparrow 13.2\%↑13.7%\uparrow 13.7\%
CoRAL33↑21.6%\uparrow 21.6\%↑17.2%\uparrow 17.2\%↑18.5%\uparrow 18.5\%↑14.0%\uparrow 14.0\%↑12.1%\uparrow 12.1\%↑13.0%\uparrow 13.0\%
CoRAL55↑20.8%\uparrow 20.8\%↑17.2%\uparrow 17.2\%↑18.0%\uparrow 18.0\%↑14.5%\uparrow 14.5\%↑12.9%\uparrow 12.9\%↑13.4%\uparrow 13.4\%
JoRAL0.50.5↑28.7%\uparrow 28.7\%↑19.9%\uparrow 19.9\%↑20.8%\uparrow 20.8\%↑14.3%\uparrow 14.3\%↑11.6%\uparrow 11.6\%↑12.6%\uparrow 12.6\%
JoRAL11↑19.3%\uparrow 19.3\%↑15.3%\uparrow 15.3\%↑17.0%\uparrow 17.0\%↑11.6%\uparrow 11.6\%↑9.1%\uparrow 9.1\%↑10.0%\uparrow 10.0\%
JoRAL22↑19.8%\uparrow 19.8\%↑15.4%\uparrow 15.4\%↑16.9%\uparrow 16.9\%↑12.9%\uparrow 12.9\%↑10.9%\uparrow 10.9\%↑11.7%\uparrow 11.7\%
JoRAL33↑18.8%\uparrow 18.8\%↑14.4%\uparrow 14.4\%↑15.5%\uparrow 15.5\%↑12.2%\uparrow 12.2\%↑10.6%\uparrow 10.6\%↑11.0%\uparrow 11.0\%
JoRAL55↑19.9%\uparrow 19.9\%↑15.4%\uparrow 15.4\%↑16.8%\uparrow 16.8\%↑12.5%\uparrow 12.5\%↑10.7%\uparrow 10.7\%↑11.1%\uparrow 11.1\%

The full comparisons with the benchmarks in Appendix C.4 further support the effectiveness of JoRAL under preference and joint distribution shifts at this scale.

Similar improvements are also observed in AlpacaEval. The table below shows the AlpacaEval results for the 8B model with δ∈0.5,1,2,3,5\delta \in\\{0.5, 1, 2, 3, 5\\}. The complete results are provided in Figure 3(b) and Table 7 of the updated manuscript.

Policyδ\deltaLC Win RateOutput Length
CoRAL0.516.81 ± 0.312054 ± 45.55
CoRAL116.15 ± 0.322099 ± 46.33
CoRAL216.88 ± 0.362094 ± 45.52
CoRAL316.04 ± 0.292323 ± 45.18
CoRAL517.70 ± 0.322320 ± 45.62
JoRAL0.517.62 ± 0.331989 ± 45.12
JoRAL116.84 ± 0.391991 ± 43.81
JoRAL218.91 ± 0.362026 ± 43.27
JoRAL318.44 ± 0.322006 ± 42.20
JoRAL518.22 ± 0.352012 ± 43.07
WDPO0.518.55 ± 0.362107 ± 44.51
WDPO118.21 ± 0.392122 ± 44.71
WDPO218.36 ± 0.322111 ± 44.34
WDPO316.20 ± 0.322106 ± 44.81
WDPO517.30 ± 0.342116 ± 44.49
KLDPO0.518.05 ± 0.382117 ± 44.34
KLDPO116.92 ± 0.362113 ± 44.57
KLDPO218.55 ± 0.362101 ± 44.51
KLDPO317.12 ± 0.352115 ± 44.68
KLDPO517.31 ± 0.312113 ± 44.64
PPO—16.79 ± 0.322104 ± 45.08
DPO—16.97 ± 0.382140 ± 44.78

[1] Erhan Xu, Kai Ye, Hongyi Zhou, Luhan Zhu, Francesco Quinzan, Chengchun Shi: Doubly Robust Alignment for Large Language Models. NeurIPS 2025

[2] Rémi Munos, Michal Valko, Daniele Calandriello, Mohammad Gheshlaghi Azar, Mark Rowland, Daniel Guo, Yunhao Tang, Matthieu Geist, Thomas Mesnard, Côme Fiegel, Andrea Michi, Marco Selvi, Sertan Girgin, Nikola Momchev, Olivier Bachem, Daniel J. Mankowitz, Doina Precup, Bilal Piot: Nash Learning from Human Feedback. ICML 2024: 36743–36768

[3] Zaiyan Xu, Sushil Vemuri, Kishan Panaganti, Dileep Kalathil, Rahul Jain, and Deepak Ramachandran. Robust llm alignment via distributionally robust direct preference optimization. arXiv preprint arXiv:2502.01930, 2025.

[4] Debmalya Mandal, Paulius Sasnauskas, and Goran Radanovic. Distributionally robust reinforcement learning with human feedback. arXiv preprint arXiv:2503.00539, 2025.

[5] Martin J Wainwright. High-dimensional statistics: A non-asymptotic viewpoint, volume 48. Cambridge university press, 2019.

[6] Zhengyuan Zhou, Susan Athey, and Stefan Wager. Offline multi-action policy learning: Generalization and optimization. Operations Research, 71(1):148–183, 2023.

Dear Reviewer, would you please let us know if our rebuttals have addressed your concerns? Thank you for your time and consideration!

Reviewer_tZpCOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies distributionally robust RLHF. The concern is that preference data is fixed, but at deployment the prompts and user preferences can shift. The authors propose two PPO-based methods with KL uncertainty sets. CORAL only lets the prompt distribution shift. JORAL lets both the prompt distribution and the preference model shift, and uses the win rate against the reference policy as the reward. By strong duality, both reduce to a one-dimensional dual problem. This gives each prompt an exponential weight that can be plugged into standard PPO. The KL penalty to the reference policy stays under the original prompt distribution. In theory, the paper shows that a log-linear policy is exactly a frozen LLM backbone with a trainable output head, and proves an O(sqrt(d/n)) regret bound for both methods. Experiments train LLaMA-3.2-1B-Instruct on HelpSteer2 and compare with PPO, DPO, W-DPO and KL-DPO. JORAL gives the best results on most ArmoRM dimensions and the best LC win rate on AlpacaEval.

Critical strengths
  1. The method is simple to use

After duality, robustness becomes a per-prompt weight in the PPO loss. Finding this weight is a one-dimensional concave problem. So the change to the standard RLHF pipeline is small.

  1. The link between theory and LLMs is made explicit

Example 2 shows exactly when the log-linear policy class matches an LLM. The paper is also honest that this only holds at the token level with a frozen backbone. Prior work often leaves this link informal.

Critical weaknesses
  1. The model is too small

All experiments use one model, LLaMA-3.2-1B-Instruct. This is very small for alignment work today. Most recent alignment papers test on 7B models or larger. A 1B model has limited ability, so it is hard to tell whether the gains would hold at a realistic scale. The gains over PPO are also small on many dimensions, often around 0.01 to 0.02 in ArmoRM score. I would like to see results on at least one 7B or 8B model, with a few values of δ

  1. The evaluation does not test real preference shift

The preference labels come from the correctness dimension of ArmoRM. The test scores come from the other dimensions of the same ArmoRM. So in setting (i), the “preference shift” is only a switch between dimensions of one reward model. These dimensions are likely highly correlated, which is far from how real user preferences change. Setting (ii) changes the prompts from HelpSteer2 to HelpSteer1, but these two datasets are very close. I would like to see test preferences from a different source, such as human labels or a judge that is not ArmoRM, and test sets with a larger shift.

Questions for authors
  1. Can the authors report results on a 7B or larger model?

  2. How large is the shift between training and test preferences in settings (i) and (ii)?

5 comments
AuthorsOpenReview ↗

Rebuttal by authors

We would like to thank the reviewer for dedicating the time to review our paper and for providing the insightful comments. We have added experiments on aligning a 8B model, please refer to our updated draft.

Weakness 1

Thank you for your helpful suggestions. We have now added alignment results using Llama-3.1-8B-Instruct, please check the latest draft. We conducted the same experiments that we've conducted on LLaMA-3.2-1B-Instruct over the distributionally robust parameter δ∈0.01,0.05,0.1,0.5,1,2,3,5\delta\in\\{0.01,0.05,0.1,0.5,1,2,3,5\\} for our CoRAL/JoRAL and all benchmarks. The full panels of all tested algorithms under all tested δ\delta's are given in Appendix C.4.

We agree that the absolute gains are modest on some dimensions. However, a small absolute difference has no uniform interpretation across the heterogeneous ArmoRM heads, which have different score distributions and calibrations. More importantly, the improvements are substantially larger on several dimensions that directly reflect the distribution shifts considered in our experiments.

For example, on the 1B backbone in setting (ii), JoRAL improves over PPO by approximately 7% on UltraFeedback helpfulness, 6.6% on UltraFeedback instruction following, 5.3% on Prometheus, and 3.6% on HelpSteer helpfulness. In the same setting, CoRAL improves UltraFeedback helpfulness by approximately 6.0% and HelpSteer complexity by approximately 8.2%. On the 8B model alignment, JoRAL achieves the best score on 16 of the 18 ArmoRM dimensions in setting (i) and leads on the large majority of dimensions in setting (ii).

The advantage of JoRAL is substantially larger in setting (iii), a test using GPT-5.6-Luna as judge. Relative to PPO, it increases the length controlled win rate from 4.94 to 6.81 on the 1B backbone (+1.87 points; +37.9%) and from 16.79 to 18.91 on the 8B backbone (+2.12 points; +12.6%), while also producing shorter responses. Please refer to our response to Weakness 2 for the complete results.

Weakness 2

We thank the reviewer for raising this important concern.

Independent judge in setting (iii)

We included a test (setting (iii)) on the AlpacaEval length-controlled (LC) win rate against GPT-4, judged by GPT-5.6-Luna (Figure 3). At the scale of our evaluation, obtaining new human labels for every response and configuration is prohibitively expensive. We therefore follow the industry's leading solution and use an independent strong language model as a scalable proxy for human preference. Specifically, LC AlpacaEval [1] in our setting (iii) evaluates general instruction-following ability while controlling for verbosity; aligned models generate responses to AlpacaEval prompts, and GPT-5.6-Luna judges them against the GPT-4. Model-based judging is now a widely adopted evaluation protocol: it is directly supported by commercial evaluation systems such as OpenAI Graders and Google Cloud's judge-model evaluation framework [4,5], and has been extensively studied academically [2,3].

Accordingly, setting (iii) changes all three relevant components: the prompts come from AlpacaEval, the preference criterion is length-controlled response quality, and the judge is GPT-5.6-Luna, which is independent of ArmoRM. We also repeated this experiment on post-training the larger Llama-3.1-8B-Instruct, in which JoRAL at δ=2\delta=2 achieves the highest LC win rate. Its average output length is also shorter than those of the strongest W-DPO and KL-DPO configurations. The table below shows the AlpacaEval results for the 8B model with δ∈0.5,1,2,3,5\delta \in\\{0.5, 1, 2, 3, 5\\}. The complete results are provided in Figure 3(b) and Table 7 of the paper.

Policyδ\deltaLC Win RateOutput Length
CoRAL0.516.81 ± 0.312054 ± 45.55
CoRAL116.15 ± 0.322099 ± 46.33
CoRAL216.88 ± 0.362094 ± 45.52
CoRAL316.04 ± 0.292323 ± 45.18
CoRAL517.70 ± 0.322320 ± 45.62
JoRAL0.517.62 ± 0.331989 ± 45.12
JoRAL116.84 ± 0.391991 ± 43.81
JoRAL218.91 ± 0.362026 ± 43.27
JoRAL318.44 ± 0.322006 ± 42.20
JoRAL518.22 ± 0.352012 ± 43.07
WDPO0.518.55 ± 0.362107 ± 44.51
WDPO118.21 ± 0.392122 ± 44.71
WDPO218.36 ± 0.322111 ± 44.34
WDPO316.20 ± 0.322106 ± 44.81
WDPO517.30 ± 0.342116 ± 44.49
KLDPO0.518.05 ± 0.382117 ± 44.34
KLDPO116.92 ± 0.362113 ± 44.57
KLDPO218.55 ± 0.362101 ± 44.51
KLDPO317.12 ± 0.352115 ± 44.68
KLDPO517.31 ± 0.312113 ± 44.64
PPO—16.79 ± 0.322104 ± 45.08
DPO—16.97 ± 0.382140 ± 44.78

In this setting, the best performing JoRAL significantly out performs the PPO baseline by 12.6% in win rate, and simultaneously reduces the PPO verbosity by 78 characters on average. CoRAL performs relatively poorly here as expected, since it is designed to hedge only against context shift.

AuthorsOpenReview ↗

Rebuttal continued

Why setting (i) is a preference shift

We agree that setting (i) is a controlled preference shift rather than a complete model of heterogeneous human populations. Nevertheless, switching between dimensions of ArmoRM already incurs a preference shift. Preference shift is formally defined by a change in the conditional preference relation. Training comparisons are induced only by the correctness coordinate: a≻tra′⇔rcorr(s,a)>rcorr(s,a′),a\succ_{\mathrm{tr}}a'\Leftrightarrow r_{\mathrm{corr}}(s,a)>r_{\mathrm{corr}}(s,a'), whereas evaluation coordinate kk induces a≻te,ka′⇔rk(s,a)>rk(s,a′).a\succ_{\mathrm{te},k}a'\Leftrightarrow r_k(s,a)>r_k(s,a'). The prompt distribution is held fixed. Unless rkr_k and rcorrr_{\mathrm{corr}} induce exactly the same ordering over every response pair with the same reward scale, the conditional preference relation changes; this is precisely a pure preference shift.

Moreover, the 17 reward dimensions tested combines vastly different criteria. A response can therefore be preferred under correctness but rejected under safety, honesty, or style dimensions. Using a common multi-objective reward such as ArmoRM is useful here because it isolates the change in the target preference from changes in evaluator calibration. Setting (iii) complements this controlled experiment by testing an entirely external evaluator and preference criterion.

Why setting (ii) has a context shift

The two prompt datasets were collected through materially different pipelines [6,7]. HelpSteer1 combines human-written Scale AI prompts. In contrast, more than 95% of HelpSteer2 prompts come from user-shared ShareGPT conversations, and approximately 29% are multi-turn.

We further computed corpus-level statistics on the full public train and validation releases after exact prompt deduplication, using the same lowercased alphanumeric tokenizer for both datasets:

StatisticHelpSteer1HelpSteer2
Mean lexical tokens per prompt423.9121.1
Median lexical tokens per prompt44441.5
Structured multi-turn prompts0%29.2%

Empirically, their mean and median prompt-token counts differ by factors of 3.5 and 10.7, respectively. The corresponding empirical unigram distributions have Jensen-Shannon divergence ≈0.160\approx0.160 bits and total-variation distance ≈0.320\approx0.320. More formally, a prompt-sequence distribution factorizes as p(s)=p(T)∏t=1Tp(xt∣x<t,T).p(s)=p(T)\prod_{t=1}^{T}p(x_t\mid x_{<t},T). It is therefore not merely ``the distribution of individual words.'' Nevertheless, a difference in either the length distribution or a token marginal is already sufficient to establish that the joint sequence distributions differ. Thus, setting (ii) combines a context shift with the preference shift studied in setting (i).

Question 1

Thank you so much for the valuable question. We now report our post-training results on Llama-3.1-8B-Instruct. Please refer to our response to Weakness 1, 2 and our updated draft.

Question 2

Thank you so much for the valuable question. Please refer to our response to Weakness 2.

[1] Dubois et al., “Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators”, COLM 2024.

[2] Zheng et al., “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena”, NeurIPS 2023.

[3] Liu et al., “G-EVAL: NLG Evaluation using GPT-4 with Better Human Alignment”, EMNLP 2023.

[4] OpenAI Graders documentation, openai api docs: guides/graders.

[5] Google Cloud judge-model evaluation documentation, google gemini-enterprise-agent-platform: models/evaluate-judge-model.

[6] Wang et al., “HelpSteer: Multi-attribute Helpfulness Dataset for SteerLM”, NAACL 2024.

[7] Wang et al., “HelpSteer2: Open-source Dataset for Training Top-performing Reward Models”, NeurIPS 2024.

Dear Reviewer, would you please let us know if our rebuttals have addressed your concerns? Thank you for your time and consideration!

Comments on the paper

1 comment

We have addressed the reviewers' main shared concern—the lack of a large-scale experiment with an 8B model—by adding a comprehensive alignment experiment on Llama-3.1-8B-Instruct in three settings: (i) preference shift on HelpSteer2; (ii) joint context and preference shift on unseen HelpSteer1 prompts; and (iii) an external AlpacaEval stress test judged by GPT-5.6-Luna using the length-controlled win rate against GPT-4. Across settings (i) and (ii), our methods, JoRAL (joint-shift robust alignment) and CoRAL (context-shift robust alignment), achieve the top score in 35 of 36 ArmoRM dimension–setting comparisons against the robust alignment baselines W-DPO and KL-DPO and the non-robust baselines PPO and DPO (29 and 6 wins, respectively). JoRAL leads in both settings, while five of CoRAL's six wins occur in setting (ii), where context shift is present. In setting (iii), JoRAL achieves a length-controlled win rate of 18.91, exceeding the best robust baseline by 0.36 points (1.9%), PPO by 2.12 points (12.6%), and DPO by 1.94 points (11.4%). CoRAL achieves 18.33, outperforming PPO by 1.54 points (9.2%) and DPO by 1.36 points (8.0%); at the matched robust parameter δ=0.1\delta=0.1, it also exceeds W-DPO and KL-DPO by 0.40 and 1.16 points, respectively. The winning JoRAL configuration also generates substantially shorter responses (75–114 fewer characters) on average than the strongest baselines, indicating that it produces higher-quality, more concise responses.

Reviewer GCfj and tZpC indicated support after our response, while Reviewer 6XVa posted no follow-up.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.