acceptodds
Under review as a conference paper at ICLR 2027

DAPO: Dual Advantage Policy Optimization for Aligning Large Language Models

Abstract

Reinforcement learning from human feedback (RLHF) has emerged as a cornerstone technique for aligning large language models (LLMs) with human preferences. However, existing approaches such as Proximal Policy Optimization (PPO) suffer from two fundamental limitations: reward hacking—where the model exploits gaps in the reward model rather than learning genuinely preferred behaviors—and conflation of distinct objectives, particularly the trade-off between linguistic fluency and semantic alignment. In this paper, we propose Dual Advantage Policy Optimization (DAPO), a novel RLHF training framework that disentangles these objectives by maintaining two separate advantage streams: a Fluency Advantage (AˆF t ), computed via a frozen language model scoring perplexity, and an Alignment Advantage (AˆA t ), derived from a trained reward model. We introduce an adaptive gating network gϕ that dynamically weighs these two streams at each decoding step, conditioned on the generation context. Extensive experiments on Anthropic HH-RLHF, AlpacaEval, MT-Bench, and TruthfulQA demonstrate that DAPO achieves state-of-the-art performance across all benchmarks, outperforming PPO by +7.1% win-rate and DPO by +8.4% win-rate on AlpacaEval, while maintaining significantly lower KL divergence from the reference policy (0.81 vs. 1.42). Ablation studies confirm the contribution of each component, and qualitative analysis reveals that DAPO generates responses that are simultaneously more fluent and more aligned with human values.

Primary area:
reinforcement learning
Keywords:
LLM

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_Y5KTOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper tries to solve reward hacking and the conflation of distinct objectives in existing RLHF approaches and proposes Dual Advantage Policy Optimization (DAPO), which is an RLHF framework that combines a Fluency Advantage and an Alignment Advantage through a context-dependent, token-level gating network within PPO. The paper uses LLaMA-2-7B as the reference model and the helpfulness subset of HH-RLHF as training data, and reports improvements over PPO, DPO, RAFT, and RRHF on four evaluation benchmarks, together with component ablations and a theoretical discussion of variance reduction.

Critical strengths
  1. The paper is easy to follow. The motivation and main components are explained in an intuitive way, making the overall approach straightforward to understand.
  2. The ablations cover the main components, including the fluency advantage, alignment advantage, adaptive fusion, and KL penalty. These comparisons help in understanding how each component contributes to the reported performance.
  3. The proposed DAPO is flexible and can be plugged into other PPO-style algorithms, such as P3O, GRPO, and RLOO.
Critical weaknesses
  1. The paper’s positioning relative to prior work could be clearer. Previous studies have explored reward decomposition and adaptive weighting. For example, Safe RLHF(Dai, et al., 2024) and Fine-Grained RLHF(Wu, et al., 2023) (this work is mentioned in the paper, howerer with the wrong name "MORLHF") consider multiple alignment objectives and decompose alignment into multiple objectives, while ArmoRM(Wang, et al., 2024) used a context-dependent gate to combine reward objectives. The paper would benefit from a more thorough review of related work and a clearer explanation of how it differs from prior approaches, particularly since dual advantage estimation and adaptive gating are presented as its main contributions.

  2. There appears to be an inconsistency between Eq. (2) and its accompanying explanation. The equation defines $ r_t^F=-\log \pi_{\mathrm{LM}}(y_t\mid s_t) +\log \pi_{\mathrm{ref}}(y_t\mid s_t). $ In this way, holding the reference term fixed, a higher token probability under the fluency model decreases this reward, which inverts the training.

  3. The method description would benefit from more detail.

    • The fluency reward $ r_t^F=-\log \pi_{\mathrm{LM}}(y_t\mid s_t) +\log \pi_{\mathrm{ref}}(y_t\mid s_t) $ is designed at the token level; however, the frozen language model $ \pi_{\mathrm{LM}} $ is GPT-2 Large, and the $ \pi_{\mathrm{ref}} $ is LLaMA-2-7B, so their tokenizations and vocabularies may differ. It is therefore unclear how the GPT-2 scores are mapped to individual LLaMA-2 tokens when computing the fluency reward in Eq. (2).
    • There are two rewards and two advantages, but the same scalar value is used to compute their GAE advantages. However, fluency and alignment rewards generally induce different value functions, making it unclear which return the shared critic is trained to predict.
  4. The experiments are not broad enough, and some information is omitted.

    • The four evaluation benchmarks provide useful coverage, but training uses only the helpfulness subset of HH-RLHF, and the policy backbone is limited to LLaMA-2-7B. Consequently, it remains unclear how consistently the improvements would transfer across training datasets, model families, or scales.
    • For AlpacaEval, please specify the benchmark version, reference model, and whether the win rate is raw or length-controlled, as the new AlpacaEval compares the response with the GPT-4 Turbo response reference.
    • Section 4.3 states that results include means and standard deviations over three seeds, but Tables 1 and 2 omit the standard deviations.
  5. The presentation may need further refinement.

    • The figures could be substantially improved in presentation quality. Some plots(Figure 2) and architectural diagrams(Figure 1) are visually rough.
    • The appendix contains an unresolved reference(Line 546).
Questions for authors
  1. Could the authors please clarify which exact AlpacaEval version and evaluation configuration were used? What are the reference model, judge model, raw win rate, and length-controlled win rate?
  2. The paper states that "All experiments are run with 3 random seeds; we report the mean and standard deviation. " but there is no standard deviations shown in Tables 1 and 2?
  3. Could the authors clarify how GPT -2 token probabilities are aligned with LLaMA-2 tokens when constructing $ r_t^F $, given that the two models use different tokenizers and vocabularies?
  4. Could the authors please clarify whether the sign in Eq. (2) is correct? Was the reported experiment run using Eq. (2) or the opposite sign?
  5. Could the authors clarify what target the shared value network is trained to predict when it is used for both the fluency and alignment GAE calculations?
  6. Could the authors provide more direct evidence that DAPO reduces reward hacking, beyond lower KL divergence and reward/judge correlation, as lower KL only indicates that the learned policy remains closer to the reference policy and does not directly measure exploitation of reward-model misspecification?
Reviewer_7AjmOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes a method called “Dual Advantage Policy Optimization” (DAPO), which is an extension of the PPO algorithm for reinforcement learning with human feedback (RLHF). This method explicitly distinguishes between linguistic fluency and alignment with human preferences. It utilizes a frozen language model and a learned reward model to estimate two token-level advantage flows, respectively. A context-based gated network dynamically combines these advantage flows to guide policy updates.

Critical strengths
  1. The proposed context-based gating mechanism offers a potentially useful alternative to fixed reward weights, allowing optimization priorities to vary depending on the generation context.

  2. Ablation studies indicate that both dominant flow and adaptive fusion contribute to performance, and that the model performs well compared to the selected baselines.

Critical weaknesses
  1. Outdated models and benchmarks. The experiments rely solely on LLaMA-2-7B and relatively outdated benchmarks. Evaluation on stronger models (e.g., Qwen3) and more challenging benchmarks, such as AlpacaEval 2.0 and Arena-Hard, is needed to establish the method's relevance and generalizability.

  2. Limited novelty and technical rigor. The method largely combines existing techniques with limited methodological innovation. The paper also has a formulaic, AutoResearch-like presentation, with insufficient theoretical justification and experimental validation, raising concerns about the depth and rigor of the contribution.

Questions for authors
  1. Page utilization: Why not make better use of the available main-paper space? There appears to be room for additional experimental details and analysis.

  2. Outdated baselines: Why does the paper primarily compare against baselines from around 2023, given that this is an ICLR 2027 submission? Comparisons with more recent and competitive methods would strengthen the evaluation.

Reviewer_qgXrOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces a linearly combined training signals by fluency scores, perplexity measured by a frozen GPT-2 Large, and reward model scores, training on a standard preference pair dataset. The paper tries to show that adding the fluency scores and dynamically adjust the weights throught GAE can help to improve alignment during RLHF.

Critical strengths

The author honestly shows the details of the experiments.

Critical weaknesses
  1. The paper seems to be a more appropriate submission in 3 years ago. The whole paper, including motivation, related work, baselines, experiment settings across base model, datasets, benchmarks, and eval judge. The 7-year-old GPT-2 is also used in a core position in the paper. The abbreviation DAPO is also a wide-known variant of GRPO in the last year. Overall, the paper really needs to be updated to the current venue.
  2. The motivation and claims don't match the experiments and results. The paper claims two major drawbacks in Abstract and Introduction that 1) reward hacking, 2) linguistic fluency and semantic alignment trade-off. While the first is a general issue applies to all RL application, the second is not explained across the paper. The paper just doesn't clarify how to solve the first problem or the linguistic fluency truly exists. The fluency issue may happen in early RLHF due to a less capable reward model and should be further justified with new generations of powerful LLMs nowadays.
  3. The writing, and the figures, are awful. They look like generated with LLMs but without any careful adjustment. Abstract contains a lot of technical words and main figures like Figure 1, 2 are consistently overlapping.
Questions for authors

Is Eq. 2 conflicting with the accompanying explanation?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.