- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper tries to solve reward hacking and the conflation of distinct objectives in existing RLHF approaches and proposes Dual Advantage Policy Optimization (DAPO), which is an RLHF framework that combines a Fluency Advantage and an Alignment Advantage through a context-dependent, token-level gating network within PPO. The paper uses LLaMA-2-7B as the reference model and the helpfulness subset of HH-RLHF as training data, and reports improvements over PPO, DPO, RAFT, and RRHF on four evaluation benchmarks, together with component ablations and a theoretical discussion of variance reduction.
Critical strengths
- The paper is easy to follow. The motivation and main components are explained in an intuitive way, making the overall approach straightforward to understand.
- The ablations cover the main components, including the fluency advantage, alignment advantage, adaptive fusion, and KL penalty. These comparisons help in understanding how each component contributes to the reported performance.
- The proposed DAPO is flexible and can be plugged into other PPO-style algorithms, such as P3O, GRPO, and RLOO.
Critical weaknesses
-
The paper’s positioning relative to prior work could be clearer. Previous studies have explored reward decomposition and adaptive weighting. For example, Safe RLHF(Dai, et al., 2024) and Fine-Grained RLHF(Wu, et al., 2023) (this work is mentioned in the paper, howerer with the wrong name "MORLHF") consider multiple alignment objectives and decompose alignment into multiple objectives, while ArmoRM(Wang, et al., 2024) used a context-dependent gate to combine reward objectives. The paper would benefit from a more thorough review of related work and a clearer explanation of how it differs from prior approaches, particularly since dual advantage estimation and adaptive gating are presented as its main contributions.
-
There appears to be an inconsistency between Eq. (2) and its accompanying explanation. The equation defines $ r_t^F=-\log \pi_{\mathrm{LM}}(y_t\mid s_t) +\log \pi_{\mathrm{ref}}(y_t\mid s_t). $ In this way, holding the reference term fixed, a higher token probability under the fluency model decreases this reward, which inverts the training.
-
The method description would benefit from more detail.
- The fluency reward $ r_t^F=-\log \pi_{\mathrm{LM}}(y_t\mid s_t) +\log \pi_{\mathrm{ref}}(y_t\mid s_t) $ is designed at the token level; however, the frozen language model $ \pi_{\mathrm{LM}} $ is GPT-2 Large, and the $ \pi_{\mathrm{ref}} $ is LLaMA-2-7B, so their tokenizations and vocabularies may differ. It is therefore unclear how the GPT-2 scores are mapped to individual LLaMA-2 tokens when computing the fluency reward in Eq. (2).
- There are two rewards and two advantages, but the same scalar value is used to compute their GAE advantages. However, fluency and alignment rewards generally induce different value functions, making it unclear which return the shared critic is trained to predict.
-
The experiments are not broad enough, and some information is omitted.
- The four evaluation benchmarks provide useful coverage, but training uses only the helpfulness subset of HH-RLHF, and the policy backbone is limited to LLaMA-2-7B. Consequently, it remains unclear how consistently the improvements would transfer across training datasets, model families, or scales.
- For AlpacaEval, please specify the benchmark version, reference model, and whether the win rate is raw or length-controlled, as the new AlpacaEval compares the response with the GPT-4 Turbo response reference.
- Section 4.3 states that results include means and standard deviations over three seeds, but Tables 1 and 2 omit the standard deviations.
-
The presentation may need further refinement.
- The figures could be substantially improved in presentation quality. Some plots(Figure 2) and architectural diagrams(Figure 1) are visually rough.
- The appendix contains an unresolved reference(Line 546).
Questions for authors
- Could the authors please clarify which exact AlpacaEval version and evaluation configuration were used? What are the reference model, judge model, raw win rate, and length-controlled win rate?
- The paper states that "All experiments are run with 3 random seeds; we report the mean and standard deviation. " but there is no standard deviations shown in Tables 1 and 2?
- Could the authors clarify how GPT -2 token probabilities are aligned with LLaMA-2 tokens when constructing $ r_t^F $, given that the two models use different tokenizers and vocabularies?
- Could the authors please clarify whether the sign in Eq. (2) is correct? Was the reported experiment run using Eq. (2) or the opposite sign?
- Could the authors clarify what target the shared value network is trained to predict when it is used for both the fluency and alignment GAE calculations?
- Could the authors provide more direct evidence that DAPO reduces reward hacking, beyond lower KL divergence and reward/judge correlation, as lower KL only indicates that the learned policy remains closer to the reference policy and does not directly measure exploitation of reward-model misspecification?