- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes APAF, an advantage estimator for multi-reward GRPO training. Within each rollout group, it blends a magnitude-based advantage with a Pareto-dominance score. A per-prompt weight controls the blend that it grows when responses differ clearly and shrinks for near-ties. Experiments on tool calling and helpfulness–safety alignment report gains over GRPO, GDPO, and GD2PO-Hard.
Critical strengths
Making within-group Pareto relations explicit in the advantage, without discarding conflicting rollouts, is a reasonable and well-motivated idea.
Critical weaknesses
Missing stronger baselines. The paper compares only against GRPO-style baselines, not against established MOO and MTL methods such as MGDA, Nash-MTL, or PAMA. Several of these have already been adapted to LLM alignment (https://arxiv.org/abs/2602.06869), and is proved to be working. Its dominance-preservation guarantee also adds nothing over the existing baselines, since linear advantages already preserve dominance.
The evidence does not support the core claim. The adaptive controller, the paper’s core contribution, is not used in the safety main results. In the tool task, the length reward is saturated for all methods, leaving little trade-off to resolve. The remaining gains are small (about 2 points on API-Bank), and the authors concede they are not statistically reliable.
Questions for authors
- The main experiments use two rewards, and three (correctness, format, length) appear only in Appendix D on one model. Does APAF remain effective with more objectives, where most pairs become incomparable and dominance relations grow sparse