- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes PODA, an advantage estimator for multi-reward GRPO-style RL. Instead of linearly combining rewards, it builds a Pareto-dominance graph over each prompt’s rollouts. Each rollout is scored by its inverse dominator count plus a signed dominance margin, and the scores are whitened. On math reasoning and BFCL-v3 tool calling with Qwen models, the authors report gains in both accuracy and constraint satisfaction over GDPO, DAPO, and tuned linear and indicator-based estimators.
Critical strengths
The paper studies an under-explored design choice in multi-reward RL for LLMs, that is, how a reward vector is turned into a single scalar advantage, especially when rollouts compete on different objectives. This is a useful framing, given how often reasoning and tool-use training now combine several rewards. However, as the weaknesses below explain, the proposed method and its evaluation do not answer this question convincingly.
Critical weaknesses
Narrow multi-objective evaluation. The main results use only two objectives per task. Prior multi-objective math RL typically uses three or more objectives (e.g., https://arxiv.org/abs/2509.11452). There is also no hypervolume or Pareto-front comparison, which is essential for a method motivated by Pareto dominance.
Small and fragile gains. Gains over the strongest tuned baselines are small (0.91 on math, 0.55 on tools, and even less over DAPO on tools). Confidence intervals are reported only against GDPO.
Questions for authors
Please define each reward function and “joint success.” If the math rewards are binary, no can change the dominance graph. What then explains the 1.3–1.7 variation of math avg acc in Table 15?