acceptodds
Under review as a conference paper at ICLR 2027

PODA: Partial-Order Advantage Estimation for Multi-Reward LLM Alignment

Abstract

Multi-reward reinforcement learning for large language models usually compresses every reward vector into a scalar advantage with a fixed linear rule. This is convenient, but it inserts pairwise preferences even when two rollouts trade off different objectives. We introduce PODA, a prompt-local advantage estimator that instead constructs a dominance graph, combines an inverse dominance-count score with an antisymmetric dominance margin, and leaves the PPO/GRPO policy objective unchanged. PODA is a credit-assignment rule applied after rewards and any task priorities have been specified; it does not infer user preferences or learn a Pareto coverage set. Across mathematical reasoning and tool calling with two and three rewards, and across Qwen2.5, Qwen3, and Qwen3.5 models, PODA improves both primary performance and constraint satisfaction over equally tuned linear, lexicographic, rank-sum, additive-indicator, hypervolume, GRPO, GDPO, and DAPO comparators. On the representative two-reward settings, it improves the strongest matched indicator baseline by 0.91 math-accuracy points and 0.55 tool-accuracy points; the corresponding three-reward gains are 0.9 and 0.8 points. Paired three-seed confidence intervals are positive in the representative settings, while the estimator adds about 1% step time at group size 16. Training and dominance diagnostics show the largest gains when reward conflict and graph sparsity make fixed scalarization or count-only scoring least informative.

Primary area:
reinforcement learning
Keywords:
multi-reward reinforcement learning, large language model alignment, advantage estimation, Pareto dominance, group-relative policy optimization, multi-objective optimization

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_kSKtOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes PODA, an advantage estimator for multi-reward GRPO-style RL. Instead of linearly combining rewards, it builds a Pareto-dominance graph over each prompt’s rollouts. Each rollout is scored by its inverse dominator count plus a signed dominance margin, and the scores are whitened. On math reasoning and BFCL-v3 tool calling with Qwen models, the authors report gains in both accuracy and constraint satisfaction over GDPO, DAPO, and tuned linear and indicator-based estimators.

Critical strengths

The paper studies an under-explored design choice in multi-reward RL for LLMs, that is, how a reward vector is turned into a single scalar advantage, especially when rollouts compete on different objectives. This is a useful framing, given how often reasoning and tool-use training now combine several rewards. However, as the weaknesses below explain, the proposed method and its evaluation do not answer this question convincingly.

Critical weaknesses

Narrow multi-objective evaluation. The main results use only two objectives per task. Prior multi-objective math RL typically uses three or more objectives (e.g., https://arxiv.org/abs/2509.11452). There is also no hypervolume or Pareto-front comparison, which is essential for a method motivated by Pareto dominance.

Small and fragile gains. Gains over the strongest tuned baselines are small (0.91 on math, 0.55 on tools, and even less over DAPO on tools). Confidence intervals are reported only against GDPO.

Questions for authors

Please define each reward function and “joint success.” If the math rewards are binary, no τ≤0.1\tau \leq 0.1 can change the dominance graph. What then explains the 1.3–1.7 variation of math avg acc in Table 15?

Reviewer_yta1OpenReview ↗
Rating
1
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PODA, a partial-order advantage estimator for multi-reward LLM alignment that addresses the limitations of fixed linear reward scalarization. PODA constructs a prompt-local Pareto dominance graph and combines inverse dominance counts with signed dominance margins to generate scalar advantages, while keeping the PPO/GRPO optimization objective unchanged. The method is evaluated on mathematical reasoning and tool-calling tasks with two and three reward objectives across Qwen2.5, Qwen3, and Qwen3.5 models. Experimental results show consistent improvements over GRPO, GDPO, DAPO, and several scalarization and indicator-based baselines, with approximately 1% additional training overhead. The authors conclude that partial-order-based credit assignment provides an efficient alternative to fixed reward scalarization.

Critical strengths
  1. PODA introduces a graph-based advantage estimator that addresses the limitations of fixed reward scalarization while remaining compatible with existing PPO/GRPO training pipelines. Its lightweight design, with approximately 1% training overhead, makes it practically appealing for multi-reward LLM alignment.
  2. The paper evaluates PODA across mathematical reasoning and tool-calling tasks, multiple model scales, and two- and three-reward settings. Comparisons against strong baselines (GRPO, GDPO, DAPO, and indicator-based methods), together with component ablations and training diagnostics, provide meaningful evidence of consistent performance improvements.
Critical weaknesses
  1. The tolerance-based dominance relation in Eq. (3) is not necessarily transitive and may produce cycles, potentially violating the partial-order assumption underlying PODA. This weakens the theoretical justification of the proposed graph-based advantage estimator and requires clarification.
  2. Although PODA consistently outperforms GDPO, its gains over stronger alternatives are relatively small (e.g., +0.20 tool accuracy over DAPO in Table 4). The reported confidence intervals primarily compare PODA with GDPO, leaving the statistical reliability of improvements over the strongest baselines insufficiently established. This limits confidence in the method's incremental advantage.
Questions for authors
  1. The tolerance-based dominance relation in Eq. (3) may violate transitivity and introduce cycles in the dominance graph. How do the authors reconcile this with the partial-order formulation? Could such cycles affect the consistency of advantage estimation, and how are they handled?
  2. Table 4 reports relatively small improvements over the strongest baselines, particularly DAPO and the additive ε-indicator. Can the authors provide paired seed-level results or confidence intervals from the existing experiments to establish whether these improvements are statistically reliable?
Reviewer_gReNOpenReview ↗
Rating
2(−1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PODA (Partial-Order Dominance Advantage Estimation), a multi-reward advantage estimator for LLM reinforcement learning. It addresses scalarization bias in conventional GRPO-style methods, where weighted reward aggregation imposes arbitrary preferences between rollouts with conflicting objectives. PODA instead uses Pareto dominance to construct a prompt-local comparison graph, assigning advantages based on graph structure and dominance strength.

Critical strengths

Simple and practical approach to multi-reward credit assignment. The paper identifies a meaningful limitation of conventional linear reward scalarization and proposes a Pareto dominance-based advantage estimator that avoids imposing direct preferences on conflicting reward vectors. Importantly, PODA can be integrated into existing PPO/GRPO pipelines without modifying the optimization objective, with only approximately 1% computational overhead. This combination of methodological simplicity, practical relevance, and ease of adoption is a key strength.

Critical weaknesses

Limited methodological novelty and theoretical justification. The proposed estimator primarily combines established Pareto dominance principles with heuristic inverse-count and margin-based scoring. Although practically useful, the theoretical analysis mainly establishes basic properties rather than explaining why this particular formulation should yield superior multi-objective optimization. Moreover, the tolerance-relaxed dominance relation is not necessarily transitive, potentially undermining its partial-order interpretation. These limitations weaken the paper's methodological novelty and theoretical contribution.

Questions for authors
  1. The theoretical analysis establishes pairwise order preservation only for two-rollout groups, while the experiments use larger groups. Moreover, the tolerance-relaxed dominance relation is not necessarily transitive, and the margin term may alter the ordering. Can the authors clarify whether PODA preserves dominance-consistent advantage ordering for larger groups and report how frequently violations occur? This is important for assessing whether the estimator fulfills its central motivation.
  2. The paper provides paired confidence intervals primarily against GDPO, whereas improvements over stronger baselines, such as the additive ε-indicator and DAPO, are relatively modest. Can the authors provide seed-level results or paired confidence intervals against these strongest competitors? This would help determine whether PODA's reported advantages are statistically reliable rather than attributable to experimental variability.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.