acceptodds
Under review as a conference paper at ICLR 2027

APAF: Adaptive Pareto Advantage Fusion for Multi-Reward Policy Optimization

Abstract

As large language models (LLMs) become increasingly capable, post-training reinforcement learning (RL) often relies on multi-dimensional reward signals to shape model behavior. However, fusing potentially conflicting rewards into a single update signal remains a central challenge. Common reward-fusion approaches either linearly combine reward dimensions into a scalar, allowing gains in one dimension to be offset by losses in another, or filter out conflicted rollouts, discarding supervision that can still provide useful learning signal. We propose Adaptive Pareto Advantage Fusion (APAF), a reward-fusion algorithm for multi-reward policy optimization. Within each rollout group, APAF uses Pareto dominance to capture the relative structure among samples while retaining a magnitude-based advantage that preserves reward magnitude information. Its core is a relative-margin control mechanism: for each query, APAF estimates the strength of pairwise Pareto evidence and adaptively sets a bounded mixing weight, determining how much structural signal to trust. In this way, APAF exploits reliable group-level Pareto relations without discarding rollouts or suppressing reward dimensions. Experiments on tool use and safety alignment show that APAF consistently outperforms representative reward-fusion baselines, demonstrating that Pareto-aware advantage fusion provides a principled, general, and effective mechanism for multi-objective post-training.

Primary area:
reinforcement learning
Keywords:
Reward fusion, Multi-objective optimization, Reinforcement learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_EbwBOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes APAF, an advantage estimator for multi-reward GRPO training. Within each rollout group, it blends a magnitude-based advantage with a Pareto-dominance score. A per-prompt weight controls the blend that it grows when responses differ clearly and shrinks for near-ties. Experiments on tool calling and helpfulness–safety alignment report gains over GRPO, GDPO, and GD2PO-Hard.

Critical strengths

Making within-group Pareto relations explicit in the advantage, without discarding conflicting rollouts, is a reasonable and well-motivated idea.

Critical weaknesses

Missing stronger baselines. The paper compares only against GRPO-style baselines, not against established MOO and MTL methods such as MGDA, Nash-MTL, or PAMA. Several of these have already been adapted to LLM alignment (https://arxiv.org/abs/2602.06869), and is proved to be working. Its dominance-preservation guarantee also adds nothing over the existing baselines, since linear advantages already preserve dominance.

The evidence does not support the core claim. The adaptive controller, the paper’s core contribution, is not used in the safety main results. In the tool task, the length reward is saturated for all methods, leaving little trade-off to resolve. The remaining gains are small (about 2 points on API-Bank), and the authors concede they are not statistically reliable.

Questions for authors
  1. The main experiments use two rewards, and three (correctness, format, length) appear only in Appendix D on one model. Does APAF remain effective with more objectives, where most pairs become incomparable and dominance relations grow sparse
Reviewer_YjH3OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes APAF, a reinforcement learning algorithm that addresses the challenge of combining multiple, potentially conflicting rewards in LLM RL. APAF incorporates pairwise Pareto dominance into the advantage estimation and adaptively combines the magnitude information with the Pareto dominance information. The method is evaluated on tool calling and alignment benchmarking. Experimental results show that APAF consistently achieve higher scores than other RL baseline without additional networks or reward models.

Critical strengths
  • The paper is well motivated by the need to address multi-reward mixing during RL.
  • The method doesn't require additional reward model, forward passes, or value functions for the reward assignment.
  • The method provides consistent improvement on benchmarks over other multi-reward RL baselines.
Critical weaknesses
  • It's unclear how magnitude base and bounded Pareto correction is obtained. The paper lacks detail in the mathematical formulation.
  • While the paper conducts ablation on different design choices, it doesn't provide any numerical values. The gap looks extremely small in figure 5c. It's hard to understand what each design brings by looking at the figure.
  • The experiments are conducted on relatively old models. It would strengthen the paper to include newer models like Qwen3.5.
  • The experiments focus on only tool calling and safety alignment and lacks diversity on a broader set of tasks.
  • In Appendix B.1, the paper states that "The Safety main results evaluate this base". Does this mean for the safety benchmarks, the adaptive fusion mechanism is not used, and what would be the reason for it?
Questions for authors

N/A

Reviewer_thYZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes APAF, a method for reinforcement learning with multiple conflicting rewards in LLM post-training. APAF combines a magnitude-sensitive base advantage with a Pareto-structured proposal derived from within-group dominance relations and bottleneck-based ranking. Experiments on API-Bank and helpfulness–harmlessness alignment tasks demonstrate consistent improvements. Further analyses suggest that controlled Pareto-structural correction is a key contributor to these gains.

Critical strengths

The paper introduces a principled approach to incorporating within-group Pareto structure into scalar advantages without discarding conflicting rollouts.

It also proposes a per-query adaptive controller based on relative pairwise margins, which helps mitigate the amplification of reward noise introduced by Pareto-structured corrections.

Provides ablations and sensitivity analyses that probe the role of the structural dose.

Critical weaknesses

The magnitude-sensitive base advantage BnB_n is a central component of APAF, but its mathematical formulation is not explicitly provided. The authors should clarify how BnB_n is computed to improve the clarity and reproducibility of the proposed method.

The paper does not adequately discuss several closely related studies on Pareto-based multi-objective optimization for LLMs. In particular, [1] explores the use of Pareto dominance relationships to construct learning signals, while [2] also incorporates Pareto ranking into advantage computation for multi-objective LLM reinforcement learning.

APAF is motivated by multi-objective optimization and explicitly incorporates Pareto dominance into advantage estimation, the experimental evaluation primarily relies on individual reward scores and their fixed-weight aggregates. I would encourage the authors to include more explicit multi-objective analyses, such as Pareto-front or hypervolume evaluations across multiple operating points. Such experiments would provide stronger evidence that APAF improves multi-objective optimization.

APAF introduces several interacting components, including a Pareto-derived residual in the magnitude base, residual orthogonalization and bounding, dominance counts, bottleneck ranking, and a relative-margin-based adaptive fusion mechanism. However, the current ablation studies do not sufficiently isolate the contributions of these individual components.

Figure 5(b) exhibits a non-monotonic and seemingly sensitive relationship between structural control strength and performance. In particular, a small control strength of 0.05 decreases the Overall score relative to zero correction (−0.029), whereas increasing the strength to 0.10 produces the largest improvement (+0.035). This behavior is not fully explained by the authors' interpretation that moderate structural correction is beneficial while weaker interventions are merely insufficient.

[1] Zhao, Guang, et al. "Pareto prompt optimization." International Conference on Learning Representations. Vol. 2025. 2025. [2] Li, Junyi, et al. "Towards Pareto-Optimal Tool-Integrated Agents with Pareto Ranking Policy Optimization." arXiv preprint arXiv:2606.16111 (2026).

Questions for authors

See Weakness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.