acceptodds
Under review as a conference paper at ICLR 2027

SAW: Stage-Aware Weighting for Multi-Reward Reinforcement Learning

Abstract

Multi-reward reinforcement learning for language models combines objectives that can improve at different rates. Fixed aggregation does not adapt their weights as training progresses, while gradient-based weighting introduces additional gradient computation. We propose Stage-Aware Weighting (SAW), a lightweight method that adjusts objective weights using coefficients of variation computed from the current rollout batch. SAW operates at the reward level in GRPO and the advantage level in GDPO. Its weighting rule requires neither additional backward passes nor statistics from previous batches. We evaluate SAW on tool calling and summarization. Across six BFCL-v4 settings, SAW improves overall accuracy over default fixed aggregation by 1.09–7.37 percentage points and achieves higher Overall point estimates than the evaluated gradient-based weighting baseline in all six settings. In the 1.5B two-reward setting, SAW also exceeds all three tested fixed ratios in Overall. Summarization gains persist under two evaluation judges not used for training. These results support current-batch reward statistics as a practical and computationally lightweight basis for adaptive multi-reward optimization.

Primary area:
reinforcement learning
Keywords:
large language models, reinforcement learning, multi-reward optimization, adaptive reward weighting

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_sWCaOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Stage-Aware Weighting (SAW), a lightweight method that adjusts objective weights using coefficients of variation computed from the current rollout batch. SAW computes weights using the coefficient of variation of each reward in the current rollout batch and applies them to reward or advantage aggregation. It can be integrated with both GRPO and GDPO without additional gradient computations. The paper further evaluates its effectiveness through ablation studies comparing alternative weighting statistics and fixed reward-weight ratios.

Critical strengths

The paper introduces a stateless weighting rule based on current-batch reward dispersion, integrated at the reward level in GRPO and the advantage level in GDPO.

Critical weaknesses
  1. The writing needs improvement. The paper should be organized in a more logical order, and the motivation should be made clearer.
  2. The effectiveness of SAW remains insufficiently explored. The experiments are limited to tool calling and summarization, without investigating how the proposed weighting mechanism performs as task difficulty increases.
Questions for authors
  1. More details about existed work are necessary and show me the clear motivation and advantage of the proposed approaches.
  2. SAW derives adaptive weights solely from the current rollout batch which is sensitive to batch size. Small batches may yield noisy estimates, causing weight fluctuations that could destabilize training without historical statistics or moving averages. I want to know if we can see varying batch size to assess SAW’s robustness.
Reviewer_y2YBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Multi-reward reinforcement learning involves objectives that improve at different rates, making fixed reward weights potentially unsuitable as training progresses. The paper proposes Stage-Aware Weighting (SAW), which adjusts objective weights using coefficients of variation computed from the current rollout batch. Its main contribution is a stateless weighting rule integrated at the reward level in GRPO and the advantage level in GDPO, without additional per-objective backward passes. Across six tool-calling configurations, SAW improves Overall accuracy over equal weighting by 1.09–7.37 percentage points and achieves higher point estimates than the evaluated gradient-based baseline.

Critical strengths
  1. Simple and lightweight method. SAW adjusts reward weights using readily available batch statistics, introducing little computational overhead and requiring no additional per-objective backward passes or cross-batch state. Its integration into both GRPO and GDPO does not require changing the underlying policy-update rule. This simplicity makes it a practical alternative to more expensive weighting strategies and reduces reliance on manually specified fixed reward weights. 2.Clear presentation and useful empirical validation. The paper is well organized, with a clear progression from the motivation for adaptive weighting to the proposed method and its evaluation. Experiments cover both tool calling and summarization, as well as two optimization frameworks, providing evidence that the approach is not restricted to a single setting. Comparisons with fixed weighting and alternative weighting statistics further help contextualize the reported improvements and support the method’s practical relevance.
Critical weaknesses
  1. The weighting signal has limited justification. The method uses a higher coefficient of variation as a reason to allocate more weight to an objective, but this statistic does not necessarily indicate a greater opportunity for learning. High variability may instead reflect reward noise, heterogeneous prompt difficulty, or an objective that remains difficult for the current model to improve. Although the paper acknowledges some of these limitations, it does not sufficiently establish when CV captures useful learning opportunities rather than incidental variation. This weakens the justification for the weighting rule and leaves its robustness in more complex reward settings uncertain.
  2. Sensitivity to reward specification. SAW’s weighting rule depends on the reward mean relative to a specified lower bound. Consequently, changing the reward offset or the chosen lower bound can alter the relative objective weights, even when the underlying ranking of outputs remains unchanged. The paper does not sufficiently establish how sensitive its results are to these choices.
  3. Unclear robustness to heterogeneous reward coverage. The method relies on batch-level statistics, but its applicability is less clear when different reward objectives are valid for different subsets of samples. Objectives with fewer valid samples may have less reliable CV estimates, potentially resulting in unstable weight assignments. The paper does not explain how missing or unevenly available rewards should be handled, limiting confidence in its applicability to heterogeneous multi-task training.
Questions for authors

The rollout number n is not clearly specified in the experimental setup. Could the authors report this value and evaluate SAW’s sensitivity to different rollout numbers? A smaller n may lead to noisier estimates of the reward statistics used for weighting, depending on how they are aggregated across the batch. It would be helpful to examine how this affects both weight stability and final performance. 2. How sensitive is SAW to the mixture of easy and difficult prompts within a batch? When reward statistics are computed across prompts, a high CV may reflect differences in prompt difficulty rather than variation in response quality for a given prompt. Could the authors investigate whether changing the proportion of easy and difficult prompts substantially affects the assigned weights and SAW’s gains over fixed-weight baselines? 3. How would SAW handle heterogeneous batches in which reward objectives have different numbers of valid samples? For example, in multi-domain training, some rewards may apply only to particular domains, leaving certain objectives with very few valid observations when those domains are underrepresented in a batch. How are inapplicable rewards handled, and can SAW still produce reliable weight estimates under such uneven reward coverage? Does the current formulation support this setting, or does it assume that every sample is evaluated using the same set of reward objectives? 4. There exist other methods like Focal, what is the advantage that compare with such baseline?

Reviewer_RmdTOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work studies the multi-reward RL. The gap is the slope differences across objectives of fixed weights and gradient based methods would introduce additional computational budgets.

SAW solve this issue by edtiing GRPO group reward/advantage based on per-batch variants.

SAW beats the GRPO/gradient-based baseline of multi-objective reward RL on Qwen2.5-1.5B/3B.

Critical strengths
  1. Gradient free method is attractive to community for multi-reward RL design. It potentially provide solutions for those tasks that cannot be measured within 5 optimization goals.
Critical weaknesses
  1. For each of the 3 tasks, I find that the authors design only 2-3 reward objectives. I regard this might be biased to real scenarios where objectives are much more. For example, on creative writing domain, such as creative writing (WritingBench, Wu et al, 2025; HowToBench, Feng et al, 2026), multimodal image generation (MMIG-Bench, Hua et al, 2025), Complex Instruction Following (ComplexBench, Wen et al, 2024), it is easy to find 4-5 optimization objectives. Current composition of correctness, format, length, conciseness are somehow toy for validating the methods.

  2. SAW might not be robust to ill-designed reward functions. For very-hard designed reward directions, or for reward with noise (such as LLM-judge self-biases), the estimation of CV is not validated to be robust.

Questions for authors
  1. For the following reward designs, how would the authors design to integrate into SAW
    1. pass@k method on math/code
    2. unit-test based pass-to-pass/fail-to-pass on SWE-bench series patch valiation
    3. LLM-as-a-judge scores, ranging from 0-10
    4. LLM-as-a-judge based pairwise scores showing A better/B better/Tie
  2. Do the authors believe that the conclusion on Qwen-2.5 1.5/3B models can scale to 10~1000B models and why?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.