acceptodds
Under review as a conference paper at ICLR 2027

Multi-Reward Hacking: From Aggregation-Induced Drift to Trust-Anchored Defense

Abstract

Modern LLM post-training increasingly combines several reward signals into one training objective. We show that symmetric aggregation of such signals can induce multi-reward hacking: the aggregate reward improves while performance on a primary criterion degrades, as the policy increasingly adopts a behavior that satisfies an auxiliary channel without satisfying the primary one. Formulating post-training as a one-step multi-reward MDP, we analyze a two-channel, three-action instance and prove that, under symmetric aggregation, the deterministic conditional-mean GRPO dynamics select the auxiliary action whenever the primary action is rewarded less often outside a joint-success component that both actions share; equal access to joint success does not remove the drift that this reward-frequency gap induces. Whether this selection is harmful depends on the unknown true reward, which we keep separate from the channels. The failure raises two questions: which channel should receive priority and how. Trust-Anchored Gating (TAG) first elects an anchor channel from the drift that each channel induces when optimized alone, then conditions auxiliary credit on the elected anchor structurally; static reweighting toward the anchor is instead calibration-sensitive, and every fixed weight we test still collapses. In the same model TAG selects the primary action from every initialization whenever the objectives are not always compatible, for every finite gating weight. Across Qwen and Llama models on mathematics and program synthesis, symmetric, constrained and regularized aggregators show the predicted degradation while TAG keeps primary performance while retaining useful auxiliary gains when the rewards are compatible.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, reinforcement learning, reward hacking, reward aggregation, multi-reward hacking, GRPO

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_fezMOpenReview ↗
Rating
2
Confidence
1(−2)
Informativeness
2
Presentation
3
Soundness
2(+1)
Read review
Paper summary

This paper studies how to aggregate multiple rewards in LLM RL, which is an interesting problem and has been explored in quite a few previous works. The main argument is that simply adding signals together can cause secondary signals to dominate training. Here, secondary signals refer to behaviors that are easier for the model to learn (e.g. shortening the output is much easier than improving accuracy). As a result, training can end up going in the wrong direction. To address this, the paper proposes a conditional design: first check correctness, and only consider length once correctness is satisfied. The authors show that this leads to better training outcomes.

Critical strengths

This paper tackles a relatively interesting problem and provides reasonable analyses from both theoretical and empirical perspectives. The overall writing is quite coherent: it starts with the phenomenon of multi-reward hacking, highlights the limitations of simply adding reward signals together, and eventually motivates the conditional design.

Critical weaknesses
  1. The techniques discussed in this paper have substantial overlap with prior work. For example, GDPO [1] (Section 4.2.1) already covers many of the ideas discussed here: an easy-to-optimize length reward can hurt the harder correctness objective; simply reducing the weight of the length reward does not necessarily ensure that correctness is prioritized; and the length reward should only be given when the answer is both correct and satisfies the length constraint. Given this overlap, I feel that the contribution of this paper is relatively incremental.

  2. The practical value of the Trust-Anchored Gating (TAG) method seems rather limited. In LLM RLVR, the choice of the primary reward is usually pretty obvious: it's the success rate. The paper also doesn't seem to analyze any setting where success rate is not chosen as the primary reward. This, in my view, significantly weakens the practical motivation for TAG.

[1] GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimization (ICML 2026)

Questions for authors

Could the authors elaborate more on TAG? While it seems like a reasonable design, I'm not sure whether there are practical scenarios in LLM RL where it would actually be useful.

5 comments
AuthorsOpenReview ↗

Response to Reviewer fezM (Part 1 of 2)

We thank the reviewer for recognizing the theoretical and empirical analyses and the coherence of the argument from multi-reward hacking to the conditional design. The responses below clarify how TAG relates to GDPO's conditional reward and the two respects in which our paper differs, the election of the anchor and the theoretical explanation (W1), and show two practical uses of TAG: in safety and compliance training, where the elected anchor is safety, and with a process reward model as the auxiliary reward (W2 & Q1).

W1: Relationship to GDPO

With a binary correctness anchor, TAG and GDPO's conditional reward use the same two terms, correctness and a correctness-conditioned auxiliary reward. GDPO normalizes each as a separate channel, where TAG normalizes their weighted sum. Our paper cites GDPO for its per-channel normalization of the advantage, and the revision adds a discussion of its conditional reward. Our paper differs in two respects: it elects the anchor from data, where GDPO takes the primary reward as given, and it explains through theory why symmetric aggregation fails and why the gate selects the primary behavior.

First, the selection of the anchor. GDPO uses a predetermined anchor, without looking at the training dynamics of the reward channels. Our anchor election requires no knowledge of what each channel measures: it selects the anchor from the drift each channel induces when optimized alone.

Second, the theory: which behavior aggregation selects, and why. In the two-channel model, Theorem 1 characterizes the behavior that symmetric aggregation selects from the rates at which the two rewards are realized, and Theorem 2 shows that the gate changes the selected behavior from every initialization. The credit bias is measured in the training rollouts: over the first 20 training steps of Correct & negative on Qwen3-1.7B, auxiliary-only responses receive a positive mean advantage under (R_P + R_A)/2 and a negative one under TAG, and the training trajectories show the same tendency (Appendix E.1, Table 13, lines 1736–1756). Our paper already includes MO-GRPO, which like GDPO computes the advantage per channel before summing, as a baseline in Table 1, where it collapses in all four settings.

We also trained GDPO as specified, with per-channel group normalization followed by batch-wise normalization of the summed advantage, and GDPO with the conditional reward, under the Table 1 protocol on Qwen3-1.7B math (Appendix E.6, Table 19, lines 1907–1927 of the revision). The table follows Table 1, with pass@4 and response length in Correct & brief, and pass@4 and the negative-answer rate in Correct & negative:

MethodCorrect & brief: pass@4 ↑len ↓Correct & negative: pass@4 ↑neg. rate ↓
R_P-only (GRPO)48.1%32848.1%3.6%
MO-GRPO15.1%93.3%99.9%
GDPO14.9%91.7%100%
GDPO (conditional reward)40.0%19546.0%3.5%
TAG45.3%22049.4%3.3%

GDPO collapses like (R_P + R_A)/2 and MO-GRPO in Table 1. With the conditional reward, GDPO avoids the collapse as TAG does: conditioning the auxiliary reward on correctness protects correctness under both methods. A likely reason for the difference in pass@4 is that TAG standardizes the gated reward as one scalar, while GDPO standardizes the bonus as a separate channel and gives it the same weight as correctness.

AuthorsOpenReview ↗

Response to Reviewer fezM (Part 2 of 2)

W2 & Q1: Practical motivation and applications of TAG

TAG is for training that must protect primary performance while optimizing an auxiliary reward, and the primary reward can be something other than correctness. With verifiable rewards the primary reward is correctness; with reward models it is often safety. The revision adds two experiments in more realistic settings. In the first, the two rewards are a safety reward model and a compliance reward model, the primary reward is safety and no success rate is involved. In the second, the auxiliary reward is a process reward model (PRM), a learned model that scores the intermediate steps of a math solution. In both, anchor election selects the primary reward: safety in the first and correctness in the second.

In safety and compliance training, TAG reaches the best balance of the two objectives, with safety elected as the anchor. The task is to train an assistant that helps with benign requests and stays safe on harmful ones, and both rewards come from reward models (Appendix E.4 of the revision, lines 1830–1857). Qwen3-1.7B is trained on Aegis 2.0 prompts for 100 steps with R_P = appropriate safety (judged by Qwen3Guard-Gen-8B) and R_A = instruction compliance (judged by Qwen3-32B); the two conflict on harmful requests. Anchor election selects safety, the probe with the least drift at step 100. Because a policy can hack the reward models it is trained on, evaluation uses different models: Granite Guardian 3.1-8B scores safety on harmful requests and Claude Opus 4.7 scores compliance on benign requests.

MethodSafety ↑Compliance ↑Mean ↑
Qwen3-1.7B93.8%80.3%87.0%
R_P-only (safety)96.0%88.8%92.4%
R_A-only (compliance)71.9%96.0%83.9%
(R_P + R_A)/288.4%94.0%91.2%
0.9 R_P + 0.1 R_A91.2%92.4%91.8%
TAG93.9%92.4%93.1%

TAG has the highest mean score: it keeps safety on harmful requests at the level of the base policy while raising compliance on benign requests, where (R_P + R_A)/2 trades safety for compliance. The training signal shows the mechanism: under (R_P + R_A)/2 some responses that the safety reward model labels unsafe receive positive advantage; under TAG none do.

With a PRM as the auxiliary reward, TAG gains 2.3–3.9 points over R_P-only on the five held-out benchmarks; (R_P + R_A)/2 and 0.9 R_P + 0.1 R_A with the same PRM collapse. A PRM scores each reasoning step of a solution, which makes it a natural auxiliary reward and also a hackable one: a policy can raise its score with plausible steps that do not reach the answer. We train Qwen3-1.7B with GRPO for 500 steps, with R_P = final-answer correctness and R_A = a frozen PURE-PRM-7B step score; the table reports held-out pass@1:

Methodtrace-val (in-domain)MATH-500OlympiadBenchMinerva MathOmni-MATHBigMath-hard
R_P-only37.8%70.1%34.5%22.4%20.6%21.1%
R_A-only0.0%0.9%0.0%0.5%0.3%0.0%
(R_P + R_A)/215.8%56.6%21.1%20.5%12.5%10.4%
0.9 R_P + 0.1 R_A17.3%58.7%21.9%21.3%12.7%11.8%
TAG39.3%72.5%37.1%26.3%22.9%23.6%

The policies trained with (R_P + R_A)/2 and 0.9 R_P + 0.1 R_A learn to write plausible steps and focus less on reaching an answer; TAG uses the same PRM to rank correct responses and gains from it on every benchmark in the table.

Dear Reviewer fezM,

As the author–reviewer discussion period draws to a close on Nov. 18, we would appreciate any further thoughts you may have on whether our response addresses your main concerns.

Thank you again for your time and consideration!

Reviewer_Cm7rOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper identifies multi-reward hacking: when several reward channels are combined into one scalar for GRPO-style post-training, the aggregate reward can rise while the primary criterion degrades, because the policy increasingly adopts behavior that satisfies only an auxiliary channel. The authors model post-training as a one-step, two-channel, three-action MDP with an unknown true reward kept separate from the channels. They prove (Theorem 1) that under any symmetric, monotone aggregator, the conditional-mean GRPO dynamics converge to the auxiliary action whenever the primary action is rewarded less often outside a shared joint-success component (η,ρ<1\eta, \rho < 1), and they characterize when this selection constitutes reward hacking (e.g., ρwP>wA\rho w_P > w_A for a linear true reward). As a defense, they propose two steps. Anchor election trains each channel alone briefly and elects the one with the smallest per-token reverse KL from the base policy as the anchor. Trust-Anchored Gating (TAG) then credits auxiliary rewards only when the anchor is satisfied, via Rk⋆(1+∑kλkRk)/(1+∑kλk)R_{k^\star}(1+\sum_k \lambda_k R_k)/(1+\sum_k \lambda_k), and provably selects the primary action from every finite initialization (Theorem 2). Experiments on Qwen3 (1.7B–30B) and Llama-3 (3B/8B) cover mathematics (Big-Math) and program synthesis (DeepCoder), with both compatible (brevity, speed) and incompatible (negative answer, no-op program) auxiliary channels. Symmetric, constrained, normalized and regularized aggregators collapse, and fixed reweighting delays but does not reliably prevent collapse. TAG keeps primary performance near the RPR_P-only reference while retaining part of the auxiliary gain when the channels are compatible.

Critical strengths
  1. A clear, well-supported account of why naive aggregation fails under GRPO. The explanation in §3.2 (“Why symmetric rules fail”) is the paper’s most valuable insight. Outside joint success, the auxiliary behavior is rewarded in every sample and the primary behavior only with probability ρ\rho, so the auxiliary behavior is reinforced faster and on-policy sampling compounds its lead. It is complemented by an observation about GRPO specifically. Group z-scoring is affine-invariant, so in groups with no correct response the rewards take two values. An auxiliary-only response then receives the full advantage however small its weight, which makes static down-weighting unsafe on hard prompts (§4.2, App. B.3). This is concrete, practically relevant to anyone combining verifiable rewards with format or length terms, and it motivates gating over reweighting.
  2. Broad and candid empirical evaluation. The phenomenon is reproduced across two model families, four scales, math and code, compatible and incompatible channels, and about ten aggregation baselines. The authors also report honest results that cut against their own method. They include informative controls as well: the wrong-anchor ablation and the three-channel cascade. This honesty makes the headline results more credible
  3. TAG is simple, cheap and robust to its only hyperparameter. The method is a one-line change to the reward with no additional models, critics or training-time monitoring, so it is trivially adoptable in existing GRPO pipelines. This matters because it shifts the practitioner's burden from tuning a fragile weight to choosing an anchor, which is a more interpretable decision.
Critical weaknesses
  1. Anchor election, the most novel component, is under-validated and possibly confounded by response length. Every auxiliary probe produces short outputs: about 9-token answer templates, about 33-token negative answers, and one-line constant-printing programs. The election statistic divides each response’s log-ratio by its own length, which App. C admits is not a divergence and weights short-response modes heavily. The reported 2.8–106× separation may therefore be largely an artifact of dividing by about 9 versus about 300 tokens. The evidence base is also small. Every setting has two channels, correctness is always the correct answer, and the auxiliaries are degenerate by construction. That amounts to a handful of easy binary decisions. The method also relies on the assumption at line 314 that at least one unhackable channel exists. If the slowest-drifting channel is itself hackable, TAG gates all credit on the exploit. The wrong-anchor control shows how badly this fails. Since the guarantees of §4.2 hold only if the anchor plays its intended role, this is consequential.

Minor Weaknesses:

  1. Overclaim in the abstract. “Every fixed weight we test still collapses” conflicts with 0.9/0.1 not collapsing at 4B. The weight sweep (Fig. 8) covers only wA≥0.1w_A \ge 0.1 for one model and setting.
  2. Choice of algorithm. All experiments use (variants of) GRPO. More recent algorithms such as DAPO (dynamic sampling, token-level loss) and GSPO (sequence-level ratios) are now standard and change exactly the group-level statistics the analysis depends on. The clip-higher 0.2/0.28 setting is borrowed from DAPO, but its other components are not. DAPO’s dynamic sampling, for instance, removes zero-variance groups but not the two-level {0,wA}\{0, w_A\} groups that drive the reweighting failure. Testing whether collapse persists under these algorithms would strengthen the generality claim.
  3. Formatting.
    • Figure 2 is not a pdf so it loses quality when zoomed; please fix this.
    • Around lines 393–394, the Table 1 caption overlaps Figure 4.
    • “46925” in Fig. 1 should render as 4692546^{925}.
    • Several sections (§3.2, §4.1) are very dense and would benefit from shorter sentences.
  4. I am heavily against putting related work in the appendix. Related work is a very important component of a paper and it sets the expectation and the environment the paper is compared and placed against. I highly recommend to find room for it.
  5. Reproducibility. No code is provided and there is no reproducibility statement. This matters especially because the election’s exclusion criteria, thresholds (“frozen on Qwen3”), confirmation checks, and probe length TT are not fully specified.
Questions for authors
  1. What exactly are the exclusion criteria, thresholds, confirmation checks, and probe length TT used in election? How were the thresholds chosen?
  2. Line 314 assumes an unhackable channel exists. What happens when it does not, e.g., a hackable unit-test verifier as the primary channel alongside a legitimate auxiliary channel? Does election detect this, or does it confidently elect the hackable channel?
  3. How does TAG compare against a conditional reward that applies the auxiliary bonus or penalty only to correct responses, as in existing length-control methods? What does TAG offer beyond this?

A few suggestions:

  1. Evaluate the failure and TAG under DAPO and GSPO.
  2. Include a setting where both channels are legitimate, or where the correct anchor is not correctness.
  3. Move related work into the main text, fix the figure-quality and overlap issues, and release code with a reproducibility statement.
4 comments
AuthorsOpenReview ↗

Response to Reviewer Cm7r (Part 1 of 2)

We thank the reviewer for recognizing our account of why symmetric aggregation fails as the most valuable insight and the breadth and candor of the experiments. The responses below show that anchor election selects correctness when the auxiliary reward favors longer responses and on equal-length prefixes (W1), revise the fixed-weight statement of the abstract (W2), show that the failure persists under GSPO and DAPO while TAG avoids the collapse (W3), specify the election rule (W4 & W5, Q1), examine the case in which every channel is hackable (Q2), and compare TAG with correctness-conditioned rewards (Q3).


W1: Anchor election and response-length confounding

Anchor election selects correctness in three new settings where the auxiliary probe writes longer responses than the correctness probe. A probe is the policy trained on one channel alone, as in the paper's anchor election. The per-token statistic was designed to reduce the influence of length, and in the paper's settings every auxiliary probe shortens its responses, so short length and large drift coincide, as the reviewer observes. The new probes separate the two, since each auxiliary reward is maximized by a longer response that does not solve the problem: response length above 512 tokens and the number of reasoning steps, both rule-based rewards on Qwen3-1.7B math, and the step score of PURE-PRM-7B, the PRM used as the auxiliary reward in the revised paper (Appendix E.3, lines 1782–1828 of the revision). The table reports each probe's drift from the base policy at step 100, per token and on the first 32 tokens, with the multiple of the correctness probe's drift in parentheses:

SettingProbe rewardMean lengthPer-token driftFirst 32 tokens
Rule-based rewardsCorrectness7160.0240.081
Response length1,0030.088 (3.7×)0.290 (3.6×)
Reasoning-step count1,0210.221 (9.2×)0.721 (8.9×)
Learned PRMCorrectness7330.0060.009
PRM step score9710.085 (15×)0.189 (21×)

Every auxiliary probe writes about 1,000 tokens and loses accuracy, and both statistics elect the same anchor: dividing by length works against the longer probe, yet its per-token drift remains 3.7–15× that of the correctness probe, and the first-32-token statistic, which scores the same number of tokens for every probe, gives the same ranking. With the PRM as the auxiliary reward in full training, TAG gains over R_P-only while (R_P + R_A)/2 and 0.9 R_P + 0.1 R_A collapse (Table 16, lines 1808–1828).

The separation in Figure 5 also holds at equal length. We recompute the drift of the Figure 5 probes on the same number of leading tokens for both probes, the median length of the shorter one: the auxiliary probe still drifts 1.4–4.9× further than the correctness probe in the four math settings and 2.5× further in program synthesis, and the anchor ranking is unchanged in every setting.

The election also covers the other cases the reviewer lists. With three channels in one pool, correctness and the two rule-based rewards, it ranks correctness first, and in safety and compliance training (Appendix E.4, lines 1830–1857), where both rewards are reward models, it selects safety. Q2 covers the remaining case, an anchor that is itself hackable.


W2: Fixed-weight claims

We agree and have revised the abstract. The sentence summarized the sweep of the auxiliary weight w_A in Figure 8 (Appendix D.1 of the paper), Qwen3-1.7B in Correct & negative, where every w_A collapses. On Qwen3-4B in the same setting, 0.9 R_P + 0.1 R_A does not collapse and reaches 61.6% pass@4, below both TAG (63.2%) and R_P-only (63.4%). The revised abstract reads: static reweighting toward the anchor is calibration-sensitive, and a fixed weight that preserves the primary objective in one setting fails in others.


W3: Generalization to GSPO and DAPO

The failure persists under GSPO and DAPO, and TAG avoids the collapse under both. The table reports held-out trace-val results on Qwen3-1.7B in the format of Table 1.

OptimizerAggregationCorrect & brief: pass@4 ↑len ↓Correct & negative: pass@4 ↑neg. rate ↓
ReferenceBase model14.0%47914.0%0.7%
GSPOR_P-only49.8%36149.8%3.4%
(R_P + R_A)/228.1%793.6%99.9%
TAG48.6%23649.4%3.9%
DAPOR_P-only52.0%34652.0%3.6%
(R_P + R_A)/216.5%223.4%100%
TAG46.4%20050.9%3.8%

Under both optimizers (R_P + R_A)/2 collapses as it does under GRPO, to negative answers in one setting and to much shorter, less accurate answers in the other, while TAG avoids the collapse and, in Correct & brief, trades some accuracy for shorter responses.

AuthorsOpenReview ↗

Response to Reviewer Cm7r (Part 2 of 2)

W4 & W5, Q1: Presentation, reproducibility and the election rule

The revised manuscript states the election rule in full: the channel whose probe has the lowest drift from the base policy is the anchor, and every other channel is auxiliary. Each channel is trained alone with GRPO from the same base policy for 100 steps, and drift is the mean per-token log-ratio between the probe and the base policy on the probe's responses to held-out prompts. The rule ranks the channels and uses no absolute threshold. The exclusion criteria and confirmation checks of Appendix C applied only to excluding a high-drift channel and never changed a selection, so the revised rule is the ranking alone (Appendix C, lines 1249–1252 of the revision). The revision also adds a reproducibility statement with all training and evaluation configurations, and the code will be released with the camera-ready version (W5). It moves the core related-work discussion into the main text, fixes the rendering of Figure 2 and the math in Figure 1, removes the Table 1 / Figure 4 overlap, and shortens the theory exposition (W4).


Q2: When no reliable channel exists

When every channel is hackable, the election selects the more reliable one, and TAG performs like training on that channel alone and collapses later than (R_P + R_A)/2. We ran a new program-synthesis experiment in which both channels are hackable. One channel rewards passing the unit-test examples printed in the problem statement, which a policy exploits by hard-coding them; the other rewards finishing a large input in under 3 ms, which only a trivial program achieves, so a policy exploits it with a no-op program. Hidden tests measure true correctness. The election at step 100 selects the statement-example channel, whose probe drifts less (0.017 against 0.037 per token); R_P denotes this elected channel and R_A the runtime channel. R_P is also the more reliable channel before any training: 35.2% of the base-policy responses it rewards pass the hidden tests, against 5.6% for R_A. The table reports hidden-test pass@4 at each arm's peak over 500 steps, the no-op rate at that step, and the collapse step, the first evaluation after the peak at which pass@4 falls below the base policy:

Methodpass@4 ↑ (step)no-op rate ↓Collapse step
Base policy12.3%2.6%—
R_P-only (anchor alone)16.6% (119)4.0%199
R_A-only15.5% (39)2.4%79
(R_P + R_A)/214.7% (79)4.5%179
TAG16.6% (59)2.9%219

TAG matches the peak accuracy of R_P-only and falls below the base policy one evaluation later, at step 219 against 199. R_A-only turns to no-op programs within a hundred steps, and (R_P + R_A)/2 peaks lower and collapses before both R_P-only and TAG; TAG pays R_A only to programs that pass the statement examples. R_P is itself hackable, and every arm that uses R_P, TAG included, is eventually hacked through it.


Q3: Comparison with correctness-conditioned auxiliary rewards

TAG elects the anchor that a correctness-conditioned bonus takes as given. With a binary anchor, TAG pays the auxiliary reward only to anchor-satisfying responses, so the design the reviewer describes corresponds to TAG with correctness as the anchor. The anchor is elected rather than assumed: in safety and compliance training the election selects safety (Appendix E.4, lines 1834–1835). The theory characterizes auxiliary-behavior selection under symmetric aggregation (Theorem 1) and establishes conditions under which gating selects the primary behavior (Theorem 2).

Thank you for the additional experiments and clarifications. Based on my overall impression of the paper and the other reviews, I think my original assessment of the paper is fair and correct, therefore I will maintain my rating.

Thank you for considering our response and revised manuscript. We appreciate your recognition of the paper's contributions. We believe the additional experiments and clarifications meaningfully strengthen the empirical support for our findings, and we hope they will be valuable in the final discussion.

Reviewer_4KmGOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies reward hacking arising in multi-reward setups, specifically when aggregating the multiple reward signals. The authors analyze the failure of symmetric reward aggregation using a simplified one-step MDP and propose trust-anchored gating (TAG), which conditions auxiliary reward on satisfying a primary rewards, therby preventing auxiliary rewards from dominating the total reward. Experiments on mathematical reasoning and code generation show that TAG can mitigate performance degradation observed with several reward aggregation strategies.

Critical strengths
  • The paper addresses an interesting and relevant problem in multi-reward RL, particularly as LLM post-training increasingly incorporates multiple objectives. The paper offers some insights in that setup that could be useful to RL practitioners,
  • The empirical results demonstrate clear failure modes of reward aggregation and the proposed method appears effective in mitigating these failures across several experimental settings
Critical weaknesses
  • The theoretical results are established using a highly simplified one-step MDP with only three actions and two reward channels. While this provides some intuition about the failure mode, it is unclear to me to what extent the conclusions extend to realistic LLM RL training, where the action and reward spaces are much more complex.

  • The paper places strong emphasis on the failure of symmetric reward aggregation. However in practical LLM post-training, reward signals are typically weighted or prioritized according to their relative importance. E.g., in math reasoning, the format reward is always assigned a much lower weight than the correctness reward, which mostly resolves the issues this paper aims to solve. It is therefore unclear how representative the studied failure mode is of realistic training setups.

  • The central finding that an easier-to-optimize auxiliary reward can dominate training at the expense of a more difficult primary objective is quite intuitive and consistent with established challenges in multi-objective RL and reward hacking.

  • Limited novelty of TAG: TAG essentially conditions auxiliary reward on satisfying the primary objective. This is a relatively straightforward reward design that resembles differently weighing the rewards. The paper could better clarify its conceptual novelty relative to existing approaches. In addition, reward gating may suppress useufl learning signal especially with the primary reward is sparse.

  • Artificial experimental settings: the explored auxiliary rewards such as rewarding negative answers regardless of correctness or extremely fast program execution, create relatively obvious opportunities for reward hacking. It remains unclear whether the method offers similar benefits in more realistic settings with less trivial conflicts between objectives. E.g., in RLHF settings where helpfulness may conflict with safety.

Questions for authors
  1. What happens when the primary reward is sparse and auxiliary rewards provide useful intermediate supervision? e.g., the auxiliary reward comes from step correctness model or PRM?
4 comments
AuthorsOpenReview ↗

Response to Reviewer 4KmG (Part 1 of 2)

We thank the reviewer for recognizing the relevance of multi-reward RL to LLM post-training, the practical insights of our work, and the experiments showing clear failures of reward aggregation that TAG mitigates across multiple settings. The responses below show that training follows the same tendency as the theoretical model (W1), show how weighted aggregation behaves in practice (W2), explain the mechanism behind the failure and what is new in TAG (W3 & W4), evaluate it with a process reward model (Q1), and test it in safety and compliance training (W5).


W1: Generality of the theoretical model

We agree that the one-step MDP is a simplified model for analysis, yet the behavior it selects is the one we observe empirically in LLM RL training. The aggregation rule enters training only through the reward of a complete response, so we model the response as one action. With a terminal reward and sequence-level GRPO, a complete response receives one aggregated reward and one group-normalized advantage, shared by all of its tokens, so the rewards compete at the level of complete responses regardless of generation length. The model then simplifies the policy to a distribution over the behaviors a complete response can show: earning neither reward (a_0), pursuing the auxiliary reward (a_1) or pursuing the primary reward (a_2).

The model describes which behavior each aggregation selects: the auxiliary behavior under symmetric aggregation (Theorem 1) and the primary behavior under TAG (Theorem 2). GRPO shifts the policy toward responses with higher advantage, and the advantages in the training rollouts show the same tendency. In the Table 1 runs on Qwen3-1.7B under Correct & negative, where joint success is rare, a response that earns only the auxiliary reward (R_P = 0, R_A = 1) has a mean group-normalized advantage over the first 20 training steps of +0.61 under the additive rule (R_P + R_A)/2 (Additive for short) and −0.17 under TAG, so the update favors such responses under Additive and disfavors them under TAG. The table shows how the shares of auxiliary-only and correct (R_P = 1) responses in these runs change between the first and last 20 training steps.

AggregationAuxiliary-only responsesCorrect responses
Additive0.4% → 97.6%8.9% → 2.1%
TAG0.5% → 1.2%8.4% → 33.8%

The two runs begin with similar outcome frequencies and then diverge: (R_P + R_A)/2 concentrates on auxiliary-only responses as correctness falls, while TAG increases correctness and keeps auxiliary-only responses rare. Under GSPO and DAPO the held-out results follow the same pattern: (R_P + R_A)/2 collapses and TAG avoids the collapse (Appendix E.2, Table 14, lines 1759–1781 of the revision).


W2: Practical reward weighting

We considered weighted aggregation, (1 − w_A) R_P + w_A R_A with auxiliary weight w_A, as a baseline in our experiments, and in most settings a smaller w_A delays the collapse without preventing it. Table 1 of the paper includes 0.9 R_P + 0.1 R_A (w_A = 0.1) in all four settings, and it collapses in three of them. Figure 8 in Appendix D.1 of the paper sweeps w_A from 0.5 to 0.1 on Qwen3-1.7B in Correct & negative. For these runs, the table gives the collapse step, the first training step at which more than half of the training rollouts give a negative answer, the peak held-out pass@4 and the final value:

ObjectiveCollapse stepPeak pass@4 (step)pass@4 at step 499
R_P-only—49.9% (459)48.1%
(R_P + R_A)/218936.2% (179)2.6%
0.6 R_P + 0.4 R_A21939.4% (159)3.3%
0.7 R_P + 0.3 R_A25442.8% (219)3.3%
0.8 R_P + 0.2 R_A24641.5% (199)3.5%
0.9 R_P + 0.1 R_A42546.3% (339)4.0%
TAG—49.4% (499)49.4%

A smaller w_A generally moves the collapse later, and every w_A in the sweep still collapses. A value of w_A that protects one model and one reward fails on another and has to be tuned per setting.

AuthorsOpenReview ↗

Response to Reviewer 4KmG (Part 2 of 2)

W3 & W4: The mechanism behind the failure, and the novelty of TAG

Our paper identifies the mechanism behind this intuition and the condition under which it holds. Theorem 1 shows that, symmetric aggregation selects the auxiliary behavior whenever a response pursuing the primary reward can fail to earn it. Group normalization further explains why a smaller w_A can fail: in a group with no correct answer and varying auxiliary rewards, reducing w_A > 0 scales both the reward differences and their standard deviation, leaving the standardized advantages unchanged. The experiments show that collapse persists under both weighted aggregation and per-channel normalization.

TAG changes which responses receive credit, while weighted aggregation changes only how much. A smaller w_A still rewards an auxiliary-only response. TAG gives it zero reward, so its group-normalized advantage is non-positive, and uses the auxiliary reward to rank correct responses; Theorem 2 shows that this selects the primary behavior. Beyond conditioning a bonus on correctness, our paper explains why symmetric aggregation fails, gives the conditions under which the gate selects the primary behavior, and elects the anchor from data.

On signal sparsity, the relevant comparison is the AND rule min(R_P, R_A), the direct way to deny credit to auxiliary-only responses. AND rewards only responses that satisfy both channels, fewer than 1% of base-policy responses in the Table 1 settings. TAG rewards every correct response and uses the auxiliary reward to rank them, so its signal is as dense as the primary reward itself, and TAG is above AND in every setting of Table 1.


Q1: Intermediate supervision with a PRM when the primary reward is sparse

With a PRM as the auxiliary reward, TAG gains 2.3–3.9 points over R_P-only on the five held-out benchmarks; (R_P + R_A)/2 and 0.9 R_P + 0.1 R_A with the same PRM collapse. With a binary anchor, TAG uses the PRM score to rank the correct responses of a group, and a group with no correct response receives zero advantage, as it does under R_P-only. Around 50% of the training groups have no correct response; the PRM refines comparisons among correct responses, and this refinement improves on R_P-only on all five held-out benchmarks. We train Qwen3-1.7B with GRPO for 500 steps, with R_P = final-answer correctness and R_A = a frozen PURE-PRM-7B step score; the table reports held-out pass@1, and the setup and these results are in Appendix E.3 of the revision (Table 16, lines 1808–1828):

Methodtrace-val (in-domain)MATH-500OlympiadBenchMinerva MathOmni-MATHBigMath-hard
R_P-only37.8%70.1%34.5%22.4%20.6%21.1%
R_A-only0.0%0.9%0.0%0.5%0.3%0.0%
(R_P + R_A)/215.8%56.6%21.1%20.5%12.5%10.4%
0.9 R_P + 0.1 R_A17.3%58.7%21.9%21.3%12.7%11.8%
TAG39.3%72.5%37.1%26.3%22.9%23.6%

The policies trained with (R_P + R_A)/2 and 0.9 R_P + 0.1 R_A learn to write plausible steps and focus less on reaching an answer; TAG uses the same PRM to rank correct responses and gains from it on every benchmark in the table.

W5: A realistic setting: safety and compliance training with reward models

In safety and compliance training, TAG reaches the best balance of the two objectives, with safety elected as the anchor. The setting follows the reviewer's suggestion. Qwen3-1.7B is trained on Aegis 2.0 prompts for 100 steps with R_P = appropriate safety (judged by Qwen3Guard-Gen-8B) and R_A = instruction compliance (judged by Qwen3-32B); the two conflict on harmful requests. Anchor election selects safety, the probe with the least drift at step 100. Because a policy can hack the reward models it is trained on, evaluation uses different models: Granite Guardian 3.1-8B scores safety on harmful requests and Claude Opus 4.7 scores compliance on benign requests.

MethodSafety ↑Compliance ↑Mean ↑
Qwen3-1.7B93.8%80.3%87.0%
R_P-only (safety)96.0%88.8%92.4%
R_A-only (compliance)71.9%96.0%83.9%
(R_P + R_A)/288.4%94.0%91.2%
0.9 R_P + 0.1 R_A91.2%92.4%91.8%
TAG93.9%92.4%93.1%

TAG has the highest mean score: it keeps safety on harmful requests at the level of the base policy while raising compliance on benign requests, where (R_P + R_A)/2 trades safety for compliance. The training signal shows the mechanism: under (R_P + R_A)/2 some responses that the safety reward model labels unsafe receive positive advantage; under TAG none do.

Dear Reviewer 4KmG,

As the author–reviewer discussion period draws to a close on Nov. 18, we would appreciate any further thoughts you may have on whether our response addresses your main concerns.

Thank you again for your time and consideration!

Dear Reviewer 4KmG,

As the discussion period comes to a close today, we would greatly appreciate any further thoughts on whether our response and additional experiments have addressed your concerns.

Thank you again for your time and consideration!

Comments on the paper

1 comment

We thank all reviewers for their thoughtful and constructive feedback, and for recognizing the relevance of the problem to LLM post-training, the account of why symmetric aggregation fails, and the breadth and candor of the experiments. We characterize aggregation-induced drift and the conditions under which TAG prevents it. Symmetric aggregation drives GRPO toward the auxiliary-only behavior (Theorem 1), TAG gates the auxiliary reward on an elected anchor so that the primary behavior is selected (Theorem 2), and both behaviors appear in training across Qwen and Llama models on mathematics and program synthesis. The responses below answer three questions that recur across the reviews.

Does the theoretical model describe actual training, and does a smaller auxiliary weight prevent the collapse? Training shows the same tendency as the model across models, tasks and optimizers, and a smaller auxiliary weight delays the collapse without preventing it. In the training rollouts, auxiliary-only responses come to dominate under (R_P + R_A)/2, while under TAG they stay rare and correct responses increase, the behaviors that Theorems 1 and 2 select in the model. The sweep of the auxiliary weight w_A in Figure 8 shows the delay on Qwen3-1.7B.

Do the anchor election and TAG extend to other auxiliary rewards, tasks and optimizers? The anchor election and TAG extend to these settings. The election selects correctness when the auxiliary reward lengthens responses and on equal-length prefixes, and it selects the more reliable channel when both are hackable. With a process reward model as the auxiliary reward, TAG gains 2.3–3.9 points over R_P-only on five held-out benchmarks while (R_P + R_A)/2 and 0.9 R_P + 0.1 R_A collapse. In safety and compliance training with two reward models, the election selects safety and TAG reaches the best balance of the two objectives. Under GSPO and DAPO, (R_P + R_A)/2 collapses as it does under GRPO, while TAG avoids the collapse.

How does TAG differ from a correctness-conditioned auxiliary reward? TAG elects its anchor from data, and the theory explains why symmetric aggregation fails. A correctness-conditioned reward takes the primary reward as given. The election finds the anchor without knowing what each channel measures, and Theorems 1 and 2 characterize which behavior each aggregation selects during training.

The revised manuscript marks changes in blue. We hope the responses address the reviewers' questions and are glad to discuss any further points.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.