- Rating
- 2
- Confidence
- 1(−2)
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2(+1)
Read review
Paper summary
This paper studies how to aggregate multiple rewards in LLM RL, which is an interesting problem and has been explored in quite a few previous works. The main argument is that simply adding signals together can cause secondary signals to dominate training. Here, secondary signals refer to behaviors that are easier for the model to learn (e.g. shortening the output is much easier than improving accuracy). As a result, training can end up going in the wrong direction. To address this, the paper proposes a conditional design: first check correctness, and only consider length once correctness is satisfied. The authors show that this leads to better training outcomes.
Critical strengths
This paper tackles a relatively interesting problem and provides reasonable analyses from both theoretical and empirical perspectives. The overall writing is quite coherent: it starts with the phenomenon of multi-reward hacking, highlights the limitations of simply adding reward signals together, and eventually motivates the conditional design.
Critical weaknesses
-
The techniques discussed in this paper have substantial overlap with prior work. For example, GDPO [1] (Section 4.2.1) already covers many of the ideas discussed here: an easy-to-optimize length reward can hurt the harder correctness objective; simply reducing the weight of the length reward does not necessarily ensure that correctness is prioritized; and the length reward should only be given when the answer is both correct and satisfies the length constraint. Given this overlap, I feel that the contribution of this paper is relatively incremental.
-
The practical value of the Trust-Anchored Gating (TAG) method seems rather limited. In LLM RLVR, the choice of the primary reward is usually pretty obvious: it's the success rate. The paper also doesn't seem to analyze any setting where success rate is not chosen as the primary reward. This, in my view, significantly weakens the practical motivation for TAG.
[1] GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimization (ICML 2026)
Questions for authors
Could the authors elaborate more on TAG? While it seems like a reasonable design, I'm not sure whether there are practical scenarios in LLM RL where it would actually be useful.
5 comments
Response to Reviewer fezM (Part 2 of 2)
W2 & Q1: Practical motivation and applications of TAG
TAG is for training that must protect primary performance while optimizing an auxiliary reward, and the primary reward can be something other than correctness. With verifiable rewards the primary reward is correctness; with reward models it is often safety. The revision adds two experiments in more realistic settings. In the first, the two rewards are a safety reward model and a compliance reward model, the primary reward is safety and no success rate is involved. In the second, the auxiliary reward is a process reward model (PRM), a learned model that scores the intermediate steps of a math solution. In both, anchor election selects the primary reward: safety in the first and correctness in the second.
In safety and compliance training, TAG reaches the best balance of the two objectives, with safety elected as the anchor. The task is to train an assistant that helps with benign requests and stays safe on harmful ones, and both rewards come from reward models (Appendix E.4 of the revision, lines 1830–1857). Qwen3-1.7B is trained on Aegis 2.0 prompts for 100 steps with R_P = appropriate safety (judged by Qwen3Guard-Gen-8B) and R_A = instruction compliance (judged by Qwen3-32B); the two conflict on harmful requests. Anchor election selects safety, the probe with the least drift at step 100. Because a policy can hack the reward models it is trained on, evaluation uses different models: Granite Guardian 3.1-8B scores safety on harmful requests and Claude Opus 4.7 scores compliance on benign requests.
| Method | Safety ↑ | Compliance ↑ | Mean ↑ |
|---|---|---|---|
| Qwen3-1.7B | 93.8% | 80.3% | 87.0% |
| R_P-only (safety) | 96.0% | 88.8% | 92.4% |
| R_A-only (compliance) | 71.9% | 96.0% | 83.9% |
| (R_P + R_A)/2 | 88.4% | 94.0% | 91.2% |
| 0.9 R_P + 0.1 R_A | 91.2% | 92.4% | 91.8% |
| TAG | 93.9% | 92.4% | 93.1% |
TAG has the highest mean score: it keeps safety on harmful requests at the level of the base policy while raising compliance on benign requests, where (R_P + R_A)/2 trades safety for compliance. The training signal shows the mechanism: under (R_P + R_A)/2 some responses that the safety reward model labels unsafe receive positive advantage; under TAG none do.
With a PRM as the auxiliary reward, TAG gains 2.3–3.9 points over R_P-only on the five held-out benchmarks; (R_P + R_A)/2 and 0.9 R_P + 0.1 R_A with the same PRM collapse. A PRM scores each reasoning step of a solution, which makes it a natural auxiliary reward and also a hackable one: a policy can raise its score with plausible steps that do not reach the answer. We train Qwen3-1.7B with GRPO for 500 steps, with R_P = final-answer correctness and R_A = a frozen PURE-PRM-7B step score; the table reports held-out pass@1:
| Method | trace-val (in-domain) | MATH-500 | OlympiadBench | Minerva Math | Omni-MATH | BigMath-hard |
|---|---|---|---|---|---|---|
| R_P-only | 37.8% | 70.1% | 34.5% | 22.4% | 20.6% | 21.1% |
| R_A-only | 0.0% | 0.9% | 0.0% | 0.5% | 0.3% | 0.0% |
| (R_P + R_A)/2 | 15.8% | 56.6% | 21.1% | 20.5% | 12.5% | 10.4% |
| 0.9 R_P + 0.1 R_A | 17.3% | 58.7% | 21.9% | 21.3% | 12.7% | 11.8% |
| TAG | 39.3% | 72.5% | 37.1% | 26.3% | 22.9% | 23.6% |
The policies trained with (R_P + R_A)/2 and 0.9 R_P + 0.1 R_A learn to write plausible steps and focus less on reaching an answer; TAG uses the same PRM to rank correct responses and gains from it on every benchmark in the table.
Dear Reviewer fezM,
As the author–reviewer discussion period draws to a close on Nov. 18, we would appreciate any further thoughts you may have on whether our response addresses your main concerns.
Thank you again for your time and consideration!
Response to Reviewer fezM (Part 1 of 2)
We thank the reviewer for recognizing the theoretical and empirical analyses and the coherence of the argument from multi-reward hacking to the conditional design. The responses below clarify how TAG relates to GDPO's conditional reward and the two respects in which our paper differs, the election of the anchor and the theoretical explanation (W1), and show two practical uses of TAG: in safety and compliance training, where the elected anchor is safety, and with a process reward model as the auxiliary reward (W2 & Q1).
W1: Relationship to GDPO
With a binary correctness anchor, TAG and GDPO's conditional reward use the same two terms, correctness and a correctness-conditioned auxiliary reward. GDPO normalizes each as a separate channel, where TAG normalizes their weighted sum. Our paper cites GDPO for its per-channel normalization of the advantage, and the revision adds a discussion of its conditional reward. Our paper differs in two respects: it elects the anchor from data, where GDPO takes the primary reward as given, and it explains through theory why symmetric aggregation fails and why the gate selects the primary behavior.
First, the selection of the anchor. GDPO uses a predetermined anchor, without looking at the training dynamics of the reward channels. Our anchor election requires no knowledge of what each channel measures: it selects the anchor from the drift each channel induces when optimized alone.
Second, the theory: which behavior aggregation selects, and why. In the two-channel model, Theorem 1 characterizes the behavior that symmetric aggregation selects from the rates at which the two rewards are realized, and Theorem 2 shows that the gate changes the selected behavior from every initialization. The credit bias is measured in the training rollouts: over the first 20 training steps of Correct & negative on Qwen3-1.7B, auxiliary-only responses receive a positive mean advantage under (R_P + R_A)/2 and a negative one under TAG, and the training trajectories show the same tendency (Appendix E.1, Table 13, lines 1736–1756). Our paper already includes MO-GRPO, which like GDPO computes the advantage per channel before summing, as a baseline in Table 1, where it collapses in all four settings.
We also trained GDPO as specified, with per-channel group normalization followed by batch-wise normalization of the summed advantage, and GDPO with the conditional reward, under the Table 1 protocol on Qwen3-1.7B math (Appendix E.6, Table 19, lines 1907–1927 of the revision). The table follows Table 1, with pass@4 and response length in Correct & brief, and pass@4 and the negative-answer rate in Correct & negative:
GDPO collapses like (R_P + R_A)/2 and MO-GRPO in Table 1. With the conditional reward, GDPO avoids the collapse as TAG does: conditioning the auxiliary reward on correctness protects correctness under both methods. A likely reason for the difference in pass@4 is that TAG standardizes the gated reward as one scalar, while GDPO standardizes the bonus as a separate channel and gives it the same weight as correctness.