acceptodds
Withdrawn from ICLR 2027

FACTOR-OPD: Allocate Reward-Specific Credit Before Combining Policy Gradients

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multi-reward reinforcement learning must balance objectives that can require different updates within the same response. Sequence-level reward aggregation leaves this token-credit problem unresolved, while teacher feedback from uncontrolled references can mix the effects of several reward properties. We introduce FactorOPD, which allocates reward-specific token credit before combining objective gradients. Verifier-checked factorial references contrast high and low levels of each reward property while balancing the other verified factors. A frozen teacher scores identical student-sampled tokens under these references, and constrained allocation converts the resulting contrasts into positive, bounded, mean-one token weights. This preserves each component advantage's sign and response-wise mean. Parameter-space MGDA then combines the induced objective gradients, accounting for their geometry in the shared model parameters. Across six benchmarks spanning mathematical reasoning, code generation, and instruction following, FactorOPD outperforms GRPO, GDPO, and GD²PO with both Qwen3-4B and Qwen3-8B. Domain-average accuracy gains over GD²PO range from 1.73 to 1.84 percentage points. Ablations support complementary benefits from token allocation and gradient coordination, including the contribution of token alignment. Reward-wise analyses further show higher correctness without reduced format compliance on Math and Code.

Primary area:
reinforcement learning
Keywords:
multi-objective optimization, Pareto front, preference learning, decision making

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_NcjmOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces FACTOR-OPD, a multi-reward reinforcement learning framework that addresses token-level credit assignment and gradient interference in large language models. By leveraging verifier-checked factorial reference panels, a frozen teacher provides controlled token contrasts that are converted into bounded, mean-one weights preserving component advantage signs and means. It then coordinates the resulting policy gradients using parameter-space MGDA. Across mathematical reasoning, code generation, and instruction following benchmarks, FACTOR-OPD consistently outperforms existing methods.

Critical strengths
  1. Novel Token-Credit Allocation Mechanism: this work Introduces verifier-checked factorial panels and bounded, mean-one allocation to precisely distribute feedback within a response while strictly preserving each component advantage's sign and response-wise mean.
  2. Parameter-Space Gradient Coordination: the proposed method employs parameter-space Multiple-Gradient Descent Algorithm (MGDA) to combine objective-specific policy gradients, accounting for the model's shared-parameter geometry rather than relying on heuristic credit-space weighting.
  3. Comprehensive Empirical Superiority: this work outperforms strong baselines across six benchmarks spanning mathematical reasoning, code generation, and instruction following over two model scales.
Critical weaknesses
  1. Insufficient Implementation Details for Hint Guidance: The paper lacks comprehensive implementation details in Appendix B regarding how the constructed hints meaningfully influence and shift student token probabilities during training.
  2. Unclear Baseline Formulation in Ablations: In Table 3, the baseline configuration (Uniform allocation + Fixed weights) unexpectedly outperforms certain strong baselines from the main evaluation table, making the formulation and relative superiority of these baselines unclear.
Questions for authors
  1. Question on Hint Influence and Implementation: The paper provides a high-level pipeline for hint construction in Appendix B, but lacks sufficient implementation details. Could the authors provide a concrete example showing how the high and low hints are generated, explain how the framework ensures these hints meaningfully shift student token probabilities during training, and discuss what happens to model performance if the hints have little or no influence?
  2. Question on Baseline Formulation in Table 3: In Table 3, the ablation baseline (Uniform allocation + Fixed weights) achieves higher scores than certain strong baselines reported in the main evaluation table. Could the authors clarify how this baseline is formulated and why it surpasses those main results?
Reviewer_j7bROpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

N/A -- see critical weaknesses

Critical strengths

N/A -- see critical weaknesses

Critical weaknesses

I strongly believe the results in (at least part) of this paper are fabricated, and not based on real training runs.

The gaps between FACTOR-OPD, GD2PO and GDPO are the same for both the 4B and 8B models:

ComparisonModelAIME24AIME25LCB v5LCB v6IFEvalIFBench
GD2PO − GDPOQwen3-4B0.120.730.810.251.654.34
GD2PO − GDPOQwen3-8B0.120.730.810.251.654.34
FACTOR-OPD − GDPOQwen3-4B1.832.703.041.483.506.00
FACTOR-OPD − GDPOQwen3-8B1.832.703.041.483.506.00

Meanwhile, these are the gaps between GRPO and Base, which seem like potentially real values:

ComparisonModelAIME24AIME25LCB v5LCB v6IFEvalIFBench
GRPO − BaseQwen3-4B0.840.212.700.500.742.67
GRPO − BaseQwen3-8B0.630.000.001.434.990.33

This looks like the first few baselines were perhaps real training runs, and then a constant was added to each value, with the same constants being used for both models. I don't know if these are placeholder values that should've been updated or are fraudulent, but in either case I will not be reviewing the rest of the paper.

Questions for authors

N/A

Reviewer_rQEkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes FACTOR-OPD for multi-reward RL in language models. First, a frozen teacher scores student-sampled tokens under a verifier-checked 2K2^K factorial reference panel, and constrained entropy-regularized allocation maps the resulting contrasts into bounded ([1−γ,1+γ][1-\gamma, 1+\gamma]), mean-one token weights that preserve each verifier advantage's sign and mean (Proposition 1). Second, parameter-space MGDA combines the separately clipped per-reward policy gradients. Experiments on Qwen3-4B/8B across Math, Code, IF, and Tool Calling show 1.23–2.23 percentage point gains over GRPO, GDPO, and GD2PO.

Critical strengths

The question is important, and credit assignment is challenging in RL. The 2×22\times 2 crossed ablation (Table 3) cleanly isolates token allocation from parameter MGDA under matched clipping paths, while the permutation control (Table 4) preserves the exact weight distribution and mean budget to prove that token-position alignment drives the gain (0.69–0.77 points)

Critical weaknesses
  1. Balancing verified reward labels does not hold the references' semantic and lexical content fixed. Thus τi,k,t\tau_{i,k,t} measures teacher sensitivity to the entire hint intervention—not a token's contribution to reward kk, and may largely reflect answer leakage or lexical overlap. The permutation ablation cannot exclude this explanation because such overlap is also token-aligned.
  2. The paper never specifies how the 2K2^K reference "editor" (Appendix B) is implemented for Math, Code, and IF, or whether high-correctness hints introduce privileged ground-truth supervision unavailable to outcome-only baselines (GRPO/GDPO/GD2PO). It is unknown how often factorial allocation is actually active. Core hyperparameters (γ,η,si,k\gamma, \eta, s_{i,k} in Equations 6–8) and the active MGDA variant (Appendix B) are also omitted.
  3. Requiring 2K2^K teacher forward passes and KK backward passes per update is only tested at K=2K=2 (where Code format compliance saturates to 100% in Table 6, degenerating Equation 10 to a single active objective). Statistically, Table 2 lacks cross-seed variance, Tables 3–4 omit across-seed standard deviations (Appendix D), and Appendix E (Page 17) reveals that FACTOR-OPD's aggregate Correct Acc. means and standard deviations in Table 5 were reconstructed from rounded table entries via a zero-covariance approximation rather than computed from raw runs.
Questions for authors
  1. How are the 2K=42^K=4 references hx(z)h_x(z) generated in Math, Code, IF, and Tool Calling (especially when all group rollouts fail), does this use privileged ground truth unavailable to baselines, and what is the exact valid panel coverage rate (si,k>0s_{i,k}>0) in each domain?
  2. What are the exact values of γ,η,si,k\gamma, \eta, s_{i,k} (Equations 6–8), and which MGDA variant in Appendix B (raw, unit-gradient, or anchored) was used in Tables 2–6?
  3. What is the wall-clock time/memory overhead relative to GD2PO (and performance under equal GPU hours)?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.