acceptodds
Under review as a conference paper at ICLR 2027

DuoPO: Joint Two-Axis Adaptation for Reinforcement Learning with Verifiable Rewards

Abstract

Reinforcement learning with verifiable rewards (RLVR) drives modern post-training of reasoning-capable large models, yet its dominant realization (Group-Relative Policy Optimization, GRPO) applies the same update rule to every rollout in a batch, overlooking two axes of rollout-level heterogeneity: the magnitude of the group-relative advantage and the correctness flag . Through controlled pilot experiments, we establish that these two attributes are empirically near-independent during training, yet interact non-separably in their contribution to training utility: a high- incorrect rollout carries markedly more per-sample signal than a high- correct rollout of matched magnitude. Motivated by this observation, we propose DuoPO, a policy-optimization framework that instantiates a generalized RLVR objective with a non-separable joint weight, a signal-axis selector, and a correctness-conditioned loss family; DuoPO reduces to GRPO under a single degenerate choice of hyperparameters and provably dominates its separable and single-axis alternatives. On six multimodal reasoning benchmarks across two model scales, DuoPO achieves new state-of-the-art performance with super-additive gains: the joint treatment exceeds the sum of its single-axis contributions over GRPO on 4 of 6 benchmarks.

Primary area:
reinforcement learning
Keywords:
LLM

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_mBAjOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DuoPO, a policy-optimization framework for Reinforcement Learning with Verifiable Rewards (RLVR) that jointly adapts to group-relative advantage magnitude (∣Ai|A_i|) and correctness flag (cic_i). The method combines a signal-axis selector that retains high-contrast rollout pairs, a non-separable joint weight incorporating advantage magnitude and self-teacher entropy, and a correctness-conditioned loss family that routes between the GRPO surrogate, dense self-distillation, and the GRPO fallback. The authors report improvements across multimodal reasoning benchmarks, with additional experiments on 32B scaling, soft-reward transfer, and text-only reasoning.

Critical strengths
  • The problem is practically relevant. Rollouts can differ in both training-signal strength and available supervision. Jointly studying rollout selection, sample weighting, and loss routing is a reasonable direction for improving sample utilization in RLVR.

  • The reported performance is promising. Under the 30k-sample, 150-step setting in Table 11, DuoPO achieves a six-benchmark average of 65.0, compared with 62.5 for GRPO, 63.4 for DAPO, and 62.8 for GSPO.

  • The experiments cover several settings and attempt mechanism analysis. Beyond the main 3B/7B multimodal experiments, the paper includes 32B scaling, Referring Expression Comprehension, and text-only reasoning. Component ablations, composition-order comparisons, and hyperparameter sweeps provide preliminary material for understanding the method.

Critical weaknesses
  • One relevant baseline is missing. DuoPO incorporates SDPO-style dense self-distillation, but the experiments do not report standalone SDPO or a simple GRPO+SDPO combination using the same teacher information. SDPO already uses a feedback-conditioned self-teacher and can use successful rollouts as feedback for failed attempts

  • Some similar new methods are not discussed in the work. CPPO (NeurIPS 2025) performs completion pruning based on absolute advantage, while D³S (ICLR 2026) combines sample-level and token-level selection.

Questions for authors
  • Main concern is about the novelty, so the discussion to highlight the contribution of this work is appreciated.
Reviewer_TskdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DuoPO argues that GRPO applies one update rule to all rollouts, ignoring the advantage magnitude ∣Ai∣|A_i| and correctness cic_i. From a pilot study it claims that the two are nearly independent but interact non-separably, with high-∣A∣|A| incorrect rollouts being more informative than equally high-∣A∣|A| correct ones. The method combines a contrastive pair selector with importance resampling, a non-separable token weight ∣Ai∣1+Δ(1−ci)e−β(1−ci)Hi,t|A_i|^{1+\Delta(1-c_i)}e^{-\beta(1-c_i)H_{i,t}}, and correctness-routed losses (self-distillation for incorrect rollouts with teacher information, the GRPO surrogate otherwise). Qwen2.5-VL-7B trained on 30k samples reaches a 65.0 six-benchmark average, +2.5 over GRPO.

Critical strengths
  1. A general objective that contains GRPO as a special case, with consistent gains across model sizes, training sets, a soft-reward grounding task and text-only models.

  2. Informative selector ablations: contrastive pairing beats top-k, bottom-k and random selection, and select-then-route beats route-then-select (48.12 vs 44.78).

Critical weaknesses
  1. The central premise is unsupported.All key quantities of the Sec. 4.1 pilot — the number of training steps, the correlation between ∣A∣|A| and cc, and the utility ratio at large ∣A∣|A| — are unfilled "[xx]" placeholders, with no figure or table, yet the abstract claims that the pilot "confirms" them.

  2. The motivation does not lead directly to the design, and the design is heavy. The weight form is posited rather than derived from measured utilities, and it does more than encode the claimed asymmetry: it amplifies failures with ∣A∣>1|A|>1 but suppresses those with ∣A∣<1|A|<1, so the Δ=0\Delta=0 vs Δ=0.5\Delta=0.5 gap (46.02 vs 48.12) does not isolate it. Near-independence is also doubtful, since with binary rewards ∣A∣|A| within a group is determined by cc and the group accuracy. Meanwhile, the method adds four hyperparameters, importance resampling and a second loss family, although the selector alone already yields most of the gain (+3.66 of +5.48 on Geo3K).

  3. The theoretical claims are overstated. The abstract says DuoPO "provably" outperforms separable and single-axis alternatives, but no formal result compares utility or performance: Lemma G.1 only shows non-separability, and Prop. G.2 is a sketch about pair contrast. The degenerate setting given in Appendix G.2 yields w=∣Ai∣w=|A_i| rather than w=1w=1.

  4. Key details are missing. The teacher information fif_i for the self-distillation branch is never specified, so it is unknown how often that branch is used; the "+1.10" and "+0.72" increments have no corresponding rows in Table 4; and the advantage histograms, training curves and GPU-time overhead mentioned in the text are not shown.

Questions for authors
  1. Please fill in Sec. 4.1 with the actual numbers and plots, including U(∣A∣,c=0)/U(∣A∣,c=1)U(|A|,c=0)/U(|A|,c=1) as a function of ∣A∣|A|, and show whether the form ∣A∣1+Δ|A|^{1+\Delta} matches it.

  2. What is fif_i in your experiments, what fraction of incorrect rollouts is routed to self-distillation, and how much of the gain remains without it?

Reviewer_4JkmOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes DuoPO, which jointly adapts RLVR updates to advantage magnitude and rollout correctness. It combines contrastive rollout selection, non-separable weighting, and correctness-conditioned GRPO/self-distillation losses. The authors report improvements over GRPO and other baselines across multimodal reasoning benchmarks, with additional scaling and transfer experiments.

Critical strengths
  • Promising reported improvements against relevant baselines.
Critical weaknesses
  • Section 4.1 still contains “[xx]” placeholders for the pilot duration, correlation statistics, and utility ratio, leaving the claimed near-independence and asymmetric utility without the measurements needed to assess them.
  • Two theoretical claims need correction: Eq. (21) should give zero under the stated symmetry assumptions, and the parameter setting in Appendix G.2 leaves (w=|A|), so it does not recover GRPO as claimed.
  • Table 4 does not isolate the contributions of weighting and loss routing, so it does not substantiate the stated separate gains of +1.10 and +0.72 or establish the claimed three-component superadditivity.
Questions for authors

What are the missing pilot statistics in §4.1, and how was the held-out-loss reduction attributable to an individual rollout measured?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.