- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes DuoPO, a policy-optimization framework for Reinforcement Learning with Verifiable Rewards (RLVR) that jointly adapts to group-relative advantage magnitude (|) and correctness flag (). The method combines a signal-axis selector that retains high-contrast rollout pairs, a non-separable joint weight incorporating advantage magnitude and self-teacher entropy, and a correctness-conditioned loss family that routes between the GRPO surrogate, dense self-distillation, and the GRPO fallback. The authors report improvements across multimodal reasoning benchmarks, with additional experiments on 32B scaling, soft-reward transfer, and text-only reasoning.
Critical strengths
-
The problem is practically relevant. Rollouts can differ in both training-signal strength and available supervision. Jointly studying rollout selection, sample weighting, and loss routing is a reasonable direction for improving sample utilization in RLVR.
-
The reported performance is promising. Under the 30k-sample, 150-step setting in Table 11, DuoPO achieves a six-benchmark average of 65.0, compared with 62.5 for GRPO, 63.4 for DAPO, and 62.8 for GSPO.
-
The experiments cover several settings and attempt mechanism analysis. Beyond the main 3B/7B multimodal experiments, the paper includes 32B scaling, Referring Expression Comprehension, and text-only reasoning. Component ablations, composition-order comparisons, and hyperparameter sweeps provide preliminary material for understanding the method.
Critical weaknesses
-
One relevant baseline is missing. DuoPO incorporates SDPO-style dense self-distillation, but the experiments do not report standalone SDPO or a simple GRPO+SDPO combination using the same teacher information. SDPO already uses a feedback-conditioned self-teacher and can use successful rollouts as feedback for failed attempts
-
Some similar new methods are not discussed in the work. CPPO (NeurIPS 2025) performs completion pruning based on absolute advantage, while D³S (ICLR 2026) combines sample-level and token-level selection.
Questions for authors
- Main concern is about the novelty, so the discussion to highlight the contribution of this work is appreciated.