DuoPO: Joint Two-Axis Adaptation for Reinforcement Learning with Verifiable Rewards

Reinforcement learning with verifiable rewards (RLVR) drives modern post-training of reasoning-capable large models, yet its dominant realization (Group-Relative Policy Optimization, GRPO) applies the same update rule to every rollout in a batch, overlooking two axes of rollout-level heterogeneity: the magnitude of the group-relative advantage $|A_i|$ and the correctness flag $c_i$. Through controlled pilot experiments, we establish that these two attributes are empirically near-independent during training, yet interact non-separably in their contribution to training utility: a high-$|A|$ incorrect rollout carries markedly more per-sample signal than a high-$|A|$ correct rollout of matched magnitude. Motivated by this observation, we propose DuoPO, a policy-optimization framework that instantiates a generalized RLVR objective with a non-separable joint weight, a signal-axis selector, and a correctness-conditioned loss family; DuoPO reduces to GRPO under a single degenerate choice of hyperparameters and provably dominates its separable and single-axis alternatives. On six multimodal reasoning benchmarks across two model scales, DuoPO achieves new state-of-the-art performance with super-additive gains: the joint treatment exceeds the sum of its single-axis contributions over GRPO on 4 of 6 benchmarks.