- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes Advantage-Based Control Variates (ABC), an actor-critic method for reinforcement learning with verifiable rewards (RLVR). The method reformulates trajectory-wise control variates using learned advantage and value functions, allowing the policy gradient to remain unbiased even with an imperfect critic. The authors further connect ABC with Direct Advantage Estimation (DAE) and propose an offline-to-online training framework, where the critic is pretrained on offline trajectories and updated during online RL. Experiments on mathematical reasoning benchmarks with Qwen3 models show reduced gradient variance and performance competitive with Dr. GRPO using fewer online actor updates.
Critical strengths
-
I like the general motivation of the paper, the paper revisits an important question: whether learned critics can improve RLVR despite the strong performance of critic-free methods such as GRPO. The proposed ABC estimator combines learned advantages with an unbiased correction, providing a principled way to reduce variance without relying entirely on critic predictions.
-
The experiments show that ABC can reduce gradient variance and achieve competitive performance with Dr. GRPO across several math benchmarks. The comparison with directly using learned advantages is also informative, highlighting the importance of correcting critic-induced bias.
Critical weaknesses
-
The paper emphasizes that ABC achieves competitive performance with only 100 actor updates, compared with approximately 6,000 updates for Dr. GRPO. However, ABC requires pretraining an 8B critic using approximately 8.2 million offline trajectories, which introduces substantial additional computation. The online batch sizes also differ significantly. Therefore, comparing the number of optimizer steps alone is not sufficient to establish training efficiency.
-
The experiments focus only on mathematical reasoning with models up to 4B, making it unclear whether the method generalizes to other RLVR tasks or larger models. I think the proposed method may be more beneficial for long-horizon tasks involving multiple consequential decisions, where trajectory-level rewards provide limited credit assignment. Although mathematical reasoning can involve long token sequences, it remains unclear whether the method can effectively capture the benefits of fine-grained advantage estimation in more complex settings, such as multi-step code generation, debugging, or agentic tasks. I would like to see experiments on these tasks, or at least an analysis of how trajectory length affects gradient variance, critic estimation error, and final performance. This would help establish when the proposed control variates are most useful compared with standard GRPO.
-
I like the motivation of this paper, and I think revisiting critic-based methods for RLVR is an interesting direction. However, I feel that the overall presentation and experimental evaluation still need further improvement. Some design choices and empirical observations are not sufficiently explained and deserve deeper investigation. For example, why does ABC fail to consistently match Dr. GRPO in the larger 4B model setting, particularly on MATH500 and AIME pass@32? Does this suggest that the benefit of learned advantages becomes less significant as model size increases? How does the performance change as pass@K is evaluated with increasing values of K?
I would like to see a more detailed analysis of when and why ABC provides advantages over existing methods, rather than primarily demonstrating competitive performance.
Questions for authors
See weakness