- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes COBRA, a batch construction and advantage normalization method for single-response RLVR. It groups trajectories by response completion order and switches from local to global reward normalization during training to improve early learning efficiency and mitigate response length inflation. Experiments show improved early mathematical reasoning performance on a specifically initialized Qwen2.5-7B model, with additional gains on logical reasoning tasks.
Critical strengths
- The paper is well-written and easy to follow.
- The method is simple and easy to implement.
Critical weaknesses
- The paper lacks efficiency comparisons with mainstream RLVR methods, such as GRPO, DAPO, and REINFORCE++, under the same training budget.
- COBRA's performance gains are primarily concentrated in the early stages of training, while its advantages in final performance are limited.
- The paper lacks sensitivity analyses of key hyperparameters, such as the number of groups, grouping granularity, and rollout pool size.
- COBRA's early performance improvements are highly sensitive to model initialization. The lack of consistent improvements across different models and initialization settings limits the generalizability of the method.
- Long-term training stability remains a concern. At update 400, DeepMath accuracy drops to 31.64% for one training seed, with a truncation rate of 52.25%. This suggests that the fixed normalization-switching strategy cannot reliably ensure long-term training stability.
Questions for authors
- What is the main motivation behind COBRA, and are there concrete application scenarios? How does it compare with mainstream RLVR methods such as DAPO, GRPO, and REINFORCE++ under the same training budget?
- How sensitive is COBRA to key hyperparameters, such as the number of groups, grouping granularity, and pool size?
- Are the early performance gains mainly due to longer reasoning rather than improved learning efficiency? Could COBRA simply introduce an implicit length bias that improves early performance, with its advantage disappearing once other methods reach similar response lengths?
- Why does training still collapse in the long run even after switching to global normalization?
- Please justify the necessity of token-weighted reward normalization.