- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper investigates the conditions under which Reinforcement Learning with Verifiable Rewards (RLVR) can sustain capability growth beyond the initial policy. The authors hypothesize that sustained improvement requires training instances that remain informative as the policy evolves. Using combinatorial optimization (CO) tasks with controllable difficulty, they observe that RL progressively shifts toward harder instances and identify mean thinking length as an intrinsic indicator of the model’s capability boundary. Based on these findings, they propose Adaptive Boundary RL (ABRL), which dynamically concentrates training near this boundary. Empirical results show that ABRL substantially outperforms uniform RL in average normalized performance.
Critical strengths
- Empirical results show that ABRL provides strong performance gains over best-of- initial policy.
Critical weaknesses
- The paper's observation that the capability boundary shifts toward harder instances is somewhat similar to Boundary-aware Curriculum Learning [1] and the edge-of-competence phenomenon [2]. While the authors claim that ABRL differs by tracking the boundary using intrinsic response lengths, this metric requires further justification beyond combinatorial optimization (CO) tasks.
- Although the paper aims to understand the conditions under which RL can sustain capability growth beyond the initial policy, I find it difficult to connect its insights to previous observations. For example, ProRL expands the reasoning boundary without dynamically allocating training data across difficulty levels. Furthermore, the proposed approach relies on a known problem difficulty axis, limiting its applicability to domains where estimating problem difficulty is challenging.
- The controlled tasks with varying difficulty levels demonstrate cross-difficulty transfer. However, the underlying assumption of gradient alignment may not hold in practice, where gradient interference can be more severe and has been identified as a potential explanation for capability boundary shrinkage [3,4,5].
Questions for authors
- How can the paper's insights help explain the conditions under which ProRL achieves capability boundary expansion?
- Does the observed pattern, where thinking length initially increases with problem size and then decreases, also hold for mathematical reasoning or general tasks? Prior work suggests that harder problems generally actually exhibit longer reasoning chains [6]
- Could the authors further clarify the key differences between ABRL, Boundary-aware Curriculum Learning, and the edge-of-competence phenomenon?
- Could the authors clarify the meaning of reward shaping in Lines 882–888? The current manuscript provides little explanation of this term.
[1] Curriculum Reinforcement Learning Can Incentivize Reasoning Capacity in LLMs Beyond the Base Model. arxiv 2606.22317.
[2] On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models. ICML 2026 Spotlight.
[3] POPE: Learning to Reason on Hard Problems via Privileged On-Policy Exploration. arxiv 2601.18779.
[4] Prompt Interference in LLM Post-Training: Why Pass@k Optimization Can Degrade Pass@1. arxiv 2602.21189v3.
[5] The Reasoning Boundary Paradox: How Reinforcement Learning Constrains Language Models. arxiv 2510.02230.
[6] Wait, Wait, Wait... Why Do Reasoning Models Loop? ICML 2026 Spotlight.
Thank you for your response. The rebuttal has partially addressed my concerns. However, Fig. 16 does not clearly establish how thinking length can serve as a reliable metric for identifying the capability boundary, especially on OpenMath dataset. Furthermore, the connection between the conditions under which ProRL is effective and its ability to expand model capabilities remains unclear.
Overall, I find the contribution somewhat incremental and conceptually similar to [1]. In contrast, [1] provides strong empirical evidence on standard benchmarks, whereas the current work is evaluated only in controlled settings, raising concerns about the generalizability of its findings. Therefore, I will maintain my original score.