acceptodds
Withdrawn from ICLR 2027

COBRA: Completion-Order Batching as a Reward-stratified Advantage

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

COBRA improves early learning from verifiable training problems in single-response reinforcement learning. We retain every rollout,form update blocks by completion order, and switch from local to pooled reward moments after an initial learning phase. From a sharedformat-calibrated Qwen2.5-7B start, two runs improve update-80 accuracy over random batching by 3.15–3.49 percentage points onDeepMath and 4.20–7.18 on MATH500. Each arm has seen about 45,900 distinct training problems in 61,440 prompt presentations. On theretained evaluation grid, mean MATH500 accuracy first exceeds 70% at update 80 versus 120–160 for random/local, after 24–35% fewerdistinct training problems. Completion grouping creates length and reward strata; an exploratory, single-seed centering interventionsupports local centering as a contributor to subsequent response growth. Switching to pooled moments limits that growth through 200updates on Qwen2.5-7B and Qwen3-4B. On held-out three-to-seven-person logic puzzles, the same recipe gains 2.35–6.13 percentagepoints over scheduled random grouping.

Primary area:
reinforcement learning
Keywords:
reinforcement learning with verifiable rewards, large language models, asynchronous reinforcement learning, policy gradients, advantage estimation, reward stratification, variance reduction, completion-order batching

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_SXCrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes COBRA, a batch construction and advantage normalization method for single-response RLVR. It groups trajectories by response completion order and switches from local to global reward normalization during training to improve early learning efficiency and mitigate response length inflation. Experiments show improved early mathematical reasoning performance on a specifically initialized Qwen2.5-7B model, with additional gains on logical reasoning tasks.

Critical strengths
  1. The paper is well-written and easy to follow.
  2. The method is simple and easy to implement.
Critical weaknesses
  1. The paper lacks efficiency comparisons with mainstream RLVR methods, such as GRPO, DAPO, and REINFORCE++, under the same training budget.
  2. COBRA's performance gains are primarily concentrated in the early stages of training, while its advantages in final performance are limited.
  3. The paper lacks sensitivity analyses of key hyperparameters, such as the number of groups, grouping granularity, and rollout pool size.
  4. COBRA's early performance improvements are highly sensitive to model initialization. The lack of consistent improvements across different models and initialization settings limits the generalizability of the method.
  5. Long-term training stability remains a concern. At update 400, DeepMath accuracy drops to 31.64% for one training seed, with a truncation rate of 52.25%. This suggests that the fixed normalization-switching strategy cannot reliably ensure long-term training stability.
Questions for authors
  1. What is the main motivation behind COBRA, and are there concrete application scenarios? How does it compare with mainstream RLVR methods such as DAPO, GRPO, and REINFORCE++ under the same training budget?
  2. How sensitive is COBRA to key hyperparameters, such as the number of groups, grouping granularity, and pool size?
  3. Are the early performance gains mainly due to longer reasoning rather than improved learning efficiency? Could COBRA simply introduce an implicit length bias that improves early performance, with its advantage disappearing once other methods reach similar response lengths?
  4. Why does training still collapse in the long run even after switching to global normalization?
  5. Please justify the necessity of token-weighted reward normalization.
Reviewer_oXqTOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper is about sorting responses into batches based on when they are completed for training LLMs with RL.

Critical strengths

The paper has large scale experiments with 4B and 7B models.

Critical weaknesses

Introduction: The abstract and introduction do not sufficiently motivate the problem being addressed or explain why the proposed approach should help. The three listed contributions are also difficult to interpret. “Early useful accuracy,” is not well-defined. “Normalization intervention that limits growth:” should explicitly identify response length growth and explain its relationship to downstream performance. “Held-out logic benefit with explicit costs,” should explain what benefits and under what computational cost.

Section 2. Some crucial definitions are unclear in the main text. In particular, how exactly is cic_i defined? Furthermore, the reader is left unsure about which policy optimization objective is being used, as the text only states “token-level surrogate from proximal policy optimization.” Here, the authors should state what objective is being used for training. Appendix A.2 presents the objective, but there should be explicit references that propose and use this objective, unless this is a new one proposed by the authors. Furthermore, authors should motivate their choice of objective and its relationship to the problem being addressed by the paper.

Section 3. The experimental setup and baselines are difficult to follow. Furthermore, several choices are left unexplained. The 30-update “format calibration that rewards output structure without checking correctness” for the 7B model requires justification. To my knowledge, this is not a standard choice. This choice significantly drops MATH500 accuracy. Furthermore, it seems Table 16 reports results compared with no calibration, so why is this choice in the main paper? Without this calibration, the gains in performance from COBRA are much smaller. This brings into question whether the step-80 gains at 7B depend strongly on the calibrated initialization. The 4B model does not undergo this format calibration (at least as presented in the main paper), making this choice seem arbitrary. Lastly, this section should clearly identify what each “arm” is, what the baselines are, and what each comparison studies.

Section 4. It’s unclear to me why one should be concerned with accuracy at step 80 when the model continues to improve after that step with more training. In the end, COBRA appears to have nearly the same performance (or worse performance) as the other methods. Without the calibration initialization for the 7B model, early accuracy is not much better. Overall for 4B, based on Table 17, I see no clear gains either early in training or at the end of training. Thus I am unconvinced of the benefits of this method.

Questions for authors

See above

Reviewer_CUxzOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes COBRA, a batching scheme for critic-free RLVR with a single response per prompt. Each round generates single response for N(=3k) prompts under a fixed policy, sort the pool by completion order, and consumes it in K(=4) sequential updates; reward moments are block-level through update 80 and pooled afterward. Starting from a format-calibrated Qwen2.5-7B, COBRA improves update-80 accuracy over random/local batching by 3.15--3.49 points on DeepMath and 4.20--7.18 on MATH500 across two random seeds. A single seed intervention (Table 3) identifies block-level reward centering as a contributor to response growth, and switching to pooled statistics limits this growth through update 200. On Knights and Knaves puzzles, COBRA gains 2.35--6.13 points over random grouping.

Critical strengths

None

Critical weaknesses

W1. The writing obscures both the motivation and the principal findings.

The main text neither states the problem being solved nor reports the conditions under which the gains disappear, the contribution cannot be assessed as presented.

  • Neither the abstract nor the introduction articulates the problem COBRA addresses or its significance.
  • The training setting (single response per prompt and a baseline computed across different prompts) is stated without motivation and justification.
  • The introduction attributes the core ideas to CANON and Stratified GRPO and defines the contribution as "the measured behavior of these components", which is inconsistent with the title and abstract presenting COBRA as a new method.
  • Poorly written abstract. The abstract uses undefined terminology (e.g., "retained evaluation grid", "scheduled random grouping") and refers to "growth" without specifying the quantity. Moreover, it includes experimental details better suited to the experimental section (e.g., "about 45,900 distinct training problems in 61,440 prompt presentations").
  • I cannot understand why the exact seed number keeps appearing in the text.

W2. Missing baselines and the disappearance of early gain once the comparison change.

  • No established method is compared (e.g., GRPO under a matched budget, REINFORCE++, CANON, Stratified GRPO).
  • The gain holds only at matched update counts. At a matched token budget, the paper reports equality (49.44% vs. 49.46%). Moreover, at update 200, COBRA is at or below both random controls on both math datasets (Table 17) while using more training tokens (Table 9).
  • Starting from official Qwen2.5-7B Base, the early gain (update 80) is not statistically significant (Table 15), and on Qwen3-4B they are negative. The main experiments instead start from a format-calibrated model whose MATH500 acc. is ~18 points below the official Base.
  • The most problematic aspect is that all these limitations are relegated to the appendix and merely referenced in the main text.
Questions for authors

Q1. The completion rank correlates 0.987 with length (App. B) and length sorting reproduces the early result (Table 8). What does the completion-order framing add over explicit length stratification, and do the gains persist when blocks are consumed as they complete, under the staleness of asynchronous pipelines such as APRIL?

Q2. Why is the single-response setting required, given that the cited antecedents use multi-response groups, and how does COBRA compare with GRPO (e.g., G = 8) or global-normalization REINFORCE++ under a matched rollout budget?

[Suggestions/Questions] S1. Revise the writing to improve readability. (e.g. specific training seed IDs appear throughout the main text, tables, and figure captions. Neutral labels such as "Run 1/Run 2" would improve readability.) The research question that COBRA aims to address, and why that problem is significant, should be clearly stated by the author, rather than left for the reader to infer.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.