acceptodds
Under review as a conference paper at ICLR 2027

HAVER-GRPO: Hazard-Aware Rollout Allocation for Efficient RLVR

Abstract

Reinforcement learning with verifiable rewards can waste substantial rollout compute on prompt groups whose responses are all incorrect and yield no group-relative learning signal. We introduce , a sequential allocation strategy that decides whether an all-wrong group merits another rollout. HAVER estimates verifier-success hazard and response-token cost, then allocates fresh rollouts by marginal recovery yield under a shared budget price. The first verified response is inserted into a fixed-size GRPO group, while every attempted token is charged to the compute budget. We analyze recovery probability, expected cost, and sensitivity to estimation error. Across five mathematical reasoning benchmarks, our main experiments achieve mean accuracies of and for the 1.5B and 7B models, respectively. Tokens-to-target decrease by and relative to uniform GRPO, demonstrating improved training efficiency at the same accuracy targets.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning with Verifiable Rewards, Group Relative Policy Optimization, Adaptive Rollout Allocation, Sequential Decision Making, Mathematical Reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_N7s8OpenReview ↗
Rating
1(−1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
1(−1)
Read review
Paper summary

The paper introduces HAVER-GRPO which is an adaptive rollout allocation strategy. The authors observe that when a group of rollouts has no correct solution, then there is no learning signal since the advantages have no differences. To counter this, they use a simple allocation strategy which is dependent on the cost of each rollout. Then, if a successful rollout is sampled, they replace it in the group that is used for advantage computation and the policy gradient. They demonstrate improved performance over strategies like DAPO which uses oversampling and regular GRPO.

Critical strengths
  1. Clarity and simplicity: The paper is easy to read and understand, and the method proposed is simple to implement and poses little overhead. Figure 1 is enough to understand the core contributions of the paper.
  2. Strength and broadness of results: The experiments are performed over multiple models like the Qwen2.5-1.5B and Qwen2.5-7B Instruct models with evaluations across multiple datasets such as AIME, AMC, MATH500 and OlympiadBench which makes the comparisons sound and significant.
Critical weaknesses
  1. What is the training dataset: The authors do not name the training dataset in the paper or in the Appendix.
  2. Insufficient baseline comparison: The authors cite work such as VIP (Nguyen et. al), however do not compare with it directly. Since it is prior art at this time (published in ICLR 2026), it would benefit to have a direct comparison since both works aim to maximize performance within a training rollout budget. The argument of returning K responses to the optimizer in Section 2 doesn't give a clear reason why the baselines would not be relevant.
Questions for authors

I have the 3 following questions:

  1. Could the authors remake figure 2 with x axis being the number of tokens generated in the rollout phase? It is hard to interpret the plot and Table 2 and Figure 3 all at the same time. Making the figure with x axis being number of tokens would make the comparisons more faithful to the intended use case of the algorithm.
  2. Why do you only choose the 'all-wrong' group for adaptive allocation of compute and not the 'all-correct' group? This aspect of the paper is under-discussed
  3. Why is the advantage still decided based on the original group size if you do adaptive allocation? As an example, say it consumed 32 rollouts to obtain 1 successful rollout. Then, the advantages will still be decided as if there was 1 success out of 8. Does this change the population level objective? Does this have some relations with up-weighting harder prompts like MaxRL (Tajwar et. al) or REINFORCE-ADA (Xiong et. al). A discussion around the population objective or relations to re-weighting groups would be beneficial.
Reviewer_xKiPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

HAVER-GRPO initially generates KK responses per prompt and performs additional sampling only for groups whose responses are all incorrect. The controller ranks eligible groups using estimated recovery utility per generated token:

Ig(t)=Ugh^g(t)c^g(t),I_g(t)=\frac{U_g\widehat h_g(t)}{\widehat c_g(t)},

where h^g(t)\widehat h_g(t) estimates the probability that the next response succeeds and c^g(t)\widehat c_g(t) estimates its token cost. Sampling continues when the index exceeds a shared price λ\lambda, subject to a rollout cap and token allowance. The first successful response replaces a predetermined failed slot, preserving the optimizer-facing group size.

The experiments use Ug=1U_g=1 and the same Beta(1,1) prior for every group upon entering recovery. Before success, the estimated hazard is therefore 1/(t+2)1/(t+2). The paper analyzes recovery probabilities, expected costs, a local stopping rule, and sensitivity to estimation error.

Experiments cover two Qwen2.5-Math model sizes, five mathematical reasoning benchmarks, and three training seeds. Table 2 reports reductions in tokens-to-target of 34.7% and 21.4% relative to Uniform GRPO. Over the complete 300-update runs, however, HAVER-GRPO consumes more generated tokens than Uniform GRPO while achieving higher final accuracy.

Critical strengths
  1. The recovery mechanism is simple, and the computational accounting is carefully distinguished.

    Initial responses, successful additional responses, and discarded failures are all charged to the generated-token budget. Table 2 and Appendix C distinguish generated tokens, retained optimizer tokens, additional recovery tokens, and tokens-to-target. This avoids treating a fixed optimizer-facing group size as equivalent to fixed computation and provides a meaningful basis for assessing efficiency.

  2. The evaluation reports both downstream accuracy and cost across model scales and training seeds.

    Tables 1–3 evaluate the learned policies rather than relying solely on recovery rates. The reported improvements cover two model sizes and five mathematical benchmarks, with averages over three training seeds. The paper also explicitly limits the theoretical claim: the greedy index is a local allocation rule, not a globally optimal policy under arbitrary rollout dependence. These are useful strengths of the evaluation and analysis.

Critical weaknesses
  1. The recovery baseline does not isolate adaptive allocation from the simpler benefit of stopping after success.

    Fixed-cap generates all CC additional responses even when an earlier response has already succeeded, while retaining only the first success (pages 4 and 15). Consequently, HAVER-GRPO’s cost advantage includes the direct benefit of first-success stopping. A fixed-cap recovery baseline that stops at the first success, while preserving the same replacement mechanism, is needed to isolate the contribution of hazard- and token-aware allocation. Moreover, all groups share the same entry prior, so the implemented hazard varies only with the number of observed failures rather than predicting prompt-specific success differences. In Table 6, the 1.5B model’s additional token cost per recovered group is actually higher for HAVER-GRPO than Fixed-cap: 50.902K versus 48.116K, despite better final accuracy. This makes the connection between recovery efficiency and downstream training gains particularly important to explain. Closely related adaptive allocation methods such as VIP and SARA are discussed but not evaluated.

  2. The main efficiency results lack essential reproducibility and uncertainty information.

    Appendix C.3 requires a positive estimate of the next response’s token cost but does not specify how c^g(t)\widehat c_g(t) is computed, fitted, or validated. Concrete values and selection procedures for the shared price and fresh-token allowance are also missing. Appendix C.5 identifies the training data through the project path lora_quarantine_v3, without adequately specifying its source, size, or construction. The three-seed results report means without variability or individual-seed tokens-to-target. Furthermore, the reported target cost is determined by the first crossing of the three-seed mean accuracy curve, which is different from averaging each seed’s cost to reach the target. These omissions directly affect the reproducibility and interpretation of the central efficiency claims; describing the existence of JSONL ledgers does not resolve them.

Questions for authors
  1. Assessment-critical validation: Under a matched token budget, how does HAVER-GRPO compare with a recovery baseline that allows at most CC additional responses but stops immediately after the first success? Please retain the same replacement mechanism. This comparison is necessary to identify how much of the improvement comes from adaptive allocation itself.

  2. Clarification: Please specify the response-cost estimator, shared price λ\lambda, fresh-token allowance, development-set selection procedure, and actual training-data manifest. Is Beta(1,1) explicitly a prior for groups already known to be all wrong, or is it carried over from before screening? The paper permits the former interpretation, so the absence of an additional KK failures is not, by itself, an established error.

  3. Assessment-critical validation: Can you provide per-seed accuracy-versus-token curves and explain why the 1.5B setting improves downstream accuracy despite a higher additional token cost per recovered group? Does a matched-token comparison with at least one closely related adaptive allocation method support the reported advantage? Reproducing every related method is unnecessary, but one direct comparison would materially strengthen the evidence.

Reviewer_VhBKOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose a controller that, for GRPO groups whose KK responses are all wrong, decides whether to draw another response by comparing an estimated success hazard per predicted token cost against a shared shadow price. The first verified success replaces a failed slot, so the optimizer still sees KK responses, and every generated token is charged to the budget. With Qwen2.5-Math-1.5B/7B-Instruct (LoRA, 300 updates, three seeds), HAVER-GRPO improves the five-suite mean over DAPO by 1.4–1.6 points and reduces tokens-to-target by 34.7% and 21.4% relative to uniform GRPO.

Critical strengths
  • A practical question. Asking whether one more response is worth its token cost refines dynamic sampling, and the fixed-KK reconstruction keeps the optimizer interface unchanged.
  • Careful accounting. All initial, successful and discarded responses are charged, and total tokens are distinguished from tokens-to-target.
Critical weaknesses
  • The hazard is not group-specific. All groups start from Beta(1,1) and see no success before recovery, so h^g(t)=1/(2+t)\hat h_g(t) = 1/(2+t) is identical across groups. The hazard component therefore reduces to an implicit cap, and the KK initial failures are not even used.
  • The gains are not explained by the mechanism. At 1.5B, HAVER recovers about half as many groups as Fixed-cap (653 against 1,240), yet it is better on every benchmark. At 7B, both methods recover about the same number of groups.
  • Weak baselines. Fixed-cap keeps sampling after a success, so the token savings over it hold by design. A first-success baseline (λ=0\lambda = 0) and compute-matched baselines are missing.
  • Reporting and reproducibility. No error bars or per-seed results are given, although the differences between arms are of 0.2–2.5 points. In addition, the training data ("lora_quarantine_v3") is not described, and the 7B model surprisingly has more all-wrong groups (about 60%) than the 1.5B one (about 43%).
Questions for authors
  1. In what sense is the hazard group-specific? Why is the prior not conditioned on the KK initial failures?
  2. Why does HAVER outperform Fixed-cap while recovering half as many groups? Is the cost index biased towards short responses?
  3. Could you add the λ=0\lambda = 0 and compute-matched baselines, together with per-seed standard errors?
  4. What is the training data, and how was it decontaminated against the evaluation suites?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.