acceptodds
Withdrawn from ICLR 2027

CABRA: Cost-Aware Bayesian Rollout Allocation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Group-relative reinforcement learning can spend much of its computation on response groups whose rewards barely differ. Discarding such groups after generation does not recover their sampling cost, and searching until a fixed number of useful groups is found can require many additional prompts. We introduce Cost-Aware Bayesian Rollout Allocation (CABRA), which fixes each prompt's number of responses before generation. From each prompt's past outcomes, it estimates, per expected token, the probability that a group will contain both correct and incorrect responses. A fixed number of top-ranked prompts receive complete groups of the full dense size. Skipped prompts with past outcomes then receive two-response refills if their expected token cost fits the remaining budget. All groups are generated once, without searching for replacements. On Qwen2.5-1.5B and GSM8K, CABRA uses 38.5% less adapter-backward work (LoRA backward-pass FLOPs) than dense DAPO across five controlled seeds, with 623.2 versus 621.4 mean correct answers out of 1,319. Without refills, work falls by 52.2% with 622.2 mean correct answers, so most of the saving comes from limiting the number of complete groups, and refills spend part of it. Generated tokens follow the same ordering. The accuracy differences are not statistically significant. We also compare CABRA with five external allocators in an eight-prompt protocol.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, verifiable rewards, rollout allocation, Bayesian estimation, computational efficiency

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_LuvmOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes a rollout allocation method called CABRA(Cost-Aware Bayesian Rollout Allocation), which determines the group size for each prompt based on a Beta-Bernoulli estimation of its reward history. The core motivation is that existing methods either filter out zero-variance prompts after generation or resample candidate prompts until a sufficient number of mixed groups are obtained. CABRA first reads the history of each prompt and selects prompts to recieve complete group. After that, the remaining budget is refilled based on previous observations. With Qwen2.5-1.5B-Instruct on the GSM8k dataset, CABRA achieves comparable performance while saving significant amount of computation compared to the other baselines.

Critical strengths

The proposed method CABRA uses a simple and deterministic rollout allocation strategy with a well-formulated problem definition. Since the group size for every prompt is determined in advance and no additional model is required, the cost of each step is predictable without excessive additional cost.

Critical weaknesses

My major concern is the validity of training and evaluation settings. As far as I know, Qwen2.5-1.5B-Instruct acheives accuracy over 70% even without any training. However, the reported score after RLVR is around 50%, which is significantly lower than the base model’s score. I suspect that this result comes from the short response length or that the model became worse after training. Since the analysis relies on this experimental setting, it is hard to agree with the argued effectiveness of this method. In addition, using only 256 prompts from a simple dataset, GSM8K and only Qwen2.5-1.5B-Instruct and a short 320 token limit, and small train batch sizes (2 and 8) make the overall results questionable.

My second concern is that it is unclear whether the cost savings come from the proposed components. I think random selection at the same budget, length only selection, or small batch sized DAPO should be compared against the proposed method to claim the contribution of the Bayesian scroing or the refill.

Questions for authors

Can you clarify the evaluation setting and score of Qwen2.5-1.5B-Instruct?

Can you provide the experiment result on more reliable settings?

Can you conduct additional experiment related to my second concern?

Reviewer_HKSqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents CABRA (Cost-Aware Bayesian Rollout Allocation), a cost-aware bayesian rollout allocation method for accelerating the RL training process. The key observation is that generating response groups with identical correctness outcomes can waste computation because such groups provide little relative learning signal. CABRA uses historical prompt correctness and response-length estimates to allocate rollouts before generation. It ranks prompts according to the Bayesian posterior probability of obtaining mixed-correctness groups per expected token, assigns full response groups to selected prompts, and optionally allocates two-response refills to previously observed prompts using the remaining token budget. Experiments using Qwen2.5-1.5B-Instruct on GSM8K show that CABRA reduces estimated LoRA adapter-backward FLOPs by 38.5% and generated tokens by 37.9% relative to dense DAPO, with similar final accuracy across five seeds.

Critical strengths

S1: Reducing uninformative rollouts is an important direction for improving the RL computational efficiency.

S2: The paper includes comparisons with multiple relevant rollout allocation approaches and achieves better performance.

Critical weaknesses

W1: All experiments use a single 1.5B-parameter model, one GSM8K task, and a very small training pool of only 256 prompts. These settings are considerably smaller than typical post-training workloads.

W2: The key idea of prioritizing prompts according to historical success probabilities and expected computational costs is highly related to existing selective rollout and adaptive allocation methods. It would be better to discuss more about the differences.

Questions for authors
  1. Can CABRA maintain its efficiency advantages when applied to larger reasoning models and other datasets such as MATH, AIME, or coding benchmarks? How sensitive is the approach to longer responses and different reward distributions?

  2. Could the authors provide training curves showing validation accuracy against cumulative generated tokens, total FLOPs, and wall-clock time?

Reviewer_A6vKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

CABRA decides how many responses to generate for each prompt before sampling begins. It keeps a history of correctness and response length for each prompt. These estimates guide the selection of prompts that receive full response groups. Some skipped prompts receive two-response refills when the remaining budget allows.

The experiments use Qwen2.5-1.5B-Instruct and a 256-prompt GSM8K training pool. CABRA reduces estimated adapter-backward work by 38.5% relative to dense DAPO. Mean test accuracy is close: 623.2 versus 621.4 correct answers out of 1,319. Generated tokens fall by 37.9% across four confirmation seeds. The paper also compares six allocators in an eight-prompt setting.

Critical strengths
  1. The method is simple and inexpensive to use. Allocation relies on stored prompt histories. It needs no extra model inference or replacement search. This makes it straightforward to add to an existing group-relative training setup.

  2. The experiments explain where the savings come from. The no-refill and no-history ablations are useful, as are the generated-token counts. I also appreciate the seed-level results and paired confidence intervals. The authors are appropriately cautious about the small accuracy differences.

Critical weaknesses
  1. The main comparison with simpler allocation rules is missing. I would like to know whether Bayesian selection does better than choosing prompts randomly or by response length at the same budget. The current experiments do not answer this.

    The refill results raise a related question. In Table 1, removing refills saves 52.2% of adapter-backward work, compared with 38.5% for full CABRA. Mean correct counts are 622.2 and 623.2, respectively. The confidence interval for this one-answer difference is [-6.02, 8.02] (Table 11). These results leave the benefit of refills unclear. There is also no no-refill ablation in the eight-prompt setting, where the refill ranking can affect which prompts are selected. The paper acknowledges these gaps. They are central to judging what the proposed method adds.

  2. It is unclear how much improvement from RL is being preserved. The paper reports final accuracy without reporting the initial checkpoint's accuracy under the same decoding setup. All runs use one model and a small recurring training pool. I therefore cannot tell how much the methods improve over initialization. Similar final scores are difficult to interpret without that reference point. Accuracy-versus-training-cost curves would also help establish whether CABRA reaches a useful level of performance more cheaply.

Questions for authors
  1. Can you compare CABRA with random and length-only selection at a matched budget? Can you also report a no-refill ablation for the eight-prompt setting? These comparisons could change my assessment.

  2. What accuracy does the initial checkpoint achieve under the same evaluation procedure? How much does each method improve over it? Learning curves for the existing configurations would be helpful.

  3. How often does the eight-prompt setting produce several eligible refill candidates with room for only some of them? I would like to understand how often the refill ranking affects the decision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.