- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper proposes a rollout allocation method called CABRA(Cost-Aware Bayesian Rollout Allocation), which determines the group size for each prompt based on a Beta-Bernoulli estimation of its reward history. The core motivation is that existing methods either filter out zero-variance prompts after generation or resample candidate prompts until a sufficient number of mixed groups are obtained. CABRA first reads the history of each prompt and selects prompts to recieve complete group. After that, the remaining budget is refilled based on previous observations. With Qwen2.5-1.5B-Instruct on the GSM8k dataset, CABRA achieves comparable performance while saving significant amount of computation compared to the other baselines.
Critical strengths
The proposed method CABRA uses a simple and deterministic rollout allocation strategy with a well-formulated problem definition. Since the group size for every prompt is determined in advance and no additional model is required, the cost of each step is predictable without excessive additional cost.
Critical weaknesses
My major concern is the validity of training and evaluation settings. As far as I know, Qwen2.5-1.5B-Instruct acheives accuracy over 70% even without any training. However, the reported score after RLVR is around 50%, which is significantly lower than the base model’s score. I suspect that this result comes from the short response length or that the model became worse after training. Since the analysis relies on this experimental setting, it is hard to agree with the argued effectiveness of this method. In addition, using only 256 prompts from a simple dataset, GSM8K and only Qwen2.5-1.5B-Instruct and a short 320 token limit, and small train batch sizes (2 and 8) make the overall results questionable.
My second concern is that it is unclear whether the cost savings come from the proposed components. I think random selection at the same budget, length only selection, or small batch sized DAPO should be compared against the proposed method to claim the contribution of the Bayesian scroing or the refill.
Questions for authors
Can you clarify the evaluation setting and score of Qwen2.5-1.5B-Instruct?
Can you provide the experiment result on more reliable settings?
Can you conduct additional experiment related to my second concern?