- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper is about the trade-off between spending RL rollouts on the same task as compared to a diversity of tasks. Using Qwen3.5-4B with group-normalized on-policy policy gradient, the authors take two tasks (SQL and math) and fix the amount of rollouts the model is allowed to train with. They then vary the amount of each task type the model sees: 64, 256, and 1024. The experiments showed the best performance on the training set at 256, but that the differences were negligible on the test set. They also noticed that despite fixed rollouts, the actual amount of wall clock time and tokens used varies a lot based on the number of tasks. The key takeaway is that task set size in isolation is not a good metric of how to allocate training time.
Critical strengths
- Careful experimental design. As an empirical study, this paper is well designed and reports comprehensively. They report task coverage, generated tokens, active optimizer updates, wall-clock time, and acquisition costs rather than relying solely on task counts or rollout budgets. The comparison between rollout count and number of tokens shows how different budget definitions can lead to different experimental comparisons. This is a useful methodological contribution for evaluating execution-feedback post-training. I'm also happy they reported negative results with respect to the effect of the adaptation methods.
Critical weaknesses
- Limited insight into the central research question. The paper aims to understand how the allocation of a fixed training budget between task coverage and repeated exposure affects learning. The experiments show little difference on the test set, even in comparison to the base model (which is a little suspicious: we know broadly that RLVR works, so I suspect there's some training issue that is worth double-checking). Consequently, the experiments don't establish a clear or generalizable relationship between task allocation and downstream performance, nor do they provide much insight into when additional task coverage should be preferred over repeated exposure. The principal conclusions instead concern the importance of accounting for training costs and evaluation endpoints, which, while interesting, offer limited new scientific insight into the motivating question. If they were to make this a key contribution of the paper by pointing out this as a common issue in research, I would like a data-driven study of the literature.
Questions for authors
-
The main positive finding is that achieves higher SQL/GSM training accuracy. However, the reserved final evaluation uses GSM8K and MBPP, excluding SQL, which constitutes half of the training tasks. Could the authors clarify the rationale for this difference in evaluation domains? Is there independent held-out SQL evidence indicating whether the training set improvement generalizes?
-
The three allocations achieve nearly identical performance on the test set, with none consistently improving over the base model. Why didn't the training change performance relative to the base model? Also, do the results indicate that the different allocations produce essentially the same predictions, or that different allocations do well on a statistically significantly different set of problems?