- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
In this paper, the authors introduce a sampling method called treerollout for RL training. They find that GPU and tool calling can bring one idle when another is full-used. They push the model to generate additional trajectory when this appear. I find the idea is interesting and the studied problem is important. However, some details are confusing and the method seems to have biased estimation on advantage of GRPO. I still doult the stablity of the method. I think more details should be claimed well.
Critical strengths
- The paper is well motivated. TreeRollout addresses resource underutilization during RL training, a common and practically important problem.
- The paper is well organized, and the figures help convey the approach clearly.
Critical weaknesses
- The proposed sampling procedure alters the distribution of model-generated outputs. Consequently, the resulting advantage estimates may be biased relative to those obtained with standard GRPO. The implications of this potential bias remain unclear.
- During batch rollout, greater resource contention can lead to more responses being sampled for certain instances. This allocation depends on runtime conditions and may devote excessive resources to particularly easy or difficult instances.
- The improvements reported in Table 1 appear marginal. Moreover, the evaluation compares only rollout time and does not report training time. Because TreeRollout generates additional trajectories, it may increase the computational cost of policy updates.
Questions for authors
- During policy updates, shared prefixes may contribute to the optimization objective multiple times. Could this disproportionately weight these prefixes, introduce optimization bias, or destabilize training? Have the authors observed such effects, and does the method require a smaller learning rate to remain stable?
- Generating alternative reasoning continuations after tool calls when the GPU is idle seems reasonable. However, what justifies selecting random branching points for tool calls when tool-execution resources are idle? Although the method imposes constraints on point selection, could this procedure still produce low-quality reasoning trajectories and adversely affect model performance?
- How many additional trajectories does TreeRollout generate, and which factors determine this number? To what extent do the performance gains arise from the sampling strategy itself, rather than simply from using more training data? If standard GRPO were trained on a comparable number of trajectories, how would its runtime and performance compare? The current ablation studies seem insufficient to distinguish these effects.
- What is the total end-to-end training time of TreeRollout, including both rollout generation and policy updates, and how does it compare with standard GRPO?