- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
TreeRL demonstrated that token credit can be allocated in GRPO by grouping rollouts in a tree, and that the training signal can be made more powerful by generating rollouts only at nodes in the tree with high entropy. Instead of depending on entropy, this paper proposes choosing the next node uniformly at random from among all nodes whose child rollouts include some correct and some incorrect responses, then allocating value to each node using a linear regression formula that corrects for an a priori estimate of the correlation between pairs of nodes.
Critical strengths
- Expanding only nodes with divergent child-rollout outcomes guarantees that all tokens spent generating new rollouts are useful to generate meaningful reward signals.
- The linear regression formula allocating rewards to nodes is theoretically well-founded and improves performance in practice.
Critical weaknesses
- A node is chosen for expansion only if it has zero consistency, which is achieved only if it is already part of at least two previously-generated rollouts. Isn't it problematic to base the entire algorithm on the assumption of nonzero redundancy? A model with sufficiently high temperature could easily generate B=6 rollouts with zero redundancy; in that case, wouldn't the proposed algorithm fallback to unmodified GRPO?
- Nodes are defined by dividing each rollout into D=5 segments of roughly uniform token count. Does this mean that rollouts of different length could have identical token strings but different segments?
- If the rollouts through node u have a success probability of p, and if previous rollouts only include two that pass through node u, then the probability that node u has zero consistency is only p^2+(1-p)^2. With the complimentary probability, 2p(1-p), node u is marked as solved, so it would not get expanded.
Questions for authors
See "Weaknesses"
2 comments
A kind reminder as the deadline is approaching
Dear Reviewer,
We again express our deep gratitude to you for spending the time and efforts on our submission. Because it is quite close to the end of rebuttal, we deeply appreciate that the remaining questions / concerns, if any, can be posted earlier, such that we have enough time to address them. We truly hope that our response can clarify your concerns.
We fully understand that it may take time to read our rebuttal in-depth. If there is any additional explanation or experiments that can save your time to understand our paper and clarify the concerns, we will be more than happy to do so.
Respectfully,
Authors
Thank you for your valuable feedback and for recognizing the theoretical foundation of our method. We apologize for the misunderstanding regarding the tree construction part of our method, and we appreciate the opportunity to clarify CAST’s workflow.
Selective Tree Construction: Rather than passively grouping independent outputs, CAST actively grows trees via top-down, prefix-conditioned expansion (Section 3.1). Each expansion step constitutes one cycle of the following process:
How continuations become tree nodes. When CAST samples from a node at depth d, it reuses the existing prefix segments and their boundaries. It splits only the newly generated suffix into D−d segments of approximately equal token length. Each new segment becomes one node along the new path.
Topology-Aware Credit Assignment: Rather than computing simple heuristic averages, CAST treats intermediate node values as joint posterior inference under a Gaussian tree covariance model, pooling all observed leaf rewards across the tree to derive closed-form edge advantages for DAPO policy updates (Sections 3.2–3.3).
Below, we address each of your specific questions in detail.
Q1: Does CAST rely on accidental prefix overlaps or fallback to GRPO?
In CAST, shared prefixes do not rely on accidental overlaps among independent rollouts. As outlined in our workflow above, each child branch is explicitly expanded by feeding its prefix as the prompt input.
Higher sampling temperatures only diversify newly generated suffixes, while the shared prefix remains fixed in the prompt. Thus, CAST does not degenerate into standard GRPO even at high temperatures. This forward construction preserves strong learning signals at depth: CAST maintains a 4.3× higher density of non-zero advantage segments at depth 5 (Figure 5(a)), and reflection segments achieve an 18.3% error-correction rate versus 4.8% for the baseline (Table 4).
Q2: Will the same prefix be split into different segments if rollout lengths vary?
In CAST, segment boundaries remain fixed once established, so differences in rollout length do not alter the partitioning of earlier prefixes. Specifically, when expanding a node n at depth d, its prefix segments from depth 0 to d are already determined; equal-length segmentation applies only to the newly generated continuation, dividing it into the remaining D−d segments. Therefore, all branches derived from the same node share the same prefix tokens, tree depth, and segment boundaries, regardless of how long their subsequent suffixes are.
Q3: Can CAST stop expanding a branch too early when only a few rollouts are available (e.g., two)?
In CAST, candidate branches are structurally guaranteed to have at least 6 rollouts: as specified in Section 3.1 and Algorithm 1 (lines 9–11), for every candidate child branch derived from a selected node, CAST actively samples continuations directly from its prefix until gathering at least M=6 completed rollouts before evaluating outcome consistency.
On this basis, the choice of completion budget M balances statistical sufficiency in branch evaluation against rollout compute cost.
Existing Ablation in the Paper (Figure 5(c), M∈2,4,6): We ablated leaf-path multiplicity M on Qwen3-1.7B-Base in Figure 5(c). The results show that smaller budgets (M=2 and M=4) suffer from inadequate sample support and achieve noticeably lower validation accuracy, whereas M=6 provides reliable statistical guidance and achieves the best performance among the three settings.
Supplementary Ablation on Qwen3-1.7B-Base (M=6 vs M=10): To examine whether an even larger budget is beneficial, we evaluate M=10 on Qwen3-1.7B-Base under the same computational resources.
At matched total training compute, although M=10 provides better estimation performance, it incurs a significant increase in computational cost. Therefore, M=6 achieves higher downstream accuracy than M=10 while avoiding the insufficient sample support of M=2, thereby providing a better balance between performance and training efficiency.
We would like to once again thank the reviewer for the constructive comments and hope that our responses have addressed the reviewer’s concerns.