acceptodds
Under review as a conference paper at ICLR 2027

CAST: Credit Assignment on Selective Reasoning Trees for Long-Chain Reasoning

Abstract

Reinforcement learning with verifiable rewards has become the standard recipe for improving long-chain reasoning. However, current methods struggle to put compute and training signal where they matter most in a trajectory. On hard problems, an early mistake often derails the trajectory, and later reflection rarely fixes it. On easier problems, once the model has enough information to determine the correct answer, it keeps generating redundant verification and self-checks. Consequently, training signals get diluted across uninformative steps, and rollout compute is wasted on reasoning that no longer changes the outcome. To resolve both bottlenecks, we propose Credit Assignment on Selective Reasoning Trees (CAST), a tree-structured RL framework for long-chain reasoning. During rollout, CAST builds selective reasoning trees, expanding only at unresolved branch points where sampled continuations yield mixed outcomes (both correct and incorrect). It then pools all leaf rewards and the tree topology to estimate intermediate step values in closed form, deriving fine-grained edge advantages. Across five diverse backbones, ranging from 1.7B base models to a 35B mixture-of-experts, CAST consistently improves accuracy while generating shorter responses. On Qwen3-4B-Base, CAST outperforms DAPO by 4.03 points on average, cuts response length by 25.8%, and matches AIME24 accuracy with a 1.8× training speedup. By focusing rollouts on decisive steps and sharing credit across the tree, CAST shows that targeted exploration and structured credit assignment make long-chain reasoning both stronger and more efficient. Upon acceptance, we will release our code and model checkpoints.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, chain-of-thought reasoning, credit assignment, LLM reasoning, tree search

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_wEFZOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

TreeRL demonstrated that token credit can be allocated in GRPO by grouping rollouts in a tree, and that the training signal can be made more powerful by generating rollouts only at nodes in the tree with high entropy. Instead of depending on entropy, this paper proposes choosing the next node uniformly at random from among all nodes whose child rollouts include some correct and some incorrect responses, then allocating value to each node using a linear regression formula that corrects for an a priori estimate of the correlation between pairs of nodes.

Critical strengths
  • Expanding only nodes with divergent child-rollout outcomes guarantees that all tokens spent generating new rollouts are useful to generate meaningful reward signals.
  • The linear regression formula allocating rewards to nodes is theoretically well-founded and improves performance in practice.
Critical weaknesses
  • A node is chosen for expansion only if it has zero consistency, which is achieved only if it is already part of at least two previously-generated rollouts. Isn't it problematic to base the entire algorithm on the assumption of nonzero redundancy? A model with sufficiently high temperature could easily generate B=6 rollouts with zero redundancy; in that case, wouldn't the proposed algorithm fallback to unmodified GRPO?
  • Nodes are defined by dividing each rollout into D=5 segments of roughly uniform token count. Does this mean that rollouts of different length could have identical token strings but different segments?
  • If the rollouts through node u have a success probability of p, and if previous rollouts only include two that pass through node u, then the probability that node u has zero consistency is only p^2+(1-p)^2. With the complimentary probability, 2p(1-p), node u is marked as solved, so it would not get expanded.
Questions for authors

See "Weaknesses"

2 comments

Thank you for your valuable feedback and for recognizing the theoretical foundation of our method. We apologize for the misunderstanding regarding the tree construction part of our method, and we appreciate the opportunity to clarify CAST’s workflow.

  • Selective Tree Construction: Rather than passively grouping independent outputs, CAST actively grows trees via top-down, prefix-conditioned expansion (Section 3.1). Each expansion step constitutes one cycle of the following process:

    1. Branching from Current Node: For selected node nn at depth dd, sample continuations from prefix qnq_n until nn has BB branch paths.
    2. Expanding Child Prefixes: For each nonterminal child cc of the current node nn, keep its existing rollouts and sample more from its prefix qcq_c until it has at least M=6M=6 completed rollouts.
    3. Advancing Depth: Evaluate outcome consistency across each child's rollouts. Uniformly sample one unresolved child (with mixed outcomes) to advance to depth d+1d+1, up to maximum depth DD.

    How continuations become tree nodes. When CAST samples from a node at depth dd, it reuses the existing prefix segments and their boundaries. It splits only the newly generated suffix into D−dD-d segments of approximately equal token length. Each new segment becomes one node along the new path.

  • Topology-Aware Credit Assignment: Rather than computing simple heuristic averages, CAST treats intermediate node values as joint posterior inference under a Gaussian tree covariance model, pooling all observed leaf rewards across the tree to derive closed-form edge advantages for DAPO policy updates (Sections 3.2–3.3).

Below, we address each of your specific questions in detail.


Q1: Does CAST rely on accidental prefix overlaps or fallback to GRPO?

In CAST, shared prefixes do not rely on accidental overlaps among independent rollouts. As outlined in our workflow above, each child branch is explicitly expanded by feeding its prefix as the prompt input.

Higher sampling temperatures only diversify newly generated suffixes, while the shared prefix remains fixed in the prompt. Thus, CAST does not degenerate into standard GRPO even at high temperatures. This forward construction preserves strong learning signals at depth: CAST maintains a 4.3× higher density of non-zero advantage segments at depth 5 (Figure 5(a)), and reflection segments achieve an 18.3% error-correction rate versus 4.8% for the baseline (Table 4).

Q2: Will the same prefix be split into different segments if rollout lengths vary?

In CAST, segment boundaries remain fixed once established, so differences in rollout length do not alter the partitioning of earlier prefixes. Specifically, when expanding a node nn at depth dd, its prefix segments from depth 0 to dd are already determined; equal-length segmentation applies only to the newly generated continuation, dividing it into the remaining D−dD-d segments. Therefore, all branches derived from the same node share the same prefix tokens, tree depth, and segment boundaries, regardless of how long their subsequent suffixes are.

Q3: Can CAST stop expanding a branch too early when only a few rollouts are available (e.g., two)?

In CAST, candidate branches are structurally guaranteed to have at least 6 rollouts: as specified in Section 3.1 and Algorithm 1 (lines 9–11), for every candidate child branch derived from a selected node, CAST actively samples continuations directly from its prefix until gathering at least M=6M=6 completed rollouts before evaluating outcome consistency.

On this basis, the choice of completion budget MM balances statistical sufficiency in branch evaluation against rollout compute cost.

  • Existing Ablation in the Paper (Figure 5(c), M∈2,4,6M\in\\{2,4,6\\}): We ablated leaf-path multiplicity MM on Qwen3-1.7B-Base in Figure 5(c). The results show that smaller budgets (M=2M=2 and M=4M=4) suffer from inadequate sample support and achieve noticeably lower validation accuracy, whereas M=6M=6 provides reliable statistical guidance and achieves the best performance among the three settings.

  • Supplementary Ablation on Qwen3-1.7B-Base (M=6M=6 vs M=10M=10): To examine whether an even larger budget is beneficial, we evaluate M=10M=10 on Qwen3-1.7B-Base under the same computational resources.

MMAIME24 (%)AIME25 (%)MATH500 (%)Avg. (%)
M=6M=617.2911.4674.6134.45
M=10M=1016.8810.8373.6733.79

At matched total training compute, although M=10M=10 provides better estimation performance, it incurs a significant increase in computational cost. Therefore, M=6M=6 achieves higher downstream accuracy than M=10M=10 while avoiding the insufficient sample support of M=2M=2, thereby providing a better balance between performance and training efficiency.

We would like to once again thank the reviewer for the constructive comments and hope that our responses have addressed the reviewer’s concerns.

AuthorsOpenReview ↗

A kind reminder as the deadline is approaching

Dear Reviewer,

We again express our deep gratitude to you for spending the time and efforts on our submission. Because it is quite close to the end of rebuttal, we deeply appreciate that the remaining questions / concerns, if any, can be posted earlier, such that we have enough time to address them. We truly hope that our response can clarify your concerns.

We fully understand that it may take time to read our rebuttal in-depth. If there is any additional explanation or experiments that can save your time to understand our paper and clarify the concerns, we will be more than happy to do so.

Respectfully,

Authors

Reviewer_KLLPOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

CAST builds selective reasoning trees by expanding prefixes with mixed sampled verifier outcomes, estimates intermediate node values using a Gaussian-tree covariance model, and trains on child-parent value differences with DAPO. A separate successful-suffix comparison encourages shorter outputs. Experiments across five backbones report improved math accuracy, shorter responses and better time-to-target performance. Component ablations and an independent-continuation value-estimation test support contributions from both selective sampling and topology-aware credit.

Critical strengths

Principled unification of selective tree rollouts and closed-Form Topology-aware credit assignment: Unlike prior tree-based RL methods that expand trees uniformly or by token entropy and rely on local child-mean averages, CAST's verifier-guided expansion rule maintains a 4.3x higher density of non-zero advantage segments at depth 5 while avoiding exponential tree blowup. Strong empirical validation and Insightful Behavioral Diagnostics: The experimental evaluation is thorough, spanning five models up to a 35B MoE and challenging benchmarks, and includes a compute-matched DAPO baseline.

Critical weaknesses

None

Questions for authors

What happens if you apply the exact same contrastive suffix regularization directly to standard DAPO?

1 comment

Thank you for your constructive review and for recognizing our tree construction, credit assignment, and empirical evaluation. We have carefully considered your question and provide our response below.

Q1: What happens if contrastive suffix regularization is applied directly to standard DAPO?

Direct Answer

In short, applying contrastive suffix regularization directly to standard DAPO would effectively act as a whole-response length penalty on nearly the entire response, which prior math RLVR work shows can reduce reasoning accuracy.

CAST avoids this problem because its tree structure protects the core reasoning chain from length pressure, applying shortening incentives only to redundant verification after an answer is reached.


1. Why standard DAPO fails: Collapse into a whole-response length penalty

  • Minimal prefix overlap: Standard DAPO samples each response independently from the root prompt. On our 35B baseline across 214 competition math problems (16 rollouts per problem), correct responses share an average prefix of only 0.50% of their length. Because independent rollouts diverge almost from the start, the "suffix" is effectively the entire response. As a result, suffix comparison becomes a standard whole-response length penalty.

  • Accuracy drop in prior literature: Recent studies confirm that applying whole-response length penalties to mathematical reasoning leads to clear performance drops:

    • Han et al. [1] (ICLR 2026, GRPO-O1): Adding a sequence-level length reward to Dr. GRPO drops mean math accuracy on Qwen3-4B from 69.07% to 67.75% (-1.32%, Table 3).
    • Chen et al. [2] (ACL 2026, Step-GRPO): Adding a group-normalized length penalty to GRPO drops AIME 2025 accuracy on Qwen3-8B from 66.7% to 60.0% (-6.70%, Table 1).

    This happens because whole-response length penalties propagate negative updates indiscriminately across the entire sequence. As a result, essential derivations and helpful reasoning prefixes in longer correct responses are wrongly penalized, impairing reasoning performance.


2. How CAST solves this: Decoupling derivation from redundant verification

CAST's tree structure eliminates this penalty on helpful prefixes directly (Section 3.3). When expanding an all-correct internal node where sampled continuations reach correct answers, CAST keeps the shared reasoning prefix fixed in the prompt context, ensuring that its derivation tokens receive zero length penalty. CAST then extracts contrastive pairs by assigning +1+1 and −1-1 advantages to the shortest and longest continuations (τshort,τlong)(\tau_{\text{short}}, \tau_{\text{long}}) branching from that node. By comparing suffixes that share the same prefix, CAST eliminates redundant verification without penalizing the core reasoning chain.


3. Empirical confirmation

Our experiments directly confirm this behavior: in our component ablation on Qwen3-4B-Base (Table 8), suffix regularization reduces mean response length by 23.0% (from 3,265 to 2,513 tokens) while keeping accuracy virtually unchanged (47.68% vs. 47.75%). Across all three Qwen3 backbones, the complete CAST framework achieves up to a 25.8% token reduction alongside higher accuracy than standard DAPO (Tables 1 and 3).

We once again thank the reviewer for the constructive feedback, and we hope our response addresses your concerns.

References

  1. Peixuan Han et al. Self-Aligned Reward: Towards Effective and Efficient Reasoners. ICLR, 2026.
  2. Benteng Chen et al. Step-GRPO: Internalizing Dynamic Early Exit for Efficient Reasoning. ACL, 2026.
Reviewer_A6hJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

CAST studies how to allocate rollout computation and assign credit during reinforcement learning for long-chain mathematical reasoning. It samples continuations from shared prefixes and continues expanding nodes only when their sampled outcomes include both correct and incorrect answers. A Gaussian tree model estimates intermediate values, and differences between child and parent values provide segment advantages for DAPO updates. The method also favors shorter correct suffixes. The experiments evaluate accuracy, response length, and training efficiency across five backbones.

Critical strengths

S1. CAST achieves modest accuracy gains over the compared baselines. On Qwen3-4B-Base, its average accuracy across AIME24, AIME25, and MATH500 is 47.68%, compared with 45.51% for DAPO with matched per-step GPU time and 46.10% for TreeRL (Table 1).

Critical weaknesses

W1. The methodological novelty is limited. VinePPO [1] estimates values from prefix continuations; SPO-tree [2] uses shared-prefix tree sampling and segment advantages; TreeRL [3] branches using token-level uncertainty. BranPO [4] and TRACE [5] also target outcome contrast from shared prefixes in multi-turn settings. These sampling and credit-assignment components are therefore not new. The main additional contribution lies in the Gaussian estimator together with the specific expansion rule.

W2. The advantage of the Gaussian design is not sufficiently established. Tables 5–6 show that it improves value estimation and downstream accuracy over child means, but this is the only reported alternative estimator. Beating this simple baseline does not establish an advantage over other reasonable estimators, such as smoothing or shrinkage methods that also share information across nodes. Since the Gaussian estimator is a main new component, these comparisons are central to assessing the contribution.

References

[1] Kazemnejad et al. VinePPO: Refining Credit Assignment in RL Training of LLMs. First posted October 2024; v2, June 2025. https://arxiv.org/abs/2410.01679v2

[2] Guo et al. Segment Policy Optimization: Effective Segment-Level Credit Assignment in RL for Large Language Models. First posted May 2025; v2, October 2025. https://arxiv.org/abs/2505.23564v2

[3] Hou et al. TreeRL: LLM Reinforcement Learning with On-Policy Tree Search. June 2025. https://arxiv.org/abs/2506.11902v1

[4] Zhao et al. BranPO: Scalable Contrastive Branch Sampling for Long-Horizon Agentic Reinforcement Learning. First posted February 2026 as Training Multi-Turn Search Agent via Contrastive Dynamic Branch Sampling; v2, June 2026. https://arxiv.org/abs/2602.03719v2

[5] Zou et al. TRACE: A Unified Rollout Budget Allocation Framework for Efficient Agentic Reinforcement Learning. First posted June 2026; v2, August 2026. https://arxiv.org/abs/2606.11119v2

Questions for authors

Q1. Which parts of CAST are new relative to VinePPO, SPO-tree, and TreeRL, and how does its sampling strategy differ from BranPO and TRACE?

Q2. Have you compared the Gaussian estimator with alternatives beyond child means, such as smoothing or shrinking node estimates toward ancestor values, using the same sampled trees and training setup? Such results would help determine whether the gains justify the particular Gaussian design.

6 comments

Response 1 of 2

Q1. Methodological novelty

Thank you for asking us to clarify what CAST contributes. CAST combines joint value estimation on a Gaussian tree with verifier-guided tree construction.

1. Value estimation

All five methods use mean rollout rewards either as value estimates or as baselines for relative advantages. CAST uses the full tree topology to estimate node values jointly.

Optimality. When node values and leaf observations are jointly Gaussian, CAST’s closed-form estimate achieves the lowest mean squared error among all estimators based on the leaf observations. If the joint distribution is non-Gaussian, the same estimate is still optimal among linear estimators when the joint means and covariances match the model (Appendix A.2). In Response 2, we compare estimators on the same trees using observed verifier rewards.

2. Tree construction

2.1 Prior methods and their limitations
  • VinePPO estimates values at selected steps along a trajectory by sampling separate Monte Carlo continuations. The prefixes on the original trajectory determine the sampling scope, leaving VinePPO without active control over where to sample. When the number of prefixes grows linearly with response length nn, resampling long suffixes can require O(n2)O(n^2) generated tokens. This makes stepwise resampling a poor fit for large-scale training on long reasoning chains. Consistent with this limitation, the reported experiments use models up to 7B, with a response cap of only 1k tokens.
  • SPO-Tree sets the branching structure, depth, and the number of tokens generated per intermediate node in advance. If segments are too long, a short answer may end before the first split. If they are too short, a shallow tree can leave most of a long answer in the final segment. Under uniform branching, increasing tree depth to preserve fine-grained credit makes the number of rollouts grow exponentially. In line with these constraints, the reported tree experiments use models up to 1.5B and a training context of at most 4k tokens.
  • TreeRL uses token entropy to choose where to branch. Entropy cannot reliably identify which branch points yield useful training signals. For example, choosing between “thus” and “therefore” can yield high entropy while leaving the reasoning and answer unchanged.
  • BranPO backtracks from a completed trajectory and stops when it finds a continuation with a contrasting outcome. The resulting local tree has a single branch point, which limits how precisely it can assign credit to individual steps after the split.
  • TRACE introduces a success predictor that requires its own training. TRACE then uses the predictor’s scores as a proxy for prefix value when allocating rollouts. These scores do not reliably reflect the actual success rates of continuations from a given prefix. For example, TRACE’s Figure 7 shows that the prefix-level Spearman correlation on DeepScaler often falls between 0.2 and 0.4 during training.
2.2 Advantages of CAST’s tree construction

CAST uses verifier feedback to focus rollout compute on reasoning segments that can provide useful training signals. It grows the tree at prefixes whose continuations yield both correct and incorrect answers, focusing segment-level credit on decisions that can affect correctness. For large-scale, long-horizon RL, this design adapts segment lengths to each answer, keeps tree growth under control, and guides branching directly with verified outcomes.

  1. Proportional segmentation. At depth dd, CAST divides only the new suffix into D−dD-d segments with roughly equal token counts. Existing prefix boundaries stay fixed. The segments are shorter for short answers and longer for long answers, so the credit assignment granularity adapts to each model’s output length within the same depth limit.
  2. Selective expansion. CAST samples complete continuations from candidate child nodes and expands at most one child with mixed outcomes per level. Combined with proportional segmentation, this keeps tree size under control and directs additional rollouts to prefixes where verified outcomes still disagree. In equally sized trees, CAST produces a higher share of segments with nonzero advantage than TreeRPO at every measured depth. At depth 5, the shares are 21.57% and 4.99%, respectively (Figure 5(a)). On AIME24, CAST’s successful correction rate among reflection segments was 3.8× DAPO’s (18.3% vs. 4.8%, Table 4).
  3. Direct verifier feedback. The verifier checks the correctness of sampled continuations and provides a direct signal for choosing where to branch, without requiring a separate predictor.

Response 2 of 2

Q1. Methodological novelty (continued)

Additional compute-matched comparison

To show how the methods compare in practice, we used the public implementations of the four baselines below and compared early checkpoints on Qwen3-4B-Base under matched training compute. CAST achieved the highest accuracy on all three benchmarks.

MethodAIME24 (%)AIME25 (%)MATH500 (%)Average (%)
CAST22.7121.4684.3342.83
VinePPO21.6720.2083.0141.63
SPO-Tree20.4218.9681.6740.35
TreeRL20.8319.9083.7441.49
BranPO20.8319.7882.9541.19

We will add these comparisons and cite BranPO and TRACE in the revised manuscript.

Q2. Comparison with alternative value estimators

Thank you for suggesting comparisons with shrinkage and smoothing estimators. To assess the contribution of CAST’s Gaussian estimator, we tested three alternative estimator families using the protocol in Section 4.4 and Table 5. We evaluated each method on the same 64 MATH500 trees against the same reference values, estimated from 200 independent continuations per internal node.

  • Root-value shrinkage combines the node’s child mean bub_u and the root’s child mean brb_r using v^uroot=(1−α)bu+αbr\hat v_u^{\mathrm{root}}=(1-\alpha)b_u+\alpha b_r.
  • Hierarchical shrinkage reduces parent–child value differences along each path using v^uHS=μr+∑(p,c)∈P(r,u)μc−μp1+λ/Np\hat v_u^{\mathrm{HS}}=\mu_r+\sum_{(p,c)\in P(r,u)}\frac{\mu_c-\mu_p}{1+\lambda/N_p}. Here μx\mu_x is the mean reward of the leaves below node xx, and NpN_p is the number of leaves below parent pp. Agarwal et al., ICML 2022
  • Graph total variation smoothing fits the child means while penalizing absolute differences between values at adjacent nodes, using v^TV=arg⁡min⁡v[12∑u(vu−bu)2+λ∑(p,c)∈E∣vp−vc∣]\hat{\boldsymbol v}^{\mathrm{TV}}=\arg\min_{\boldsymbol v}\left[\frac12\sum_u(v_u-b_u)^2+\lambda\sum_{(p,c)\in E}\lvert v_p-v_c\rvert\right]. Madrid Padilla et al., JMLR 2018
EstimatorSettingNode MSE ↓
Child meanOriginal baseline0.02371
Root-value shrinkageα=0.1\alpha=0.10.02062
Root-value shrinkageα=0.3\alpha=0.30.02121
Hierarchical shrinkageλ=5\lambda=50.02192
Hierarchical shrinkageλ=10\lambda=100.02063
Graph total variation smoothingλ=0.01\lambda=0.010.02505
Graph total variation smoothingλ=0.03\lambda=0.030.02960
CAST (Gaussian)Original fixed setting0.01843

CAST achieved the lowest node MSE across all reported methods and settings. This result supports using shared-prefix covariance to combine leaf rewards across related branches.

Optimality. As discussed in Q1, when node values and leaf observations are jointly Gaussian, CAST’s closed-form estimate minimizes MSE among all estimators based on the leaf observations. Even when the distribution is non-Gaussian, the same estimate remains optimal among linear estimators if the joint means and covariances match the model.

Key derivation (Appendix A.2). On a fixed tree, the root value and shared path determine the covariance between nodes. Let ΣVL\Sigma_{VL} be the node–leaf covariance and ΣLL\Sigma_{LL} the leaf covariance. With centered values VV and leaf observations YY, independent observation noise adds λI\lambda I to ΣLL\Sigma_{LL}. Minimizing the mean squared error of the linear estimate AYAY gives A(ΣLL+λI)=ΣVLA(\Sigma_{LL}+\lambda I)=\Sigma_{VL}. Solving for AA yields

V^=ΣVL(ΣLL+λI)−1Y.\widehat V=\Sigma_{VL}(\Sigma_{LL}+\lambda I)^{-1}Y.

CAST uses the observed verifier rewards as YY.

We will add these results to the mechanism and ablation section and describe the baseline estimators in the appendix.

AuthorsOpenReview ↗

A kind reminder as the deadline is approaching

Dear Reviewer,

We again express our deep gratitude to you for spending the time and efforts on our submission. Because it is quite close to the end of rebuttal, we deeply appreciate that the remaining questions / concerns, if any, can be posted earlier, such that we have enough time to address them. We truly hope that our response can clarify your concerns.

We fully understand that it may take time to read our rebuttal in-depth. If there is any additional explanation or experiments that can save your time to understand our paper and clarify the concerns, we will be more than happy to do so.

Respectfully,

Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global Response to Reviewers

We thank the reviewers and the area chair for their constructive feedback. We appreciate their recognition of CAST’s selective rollouts and topology-aware credit assignment (KLLP, wEFZ) and its accuracy gains across model scales (A6hJ, KLLP). We hope our replies address all three reviewers’ concerns and clarify any ambiguities in our original description. Below we summarize CAST’s contributions and the additional evidence.

Core Contributions

  1. Verifier-guided tree expansion. Our goal is to spend limited rollout compute on reasoning decisions that can affect answer correctness. CAST uses disagreement between correct and incorrect continuations from the same prefix to decide where to expand.

    • Proportional segmentation. At depth dd, CAST divides only the new suffix into D−dD-d segments of roughly equal token count and keeps existing prefix boundaries fixed. Compared with fixed-length segmentation in methods such as SPO-Tree, this adapts credit granularity to response length. Shorter answers get shorter segments, and longer answers fit within the same depth limit DD.
    • Selective expansion. CAST expands at most one child with mixed outcomes per level, limiting tree growth and preserving training signals at depth. At matched tree size, CAST has a higher share of nonzero-advantage segments than TreeRPO at every measured depth (21.57% vs. 4.99% at depth 5, Figure 5(a)). On AIME24, its successful correction rate among reflection segments is 3.8× DAPO’s (18.3% vs. 4.8%, Table 4).
    • Direct verifier feedback. Choosing branches by verified correctness avoids training a separate success predictor and prevents prediction errors from misdirecting rollouts.
  2. Topology-aware credit assignment. CAST uses shared-prefix covariance across the full tree to jointly estimate intermediate node values from leaf rewards. Child–parent value differences give segment-level advantages.

    • Optimality guarantees. Under joint Gaussianity, CAST’s closed-form estimate minimizes MSE among all estimators based on leaf observations. For non-Gaussian distributions, it remains optimal among linear estimators when joint means and covariances match the model (Appendix A.2).

Contrastive suffix regularization compares the lengths of correct continuations from the same prefix, discouraging redundancy while protecting the shared prefix from length penalties. CAST improves average reasoning accuracy on all five backbones and generates 12.5–25.8% fewer tokens than DAPO on the three Qwen3 backbones.

  • Our reply to A6hJ (Q1) compares CAST with VinePPO, SPO-Tree, TreeRL, BranPO, and TRACE.
  • Our first reply to wEFZ summarizes CAST’s workflow.
    • See Sections 3.1–3.3 for details and Algorithm 1 (Appendix A.1) for the full rollout schedule.

Additional Experiments and Clarifications

  1. Compute-matched comparison (A6hJ). At early Qwen3-4B-Base checkpoints under matched training compute, CAST outperformed all four baselines on each benchmark.

    MethodAIME24(%)AIME25(%)MATH500(%)Average(%)
    CAST22.7121.4684.3342.83
    VinePPO21.6720.2083.0141.63
    SPO-Tree20.4218.9681.6740.35
    TreeRL20.8319.9083.7441.49
    BranPO20.8319.7882.9541.19
  2. Value estimators (A6hJ). Following Section 4.4 and Table 5, we compared estimators on the same 64 MATH500 trees, with reference values from 200 independent continuations per internal node. CAST had the lowest MSE, supporting the use of shared-prefix covariance.

    EstimatorLowest reported node MSE↓
    CAST (Gaussian)0.01843
    Child mean0.02371
    Root-value shrinkage0.02062
    Hierarchical shrinkage0.02063
    Graph total variation smoothing0.02505
  3. Rollout budget (wEFZ). Our reply explains fixed-prefix sampling, unchanged segment boundaries, and how CAST checks MM rollouts for mixed outcomes. Our compute-matched Qwen3-1.7B-Base ablation and Figure 5(c) support M=6M=6 as a balance between branch assessment and deeper exploration.

    Rollouts per candidate childAIME24(%)AIME25(%)MATH500(%)Average(%)
    617.2911.4674.6134.45
    1016.8810.8373.6733.79
  4. Suffix regularization on DAPO (KLLP). In our 35B analysis, independently sampled correct answers shared only 0.50% of the shorter answer as a prefix on average. Comparing their suffixes would penalize length across nearly the whole answer, which prior work shows can hurt accuracy. CAST limits the incentive to correct continuations from a fixed prefix. On Qwen3-4B-Base, the regularizer reduced mean response length by 23.0%, with accuracy changing from 47.75% to 47.68%.

We will add the new comparisons and clarify tree expansion and shared prefixes in the revision. We welcome further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.