acceptodds
Under review as a conference paper at ICLR 2027

Learn to Reason Efficiently on Real Paths via Tree-Structured Policy Optimization

Abstract

Large reasoning models often waste substantial test-time compute, and recent length-aware reinforcement learning addresses this issue by rewarding short and correct trajectories. We reveal a previously overlooked training–inference mismatch in the prevailing trajectory-level paradigm for length-aware RL: positive learning signals are concentrated on trajectories whose early reasoning is already well directed, whereas inference requires the model to continue from its own potentially erroneous or exploratory prefixes. Through prefix-level continuation analysis, we empirically verify this mismatch and show that conditioning reinforcement learning on self-generated prefixes substantially improves both reasoning accuracy and efficiency. To address this issue, we propose **T**ree-structured **L**ength-aware **P**olicy **O**ptimization (**TLPO**), which performs prefix-conditioned credit assignment by comparing continuations branching from the same self-generated prefix and optimizes their efficiency with segment-level length-aware rewards. Across three representative reasoning models and five benchmarks, TLPO reduces average reasoning length by 56.0% to 80.3% while improving average accuracy by 0.5 to 4.1 percentage points, with out-of-domain evaluations showing consistent gains on scientific reasoning and coding tasks. These results establish prefix-conditioned optimization over self-generated reasoning paths as an effective principle for training efficient reasoning models.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Efficient Reasoning, Reinforcement Learning, Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_yFQhOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the credit assignment problem in length-aware reinforcement learning (RL). The authors argue that positive updates in conventional trajectory-level RL are primarily concentrated on trajectories with relatively high-quality early reasoning steps. However, during actual inference, models must also continue generating from self-generated prefixes that may contain exploration, detours, or intermediate errors.The authors propose TLPO, which constructs multiple continuations from shared prefixes through a three-level tree rollout, with a default branching factor of (1, 8, 8). Leaf nodes receive rewards based on final answer correctness and a length threshold relative to the remaining context.

Critical strengths
  1. The research problem is practically meaningful, focusing on the trade-off between reasoning accuracy and efficiency. The paper addresses the common issue of excessive reasoning length in large language models, aiming to reduce token consumption while maintaining or even improving accuracy. This research direction has clear practical relevance.

  2. The experiments cover multiple models and reasoning tasks, demonstrating substantial efficiency improvements. The paper conducts experiments across models of different scales and initialization settings, covering several reasoning benchmarks, including AIME, MATH500, AMC, and GPQA-Diamond. The experimental results show that TLPO can significantly reduce reasoning token consumption while maintaining or improving accuracy.

Critical weaknesses
  1. The explanation of the training–inference mismatch is not sufficiently clear.I have some concerns regarding the training–inference mismatch proposed in the paper. The authors observe that trajectories with positive advantages tend to contain higher-quality prefixes, which is understandable. However, in on-policy RL, rollouts during both training and inference are generated by the model itself. Under the same policy, input distribution, and sampling configuration, their prefix distributions should, in principle, be consistent. I would appreciate further clarification of this concept and an explanation of why this phenomenon should be considered a training–inference mismatch.

  2. The proposed method is similar to existing Tree-based GRPO approaches, and its novelty needs further clarification. The main ideas of TLPO, including generating multiple continuations from shared prefixes, estimating intermediate node values using subtree outcomes, and computing local advantages, overlap considerably with existing Tree-based GRPO methods. For example, "Tree Search for LLM Agent Reinforcement Learning" already adopts similar tree-structured sampling and fine-grained credit assignment, while "Why Tree-Style Branching Matters for Thought Advantage Estimation in GRPO" also investigates multiple continuation sampling from shared prefixes and provides related theoretical analysis.

Questions for authors
  1. Regarding Training–Inference Mismatch.Could the authors further clarify the precise definition of training–inference mismatch in the paper? In particular, under the on-policy RL setting, why should the difference in prefix quality between positive-advantage trajectories and the overall sampled trajectories be interpreted as a distribution mismatch between training and inference?

  2. Regarding Baselines and the Contributions of Individual Components.I suggest that the authors include a standard GRPO baseline, as well as a comparison with Tree-GRPO under the same length-aware reward and training budget, to better disentangle the contributions of tree branching, fine-grained credit assignment, and reward shaping.

4 comments

We thank the reviewer for the comments. We would like to clarify the precise meaning of the training–inference mismatch and further explain the novelty and contributions of TLPO, supported by additional experimental evidence.

W1 / Q1. Clarification of the Training–Inference Mismatch

We agree that on-policy RL produces the same prefix distribution during training and inference under identical conditions. However, the mismatch lies in the optimization signals rather than the sampling distribution: trajectory-level length-aware RL preferentially reinforces globally short and correct trajectories with well-directed prefixes, while successful recovery behaviors from suboptimal prefixes—also encountered at inference time—may receive insufficient positive reinforcement.

To illustrate this distinction, consider a two-stage reasoning problem where the model can initially explore strategy A or B, but cannot determine which works without further reasoning. Suppose eight trajectories are sampled:

  • 4 trajectories: Start with B and directly reach the correct answer (reward = 1.5).
  • 2 trajectories: Start with A, discover that it fails, switch to B, and reach the correct answer (reward = 1.0).
  • 2 trajectories: Start with A, continue in the wrong direction, and fail (reward = 0).

Under trajectory-level GRPO with length-aware rewards, the group mean reward is 1.0, with a standard deviation of approximately 0.612. Consequently, the successful recovery trajectories (A → B → Correct) receive zero normalized advantage, despite reaching correct answers. Their recovery tokens thus receive no direct positive reinforcement, while trajectories starting directly with B receive positive advantages.

In contrast, TLPO compares continuations conditioned on the same prefix A. Among the four continuations, two successfully recover (reward = 1.0), while two fail (reward = 0). The local group mean is 0.5, with a standard deviation of 0.5. Therefore, successful recovery behaviors receive a positive normalized advantage of +1.0, rather than zero.

This example illustrates the key distinction: the issue is not whether unfavorable prefixes are sampled during training, but whether successful recovery behaviors from these prefixes are effectively reinforced. Although negative advantages can penalize unsuccessful trajectories, they do not necessarily provide positive learning signals for successful recovery under the same prefix.

The prefix-quality differences observed in Section 2.2 illustrate this selective reinforcement, while Section 2.3 provides initial evidence that conditioning training on self-generated prefixes improves reasoning efficiency and accuracy.

Additional experiment: Continuation from fixed failure-associated prefixes.

To further directly validate our hypothesis, we conduct an experiment on MATH500 using identical prefixes extracted from unsuccessful trajectories of the original model. We provide the same prefixes to Base, LASER, and TLPO and evaluate their continuation accuracy and length.

Continuation ModelAccuracy (%)Length
Base23.139,784
LASER19.202,239
TLPO32.751,157

Compared with LASER, TLPO improves continuation accuracy by 13.56 percentage points while reducing continuation length by 48.3%. Notably, LASER decreases continuation accuracy relative to Base, whereas TLPO improves both accuracy and efficiency.

These results provide direct evidence that TLPO learns more effective continuations under unfavorable self-generated prefixes. We will clarify that the training–inference mismatch in our paper refers to an advantage-induced mismatch between training optimization signals and inference-time reasoning requirements, rather than a shift in the underlying on-policy sampling distribution.

W2 / Q2. Novelty, Baselines, and Component Analysis

We would like to clarify that TLPO goes beyond simply adopting existing tree-based GRPO methods. Our contributions and supporting evidence are summarized as follows:

1. Conceptual and Methodological Contributions. Our contribution lies in both identifying an overlooked problem and developing a targeted solution.

  • Problem: Existing trajectory-level length-aware RL preferentially reinforces globally short and correct trajectories, potentially overlooking effective recovery behaviors from suboptimal self-generated prefixes. To the best of our knowledge, we are the first to identify and systematically investigate this overlooked imbalance in optimization signals, revealing a fundamental mismatch between the reasoning behaviors reinforced during training and those required at inference time. This is an important yet previously unexplored issue in efficient reasoning RL.
  • Method: Rather than simply adopting existing Tree RL, TLPO introduces prefix-conditioned length-aware optimization, integrating tree-structured segment-level credit assignment with a specially designed reward that accounts for the existing prefix and remaining reasoning budget. This enables efficient recovery behaviors to receive positive reinforcement even under suboptimal prefixes.

2. Component Ablations. As reported in **Section 4.4 **, our ablations demonstrate that tree-structured optimization and length-aware reward shaping are complementary, and their combination achieves substantially better accuracy–efficiency trade-offs. On AIME24, removing tree sampling reduces accuracy from 32.92% to 28.54% and increases length from 2,953 to 4,327 tokens, while removing the length penalty reduces accuracy to 31.42% and increases length to 4,533 tokens.

3. Comparisons with Existing Tree-based RL. Table 5 already provides direct comparisons with SPO-tree and TreePO under matched training length constraints. On AIME25, TLPO achieves 27.29% / 2,874 tokens, outperforming SPO-tree (24.92% / 4,522) and TreePO (23.75% / 3,925). Similarly, on MATH500, TLPO achieves 87.46% / 1,341 tokens, compared with SPO-tree (87.08% / 2,045) and TreePO (84.99% / 1,765). These results demonstrate that directly applying existing tree-based RL methods under the same length constraints does not achieve TLPO's accuracy–efficiency trade-off.

4. Controlled GRPO-style Baseline with Identical Length-aware Reward. Our TLPO w/o Tree Sampling variant serves as a trajectory-level GRPO-style baseline, retaining the same base model, training data, length-aware reward design, and training settings, while removing tree sampling and prefix-conditioned credit assignment. We further extend this controlled comparison to all five benchmarks, using 32 evaluation rollouts per problem:

Benchmarkw/o Tree Acc / LenTLPO Acc / Len
AIME2428.54 / 4,32732.92 / 2,953
AIME2523.21 / 5,79027.29 / 2,874
MATH50083.84 / 2,21587.46 / 1,341
AMC202374.70 / 3,27676.41 / 2,003
GPQA-Diamond35.06 / 5,16837.06 / 2,338
Average49.07 / 4,15552.23 / 2,302

TLPO consistently improves accuracy and reduces response length across all five benchmarks, achieving a 3.16 percentage-point gain in average accuracy and a 44.6% reduction in average response length over the controlled trajectory-level baseline.

Taken together, these results demonstrate that TLPO's contribution is not tree sampling or segment-level advantage estimation in isolation, but identifying and addressing a previously overlooked credit-assignment problem in efficient reasoning RL through their integration with prefix-conditioned length-aware optimization. We will further clarify these contributions, incorporate the suggested related works, and include the expanded experimental results in the revised manuscript.

Dear Reviewer yFQh,

Thank you again for your feedback. With the discussion period ending soon, we would appreciate your thoughts on our clarification of the training–inference mismatch and the added controlled GRPO-style comparison. We would be happy to discuss any remaining concerns about the motivation or contribution.

Reviewer_yFQhOpenReview ↗

Response to rebuttal

Thank you for your detailed response; your rebuttal has addressed my concerns, and I have increased my rating accordingly.

Reviewer_PDVYOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Tree-structured Length-aware Policy Optimization (TLPO) to improve reasoning efficiency through prefix-conditioned reinforcement learning. It uses tree-structured rollouts, segment-level advantage estimation, and length-aware rewards to encourage shorter and more accurate continuations from self-generated reasoning prefixes. Experiments on mathematical reasoning and out-of-domain benchmarks show substantial reductions in reasoning length while maintaining or improving accuracy.

Critical strengths
  1. The method achieves substantial reasoning-length reductions across three models and multiple benchmarks, generally maintaining or improving accuracy.
  2. The paper identifies an interesting limitation of trajectory-level length-aware RL: training tends to favor short, successful trajectories, while models must also continue from less favorable prefixes during inference.
Critical weaknesses
  1. Tree sampling and segment-level advantage estimation have already been explored[1-4]. The main difference seems to be the length-aware reward. Table 5 should compare these methods.
  2. The claimed training–inference mismatch and improved error recovery are not directly demonstrated. Sec. 2.2 mainly shows that high-advantage trajectories have better prefixes, which may reflect selection bias. The authors should evaluate whether TLPO actually improves continuation accuracy from the same incorrect or suboptimal prefixes. It would be better to directly evaluate whether the model improves at recovering from incorrect reasoning prefixes.
  3. Sec. 3.1 randomly truncates reasoning at [0, 0.4] and [0.2, 0.6], without explaining why these ranges were chosen or whether they preserve meaningful reasoning steps. The authors examine branching factors and α\alpha, but lack ablations on truncation ranges, tree depth, and β\beta.

[1]. TreeRL: LLM Reinforcement Learning with On-Policy Tree Search
[2]. TreePO: Bridging the Gap of Policy Optimization and Efficacy and Inference Efficiency with Heuristic Tree-based Modeling
[3]. TreeRPO: Tree Relative Policy Optimization
[4]. Segment Policy Optimization: Effective Segment-Level Credit Assignment in RL for Large Language Models

Questions for authors
  1. The paper filters out groups where all continuations receive the same reward. Since difficult or incorrect prefixes may lead to all-zero rewards, how often are these groups discarded? Does this limit TLPO's ability to learn from the challenging prefixes it aims to address?
  2. TLPO uses 64 rollouts per prompt, while most baselines use only 8. Is this a fair comparison? How does the additional rollout cost affect the overall training efficiency?
4 comments

We thank the reviewer for the constructive feedback. We would like to clarify the distinction between TLPO and existing tree-based RL methods, and provide additional experimental evidence addressing the concerns.

W1. Novelty and Comparison with Existing Tree-based RL Methods

We would like to clarify that TLPO goes beyond simply incorporating a length-aware reward into existing tree-based RL methods. Our key contribution lies in identifying an overlooked yet important limitation of trajectory-level length-aware RL and developing a principled solution to address it.

First, we identify a previously overlooked problem in efficient reasoning RL. Existing length-aware RL methods primarily reinforce globally short and correct trajectories, potentially overlooking effective recovery behaviors from suboptimal self-generated prefixes. As demonstrated in Section 2, this creates a training–inference mismatch that can undermine both reasoning accuracy and efficiency. To the best of our knowledge, prior tree-based RL methods have not explicitly investigated or addressed this issue in the context of efficient reasoning.

Second, TLPO introduces prefix-conditioned length-aware optimization to address this problem. Rather than simply adopting existing tree-based RL, we integrate tree-structured segment-level credit assignment into efficient reasoning RL and design a length-aware reward that explicitly accounts for the remaining reasoning budget under each prefix. This enables the model to reinforce efficient continuations even when earlier reasoning steps are suboptimal. Our ablation studies further demonstrate that tree-structured optimization and length-aware reward shaping are complementary, and their combination achieves substantially better accuracy–efficiency trade-offs than either component alone.

Third, Table 5 already provides direct comparisons with SPO-tree and TreePO under matched training length constraints. TLPO achieves 27.29% accuracy with 2,874 tokens on AIME25, outperforming SPO-tree (24.92%, 4,522 tokens) and TreePO (23.75%, 3,925 tokens). Similarly, on MATH500, TLPO achieves 87.46% accuracy with 1,341 tokens, compared with SPO-tree (87.08%, 2,045 tokens) and TreePO (84.99%, 1,765 tokens). These results demonstrate that directly applying existing tree-based RL methods under the same length constraints does not achieve the accuracy–efficiency trade-off of TLPO.

Therefore, the novelty of TLPO is not tree sampling or segment-level advantage estimation in isolation, but the identification and resolution of a previously underexplored credit-assignment problem in efficient reasoning RL, supported by both component ablations and direct comparisons with existing tree-based methods. We will further clarify this distinction in the revised manuscript.

W2. Training–Inference Mismatch and Error Recovery

We would like to clarify that the selection bias observed in Section 2.2 is not merely an artifact of evaluation, but reflects the underlying credit-assignment bias in trajectory-level length-aware RL. Specifically, trajectories with well-directed early prefixes tend to receive stronger positive reinforcement, whereas successful recovery behaviors from suboptimal prefixes may receive zero or even negative advantages. This creates a mismatch between the reasoning paths preferentially reinforced during training and the diverse self-generated prefixes encountered at inference time.

For example, consider eight trajectories: four directly reach correct answers (reward 1.5), two recover from an initially unsuccessful strategy and reach correct answers (reward 1.0), and two fail (reward 0). The successful recovery trajectories receive zero global advantage, despite reaching correct answers. In contrast, TLPO compares continuations from the same unfavorable prefix, assigning positive advantages to successful recovery behaviors.

Section 2.3 already provides initial evidence that training on self-generated prefixes improves reasoning efficiency and accuracy. To further provide direct evidence, we conduct an additional controlled continuation experiment on MATH500, using identical prefixes extracted from unsuccessful trajectories of the original model.

Continuation ModelAccuracy (%)Length
Base23.139,784
LASER19.202,239
TLPO32.751,157

Compared with LASER, TLPO improves continuation accuracy by 13.56 percentage points while reducing length by 48.3%. Notably, LASER sacrifices accuracy relative to Base, whereas TLPO improves both accuracy and efficiency.

These results provide direct evidence that TLPO learns more effective continuations under the same unfavorable reasoning contexts, supporting our hypothesis that prefix-conditioned optimization alleviates the limitation of trajectory-level length-aware RL.

W3. Truncation Ranges and Tree Structure

We would like to clarify that our goal is not to optimize tree architectures or carefully tune truncation positions, but to validate the effectiveness of prefix-conditioned process-level credit assignment for efficient reasoning RL. The default truncation ranges [0, 0.4] and [0.2, 0.6] were chosen as simple, non-tuned settings to expose the model to diverse self-generated prefixes at different reasoning stages. TLPO does not require explicitly identified semantic step boundaries, as its objective is to learn effective continuations from partial reasoning contexts. Despite this simple design, TLPO already achieves substantial improvements in both accuracy and efficiency.

Appendix B.6 already examines different branching factors under a fixed 64-leaf budget and varying length thresholds α. We further extend our ablations to truncation ranges, tree depth, and the length-reward coefficient β, with the following results:

ConfigurationAIME24 Acc / LenAIME25 Acc / LenMATH500 Acc / Len
Truncation ranges
Earlier branching: [0, 0.2], [0, 0.4]31.38 / 3,09325.95 / 2,51586.86 / 1,455
Later branching: [0.2, 0.6], [0.4, 0.8]32.53 / 3,58726.91 / 3,03987.01 / 1,703
Tree depth
Depth 2: (8,8)32.73 / 3,93926.75 / 3,26887.19 / 1,566
Depth 4: (2,2,4,4)32.49 / 3,27525.00 / 2,32085.94 / 1,133
Length-reward coefficient
β = 1.030.14 / 2,12323.16 / 2,05385.98 / 934
β = 0.2532.99 / 4,27727.10 / 3,97887.48 / 1,950

The results demonstrate that TLPO remains effective across different truncation ranges and tree depths, without requiring carefully optimized branching positions. Earlier branching generally produces shorter responses, while later branching achieves slightly higher accuracy. Similarly, changing tree depth and β leads to different accuracy–efficiency trade-offs, rather than eliminating the benefits of TLPO.

Together with the existing branching-factor and α ablations, these results provide comprehensive evidence that TLPO's improvements are not dependent on a narrowly tuned tree configuration. We will incorporate these additional experiments and clarify the truncation strategy in the revised manuscript.

Q1. Filtering of Zero-Variance Groups

We would like to clarify that zero-variance groups are a general limitation of group-relative RL, rather than an issue specific to TLPO. In GRPO, groups with identical rewards yield zero advantages and thus provide no effective policy-gradient signal, even if retained during training. DAPO explicitly filters such groups through dynamic sampling to improve training stability and sample efficiency. Importantly, neither TLPO nor conventional group-relative RL requires every challenging prefix to produce successful continuations. Whenever continuations from the same prefix yield different rewards, TLPO can assign discriminative local advantages, even under suboptimal prefixes. Moreover, TLPO estimates intermediate segment rewards through subtree averaging, allowing learning signals to propagate across multiple tree levels rather than relying solely on individual leaf outcomes.

Empirically, after training stabilizes, the proportions of zero-variance groups are 65% for TLPO and 66% for GRPO, with all-incorrect groups accounting for 29% and 27%, respectively. These highly comparable statistics indicate that TLPO does not encounter more degenerate groups than conventional GRPO, despite explicitly optimizing continuations from challenging prefixes. Finally, as discussed in DAPO, filtering zero-gradient groups does not necessarily reduce overall training efficiency, since generation time in synchronized RL systems is often dominated by long-tail responses. We will include these statistics and clarify the filtering mechanism in the revised manuscript.

Q2. Rollout Budget and Training Efficiency

We would like to clarify that this concern has already been explicitly addressed in Section 3.1 and Appendix B.2 of our paper. Although TLPO uses 64 leaf rollouts per prompt, we reduce the prompt batch size to maintain a comparable total rollout budget.

Specifically, Appendix B.2 reports a controlled comparison with DAPO under the same number of leaf rollouts per training step:

MethodPrompts × RolloutsTime / Step
DAPO512 × 16 = 8,1922,945 s
TLPO128 × 64 = 8,1922,059 s

Under the same rollout count, TLPO achieves approximately 30.1% lower per-step training time, benefiting from shared-prefix reuse during tree sampling. Appendix B.2 also reports only modest additional peak GPU memory usage (39,580 MB vs. 35,460 MB).

Therefore, the larger number of rollouts per prompt does not imply a larger overall rollout budget or higher training cost. These results and their experimental settings are already provided in our original submission.

Dear Reviewer PDVY,

Thank you again for your comments. We would appreciate your thoughts on our clarification of novelty, the fixed-prefix continuation results, and the additional sensitivity ablations. With the discussion period ending soon, please let us know if there are any remaining concerns we can address.

Reviewer_Tsv8OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that trajectory-level length-aware RL concentrates positive learning signal on trajectories whose early reasoning is already well directed, while at inference the model must continue from its own, often suboptimal, prefixes. It proposes TLPO, which builds a (1, 8, 8) rollout tree over self-generated reasoning segments, computes advantages among siblings that share the same prefix, and uses a length reward conditioned on the remaining context budget. Across DeepSeek-R1-Distill-Qwen-1.5B/7B and DeepScaleR-1.5B-Preview, TLPO reduces average reasoning length by 56.0–80.3% while improving average accuracy by 0.5–4.1 points on four math benchmarks and GPQA-Diamond.

Critical strengths

S1. Strong and consistent results. TLPO attains the best average accuracy and the shortest average length for every base model (Table 1, L324–348; L378–379). The gains also transfer outside the math training data, to GPQA-Diamond and three code benchmarks (Table 2, L1010–1020). S2. A thorough empirical study. Beyond the main table, the paper reports accuracy–length trade-offs across budgets (Figure 7), a 49.0% shorter 90th-percentile length for incorrect responses compared with LASER-L2048 (L418–419), component and sensitivity ablations (Tables 3–4), a controlled comparison with SPO-tree and TreePO (Table 5), and training-time cost (2,059 s vs. 2,945 s per step against DAPO, L974–975). Evaluation uses 32 samples per problem with paired bootstrap tests (L990–993).

Critical weaknesses

W1. The evidence for the training–inference mismatch in Section 2.2 largely follows from how the prefix groups are defined. "Positive prefixes" and "Max prefixes" (L157–160) are selected because their trajectories were correct and, under length-aware RL, short. It is therefore expected under any policy that their continuations are more accurate and shorter than those from unfiltered prefixes (Figure 3; conclusion at L180–186). This does not show that trajectory-level RL fails to provide learning signal for unfavorable prefixes. Under GRPO-style objectives, trajectories that start from poor prefixes still receive negative advantages, and these also update the policy on those prefixes. The claim that such prefixes receive "weaker optimization signals" (Figure 1 caption, L073) needs a more direct test. For example, compare advantage or gradient magnitude at matched prefix quality, or show that standard RL fails to improve continuation from a fixed set of poor prefixes while TLPO succeeds. Section 2.3 (Figure 4, L207–208) is closer to such a test, but it covers one model on MATH500 only. W2. The main comparison is not controlled, and the controlled evidence suggests most of the compression comes from the training setup rather than the tree. Table 1 (L324–348) compares TLPO with released checkpoints (L368–371), which were trained with different data, numbers of steps and length limits. TLPO itself is trained with a 6k context for the distilled models (L964–965). L372–373 states that baseline configurations are given in Appendices B.2–B.3, but those sections (L960–970, L990–993) only describe TLPO and the evaluation protocol. The authors themselves note that SPO-tree and TreePO are shorter "partly because they are trained with the same maximum generation length" (L1130–1131). Table 3 (L1061–1067) supports this. On AIME24, "TLPO w/o Tree Sampling" already reduces length from 16,717 to 4,327 tokens, which is about 90% of TLPO's total reduction (to 2,953). The accuracy gap (28.54 vs. 32.92) comes from a single benchmark with no variance reported. A fair test of the central claim needs trajectory-level length-aware RL (e.g., a LASER-style reward, or DAPO with an overlong penalty) trained with the same data, number of steps, 6k limit and rollout budget, and compared on all five benchmarks over multiple seeds.

Questions for authors

Q1. Can you report the controlled comparison described in W2, at least for DeepSeek-R1-Distill-Qwen-1.5B on all five benchmarks? Minor (no response needed). The Figure 7 caption (L446) says "p-value < 0.05" without stating which comparison or test this refers to. The model name appears as "DeepScalerR" (L344, L363, L968) and as "DeepScaler"/"DeepScaleR" elsewhere (L155, L169). "Eqs. equation 1 and equation 2" (L300) and "Eq. equation 1/3/4" in Algorithm 1 (L942–946) are broken cross-references.

3 comments

We thank the reviewer for the constructive comments. We clarify the motivation behind TLPO and provide additional experiments addressing both concerns.

W1. Training–Inference Mismatch and Learning Signals for Suboptimal Prefixes

We agree that trajectory-level RL does not completely lack learning signals for unfavorable prefixes. Our central argument is more specific: trajectory-level advantage estimation may fail to positively reinforce successful recovery behaviors from suboptimal prefixes, even when such behaviors lead to correct answers.

Consider a two-stage reasoning problem where the model can initially explore strategy A or B, but cannot determine which strategy works without further reasoning. Suppose eight trajectories are sampled:

  • 4 trajectories: Start with B and directly reach the correct answer (reward = 1.5).
  • 2 trajectories: Start with A, discover that it fails, switch to B, and reach the correct answer (reward = 1.0).
  • 2 trajectories: Start with A, continue in the wrong direction, and fail (reward = 0).

Under trajectory-level GRPO with length-aware rewards, the group mean reward is 1.0 and the standard deviation is approximately 0.612. Consequently, the successful recovery trajectories (A →\to B →\to Correct) receive zero normalized advantage, despite reaching the correct answer. Their recovery tokens thus receive no positive reinforcement, while the trajectories starting directly with B receive positive advantages.

In contrast, TLPO compares continuations conditioned on the same prefix A. Among the four continuations, the two successful recovery trajectories receive reward 1.0, while the two unsuccessful trajectories receive reward 0. Under prefix-conditioned advantage estimation, the recovery trajectories receive a positive normalized advantage of +1.0.

This example illustrates the distinction: although trajectory-level RL can penalize unfavorable trajectories, it may fail to distinguish successful recovery from continued failure through positive learning signals once an unfavorable prefix has occurred. TLPO directly reinforces effective continuations under such prefixes.

Additional experiment: Continuation from fixed failure-associated prefixes.

To directly test this hypothesis, we conduct an experiment on MATH500 using a fixed set of self-generated prefixes extracted from unsuccessful trajectories of the original model. We provide exactly the same prefixes to Base, LASER, and TLPO and evaluate their continuation accuracy and length.

Continuation modelAccuracy (%)Length
Base23.139,784
LASER19.202,239
TLPO32.751,157

Compared with LASER, TLPO improves continuation accuracy by 13.56 percentage points while reducing continuation length by 48.3%. Notably, LASER achieves shorter continuations than Base but lower accuracy, whereas TLPO improves both metrics.

These results directly support our hypothesis that prefix-conditioned optimization better reinforces effective reasoning continuations under unfavorable, self-generated reasoning contexts. We will incorporate this analysis and clarify the corresponding claims in Section 2.

W2 / Q1. Controlled Comparison and the Contribution of Tree Sampling

We appreciate the concern about differences in training configurations among released checkpoints.

First, we clarify that LASER and TLPO use the same DeepScaleR-Preview mathematics training dataset and the same base model. Although their training configurations are not strictly identical, their overall rollout budgets are highly comparable. Specifically, LASER's released training configuration specifies 5 epochs, a batch size of 128, and 8 rollouts per prompt, corresponding to approximately 1.63M trajectory rollouts over roughly 1,590 training steps. In comparison, TLPO uses a batch size of 128 and 64 leaf rollouts per prompt over 200 steps, yielding approximately 1.64M leaf rollouts. While this does not constitute a strictly controlled comparison, it demonstrates that the existing baseline comparison is conducted under comparable rollout budgets, rather than substantially different amounts of rollout sampling.

Importantly, we now provide an expanded controlled trajectory-level RL comparison across all five benchmarks.

Our TLPO w/o Tree Sampling variant removes tree-structured sampling and prefix-conditioned segment-level credit assignment while retaining the same backbone, training data, length-aware reward design, and other training settings.

We have now extended the controlled ablation from AIME24 to all five benchmarks, with 32 independent rollouts per problem to reduce sampling variance:

Benchmarkw/o Tree Acc / LenTLPO Acc / Len
AIME2428.54 / 4,32732.92 / 2,953
AIME2523.21 / 5,79027.29 / 2,874
MATH50083.84 / 2,21587.46 / 1,341
AMC202374.70 / 3,27676.41 / 2,003
GPQA-Diamond35.06 / 5,16837.06 / 2,338
Average49.07 / 4,15552.23 / 2,302

TLPO consistently improves accuracy and reduces response length across all five benchmarks, achieving an average +3.16 percentage-point accuracy gain and 44.6% additional length reduction over the controlled trajectory-level baseline.

Regarding the reviewer's 90% observation, we note that the additional compression should be measured relative to the trajectory-level baseline rather than the original model. On AIME24, TLPO further reduces length by 31.8% while improving accuracy by 4.38 percentage points.

These results confirm the substantial contribution of tree-structured, prefix-conditioned optimization beyond trajectory-level length-aware RL. We will incorporate the expanded results into the main table and clarify the baseline configurations. We acknowledge that multiple independent training seeds would further strengthen the evidence.

Dear Reviewer Tsv8,

Thank you again for your helpful feedback. As the discussion period is nearing its end, we would appreciate your thoughts on our responses, particularly the fixed-prefix continuation experiment and the controlled comparison across all five benchmarks. Please let us know if any concerns remain; we would be happy to address them.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

We thank the reviewers for their helpful feedback. We summarize the main updates below:

  • Novelty: To the best of our knowledge, we are the first to identify and systematically investigate how trajectory-level length-aware RL can fail to positively reinforce successful recovery from suboptimal self-generated prefixes. We clarified TLPO’s credit-assignment motivation and its relation to existing tree-based RL methods.
  • More controlled comparison: We extended the controlled trajectory-level comparison to all five benchmarks. TLPO improves average accuracy by 3.16 percentage points while reducing average response length by 44.6%.
  • Fixed-prefix continuation: We added experiments using identical prefixes from failed trajectories, showing improved continuation accuracy and efficiency.
  • More sensitivity analyses: We added ablations on truncation ranges, tree depth, and the length-reward coefficient β\beta.

These additions are included in the revised manuscript. We would appreciate your feedback and would be happy to address any remaining questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.