- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates the credit assignment problem in length-aware reinforcement learning (RL). The authors argue that positive updates in conventional trajectory-level RL are primarily concentrated on trajectories with relatively high-quality early reasoning steps. However, during actual inference, models must also continue generating from self-generated prefixes that may contain exploration, detours, or intermediate errors.The authors propose TLPO, which constructs multiple continuations from shared prefixes through a three-level tree rollout, with a default branching factor of (1, 8, 8). Leaf nodes receive rewards based on final answer correctness and a length threshold relative to the remaining context.
Critical strengths
-
The research problem is practically meaningful, focusing on the trade-off between reasoning accuracy and efficiency. The paper addresses the common issue of excessive reasoning length in large language models, aiming to reduce token consumption while maintaining or even improving accuracy. This research direction has clear practical relevance.
-
The experiments cover multiple models and reasoning tasks, demonstrating substantial efficiency improvements. The paper conducts experiments across models of different scales and initialization settings, covering several reasoning benchmarks, including AIME, MATH500, AMC, and GPQA-Diamond. The experimental results show that TLPO can significantly reduce reasoning token consumption while maintaining or improving accuracy.
Critical weaknesses
-
The explanation of the training–inference mismatch is not sufficiently clear.I have some concerns regarding the training–inference mismatch proposed in the paper. The authors observe that trajectories with positive advantages tend to contain higher-quality prefixes, which is understandable. However, in on-policy RL, rollouts during both training and inference are generated by the model itself. Under the same policy, input distribution, and sampling configuration, their prefix distributions should, in principle, be consistent. I would appreciate further clarification of this concept and an explanation of why this phenomenon should be considered a training–inference mismatch.
-
The proposed method is similar to existing Tree-based GRPO approaches, and its novelty needs further clarification. The main ideas of TLPO, including generating multiple continuations from shared prefixes, estimating intermediate node values using subtree outcomes, and computing local advantages, overlap considerably with existing Tree-based GRPO methods. For example, "Tree Search for LLM Agent Reinforcement Learning" already adopts similar tree-structured sampling and fine-grained credit assignment, while "Why Tree-Style Branching Matters for Thought Advantage Estimation in GRPO" also investigates multiple continuation sampling from shared prefixes and provides related theoretical analysis.
Questions for authors
-
Regarding Training–Inference Mismatch.Could the authors further clarify the precise definition of training–inference mismatch in the paper? In particular, under the on-policy RL setting, why should the difference in prefix quality between positive-advantage trajectories and the overall sampled trajectories be interpreted as a distribution mismatch between training and inference?
-
Regarding Baselines and the Contributions of Individual Components.I suggest that the authors include a standard GRPO baseline, as well as a comparison with Tree-GRPO under the same length-aware reward and training budget, to better disentangle the contributions of tree branching, fine-grained credit assignment, and reward shaping.
4 comments
W2 / Q2. Novelty, Baselines, and Component Analysis
We would like to clarify that TLPO goes beyond simply adopting existing tree-based GRPO methods. Our contributions and supporting evidence are summarized as follows:
1. Conceptual and Methodological Contributions. Our contribution lies in both identifying an overlooked problem and developing a targeted solution.
- Problem: Existing trajectory-level length-aware RL preferentially reinforces globally short and correct trajectories, potentially overlooking effective recovery behaviors from suboptimal self-generated prefixes. To the best of our knowledge, we are the first to identify and systematically investigate this overlooked imbalance in optimization signals, revealing a fundamental mismatch between the reasoning behaviors reinforced during training and those required at inference time. This is an important yet previously unexplored issue in efficient reasoning RL.
- Method: Rather than simply adopting existing Tree RL, TLPO introduces prefix-conditioned length-aware optimization, integrating tree-structured segment-level credit assignment with a specially designed reward that accounts for the existing prefix and remaining reasoning budget. This enables efficient recovery behaviors to receive positive reinforcement even under suboptimal prefixes.
2. Component Ablations. As reported in **Section 4.4 **, our ablations demonstrate that tree-structured optimization and length-aware reward shaping are complementary, and their combination achieves substantially better accuracy–efficiency trade-offs. On AIME24, removing tree sampling reduces accuracy from 32.92% to 28.54% and increases length from 2,953 to 4,327 tokens, while removing the length penalty reduces accuracy to 31.42% and increases length to 4,533 tokens.
3. Comparisons with Existing Tree-based RL. Table 5 already provides direct comparisons with SPO-tree and TreePO under matched training length constraints. On AIME25, TLPO achieves 27.29% / 2,874 tokens, outperforming SPO-tree (24.92% / 4,522) and TreePO (23.75% / 3,925). Similarly, on MATH500, TLPO achieves 87.46% / 1,341 tokens, compared with SPO-tree (87.08% / 2,045) and TreePO (84.99% / 1,765). These results demonstrate that directly applying existing tree-based RL methods under the same length constraints does not achieve TLPO's accuracy–efficiency trade-off.
4. Controlled GRPO-style Baseline with Identical Length-aware Reward. Our TLPO w/o Tree Sampling variant serves as a trajectory-level GRPO-style baseline, retaining the same base model, training data, length-aware reward design, and training settings, while removing tree sampling and prefix-conditioned credit assignment. We further extend this controlled comparison to all five benchmarks, using 32 evaluation rollouts per problem:
| Benchmark | w/o Tree Acc / Len | TLPO Acc / Len |
|---|---|---|
| AIME24 | 28.54 / 4,327 | 32.92 / 2,953 |
| AIME25 | 23.21 / 5,790 | 27.29 / 2,874 |
| MATH500 | 83.84 / 2,215 | 87.46 / 1,341 |
| AMC2023 | 74.70 / 3,276 | 76.41 / 2,003 |
| GPQA-Diamond | 35.06 / 5,168 | 37.06 / 2,338 |
| Average | 49.07 / 4,155 | 52.23 / 2,302 |
TLPO consistently improves accuracy and reduces response length across all five benchmarks, achieving a 3.16 percentage-point gain in average accuracy and a 44.6% reduction in average response length over the controlled trajectory-level baseline.
Taken together, these results demonstrate that TLPO's contribution is not tree sampling or segment-level advantage estimation in isolation, but identifying and addressing a previously overlooked credit-assignment problem in efficient reasoning RL through their integration with prefix-conditioned length-aware optimization. We will further clarify these contributions, incorporate the suggested related works, and include the expanded experimental results in the revised manuscript.
Dear Reviewer yFQh,
Thank you again for your feedback. With the discussion period ending soon, we would appreciate your thoughts on our clarification of the training–inference mismatch and the added controlled GRPO-style comparison. We would be happy to discuss any remaining concerns about the motivation or contribution.
Response to rebuttal
Thank you for your detailed response; your rebuttal has addressed my concerns, and I have increased my rating accordingly.
We thank the reviewer for the comments. We would like to clarify the precise meaning of the training–inference mismatch and further explain the novelty and contributions of TLPO, supported by additional experimental evidence.
We agree that on-policy RL produces the same prefix distribution during training and inference under identical conditions. However, the mismatch lies in the optimization signals rather than the sampling distribution: trajectory-level length-aware RL preferentially reinforces globally short and correct trajectories with well-directed prefixes, while successful recovery behaviors from suboptimal prefixes—also encountered at inference time—may receive insufficient positive reinforcement.
To illustrate this distinction, consider a two-stage reasoning problem where the model can initially explore strategy A or B, but cannot determine which works without further reasoning. Suppose eight trajectories are sampled:
Under trajectory-level GRPO with length-aware rewards, the group mean reward is 1.0, with a standard deviation of approximately 0.612. Consequently, the successful recovery trajectories (A → B → Correct) receive zero normalized advantage, despite reaching correct answers. Their recovery tokens thus receive no direct positive reinforcement, while trajectories starting directly with B receive positive advantages.
In contrast, TLPO compares continuations conditioned on the same prefix A. Among the four continuations, two successfully recover (reward = 1.0), while two fail (reward = 0). The local group mean is 0.5, with a standard deviation of 0.5. Therefore, successful recovery behaviors receive a positive normalized advantage of +1.0, rather than zero.
This example illustrates the key distinction: the issue is not whether unfavorable prefixes are sampled during training, but whether successful recovery behaviors from these prefixes are effectively reinforced. Although negative advantages can penalize unsuccessful trajectories, they do not necessarily provide positive learning signals for successful recovery under the same prefix.
The prefix-quality differences observed in Section 2.2 illustrate this selective reinforcement, while Section 2.3 provides initial evidence that conditioning training on self-generated prefixes improves reasoning efficiency and accuracy.
Additional experiment: Continuation from fixed failure-associated prefixes.
To further directly validate our hypothesis, we conduct an experiment on MATH500 using identical prefixes extracted from unsuccessful trajectories of the original model. We provide the same prefixes to Base, LASER, and TLPO and evaluate their continuation accuracy and length.
Compared with LASER, TLPO improves continuation accuracy by 13.56 percentage points while reducing continuation length by 48.3%. Notably, LASER decreases continuation accuracy relative to Base, whereas TLPO improves both accuracy and efficiency.
These results provide direct evidence that TLPO learns more effective continuations under unfavorable self-generated prefixes. We will clarify that the training–inference mismatch in our paper refers to an advantage-induced mismatch between training optimization signals and inference-time reasoning requirements, rather than a shift in the underlying on-policy sampling distribution.