acceptodds
Under review as a conference paper at ICLR 2027

Is Tree RL Losing to Standard RL? Validation Mismatch in Tree-Structured RL

Abstract

Tree-structured reinforcement learning branches rollouts from shared prefixes, reusing generated tokens and turning outcome rewards into process supervision, and tree-RL methods have reported consistent gains over GRPO. Against DAPO, a more recent post-training RL algorithm that corrects the entropy collapse and instability of GRPO, these gains are no longer observed under standard validation: across three tree-RL methods, three model scales and four tasks, with the number of training trajectories per update matched, the tree-trained policy trails the DAPO-trained policy in all five comparisons, by 1.0 to 4.6 points. We identify a validation mismatch in this assessment, a mismatch between how training rollouts are generated and how checkpoints are validated: standard validation evaluates every checkpoint with greedy decoding or independent samples, the procedure that standard RL trains with but not the one tree RL trains with. Tree validation, which evaluates checkpoints with the branching procedure that tree RL trains with, corrects the assessment. Under tree validation, the tree-trained policy outperforms the DAPO-trained policy in all five comparisons, by 0.9 to 4.3 points, and it reveals training progress that standard validation misses: on Qwen2.5-Math-1.5B-Instruct, TreeRL shows no improvement over its initialization under standard validation but a gain of 4.5 points under tree validation. The same pattern holds at deployment: in every setting we test, a tree-trained policy deployed with a tree has the highest mean accuracy of the six combinations of training method and inference procedure, while generating fewer tokens than a majority vote over independent samples. Tree RL should therefore be validated and monitored with the procedure it is trained with, alongside standard validation, with the generator, answer selector and budget of each procedure specified.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, post-training, checkpoint validation, training-inference alignment

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_iWNbOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper argues that tree-based RL methods should be evaluated the same way they are trained, and shows that tree-trained policies outperform DAPO under tree evaluation but underperform it under standard evaluation.

Critical strengths
  1. The paper raises an interesting research question that challenges previous opinions in this line of research by rethinking how tree-based RL is evaluated.
  2. The paper is easy to follow.
Critical weaknesses
  1. The inference cost of tree evaluation is unclear. Although it generates fewer new tokens than independent sampling, tree inference requires sequential rounds of generation, and the paper reports no wall-clock latency or throughput. The paper should discuss and provide concrete results on why tree-based inference is worth adopting in practice; the current version does not sufficiently motivate this question.
  2. Limited novelty. The main finding, that a tree-trained policy performs better than DAPO under tree inference, is largely expected, since each policy is evaluated under the procedure it was trained with. Unless the paper justifies why tree-based inference should be used in practice (Weakness 1), showing that tree-based RL outperforms standard RL under tree inference alone is not a meaningful contribution.
  3. The experiments use only Qwen2.5-family models, on which RLVR has been shown to improve even with spurious or random rewards (Shao et al., 2025). Do the conclusions hold on more recent model families that are less affected by this issue?

Shao, Rulin, et al. "Spurious rewards: Rethinking training signals in rlvr." arXiv preprint arXiv:2506.10947 (2025).

Questions for authors

See weakness above.

5 comments
AuthorsOpenReview ↗

Response to reviewer iWNb

Response to Reviewer iWNb -- part A

We sincerely thank the reviewer for finding our research question interesting, one that "challenges previous opinions in this line of research," and the paper easy to follow. We are also grateful for the reviewer's questions on the practical value of tree inference, which led directly to the new latency measurements in this response.

Our main point. All five tree-RL methods we surveyed train with branching but validate without it (Table 1). This simple mismatch has been overlooked, and its effect is large: changing only the validation procedure reverses the ranking between tree RL and DAPO in all five comparisons (5.4 points on average), and changes whether training appears to help at all (TreeRL on Qwen2.5-Math-1.5B-Instruct: no gain under standard validation, +4.5 points under tree validation; §4.3). Conclusions about tree RL drawn from standard validation alone can therefore be incomplete or misleading.

W1. Inference cost of tree evaluation. We thank the reviewer for this important point. While our main focus is the validation mismatch, we fully agree that latency matters for deployment. We have therefore measured it, and we also explored a new inference procedure. The full results are in the General Response (Concern 2, Tables A and B); all procedures run on the same hardware and serving stack.

  • Tree-trained + Tree6 reaches 78.3%, against 74.7% for DAPO-trained + IID6, with 11% fewer tokens, at a median latency of 30.5 s vs 23.1 s.
  • Under equal response-time limits, DAPO + IID6 is not the most accurate procedure at any limit we tested. The tree-trained policy with tree inference is the most accurate at 30 s and 60 s (with our non-batch tree) and at 90 s and above (with Tree6).
  • Our new non-batch tree, which branches during decoding instead of building the full tree first, is faster than IID6 (21.2 s vs 23.1 s) with similar accuracy (75.1% vs 74.7%).

With dynamic branching, the non-batch tree removes the latency cost, but its accuracy is at the level of DAPO + IID6 (75.1% vs 74.7%), and with a single trajectory it is below greedy decoding (71.6% vs 73.3%). Tree RL therefore faces a dilemma: with standard decoding it trails DAPO, with batch tree inference it wins but costs latency, and with dynamic branching it is fast but loses most of its gain. This leaves room to explore how dynamic-branching tree inference can keep the accuracy gain while saving latency, which is largely a systems question.

W2. Novelty. We thank the reviewer for raising this question, which we also address in the General Response (Concern 1). We agree that the intuition is natural. But the result is not simply that a policy does best under its own training procedure:

  • Tree inference helps only tree-trained policies. The same Tree6 procedure does not help DAPO-trained policies (+0.4, −2.3 and −1.6 points over IID6), but helps tree-trained policies by 2.3 to 5.1 points (§4.5). Tree inference is not a stronger sampler; the gain comes from matching the inference procedure to the training procedure, and this had not been measured before.
  • Current evaluation misses this, whichever way tree RL is deployed. If tree RL is deployed with standard decoding, it trails DAPO in all five comparisons; prior tree-RL papers compared against GRPO, and to our knowledge this result has not been reported. If tree RL is deployed with tree inference, it outperforms DAPO, but at a latency cost, and removing this cost with dynamic branching loses most of the gain (W1). In both cases, standard validation alone hides these problems and can lead to the wrong conclusion about which method is better.
  • The intuition does not tell how large the effect is. We find that it is large enough to reverse the ranking in every comparison.

We believe that identifying a widely used evaluation practice that reverses the conclusions of method comparisons, and exposing the resulting dilemma for tree RL, is a meaningful contribution. We will make this motivation more explicit in the introduction.

AuthorsOpenReview ↗

Response to Reviewer iWNb – part B

Response to Reviewer iWNb -- part B

W3. Model families and spurious rewards. We thank the reviewer for the pointer to Shao et al. (2025). Our main result is a relative comparison, which is robust to this issue:

  • Within each pair, both policies share the same initialization, data and number of training trajectories. Any improvement from spurious reward signals in Qwen2.5 applies to both training methods.
  • The Shift is a difference between two validation procedures applied to the same checkpoints. It does not depend on the absolute RL gain over the initialization.
  • Our comparisons include general Qwen2.5 models at 3B and 7B, not only Qwen2.5-Math, and non-math tasks (LiveCodeBench, HotpotQA), and the same reversal appears in all of them.

We will add results on a non-Qwen model family in the revision.

Revision. We will (i) add the latency and response-time-limit results to §4.6, (ii) make the motivation above more explicit in the introduction, (iii) add the non-batch tree experiments and state this dilemma clearly, and (iv) add results on a non-Qwen model family.

We thank the reviewer again for these valuable questions, which have helped us strengthen the paper. We hope the new results and clarifications address the reviewer's concerns, and we would be sincerely grateful if the reviewer would reconsider the assessment in light of them.

Thank you for the detailed response and the additional experiments. After reading the rebuttal and the general response, I have decided to keep my score, for the following reasons.

  1. Inference cost. I appreciate the new latency measurements. However, they confirm my concern: batch tree inference is about 32% slower than IID6 (30.5 s vs 23.1 s) for a 3.6-point accuracy gain, and the non-batch variant removes the latency cost only by giving up most of that gain (75.1% vs 74.7%). As the authors acknowledge, tree RL faces a dilemma in deployment. Given the modest accuracy improvement, I find this latency cost concerning for practical use. In addition, these results come from a single setting without reported variance.

  2. Novelty. I still find the main finding, that a policy performs best under the procedure it was trained with, to be largely expected. The response restates the arguments in the paper, and I do not see a new insight beyond quantifying this effect.

  3. Model families. Results on a non-Qwen model family are promised for the revision but not provided, so this concern remains unaddressed. The argument that a relative comparison is robust to spurious rewards assumes that such signals affect both training methods equally, which has not been verified.

I thank the authors again for their efforts.

Reviewer_imtROpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates an evaluation mismatch in Tree RL for LLMs. The authors observe that while Tree RL methods generate training rollouts via branching, they are typically validated using standard decoding. Across five comparisons against DAPO, tree-trained policies underperform under standard validation but outperform DAPO under tree validation. It further shows that at deployment, tree training with tree inference achieves the highest mean accuracy with fewer generated tokens. The main contribution is identifying this validation mismatch and recommending that Tree RL be validated and monitored with the procedure it is trained with.

Critical strengths
  1. The paper studies a simple but consequential question that is easy to overlook.
  2. Rather than evaluating each training method only under its customary inference procedure, the authors evaluate both tree-trained and DAPO-trained policies under both standard and tree-based sampling.By using shared initializations and matching the number of training trajectories per update within each pair, this design provides a useful way to examine the interaction between training methods and evaluation procedures.
Critical weaknesses
  1. Several tree-validation gains are small and the standard deviations across seeds often overlap. The paper reports means ± std but does not provide paired significance tests or confidence intervals for each Δ.
  2. Appendix Table 6 shows that the TreeRL training configuration uses (M=6, N=2, T=2, L=1), whereas its 16-path tree-validation configuration uses (M=4, N=3, T=1, L=1). These configurations generate different tree topologies, so the evaluation is not an exact match to the training rollout configuration, even though it preserves the same general sampling method and fork rule. The paper demonstrates the mismatch but does not decompose how fork rule, tree depth, tree size, or selector contribute to the effect.
Questions for authors
  1. Could the authors report paired per-seed differences and confidence intervals for the tree-minus-DAPO accuracy gaps, particularly for TreePO on AIME and MATH500? This would help establish whether the positive gaps in all five comparisons are robust to training-seed variability.
  2. For TreeRL, why does tree validation change the root count, number of fork points, and branching factor relative to training? Do the main conclusions persist when the training configuration is retained or when the tree topology and budget are systematically varied?
1 comment
AuthorsOpenReview ↗

Response to reviewer imtR

Response to Reviewer imtR

We sincerely thank the reviewer for the careful reading, for the encouraging assessment that the question is "simple but consequential" and "easy to overlook," and for the two constructive questions, which we answer below.

Q1. Paired per-seed differences and confidence intervals. We thank the reviewer for this suggestion. Each tree-RL run and its DAPO counterpart share the same seed and initialization, so we pair them by seed. For each comparison, we compute the per-seed tree-minus-DAPO difference (Δ) at update 100 under both validation procedures, and the Shift (Δ under tree validation minus Δ under standard validation). Intervals are 95% paired t-intervals over three seeds.

ComparisonΔ, standard validationΔ, tree validationShiftShift per seed
TreeRL, 1.5B, MATH500−3.0 [−4.3, −1.7]+3.7 [2.1, 5.3]+6.7 [4.1, 9.3]+5.6, +7.0, +7.6
TreeRL, 3B, LCB−4.6 [−12.1, 3.0]+4.3 [−4.7, 13.3]+8.9 [3.9, 13.8]+7.0, +8.7, +10.9
Tree-GRPO, 3B, HotpotQA−1.8 [−3.1, −0.4]+2.4 [−2.2, 6.9]+4.1 [−1.8, 10.0]+6.1, +4.8, +1.5
TreePO, 7B, AIME−4.0 [−8.2, 0.2]+1.4 [−3.7, 6.6]+5.4 [1.9, 9.0]+5.7, +3.9, +6.7
TreePO, 7B, MATH500−1.0 [−5.0, 3.0]+0.9 [−1.9, 3.7]+1.9 [−4.9, 8.6]+2.5, −1.1, +4.2
Average of five+5.4 [4.2, 6.6]

What these results show:

  • The Shift is consistent across seeds. It is positive for 14 of the 15 seed pairs (one-sided sign test, p < 0.001). Its average over the five comparisons is 5.4 points, with a 95% interval of [4.2, 6.6].
  • The ranking reverses at the level of individual seeds. Under standard validation, DAPO leads in 14 of the 15 seed pairs; under tree validation, the tree-trained policy leads in 13 of the 15 (p = 0.004).
  • Consistent across checkpoints. For TreePO MATH500, where the gap is smallest, the direction is consistent across checkpoints: under tree validation, TreePO leads DAPO at all ten checkpoints from update 20 to 200 (Appendix, values of Figure 1).

Our main claim is that changing the validation procedure changes the ranking of the two methods, and the paired results support it. We will add this table to the paper.

Q2. Why TreeRL's validation topology differs from training, and whether the conclusions persist. We thank the reviewer for this careful observation. We changed the topology to match the budget. The training configuration (M=6, N=2, T=2) produces 30 paths, while standard validation uses 16 samples. To compare the two validation procedures at the same budget, so that they differ only in how samples are generated, we used a 16-path tree (M=4, N=3, T=1) with the same fork rule and depth (L=1).

The conclusion does not depend on this one topology. The same reversal appears at deployment (§4.5), where the tree has a different topology and budget: 6 paths (M=2, N=1, T=2 for TreeRL), with T=2, as in training. With TreeRL 1.5B at update 100, the DAPO-trained policy is more accurate under IID6 (72.5% vs 69.2%), and the tree-trained policy is more accurate under Tree6 (74.3% vs 72.9%). The same holds for TreePO 7B at updates 100 and 200. The effect also appears under all three fork rules in our comparisons (entropy-based, segment-based and agent-step).

Following the reviewer's suggestion, we will add to the revision (i) TreeRL validation with the training configuration (M=6, N=2, T=2) for both policies at a matched 30-sample budget, and (ii) a sweep over tree topology and budget, to show the contributions of the fork rule, tree depth, tree size and answer selection.

We thank the reviewer again for these thoughtful questions, which have helped us strengthen the paper. We hope the paired analysis and the topology results address the reviewer's concerns, and we would be sincerely grateful if the reviewer would take them into account in the final assessment.

Reviewer_qojXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how evaluation changes the comparison between tree RL and DAPO. The authors test both models with independent sampling and tree sampling. Across five comparisons, the average results over three training seeds favor DAPO with independent sampling, but favor tree RL with tree sampling. The paper suggests using the sampling method planned for deployment when evaluating the model.

Critical strengths

Both tree-trained and DAPO-trained models are tested with independent and tree sampling. This comparison clearly shows how evaluation changes their ranking. Three seeds and multiple checkpoints give further support to this finding.

Critical weaknesses

Table 5 shows that tree deployment generates 15%–29% fewer tokens. But I did not find measurements of latency which is more important for real-world users. Section 4.6 already discusses the sequential generation rounds in TreePO. So fewer tokens do not necessarily mean a faster answer. Without actual timing results, it is hard to judge whether the reported accuracy gains are worth the serving cost.

Questions for authors

Can you compare tree-trained models with Tree6 and DAPO-trained models with IID6 on the same hardware and under the same request load? If both methods have the same response-time limit, does tree deployment still give higher accuracy?

1 comment
AuthorsOpenReview ↗

Response to reviewer qojX

Response to Reviewer qojX

We thank the reviewer for the positive assessment of our 2×2 design and for raising the question of serving cost. We agree that latency matters more to users than token count, and that fewer tokens do not necessarily mean a faster answer. We have therefore measured wall-clock latency. The setup and full tables are in the General Response (Concern 2, Tables A and B).

Q1. Tree-trained + Tree6 vs DAPO-trained + IID6 on the same hardware and under the same request load. Both run on one H200 with the same serving stack (vLLM with CUDA graphs and prefix caching), starting from a cleared cache. Both are measured under the same load: one question at a time.

Acc. (%)Median latency (s)Tokens / q
DAPO + IID674.723.114.2K
Tree RL + Tree678.330.512.6K

Tree6 is 3.6 points more accurate and uses 11% fewer tokens, but it is 7.4 s slower. It is slower because it generates in two sequential rounds (see below). In the revision, we will also report this comparison at higher concurrent loads.

Q2. Same response-time limit. The most accurate procedure at each limit (General Response, Table B):

  • 15 s: greedy decoding (48.3%).
  • 30 s: our new non-batch tree, which branches during decoding (General Response, Concern 2) (58.5%); IID6 55.3%, Tree6 43.1%.
  • 45 s: greedy decoding (73.3%); IID6 70.4%, non-batch tree 70.3%, Tree6 66.0%.
  • 60 s: non-batch tree (75.1%); IID6 74.7%, Tree6 71.7%.
  • 90 s and above: Tree6 (78.3%); IID6 74.7%.

So, at limits of 60 s or less, Tree6 is less accurate than IID6, because some of its answers take longer than the limit. At 90 s and above, Tree6 is more accurate than IID6. The non-batch tree is more accurate than IID6 at 30 s and 60 s, and about equal at 45 s. DAPO + IID6 is not the most accurate at any limit.

Why Tree6 is slow. Tree6 must finish its root responses before it can choose fork points, so it generates in two rounds, and their latencies add up. Current engines such as vLLM and SGLang also have no operation for forking a sequence during decoding, so each branch is sent as a new request. Our non-batch tree forks during decoding and has no second round. Its median latency is 21.2 s, lower than IID6 (23.1 s), and its accuracy is 75.1%, which is 0.4 points above IID6 and 3.2 points below Tree6.

A dilemma for tree RL. Our findings point to a dilemma, which is the key point of our paper:

  • Deploy with standard decoding. It is fast, but the tree-trained policy then trails DAPO, so the benefit of tree training is lost.
  • Deploy with tree inference. This is where tree training wins (78.3% vs 74.7% for DAPO + IID6), but it costs 7.4 s of median latency.
  • Deploy with dynamic branching. Our non-batch tree is faster than IID6, but its accuracy is at the level of IID6 (75.1% vs 74.7%), so most of the advantage is lost.

This dilemma is invisible when tree RL is validated only with standard decoding, which is why we argue that tree RL must be evaluated with both procedures. It also leaves room to explore how dynamic-branching tree inference can keep the accuracy gain while saving latency, which is largely a systems question (General Response, Concern 2).

Revision. We will (i) add the latency and response-time-limit results to §4.6, (ii) state in §4.5 that fewer tokens do not mean lower latency, (iii) add the comparison at higher concurrent loads, and (iv) add the non-batch tree experiments and state this dilemma clearly.

We thank the reviewer again for this question. We hope these results address the concern about serving cost, and we would be grateful if the reviewer would take them into account in the final assessment.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response

General Response -- part 1

We sincerely thank all three reviewers for their careful reading and constructive feedback. Two concerns are shared across the reviews, and we address them here before the individual responses: whether our finding is expected (Reviewer iWNb), and whether tree inference is worth deploying once latency is counted (Reviewers qojX and iWNb).

Concern 1: Isn't it expected that a policy does best under the procedure it was trained with?

We understand why the finding can seem expected: once stated, the intuition is natural, and we agree with it. Our point is that, although expected in principle, it has not shaped how tree RL is evaluated, and its consequences had not been measured.

  • Current practice does not reflect it. All five tree-RL methods we surveyed train by branching but validate without it (greedy, maj@16 or mean@16; Table 1), and their comparisons with standard RL are drawn under that procedure. To our knowledge, no prior work compares the two kinds of policy under both procedures (§2.2).
  • The size of the effect is not obvious from the intuition. The intuition alone does not say whether the effect is a small correction or large enough to change conclusions. We find the latter: in all five comparisons, the tree-trained policy trails DAPO under standard validation and leads under tree validation, with a 5.4 ± 0.7-point shift on average (Table 2). It also changes which checkpoint is selected and whether training appears to help at all (§4.3, Figure 1).
  • It is not simply that the tree sampler is stronger. Both procedures use the same budget and scoring. A stronger sampler would lift both policies equally; instead, each procedure favors the policy trained with it, in every comparison and at 22 of 25 checkpoints (§4.4).

These results matter because validation is used to compare RL methods, monitor training progress, and select checkpoints. If the validation procedure changes these decisions, it cannot be treated as a minor implementation detail. We therefore believe that tree RL should be evaluated under both standard and tree validation, with the evaluation procedure clearly reported. We will make this motivation more explicit in the revision.

Concern 2: Is tree inference worth deploying once latency is counted?

We agree with Reviewers qojX and iWNb that token savings alone do not establish the practical value of tree inference. We have therefore measured end-to-end latency and accuracy under different response-time limits. These results help answer a practical question raised by our findings: when is the accuracy gain from tree inference worth its latency cost?

In addition, for this response we explored a new tree inference procedure, which is not part of the submitted paper: a non-batch tree that branches dynamically during decoding, without constructing a full tree. It tests whether tree inference can be made as fast as standard decoding while keeping the accuracy gain of tree training. As we show below, our results point to a dilemma for tree RL.

Setup. One H200 per run; vLLM with CUDA graphs and prefix caching; MATH500; a 7B tree-trained policy (Qwen2.5-7B, TreeRL) compared with a DAPO-trained policy. Each procedure starts from a cleared cache; latency is per question, end to end, including answer selection.

Procedures.

  • IID6: 6 independent samples + majority vote.
  • Batch tree (Tree6): the training procedure with 6 paths. It generates the root responses in full, then forks at high-uncertainty tokens and generates the branches in a second round; majority vote.
  • Non-batch tree (new in this response, not in the submitted paper): branches at run time. Each trajectory decodes normally until a token's negative log-probability exceeds a threshold calibrated to the training fork points; there it samples alternative continuations up to the next paragraph break, keeps the one with the highest mean log-probability, and continues. We run 1 trajectory, or 6 in one batch with majority vote. It uses the training fork rule but never builds a full tree, so no answer waits for a later round.

Table A. Latency and accuracy.

ProcedureAcc. (%)Median latency (s)Generated tokens / q
Reference: the RL rollout procedure
Batch tree, identical to the training rollouts (EPTree M6 N2 T2, 30 paths) + vote80.772.668K
Decoding procedures for deployment
Greedy decoding73.314.23.2K
DAPO + IID6 + vote74.723.114.2K
Tree RL + batch tree (Tree6) + vote78.330.512.6K
Tree RL + non-batch tree (1 trajectory)71.617.23.4K
Tree RL + non-batch tree (6 trajectories) + vote75.121.211.6K
AuthorsOpenReview ↗

General Response

General Response -- part 2

Table B. Accuracy under a per-question response-time limit (correct and returned within the limit).

Limit (s)1530456090120180
Greedy decoding48.354.773.373.373.373.373.3
DAPO + IID6 + vote17.555.370.474.774.774.774.7
Tree RL + batch tree (Tree6) + vote15.243.166.071.778.378.378.3
Tree RL + non-batch tree (1 trajectory)44.953.871.671.671.671.671.6
Tree RL + non-batch tree (6 trajectories) + vote16.558.570.375.175.175.175.1
Reference: RL rollout procedure (batch tree, 30 paths)0.05.020.040.063.378.380.7

Findings.

  • The batch tree is the most accurate but slow: 78.3% at 30.5 s (2.1× greedy, 1.3× IID6). It is never the most accurate under limits of 60 s or less, and is from about 90 s on.
  • The non-batch tree removes the latency, but also most of the accuracy gain: with 6 trajectories, it is faster than IID6 (21.2 s vs 23.1 s), but its accuracy (75.1%) is at the level of DAPO + IID6 (74.7%), 3.2 points below Tree6.
  • With a single trajectory, it is below greedy decoding: 71.6% vs 73.3%, at a similar latency (17.2 s vs 14.2 s).

A dilemma for tree RL. Our results show that tree RL faces a dilemma:

  • Deployed with standard decoding, it is fast, but it trails DAPO.
  • Deployed with batch tree inference, it outperforms DAPO, but costs latency.
  • Deployed with dynamic branching (our non-batch tree), it is fast again, but most of the accuracy gain is lost.

This dilemma is invisible when tree RL is validated only with standard decoding, which is why we believe our finding matters. It also leaves room for future work: how to design dynamic-branching tree inference that keeps the accuracy gain of tree training while saving latency. Much of this is a systems question, since current serving engines have no operation for forking a sequence during decoding. We will revise the manuscript to include these experiments in §4.6 and to state this dilemma clearly in the discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.