- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper argues that tree-based RL methods should be evaluated the same way they are trained, and shows that tree-trained policies outperform DAPO under tree evaluation but underperform it under standard evaluation.
Critical strengths
- The paper raises an interesting research question that challenges previous opinions in this line of research by rethinking how tree-based RL is evaluated.
- The paper is easy to follow.
Critical weaknesses
- The inference cost of tree evaluation is unclear. Although it generates fewer new tokens than independent sampling, tree inference requires sequential rounds of generation, and the paper reports no wall-clock latency or throughput. The paper should discuss and provide concrete results on why tree-based inference is worth adopting in practice; the current version does not sufficiently motivate this question.
- Limited novelty. The main finding, that a tree-trained policy performs better than DAPO under tree inference, is largely expected, since each policy is evaluated under the procedure it was trained with. Unless the paper justifies why tree-based inference should be used in practice (Weakness 1), showing that tree-based RL outperforms standard RL under tree inference alone is not a meaningful contribution.
- The experiments use only Qwen2.5-family models, on which RLVR has been shown to improve even with spurious or random rewards (Shao et al., 2025). Do the conclusions hold on more recent model families that are less affected by this issue?
Shao, Rulin, et al. "Spurious rewards: Rethinking training signals in rlvr." arXiv preprint arXiv:2506.10947 (2025).
Questions for authors
See weakness above.
5 comments
Response to Reviewer iWNb – part B
Response to Reviewer iWNb -- part B
W3. Model families and spurious rewards. We thank the reviewer for the pointer to Shao et al. (2025). Our main result is a relative comparison, which is robust to this issue:
- Within each pair, both policies share the same initialization, data and number of training trajectories. Any improvement from spurious reward signals in Qwen2.5 applies to both training methods.
- The Shift is a difference between two validation procedures applied to the same checkpoints. It does not depend on the absolute RL gain over the initialization.
- Our comparisons include general Qwen2.5 models at 3B and 7B, not only Qwen2.5-Math, and non-math tasks (LiveCodeBench, HotpotQA), and the same reversal appears in all of them.
We will add results on a non-Qwen model family in the revision.
Revision. We will (i) add the latency and response-time-limit results to §4.6, (ii) make the motivation above more explicit in the introduction, (iii) add the non-batch tree experiments and state this dilemma clearly, and (iv) add results on a non-Qwen model family.
We thank the reviewer again for these valuable questions, which have helped us strengthen the paper. We hope the new results and clarifications address the reviewer's concerns, and we would be sincerely grateful if the reviewer would reconsider the assessment in light of them.
Thank you for the detailed response and the additional experiments. After reading the rebuttal and the general response, I have decided to keep my score, for the following reasons.
-
Inference cost. I appreciate the new latency measurements. However, they confirm my concern: batch tree inference is about 32% slower than IID6 (30.5 s vs 23.1 s) for a 3.6-point accuracy gain, and the non-batch variant removes the latency cost only by giving up most of that gain (75.1% vs 74.7%). As the authors acknowledge, tree RL faces a dilemma in deployment. Given the modest accuracy improvement, I find this latency cost concerning for practical use. In addition, these results come from a single setting without reported variance.
-
Novelty. I still find the main finding, that a policy performs best under the procedure it was trained with, to be largely expected. The response restates the arguments in the paper, and I do not see a new insight beyond quantifying this effect.
-
Model families. Results on a non-Qwen model family are promised for the revision but not provided, so this concern remains unaddressed. The argument that a relative comparison is robust to spurious rewards assumes that such signals affect both training methods equally, which has not been verified.
I thank the authors again for their efforts.
Response to reviewer iWNb
Response to Reviewer iWNb -- part A
We sincerely thank the reviewer for finding our research question interesting, one that "challenges previous opinions in this line of research," and the paper easy to follow. We are also grateful for the reviewer's questions on the practical value of tree inference, which led directly to the new latency measurements in this response.
Our main point. All five tree-RL methods we surveyed train with branching but validate without it (Table 1). This simple mismatch has been overlooked, and its effect is large: changing only the validation procedure reverses the ranking between tree RL and DAPO in all five comparisons (5.4 points on average), and changes whether training appears to help at all (TreeRL on Qwen2.5-Math-1.5B-Instruct: no gain under standard validation, +4.5 points under tree validation; §4.3). Conclusions about tree RL drawn from standard validation alone can therefore be incomplete or misleading.
W1. Inference cost of tree evaluation. We thank the reviewer for this important point. While our main focus is the validation mismatch, we fully agree that latency matters for deployment. We have therefore measured it, and we also explored a new inference procedure. The full results are in the General Response (Concern 2, Tables A and B); all procedures run on the same hardware and serving stack.
With dynamic branching, the non-batch tree removes the latency cost, but its accuracy is at the level of DAPO + IID6 (75.1% vs 74.7%), and with a single trajectory it is below greedy decoding (71.6% vs 73.3%). Tree RL therefore faces a dilemma: with standard decoding it trails DAPO, with batch tree inference it wins but costs latency, and with dynamic branching it is fast but loses most of its gain. This leaves room to explore how dynamic-branching tree inference can keep the accuracy gain while saving latency, which is largely a systems question.
W2. Novelty. We thank the reviewer for raising this question, which we also address in the General Response (Concern 1). We agree that the intuition is natural. But the result is not simply that a policy does best under its own training procedure:
We believe that identifying a widely used evaluation practice that reverses the conclusions of method comparisons, and exposing the resulting dilemma for tree RL, is a meaningful contribution. We will make this motivation more explicit in the introduction.