- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
Paper Summary
This paper studies count fragmentation in MCTS for LLM reasoning: semantically similar branches maintain separate visit counts, potentially leading to redundant exploration. The proposed LPC-MCTS uses a Coin-Flipping Network on policy hidden states to estimate pseudo-counts and substitutes these estimates into the UCT exploration term. It also uses pseudo-counts to select trajectories for policy training. The paper provides theoretical analysis under idealized action equivalence assumptions and evaluates the approach on mathematical reasoning and search-based question answering across several models.
Critical strengths
Critical Strengths
-
A meaningful problem and a useful integration of learned counts into search. Count fragmentation is a plausible source of inefficiency in LLM tree search. Although the paper builds on existing count estimation techniques, using learned pseudo-counts directly in UCT, together with trajectory selection during training, constitutes a meaningful methodological and engineering contribution. The mathematical reasoning results provide encouraging evidence that this integration can improve performance.
-
Broad evaluation and substantial supporting material. The evaluation covers mathematical reasoning and search-based question answering, multiple models, and several benchmarks. The comparisons between training and inference configurations help distinguish their effects. The theoretical analysis and released implementation also make the contribution more informative and inspectable.
Critical weaknesses
Critical Weaknesses
-
The evidence for the proposed mechanism and its distinct contribution could be stronger. The paper appropriately cites closely related work, including FETCH and MERCI, but does not directly compare against these particularly relevant approaches. Moreover, the claim that learned counts share exploration information across semantically equivalent branches is supported mainly by qualitative examples rather than quantitative measurements. The trajectory-selection ablation shows that training data selection contributes substantially to the gains, leaving some uncertainty about how much improvement specifically comes from correcting count fragmentation.
-
The evaluation leaves uncertainty about robustness and computational efficiency. Although the benchmark coverage is broad, the paper does not report variability across independent training runs or confidence intervals. This is particularly relevant for search-based QA, where replacing standard MCTS with LPC-MCTS on the same LPC-trained policy yields relatively small average gains. Multiple responses sampled from a single search tree do not establish stability across independent runs. In addition, performance-versus-compute curves against standard MCTS and simpler sampling baselines would help determine whether the improvements remain compelling under matched token, tool-call, or wall-clock budgets.
Questions for authors
Questions For Authors
-
Efficiency clarification: How does the additional full-prefill request used for CFN hidden-state extraction reconcile with Appendix A.4's explanation of prefix-cache reuse? Please provide a breakdown of CFN-related backbone computation and wall-clock overhead, including inference-time costs.
-
Experimental configuration clarification: Could you provide the exact configurations used for the main tables? In particular, please clarify the consistency reward for non-majority answers and the search iteration budget, and explain the differences from the released evaluation settings.
-
Contribution attribution: Under matched generation and training budgets, how much improvement comes from replacing the UCT count, from pseudo-count-based trajectory selection, and from their combination? How stable are the relatively small inference gains on search-based QA across independent runs?
-
Mechanism validation: Do you have quantitative evidence that the learned counts transfer across semantically equivalent branches while avoiding inappropriate sharing between distinct strategies or problem contexts? A comparison with a random-head or another simple exploration-score control would help establish this.
-
Positioning: How does LPC-MCTS compare with the cited FETCH and MERCI approaches under comparable budgets? If a direct comparison is impractical, please explain the obstacles and more explicitly delimit the contribution relative to these methods.
9 comments
Response to Reviewer VnYk (2)
Q3 (W1 & W2): Contribution attribution.
Our current tables in PDF already provide a comparison that holds the trained policy fixed. For each LPC-MCTS-trained policy, replacing the UCT count with our learned pseudo-count improves the mathematical-reasoning average from 29.00 to 32.79 on Qwen2.5-7B and from 63.79 to 66.50 on Qwen3-4B. The corresponding improvements on search are from 46.25 to 46.83 and from 44.73 to 45.18. Since the policy checkpoint is unchanged within each comparison, these differences cannot be explained by a change in policy-training trajectory selection between the two evaluation conditions. They support a benefit from replacing the UCT count in the tree search at inference.
Separately, Table 9 in Appendix F.2.2 shows that removing pseudo-count-guided trajectory selection decreases mathematical-reasoning performance by 2.71 (Qwen2.5-7B) and 1.41 (Qwen3-4B) percentage points, which is a substantial contribution. Additionally, we have added an experiment on using standard UCT counts and only reserving pseudo-count-guided trajectory selection.
(a) Qwen2.5-7B
| Method | AIME24 | AIME25 | AIME26 | HMMT25 | BRUMO25 | Avg. |
|---|---|---|---|---|---|---|
| Full LPC-MCTS | 37.08 | 30.63 | 27.92 | 23.33 | 45.00 | 32.79 |
| w/o pseudo-count | 39.17 | 25.63 | 24.79 | 18.96 | 44.58 | 30.63 |
| w/o trajectory selection | 34.79 | 29.58 | 27.71 | 15.83 | 42.50 | 30.08 |
| Standard MCTS (Table 1) | 28.33 | 27.50 | 26.04 | 15.21 | 42.08 | 27.83 |
(b) Qwen3-4B
| Method | AIME24 | AIME25 | AIME26 | HMMT25 | BRUMO25 | Avg. |
|---|---|---|---|---|---|---|
| Full LPC-MCTS | 78.13 | 63.54 | 72.92 | 51.88 | 66.04 | 66.50 |
| w/o pseudo-count | 76.25 | 57.71 | 64.38 | 47.29 | 65.21 | 62.17 |
| w/o trajectory selection | 74.38 | 63.13 | 72.08 | 50.21 | 65.63 | 65.09 |
| Standard MCTS (Table 1) | 73.75 | 56.46 | 65.00 | 50.21 | 65.21 | 62.13 |
Removing either pseudo-counts or pseudo-count-guided trajectory selection reduces average performance, supporting contributions from both components.
For the multiple-run experiments, we agree that variability across independent training runs is important. However, due to the computational cost and the time available for rebuttal, we were unable to do multiple-run training. We are sorry for the absence, and will make every effort to provide these results when computational resources are available. To address your concern on the reliability of replacing standard MCTS with LPC-MCTS on the same LPC-trained policy, we repeated this comparison using 3 independent evaluation runs in both mathematical reasoning and search. The results are as follows:
Mathematical Reasoning
| Model | Method | AIME24 | AIME25 | AIME26 | HMMT25 | BRUMO25 | Overall |
|---|---|---|---|---|---|---|---|
| Qwen2.5-7B | UCT count | 33.96 ± 2.94 | 27.43 ± 2.65 | 27.29 ± 1.78 | 17.43 ± 0.32 | 43.33 ± 1.27 | 29.89 ± 0.78 |
| Qwen2.5-7B | Learned pseudo-count | 37.01 ± 1.35 | 30.90 ± 1.27 | 29.45 ± 1.68 | 21.39 ± 2.68 | 44.44 ± 0.96 | 32.64 ± 0.46 |
| Qwen3-4B | UCT count | 72.78 ± 1.57 | 58.13 ± 1.63 | 71.95 ± 2.75 | 51.60 ± 0.94 | 62.71 ± 0.75 | 63.43 ± 0.38 |
| Qwen3-4B | Learned pseudo-count | 77.57 ± 1.15 | 62.22 ± 1.94 | 74.31 ± 1.27 | 52.02 ± 0.43 | 64.24 ± 1.62 | 66.07 ± 0.71 |
Search
| Model | Method | NQ | TriviaQA | PopQA | HotpotQA | 2Wiki | MuSiQue | Bamboogle | Overall |
|---|---|---|---|---|---|---|---|---|---|
| Qwen2.5-7B | UCT count | 45.59 ± 0.26 | 58.51 ± 0.40 | 41.61 ± 0.74 | 48.05 ± 0.71 | 46.39 ± 0.30 | 33.21 ± 0.28 | 50.13 ± 0.40 | 46.21 ± 0.08 |
| Qwen2.5-7B | Learned pseudo-count | 46.28 ± 0.64 | 58.68 ± 0.57 | 41.92 ± 0.76 | 47.91 ± 0.13 | 47.72 ± 0.73 | 33.89 ± 0.47 | 50.88 ± 0.55 | 46.75 ± 0.06 |
| Llama3.2-3B | UCT count | 46.77 ± 0.40 | 56.00 ± 0.28 | 41.49 ± 0.54 | 46.76 ± 0.46 | 41.81 ± 0.14 | 31.03 ± 0.65 | 48.48 ± 0.58 | 44.62 ± 0.19 |
| Llama3.2-3B | Learned pseudo-count | 46.60 ± 0.49 | 56.32 ± 0.39 | 41.97 ± 0.12 | 47.24 ± 0.14 | 42.15 ± 0.23 | 31.79 ± 0.42 | 49.92 ± 0.16 | 45.14 ± 0.09 |
These results support stability across the repeated evaluations of the trained policies. Regarding the the relatively small inference gains on search-based QA, we have provided a detailed analysis in Section 6.3 in our manuscript. Since the tree structure for search is narrower, it suggests that the search task is simpler in our setting and tends to follow more standardized solution strategies, leaving fewer competing branches across which pseudo-counts can share exploration evidence.
Response to Reviewer VnYk (3)
Q4 (W1): Mechanism validation.
Thank you for suggesting quantitative mechanism validation. We have added a semantic-alignment diagnostic using the frozen policy backbone and trained CFN head. For each of Qwen3-4B and Qwen2.5-7B, we sample 8 distinct non-root MCTS nodes from each of 128 questions, yielding 2,048 nodes in total. Node selection does not use counts or embedding similarities.
We recompute each node’s pseudo-count and independently embed its assistant-generated text using Qwen3-Embedding-4B and QZhou-Embedding. Across the 28 within-question node pairs, we compare count closeness, , with embedding cosine similarity. We also measure whether count closeness and embedding similarity identify the same nearest neighbor.
| CFN backbone | Embedding model | Spearman ρ [95% CI] | Nearest-neighbor agreement rate |
|---|---|---|---|
| Qwen3-4B | Qwen3-Embedding-4B | 0.255 [0.197, 0.315] | 24.1% |
| Qwen3-4B | QZhou-Embedding | 0.248 [0.197, 0.299] | 24.8% |
| Qwen2.5-7B | Qwen3-Embedding-4B | 0.335 [0.279, 0.391] | 29.2% |
| Qwen2.5-7B | QZhou-Embedding | 0.358 [0.303, 0.413] | 26.9% |
These results provide quantitative evidence of a association between pseudo-count closeness and embedding-based semantic similarity: all 4 correlations are positive, with 95% confidence intervals excluding zero. Nearest-neighbor agreement rate is 24.1–29.2%, exceeding the 14.3% uniform-random baseline. These correlations suggest that learned pseudo-counts provide an exploration signal that is related to semantic similarity but does not fully match it. It is consistent with learned pseudo-counts providing a related but distinct, coverage-based exploration signal, unlike approaches such as FETCH that explicitly merge states based on semantic similarity.
We also conducted a quantitative analysis of learned count transfer. We perform a controlled update experiment on pairs of sibling responses and from the same MCTS parent. We treat as the update source and as the held-out probe. GPT-6 annotates their reasoning-pattern similarity separately from content equivalence. We measure the node-level pseudo-count of before and after 16 CFN-head updates on , using the coin-flip objective with a frozen backbone; is not used in these updates. Each trial starts from the original head and a fresh optimizer. We average log-pseudo-count changes within each question and report geometric mean percentage changes as follows:
| Probe nodes | Questions | Geometric mean pseudo-count change [95% CI] |
|---|---|---|
| Same-pattern siblings | 400 | +5.5% [+3.5%, +7.4%] |
| Distinct-pattern siblings | 126 | −3.2% [−5.0%, −1.3%] |
These results support preferential transfer across similar reasoning patterns while avoiding inappropriate sharing. Together with the qualitative examples in Appendix F.1.2, they provide complementary evidence that CFN’s learned counts generalize selectively across reasoning branches. We have included these results in our revised manuscript.
Random-head control. We additionally evaluated our LPC-MCTS against 3 randomly initialized heads, as you mentioned. We keep the policy checkpoint, benchmark questions, search settings, and scoring protocol fixed. Results on mathematical reasoning are as follows:
Qwen3-4B
| Method | AIME24 | AIME25 | AIME26 | HMMT25 | BRUMO25 | Overall |
|---|---|---|---|---|---|---|
| Trained head | 77.57 ± 1.15 | 62.22 ± 1.94 | 74.31 ± 1.27 | 52.02 ± 0.43 | 64.24 ± 1.62 | 66.07 ± 0.71 |
| Random head | 72.08 ± 4.43 | 57.02 ± 1.94 | 72.64 ± 2.09 | 48.13 ± 3.12 | 62.78 ± 2.05 | 62.53 ± 1.31 |
| Gain (pp) | +5.49 | +5.20 | +1.67 | +3.89 | +1.46 | +3.54 |
Qwen2.5-7B
| Method | AIME24 | AIME25 | AIME26 | HMMT25 | BRUMO25 | Overall |
|---|---|---|---|---|---|---|
| Trained head | 37.01 ± 1.35 | 30.90 ± 1.27 | 29.45 ± 1.68 | 21.39 ± 2.68 | 44.44 ± 0.96 | 32.64 ± 0.46 |
| Random head | 35.77 ± 1.89 | 27.71 ± 1.10 | 28.68 ± 2.19 | 18.82 ± 1.18 | 41.80 ± 0.87 | 30.56 ± 0.65 |
| Gain (pp) | +1.24 | +3.19 | +0.77 | +2.57 | +2.64 | +2.08 |
The trained head achieves higher mean@16 on all five benchmarks for both backbones. Overall accuracy improves from 62.53% to 66.07% on Qwen3-4B and from 30.56% to 32.64% on Qwen2.5-7B, corresponding to gains of 3.54 and 2.08 percentage points, respectively. These results support a contribution from learned head parameters to inference-time search performance beyond using a randomly initialized head on the same policy representation. Together with the semantic-alignment analysis, they provide evidence for both semantic association and the practical value of trained counts.
Response to Reviewer VnYk (1)
We sincerely thank you for your time and insightful review on our submission.
Q1 (W2): Backbone computation and prefix-cache reuse.
Thank you for requesting clarification. We have revised Appendix A.4 to provide a more detailed explanation. Sharing the same backbone and SGLang engine avoids maintaining a separate CFN backbone, including its weight loading and storage, and does not require switching between distinct policy and CFN backbone instances.
Specifically, the additional prefill request for CFN's feature extraction processes the node-local segment, not the entire root-to-node trajectory. In our implementation, this request does not skip cached prefixes: it obtains the complete token-level hidden states required by CFN scoring, and it does not flush the shared KV cache. Policy rollouts therefore continue to benefit from prefix-KV reuse during tree expansion. In frozen-tree online training, extracted node-level hidden states are additionally cached and reused for CFN head updates and rescoring without further backbone computation.
As you mention that more analysis on efficiency is required, we have added an experimental cost analysis. At inference time, we evaluate MCTS and LPC-MCTS on AIME 2024/2025 problems using 2 H200 GPUs, on the same checkpoint within identical search settings. The CFN head is frozen during inference.
We first report the CFN computation and runtime breakdown:
Component time shares are calculated from elapsed times summed across workers, divided by summed worker search time. Feature extraction includes the backbone request, scheduling/waiting, and hidden-state return and processing; scoring/control includes head evaluation.
We additionally report the overall inference costs. The wall-clock measurements include all search and CFN computation:
Policy-generated token counts cover the entire search, including branches not selected for the final output. We additionally report the amortized inference wall-clock time per returned candidate under the same evaluation setting: 16 candidate slots per problem. It excludes model initialization and post-processing.
These results show that CFN introduces measurable feature-extraction cost while subsequent scoring remains lightweight. Overall inference wall-clock cost remains comparable under the evaluated settings, alongside reduced policy-generated token consumption. The revised Appendix A.4 distinguishes the benefits of shared-backbone execution from the additional computation required for CFN feature extraction.
Q2: Experimental configuration clarification.
In our initial submission, we have provided some important experimental configurations that you mention. Section 6.1 specifies inference-time consistency rewards of +1 for terminal nodes in the current majority-answer group and −1 for the other terminal nodes. The backpropagation of these consistency-based rewards avoids accessing ground-truth answers during search, thereby preventing ground-truth leakage. Appendix A.2 specifies maximum search budgets of 24 MCTS iterations during training and 128 during evaluation. These iteration limits should be distinguished from the number of terminal trajectories collected. We have also provided more details on how the results in main tables were evaluated in Appendix E.2. For MCTS-based evaluation, we run a search tree per problem and collect 16 terminal trajectories for mathematical reasoning or 5 for search-based QA. Tree search stops as soon as the required number of trajectories is collected, with the iteration limit serving only as an upper bound. If the limit is reached before collecting enough trajectories, each missing candidate is counted as incorrect. we report mean@16 for mathematical reasoning and mean@5 for search-based QA.