acceptodds
Under review as a conference paper at ICLR 2027

MCTS Is Counting the Wrong Thing: Learning Exploration Counts for LLM Reasoning

Abstract

Monte Carlo tree search (MCTS) has powered landmark successes in reinforcement learning, yet its gains in LLM reasoning can remain modest despite substantial search effort. We identify an overlooked bottleneck: what MCTS counts. Standard UCT assigns independent visit counts to textual branches, allowing different formulations of the same reasoning strategy to repeatedly receive exploration bonuses. Our theoretical analysis shows that this count fragmentation can impair exploration efficiency, as quantified by increased cumulative regret. To address this problem, we introduce Learned Pseudo-Count MCTS (LPC-MCTS), which changes only the exploration statistic in UCT. A lightweight Coin-Flipping Network head on the policy's hidden representations estimates representation coverage, enabling related branches to share exploration evidence. Across 5 mathematical-reasoning and 7 search benchmarks, LPC-MCTS consistently outperforms strong MCTS baselines. Holding each LPC-MCTS-trained policy fixed, replacing standard MCTS counts at inference improves average mean@16 accuracy across 5 mathematical benchmarks by 3.79 percentage points on Qwen2.5-7B and 2.71 points on Qwen3-4B. Applying LPC-MCTS during both training and inference yields gains of 4.96 and 4.37 points over standard MCTS, respectively. Together, our theory and experiments show that changing what MCTS counts can substantially improve its exploration efficiency and reasoning performance.

Primary area:
foundation or frontier models, including LLMs
Keywords:
MCTS, LLM, exploration

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_VnYkOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Paper Summary

This paper studies count fragmentation in MCTS for LLM reasoning: semantically similar branches maintain separate visit counts, potentially leading to redundant exploration. The proposed LPC-MCTS uses a Coin-Flipping Network on policy hidden states to estimate pseudo-counts and substitutes these estimates into the UCT exploration term. It also uses pseudo-counts to select trajectories for policy training. The paper provides theoretical analysis under idealized action equivalence assumptions and evaluates the approach on mathematical reasoning and search-based question answering across several models.

Critical strengths

Critical Strengths

  1. A meaningful problem and a useful integration of learned counts into search. Count fragmentation is a plausible source of inefficiency in LLM tree search. Although the paper builds on existing count estimation techniques, using learned pseudo-counts directly in UCT, together with trajectory selection during training, constitutes a meaningful methodological and engineering contribution. The mathematical reasoning results provide encouraging evidence that this integration can improve performance.

  2. Broad evaluation and substantial supporting material. The evaluation covers mathematical reasoning and search-based question answering, multiple models, and several benchmarks. The comparisons between training and inference configurations help distinguish their effects. The theoretical analysis and released implementation also make the contribution more informative and inspectable.

Critical weaknesses

Critical Weaknesses

  1. The evidence for the proposed mechanism and its distinct contribution could be stronger. The paper appropriately cites closely related work, including FETCH and MERCI, but does not directly compare against these particularly relevant approaches. Moreover, the claim that learned counts share exploration information across semantically equivalent branches is supported mainly by qualitative examples rather than quantitative measurements. The trajectory-selection ablation shows that training data selection contributes substantially to the gains, leaving some uncertainty about how much improvement specifically comes from correcting count fragmentation.

  2. The evaluation leaves uncertainty about robustness and computational efficiency. Although the benchmark coverage is broad, the paper does not report variability across independent training runs or confidence intervals. This is particularly relevant for search-based QA, where replacing standard MCTS with LPC-MCTS on the same LPC-trained policy yields relatively small average gains. Multiple responses sampled from a single search tree do not establish stability across independent runs. In addition, performance-versus-compute curves against standard MCTS and simpler sampling baselines would help determine whether the improvements remain compelling under matched token, tool-call, or wall-clock budgets.

Questions for authors

Questions For Authors

  1. Efficiency clarification: How does the additional full-prefill request used for CFN hidden-state extraction reconcile with Appendix A.4's explanation of prefix-cache reuse? Please provide a breakdown of CFN-related backbone computation and wall-clock overhead, including inference-time costs.

  2. Experimental configuration clarification: Could you provide the exact configurations used for the main tables? In particular, please clarify the consistency reward for non-majority answers and the search iteration budget, and explain the differences from the released evaluation settings.

  3. Contribution attribution: Under matched generation and training budgets, how much improvement comes from replacing the UCT count, from pseudo-count-based trajectory selection, and from their combination? How stable are the relatively small inference gains on search-based QA across independent runs?

  4. Mechanism validation: Do you have quantitative evidence that the learned counts transfer across semantically equivalent branches while avoiding inappropriate sharing between distinct strategies or problem contexts? A comparison with a random-head or another simple exploration-score control would help establish this.

  5. Positioning: How does LPC-MCTS compare with the cited FETCH and MERCI approaches under comparable budgets? If a direct comparison is impractical, please explain the obstacles and more explicitly delimit the contribution relative to these methods.

9 comments
AuthorsOpenReview ↗

Response to Reviewer VnYk (1)

We sincerely thank you for your time and insightful review on our submission.

Q1 (W2): Backbone computation and prefix-cache reuse.

Thank you for requesting clarification. We have revised Appendix A.4 to provide a more detailed explanation. Sharing the same backbone and SGLang engine avoids maintaining a separate CFN backbone, including its weight loading and storage, and does not require switching between distinct policy and CFN backbone instances.

Specifically, the additional prefill request for CFN's feature extraction processes the node-local segment, not the entire root-to-node trajectory. In our implementation, this request does not skip cached prefixes: it obtains the complete token-level hidden states required by CFN scoring, and it does not flush the shared KV cache. Policy rollouts therefore continue to benefit from prefix-KV reuse during tree expansion. In frozen-tree online training, extracted node-level hidden states are additionally cached and reused for CFN head updates and rescoring without further backbone computation.

As you mention that more analysis on efficiency is required, we have added an experimental cost analysis. At inference time, we evaluate MCTS and LPC-MCTS on AIME 2024/2025 problems using 2 H200 GPUs, on the same checkpoint within identical search settings. The CFN head is frozen during inference.

We first report the CFN computation and runtime breakdown:

MeasurementQwen3-4BQwen2.5-7B
Mean CFN input length (tokens/sequence)2,905.481,442.85
CFN input sequences processed20,96816,512
CFN feature-extraction time share6.50%5.31%
CFN scoring/control time share0.23%0.18%
Combined CFN time share6.73%5.49%

Component time shares are calculated from elapsed times summed across workers, divided by summed worker search time. Feature extraction includes the backbone request, scheduling/waiting, and hidden-state return and processing; scoring/control includes head evaluation.

We additionally report the overall inference costs. The wall-clock measurements include all search and CFN computation:

ModelPolicy-generated tokens (M), MCTS → LPC-MCTSToken reductionAmortized wall-clock time (s/candidate), MCTS → LPC-MCTSRelative time
Qwen3-4B80.958 → 63.10622.05%16.72 → 15.980.956×
Qwen2.5-7B44.007 → 25.52542.00%12.05 → 12.831.064×

Policy-generated token counts cover the entire search, including branches not selected for the final output. We additionally report the amortized inference wall-clock time per returned candidate under the same evaluation setting: 16 candidate slots per problem. It excludes model initialization and post-processing.

These results show that CFN introduces measurable feature-extraction cost while subsequent scoring remains lightweight. Overall inference wall-clock cost remains comparable under the evaluated settings, alongside reduced policy-generated token consumption. The revised Appendix A.4 distinguishes the benefits of shared-backbone execution from the additional computation required for CFN feature extraction.

Q2: Experimental configuration clarification.

In our initial submission, we have provided some important experimental configurations that you mention. Section 6.1 specifies inference-time consistency rewards of +1 for terminal nodes in the current majority-answer group and −1 for the other terminal nodes. The backpropagation of these consistency-based rewards avoids accessing ground-truth answers during search, thereby preventing ground-truth leakage. Appendix A.2 specifies maximum search budgets of 24 MCTS iterations during training and 128 during evaluation. These iteration limits should be distinguished from the number of terminal trajectories collected. We have also provided more details on how the results in main tables were evaluated in Appendix E.2. For MCTS-based evaluation, we run a search tree per problem and collect 16 terminal trajectories for mathematical reasoning or 5 for search-based QA. Tree search stops as soon as the required number of trajectories is collected, with the iteration limit serving only as an upper bound. If the limit is reached before collecting enough trajectories, each missing candidate is counted as incorrect. we report mean@16 for mathematical reasoning and mean@5 for search-based QA.

AuthorsOpenReview ↗

Response to Reviewer VnYk (2)

Q3 (W1 & W2): Contribution attribution.

Our current tables in PDF already provide a comparison that holds the trained policy fixed. For each LPC-MCTS-trained policy, replacing the UCT count with our learned pseudo-count improves the mathematical-reasoning average from 29.00 to 32.79 on Qwen2.5-7B and from 63.79 to 66.50 on Qwen3-4B. The corresponding improvements on search are from 46.25 to 46.83 and from 44.73 to 45.18. Since the policy checkpoint is unchanged within each comparison, these differences cannot be explained by a change in policy-training trajectory selection between the two evaluation conditions. They support a benefit from replacing the UCT count in the tree search at inference.

Separately, Table 9 in Appendix F.2.2 shows that removing pseudo-count-guided trajectory selection decreases mathematical-reasoning performance by 2.71 (Qwen2.5-7B) and 1.41 (Qwen3-4B) percentage points, which is a substantial contribution. Additionally, we have added an experiment on using standard UCT counts and only reserving pseudo-count-guided trajectory selection.

(a) Qwen2.5-7B

MethodAIME24AIME25AIME26HMMT25BRUMO25Avg.
Full LPC-MCTS37.0830.6327.9223.3345.0032.79
w/o pseudo-count39.1725.6324.7918.9644.5830.63
w/o trajectory selection34.7929.5827.7115.8342.5030.08
Standard MCTS (Table 1)28.3327.5026.0415.2142.0827.83

(b) Qwen3-4B

MethodAIME24AIME25AIME26HMMT25BRUMO25Avg.
Full LPC-MCTS78.1363.5472.9251.8866.0466.50
w/o pseudo-count76.2557.7164.3847.2965.2162.17
w/o trajectory selection74.3863.1372.0850.2165.6365.09
Standard MCTS (Table 1)73.7556.4665.0050.2165.2162.13

Removing either pseudo-counts or pseudo-count-guided trajectory selection reduces average performance, supporting contributions from both components.

For the multiple-run experiments, we agree that variability across independent training runs is important. However, due to the computational cost and the time available for rebuttal, we were unable to do multiple-run training. We are sorry for the absence, and will make every effort to provide these results when computational resources are available. To address your concern on the reliability of replacing standard MCTS with LPC-MCTS on the same LPC-trained policy, we repeated this comparison using 3 independent evaluation runs in both mathematical reasoning and search. The results are as follows:

Mathematical Reasoning

ModelMethodAIME24AIME25AIME26HMMT25BRUMO25Overall
Qwen2.5-7BUCT count33.96 ± 2.9427.43 ± 2.6527.29 ± 1.7817.43 ± 0.3243.33 ± 1.2729.89 ± 0.78
Qwen2.5-7BLearned pseudo-count37.01 ± 1.3530.90 ± 1.2729.45 ± 1.6821.39 ± 2.6844.44 ± 0.9632.64 ± 0.46
Qwen3-4BUCT count72.78 ± 1.5758.13 ± 1.6371.95 ± 2.7551.60 ± 0.9462.71 ± 0.7563.43 ± 0.38
Qwen3-4BLearned pseudo-count77.57 ± 1.1562.22 ± 1.9474.31 ± 1.2752.02 ± 0.4364.24 ± 1.6266.07 ± 0.71

Search

ModelMethodNQTriviaQAPopQAHotpotQA2WikiMuSiQueBamboogleOverall
Qwen2.5-7BUCT count45.59 ± 0.2658.51 ± 0.4041.61 ± 0.7448.05 ± 0.7146.39 ± 0.3033.21 ± 0.2850.13 ± 0.4046.21 ± 0.08
Qwen2.5-7BLearned pseudo-count46.28 ± 0.6458.68 ± 0.5741.92 ± 0.7647.91 ± 0.1347.72 ± 0.7333.89 ± 0.4750.88 ± 0.5546.75 ± 0.06
Llama3.2-3BUCT count46.77 ± 0.4056.00 ± 0.2841.49 ± 0.5446.76 ± 0.4641.81 ± 0.1431.03 ± 0.6548.48 ± 0.5844.62 ± 0.19
Llama3.2-3BLearned pseudo-count46.60 ± 0.4956.32 ± 0.3941.97 ± 0.1247.24 ± 0.1442.15 ± 0.2331.79 ± 0.4249.92 ± 0.1645.14 ± 0.09

These results support stability across the repeated evaluations of the trained policies. Regarding the the relatively small inference gains on search-based QA, we have provided a detailed analysis in Section 6.3 in our manuscript. Since the tree structure for search is narrower, it suggests that the search task is simpler in our setting and tends to follow more standardized solution strategies, leaving fewer competing branches across which pseudo-counts can share exploration evidence.

AuthorsOpenReview ↗

Response to Reviewer VnYk (3)

Q4 (W1): Mechanism validation.

Thank you for suggesting quantitative mechanism validation. We have added a semantic-alignment diagnostic using the frozen policy backbone and trained CFN head. For each of Qwen3-4B and Qwen2.5-7B, we sample 8 distinct non-root MCTS nodes from each of 128 questions, yielding 2,048 nodes in total. Node selection does not use counts or embedding similarities.

We recompute each node’s pseudo-count and independently embed its assistant-generated text using Qwen3-Embedding-4B and QZhou-Embedding. Across the 28 within-question node pairs, we compare count closeness, −∣log⁡Ci−log⁡Cj∣-\left|\log C_i - \log C_j\right|, with embedding cosine similarity. We also measure whether count closeness and embedding similarity identify the same nearest neighbor.

CFN backboneEmbedding modelSpearman ρ [95% CI]Nearest-neighbor agreement rate
Qwen3-4BQwen3-Embedding-4B0.255 [0.197, 0.315]24.1%
Qwen3-4BQZhou-Embedding0.248 [0.197, 0.299]24.8%
Qwen2.5-7BQwen3-Embedding-4B0.335 [0.279, 0.391]29.2%
Qwen2.5-7BQZhou-Embedding0.358 [0.303, 0.413]26.9%

These results provide quantitative evidence of a association between pseudo-count closeness and embedding-based semantic similarity: all 4 correlations are positive, with 95% confidence intervals excluding zero. Nearest-neighbor agreement rate is 24.1–29.2%, exceeding the 14.3% uniform-random baseline. These correlations suggest that learned pseudo-counts provide an exploration signal that is related to semantic similarity but does not fully match it. It is consistent with learned pseudo-counts providing a related but distinct, coverage-based exploration signal, unlike approaches such as FETCH that explicitly merge states based on semantic similarity.

We also conducted a quantitative analysis of learned count transfer. We perform a controlled update experiment on pairs of sibling responses AA and BB from the same MCTS parent. We treat AA as the update source and BB as the held-out probe. GPT-6 annotates their reasoning-pattern similarity separately from content equivalence. We measure the node-level pseudo-count of BB before and after 16 CFN-head updates on AA, using the coin-flip objective with a frozen backbone; BB is not used in these updates. Each trial starts from the original head and a fresh optimizer. We average log-pseudo-count changes within each question and report geometric mean percentage changes as follows:

Probe nodesQuestionsGeometric mean pseudo-count change [95% CI]
Same-pattern siblings400+5.5% [+3.5%, +7.4%]
Distinct-pattern siblings126−3.2% [−5.0%, −1.3%]

These results support preferential transfer across similar reasoning patterns while avoiding inappropriate sharing. Together with the qualitative examples in Appendix F.1.2, they provide complementary evidence that CFN’s learned counts generalize selectively across reasoning branches. We have included these results in our revised manuscript.

Random-head control. We additionally evaluated our LPC-MCTS against 3 randomly initialized heads, as you mentioned. We keep the policy checkpoint, benchmark questions, search settings, and scoring protocol fixed. Results on mathematical reasoning are as follows:

Qwen3-4B

MethodAIME24AIME25AIME26HMMT25BRUMO25Overall
Trained head77.57 ± 1.1562.22 ± 1.9474.31 ± 1.2752.02 ± 0.4364.24 ± 1.6266.07 ± 0.71
Random head72.08 ± 4.4357.02 ± 1.9472.64 ± 2.0948.13 ± 3.1262.78 ± 2.0562.53 ± 1.31
Gain (pp)+5.49+5.20+1.67+3.89+1.46+3.54

Qwen2.5-7B

MethodAIME24AIME25AIME26HMMT25BRUMO25Overall
Trained head37.01 ± 1.3530.90 ± 1.2729.45 ± 1.6821.39 ± 2.6844.44 ± 0.9632.64 ± 0.46
Random head35.77 ± 1.8927.71 ± 1.1028.68 ± 2.1918.82 ± 1.1841.80 ± 0.8730.56 ± 0.65
Gain (pp)+1.24+3.19+0.77+2.57+2.64+2.08

The trained head achieves higher mean@16 on all five benchmarks for both backbones. Overall accuracy improves from 62.53% to 66.07% on Qwen3-4B and from 30.56% to 32.64% on Qwen2.5-7B, corresponding to gains of 3.54 and 2.08 percentage points, respectively. These results support a contribution from learned head parameters to inference-time search performance beyond using a randomly initialized head on the same policy representation. Together with the semantic-alignment analysis, they provide evidence for both semantic association and the practical value of trained counts.

Reviewer_RCioOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary
  1. This work investigates an overlooked bottleneck in monte carlo tree search methods in LLM reasoning, where the UCT based visit counting strategy ignoring the repetition across multiple branches, leading to ineffective exploration.
  2. To alleviate this problem, they introduce learned-pseudo-count MCTS, which uses a simple neural network to measure the similarity between different textual branches, to improve the exploration efficiency.
  3. Empirical results across both mathematical reasoning and search tasks demonstrate the effectiveness of the proposed LPC-MCTS, with obvious superiority on Qwen2.5-7B and Qwen3-4B compared with baseline methods.
Critical strengths
  1. This paper identifies a critical limitation in tree-based LLM reasoning methods. Specifically, the exploration stage might uses a wrong visit counting mechanism, ignoring the repetition between different textual branches, resulting in redundant exploration. To solve the problem, they propose a simple and straightforward method to consider the similarity between different tree branches to guide more effective exploration.
  2. Empirical results across multiple benchmarks across both reasoning and search demonstrate the superiority of the proposed strategy, with clear improvements compared with baseline methods.
Critical weaknesses

A critical weakness of this work is the lack of cost analysis, including both the token efficiency and time latency, which is important in tree-search based reasoning methods.

Questions for authors

Both Table 1 and Table 2 show the performance after training, how do the models perform before training? This provides significant insights to understand the behavior change after training.

3 comments
AuthorsOpenReview ↗

Response to Reviewer RCio

We sincerely thank you for your time and insightful review on our submission.

Weakness: Cost Analysis

Thank you for highlighting this important question. We have added a cost analysis that reports token efficiency and time latency.

At inference time, we evaluate MCTS and LPC-MCTS on AIME 2024/2025 problems using 2 H200 GPUs, on the same checkpoint within identical search settings. The CFN head is frozen during inference. Results of token efficiency are as follows:

ModelMCTS policy-generated tokens (M)LPC-MCTS policy-generated tokens (M)Reduction
Qwen3-4B80.95863.10622.05%
Qwen2.5-7B44.00725.52542.00%

We count policy-generated tokens across the entire search, including branches not selected for the final output. Under the same maximum search-iteration budget, LPC-MCTS explores using fewer policy-generated tokens, which enables more efficient search.

We additionally report the amortized inference wall-clock time per returned candidate under the same evaluation setting: 16 candidate slots per problem.

ModelMCTS (s/candidate)LPC-MCTS (s/candidate)Relative time
Qwen3-4B16.7215.980.956× (−4.44%)
Qwen2.5-7B12.0512.831.064× (+6.43%)

Under this fixed evaluation setting, our LPC-MCTS substantially reduces policy-generated token consumption while maintaining comparable inference wall-clock cost.

We have also included the above analysis in our revised manuscript.

Question: Performance before RL training

Thank you for this suggestion. We evaluated the SFT checkpoints used to initialize RL training, providing an explicit RL training baseline for our comparison.

We evaluate each initial policy under both inference modes considered in the main experiments: direct sampling and MCTS for mathematical reasoning, and greedy sampling and MCTS for search. MCTS uses standard visit counts with consistency-based backup. The tables below report per-benchmark accuracy, with Avg. denoting the average across benchmarks, consistent with Tables 1 and 2.

Mathematical reasoning before RL: mean@16 accuracy (%).

ModelInferenceAIME 2024AIME 2025AIME 2026HMMT Feb 2025BRUMO 2025Avg.
Qwen2.5-7BDirect sampling9.588.548.134.1719.5810.00
Qwen2.5-7BMCTS10.4212.088.5410.0021.0412.42
Qwen3-4BDirect sampling40.4228.3325.4220.6337.5030.46
Qwen3-4BMCTS36.4625.0018.9616.8839.5827.38

Search before RL: greedy accuracy / MCTS mean@5 accuracy (%).

ModelInferenceNQTriviaQAPopQAHotpotQA2WikiMuSiQueBamboogleAvg.
Qwen2.5-7B-InstructGreedy36.8054.4025.8043.2038.6029.2046.4039.20
Qwen2.5-7B-InstructMCTS36.9653.8425.2442.8042.9229.7247.5239.86
Llama-3.2-3B-InstructGreedy39.4051.6026.2039.4035.2027.6047.2038.09
Llama-3.2-3B-InstructMCTS38.1252.4026.8840.1635.6825.7645.9237.85

These results establish the starting performance of each policy and provide a reference for interpreting the post-RL results in Tables 1 and 2. We have included these SFT-only baselines in our revised manuscript to make the starting capabilities and subsequent changes more explicit.

We sincerely hope these clarifications adequately address your concerns.

Thank you for your thoughtful response and the additional experiments. I have carefully considered your replies and appreciate your efforts to address my concerns. I retain my positive score for this work. Thank you for your time and effort.

We are sincerely grateful for your constructive feedback and your positive assessment of our work.

Reviewer_fLhvOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Learned Pseudo-Count Monte Carlo Tree Search (LPC-MCTS) to reduce redundant exploration in LLM reasoning. The authors show that standard visit count allows different textual formulations of the same reasoning strategy to obtain an excessive exploration bonus. LPC-MCTS replaces the visit count with a learned pseudo-count using a linear Coin-Flipping Network head on the policy hidden representations. The authors evaluate LPC-MCTS on two agentic settings: tool-integrated mathematical reasoning and search. Using LPC-MCTS during both training and inference improves average tool-integrated mathematical accuracy over standard MCTS by 4.96 points on Qwen2.5-7B and 4.37 points on Qwen3-4B. On the search tasks, LPC-MCTS outperforms standard MCTS by 2.26 and 1.71 points on Qwen2.5-7B-Instruct and Llama3.2-3B-Instruct, respectively.

Critical strengths
  1. The method is simple and adds a lightweight network to standard training setups. This simplicity makes the approach easy to integrate into existing MCTS-based pipelines and supports its practical applicability.

  2. The results on challenging mathematical benchmarks are impressive. The gains over standard MCTS provide empirical evidence that reducing redundant exploration improves reasoning accuracy.

Critical weaknesses
  1. The authors used only three models and not very recent. This limited scope makes it difficult to assess whether LPC-MCTS is also effective with more recent models.
Questions for authors
  1. Could you evaluate LPC-MCTS using a model from the Qwen3.5 family? This would help to assess whether the reported improvements are consistent with recent model backbones.
4 comments
AuthorsOpenReview ↗

Response to Reviewer fLhv

We sincerely thank you for your time and insightful review on our submission.

W & Q: Experiments on the Qwen3.5 family

Thank you for this suggestion. We have extended our evaluation to Qwen3.5-4B to assess whether the benefits of LPC-MCTS generalize to a more recent backbone. For the Qwen3.5-4B experiments, we increase the maximum response length to 36,000 tokens and the maximum response length per node to 6,144 tokens. All other training and evaluation parameters remain the same as those reported in our manuscript. Within the fixed-policy inference comparison, we use identical evaluation questions, decoding parameters, and search-budget settings. All Qwen3.5-4B configurations use non-thinking mode.

Due to the limited computational resources and the time available for rebuttal, we added experiments on 3 critical configurations: (A) MCTS training with MCTS inference, (B) LPC-MCTS training with MCTS inference, and (C) LPC-MCTS training with LPC-MCTS inference. Configurations B and C use exactly the same trained policy checkpoint, isolating the effect of the inference-time search method.

Mathematical reasoning results (mean@16) on Qwen3.5-4B:

Training → InferenceAIME24AIME25AIME26HMMT25BRUMO25Avg.
SFT → Direct sampling41.4629.3832.7123.9645.0034.50
SFT → MCTS35.8330.8326.4623.9643.5432.12
MCTS → Direct sampling69.7960.8370.4246.4673.5464.21
A: MCTS → MCTS76.2560.4281.4659.1778.1371.09
LPC-MCTS → Direct sampling71.8858.5469.7948.7574.7964.75
B: LPC-MCTS → MCTS79.3862.5078.1363.7579.7972.71
C: LPC-MCTS → LPC-MCTS80.6364.5877.7162.7183.9673.92
Inference gain: C − B+1.25+2.08−0.42−1.04+4.17+1.21
End-to-end gain: C − A+4.38+4.16−3.75+3.54+5.83+2.83

Avg. is the average across 5 benchmarks.

These results provide additional evidence that the average fixed-policy benefit of LPC-MCTS extends to Qwen3.5-4B. Holding the trained policy checkpoint fixed, replacing MCTS inference with LPC-MCTS (C vs. B) increases average mean@16 accuracy from 72.71% to 73.92% (+1.21 points). Under the same MCTS inference method, LPC-MCTS training also improves average accuracy by 1.62 points (B vs. A). Together, these comparisons support benefits from both training-time and inference-time use of LPC-MCTS on this more recent backbone.

We sincerely hope these clarifications adequately address your concerns.

We are very grateful for your thoughtful feedback on our manuscript. We believe that incorporating your suggestions has substantially improved the quality and clarity of our work.

With limited hours in the discussion period and having not yet received your feedback, we would like to kindly confirm whether our response has addressed all your concerns. We would be grateful for any additional discussion or suggestions. We thank you once more for your invaluable contribution to our work.

Best Regards,

Authors

I thank the authors for their reply. My rating remains positive and unchanged.

We are sincerely grateful for your constructive feedback and your positive assessment of our work.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Our Rebuttal and Reviewers' Feedback

Dear ACs,

Thank you for your time and effort in evaluating our submission. We briefly summarize the reviews and the discussion.

Our submission received 3 positive initial ratings: 3/3/3. The reviews highlighted the meaningful problem of count fragmentation, the simplicity of LPC-MCTS, and its empirical improvements. Reviewers mainly requested additional experiments and analyses, all of which we provided during the rebuttal. All reviewers responded positively to these additions and confirmed that they would maintain their positive ratings.

We sincerely appreciate the reviewers’ constructive suggestions, which have strengthened our work and improved the manuscript, and thank you for your consideration.

Best regards,

ICLR 2027 Conference Submission43020 Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.