acceptodds
Under review as a conference paper at ICLR 2027

When Does Self-Evaluation Help in Reasoning? An Anatomy of ToT on Mathematical Tasks

Abstract

Step-wise reasoning, as a fundamental capability of large language models (LLMs), has significantly advanced complex reasoning through decomposing a problem into a sequence of coherent intermediate thoughts conditioned on context information and prior knowledge. As two representatives, sampling-based reasoning paradigms (e.g., Self-Consistency Chain-of-Thought, SC-CoT) achieve impressive performance on tasks with deterministic answers by aggregating multiple trajectories via majority voting, while search-based reasoning paradigms (e.g., Tree-of-Thoughts, ToT) excel on search-intensive tasks that require explicit step-wise evaluations. However, despite explicit step-wise evaluations, ToT often falls short in mathematical reasoning tasks. Rather than simply attributing this problem to unreliable self-evaluations, in this paper, we revisit ToT and explore the properties of its evaluation behaviors from both theoretical and empirical perspectives. Different from SC-CoT, which marginalizes out thoughts and benefits from a concentrated distribution around the correct answer (i.e., low answer diversity), the performance of ToT is correlated with the coverage of branches in a given budget and the distributions of self-evaluations. Motivated by these findings, we propose a tool, entropy-guided adaptive reasoning tree (EGART), to further investigate whether evaluations can provide effective signals to select the promising branches. Overall, our findings uncover the behaviors of ToT in reasoning tasks and provide an insight into why explicit self-evaluation steps fail in search-based stepwise reasoning.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
LLM Reasoning, ToT

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_PfcFOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper investigates why Tree-of-Thoughts (ToT), despite incorporating explicit step-wise self-evaluation, frequently underperforms Self-Consistency Chain-of-Thought (SC-CoT) on mathematical reasoning tasks. The authors analyze the two paradigms from theoretical and empirical perspectives, emphasizing the role of answer concentration in SC-CoT and candidate coverage and evaluation distributions in ToT. Their theoretical analysis characterizes the conditions under which SC-CoT benefits from additional sampling and ToT successfully preserves correct intermediate reasoning steps.

Based on these analyses, the paper introduces Entropy-Guided Adaptive Reasoning Tree (EGART), a variant of ToT that adaptively determines the number of intermediate reasoning branches to preserve using the entropy of self-evaluation signals. The authors further investigate different evaluation and selection strategies, including voting, independent scoring, stochastic sampling, and top-k selection.

Experiments are conducted using Qwen2.5-3B-Instruct and Qwen2.5-7B-Instruct on GSM8K, MATH, AIME24, and AIME25. The results suggest that evaluation-derived signals, particularly ranking and entropy, can help characterize reasoning behavior and guide adaptive branch preservation. EGART achieves performance comparable to conventional ToT while preserving fewer intermediate reasoning paths in some settings. However, its improvements are generally modest and inconsistent, and SC-CoT remains substantially stronger on most evaluated mathematical reasoning benchmarks.

Critical strengths
  1. Useful empirical characterization of SC-CoT and ToT. The paper investigates a relevant discrepancy between sampling-based and search-based reasoning: despite explicitly evaluating intermediate thoughts, ToT often underperforms SC-CoT on mathematical reasoning tasks. The analyses of answer concentration, candidate coverage, and self-evaluation behavior provide useful insights into the practical limitations of these paradigms. In particular, the observation that self-evaluation rankings may retain useful information even when fine-grained evaluation scores are unreliable is informative. These analytical findings are arguably more compelling than the proposed EGART method itself.

  2. An intuitive adaptive branch-preservation mechanism with improved theoretical formulation. EGART uses evaluation entropy to dynamically adjust the number of preserved intermediate reasoning branches. The mechanism is straightforward and relatively easy to implement. Compared with the earlier formulation, the theoretical analysis now more explicitly distinguishes the population-level answer distribution in SC-CoT from empirical sampling outcomes, and characterizes ToT selection through a conditional single-step analysis. These revisions improve the mathematical clarity of the paper, although the theoretical justification for EGART itself remains incomplete.

Critical weaknesses
  1. Limited methodological novelty and practical improvement. EGART remains fundamentally an adaptive-width variant of ToT, where the preservation budget is controlled by evaluation entropy. Although the paper provides theoretical and empirical motivation for this design, the mechanism does not appear substantially different from existing adaptive beam-width or confidence-based pruning approaches. The experimental evidence does not convincingly establish that evaluation entropy provides a meaningful advantage over simpler adaptive search strategies. Moreover, the improvements over conventional ToT remain modest and inconsistent, while SC-CoT continues to substantially outperform ToT-based approaches on most mathematical reasoning benchmarks. Consequently, the practical significance of the proposed method remains limited.

  2. Limitations in the theoretical analysis and its connection to EGART. Although the revised theoretical formulation addresses some previously identified mathematical issues, several limitations remain. First, Theorem 2.2 characterizes SC-CoT through the population-level probability margin between the correct answer and the most probable incorrect answer. However, the empirical analysis primarily uses the number of unique sampled answers as a diversity measure, which is not equivalent to the probability margin appearing in the theorem. Consequently, the relationship between the theoretical characterization and empirical observations remains indirect. Second, Theorem 2.4 assumes that the candidate pool already contains at least one correct intermediate thought and characterizes the probability of preserving a correct candidate in a single reasoning step. This conditional result does not capture generation-stage failures, where no correct candidate is produced, nor does it establish end-to-end reasoning guarantees for the full search tree. More importantly, neither theorem directly establishes that evaluation entropy is an appropriate estimator of the required preservation budget or that the proposed mapping from entropy to search width is theoretically optimal. Low evaluation entropy may also arise when the evaluator confidently favors an incorrect intermediate thought, potentially causing EGART to prune valid reasoning paths. Therefore, the theoretical analysis provides useful intuition for candidate coverage and evaluation behavior, but does not sufficiently justify the central entropy-guided mechanism.

  3. Insufficient experimental coverage and statistical evidence. The evaluation remains restricted to Qwen2.5-3B-Instruct and Qwen2.5-7B-Instruct, two relatively small models from the same family. This limits the generalizability of conclusions regarding LLM self-evaluation and reasoning behavior. The benchmark selection also remains problematic: GSM8K and MATH are relatively manageable under repeated sampling, whereas AIME24 and AIME25 are considerably more challenging for models of this scale and each contain only 30 problems. Although the revised manuscript includes additional experimental configurations and reports averages and standard deviations across two seeds, these changes do not fully resolve the statistical limitations. Several reported AIME differences correspond to only one or two correctly solved problems. Furthermore, the primary preservation-budget metric does not directly represent total inference computation, making the practical efficiency advantage of EGART difficult to establish without more comprehensive compute-matched comparisons.

Questions for authors
  1. Novelty and comparison with adaptive-width methods. EGART appears to be an adaptive-width implementation of ToT, with evaluation entropy serving as the control signal for branch preservation. Beyond using a different uncertainty metric, what is the key methodological distinction between EGART and existing adaptive beam-width or confidence-based pruning approaches? Can the authors provide a controlled comparison against a simpler adaptive-width ToT baseline under comparable computational budgets?

  2. Theoretical justification of entropy-guided preservation. Theorem 2.4 characterizes the probability of preserving a correct intermediate thought in terms of the evaluation margin and preservation budget, under the assumption that the candidate pool contains at least one correct thought. However, EGART determines the preservation budget using evaluation entropy, and the theoretical analysis does not explicitly establish a relationship between entropy and the evaluation margin. Can the authors provide a theoretical justification for the proposed entropy-to-width mapping, or demonstrate that it provides a more reliable preservation strategy than alternative uncertainty measures? In particular, how does the proposed mechanism behave when the evaluator confidently assigns high scores to incorrect thoughts, resulting in low entropy despite the presence of potentially correct alternatives?

  3. Relationship between theoretical assumptions and empirical evaluation. Theorem 2.2 characterizes SC-CoT through the probability margin between the correct answer and the most probable incorrect answer, while the empirical analysis primarily considers the number of unique sampled answers. Can the authors clarify how this empirical diversity measure validates the theoretical result? Additionally, given that Theorem 2.4 conditions on the presence of at least one correct candidate, can the authors provide empirical evidence separating failures caused by incorrect candidate generation from failures caused by evaluation-based pruning? Such an analysis would help establish whether EGART addresses the dominant source of error in ToT.

  4. Robustness and generalizability. The experiments continue to rely exclusively on Qwen2.5-3B/7B, and the improvements on AIME24/25 remain difficult to interpret because of their small test-set sizes. Can the authors provide additional evidence of robustness, such as evaluations across more seeds, confidence intervals, or experiments involving another model family or larger model? In particular, are the reported improvements over ToT statistically distinguishable from evaluation variance?

  5. Actual computational efficiency. The paper emphasizes that EGART can preserve fewer intermediate reasoning paths while maintaining comparable accuracy. However, reducing the number of preserved paths does not necessarily translate into lower overall inference cost, given the additional self-evaluation operations. Although the appendix provides runtime and token-consumption analyses, can the authors establish the benefits of EGART using accuracy-versus-total-compute comparisons against fixed-width ToT and SC-CoT?

12 comments
AuthorsOpenReview ↗

A Clarification regarding the Position of Our Paper

We would like to first clarify that, in our paper, EGART is a probing tool, instead of a method for SoTA performance, to validate the efficacy of ranking information and evaluation entropy signals.

  • The main goal of this paper is to understand the reliability of self-evaluation and discuss potential signals that are meaningful for search-based reasoning paradigms.
  • As a probing intervention tool, EGART aims to validate whether evaluation entropy, along with appropriate selection strategies, helps preserve viable intermediate thoughts for reasoning, instead of pursuing SoTA performance.
  • The main contributions of this paper are comprehensive and detailed empirical analyses on self-evaluation and exploration and validation of ranking and evaluation entropy in search-based reasoning paradigms.
AuthorsOpenReview ↗

Response to Weakness 1 & Question 1. Concern about Novelty of EGART and Limited Improvement.

Response: We appreciate your positive attitudes towards our theoretical and empirical analyses. For your concerns, we think you might have a misunderstanding of our paper. To address your concerns, we first summarize your concerns as follows:

  1. [Similar to Existing Methods] The main idea of EGART is equivalent to existing adaptive beam-width and confidence-based pruning;
  2. [Limited Improvements on Performance] The reported results show limited improvements compared to ToT;
  3. [Comparison to Other Methods] Lack comparison with existing adaptive ToT methods for difference.

For the concern 1 & 2, please check our clarification above.


About Comparison with Existing Methods

For the concern 3, we here consider a confidence-based pruning strategy, DeepConf [1]. There are two main differences between DeepConf and our intervention tool, EGART.

  • DeepConf is designed for sampling-based reasoning paradigms, such as CoT-SC, while our intervention tool, EGART, is designed for search-based reasoning paradigms;
  • From the perspective of thought evaluation, DeepConf evaluates thoughts via a confidence derived from log-probabilities, while EGART adopts the subjective evaluations of LLMs.
  • From the perspective of selection strategy, DeepConf preserves the branches by ranking all intermediate thoughts with confidence and taking the top 10%. However, EGART selects intermediate thoughts based on the entropy of evaluation results. In such a case, the preservation budget is manually set in DeepConf, while adaptively determined by evaluation entropy.

To address your concern, we apply the pruning strategy in DeepConf in both ToT and EGART. In this way, on the one hand, we can compare the two self-evaluation strategies; on the other hand, we can further validate the effectiveness of entropy-guided adaptive selection strategy.

Specifically, we first have LLMs generate intermediate thoughts and calculate the confidence with bottom 10% Group Confidence and top-k=20. Then, we preserve branches for ToT with fixed width and for EGART with adaptive width. Results are reported in the following tables (with τ=1.0\tau=1.0). According to the table,

  • [Quantitative Results.] In the Qwen2.5-3B case, DeepConf achieves inferior performance to ToT and EGART on simple tasks (e.g., GSM8K and MATH), while comparable or even better performance on hard tasks (e.g., AIME tasks). In the Qwen2.5-7B case, although DeepConf achieves better performance on GSM8K, it underperforms ToT and EGART on other tasks.
  • [Resource Consumption.] Meanwhile, since DeepConf merely relies on log-probabilities of LLMs for pruning and only preserves top 10% branches, fewer tokens are consumed.

Qwen2.5-3B

MethodsGSM8K Acc. (%) ↑GSM8K kTok ↓MATH Acc. (%) ↑MATH kTok ↓AIME 24 Acc. (%) ↑AIME 24 kTok ↓AIME 25 Acc. (%) ↑AIME 25 kTok ↓
ToT Vote TopK88.7±0.164.2660.1±1.5101.158.4±2.3144.241.7±2.3151.10
ToT Score TopK89.1±0.2129.6659.5±0.1185.228.4±2.3249.775.0±2.4259.77
EGART Vote TopK88.1±0.164.3658.2±1.1106.368.4±2.3152.781.7±2.3158.64
EGART Score TopK87.2±0.2117.2358.0±0.7158.376.7±0.0197.241.7±2.3208.59
DeepConf-ToT78.0±1.618.7851.1±1.128.315.0±2.440.013.3±0.040.67
DeepConf EGART78.4±0.425.6652.8±0.438.5110.0±4.754.485.0±2.455.25

Qwen2.5-7B

MethodsGSM8K Acc. (%) ↑GSM8K kTok ↓MATH Acc. (%) ↑MATH kTok ↓AIME 24 Acc. (%) ↑AIME 24 kTok ↓AIME 25 Acc. (%) ↑AIME 25 kTok ↓
ToT Vote TopK82.9±0.146.0558.8±0.666.5211.7±2.398.963.4±4.7102.15
ToT Score TopK80.5±0.398.9660.0±0.1125.0816.7±4.7170.178.4±2.3183.05
EGART Vote TopK80.9±1.738.9358.2±0.556.5511.7±2.388.446.7±0.089.99
EGART Score TopK81.2±0.296.2758.0±0.1118.0313.3±0.0138.935.0±2.4156.97
DeepConf-ToT81.9±0.015.2854.9±0.620.966.7±4.730.433.3±0.033.63
DeepConf EGART83.0±0.520.1057.5±0.127.868.3±2.441.196.7±0.044.18

[1] Fu et al., Deep Think with Confidence. ICLR 2026

AuthorsOpenReview ↗

Response to Weakness 3 & Question 1. Experiments with Matched Computational Budget

Response: According your comments in Question 1, you suggested us to perform tasks with the similar token budget. We agree that token consumption is an important consideration in LLM reasoning. We first explain why more tokens are consumed in ToT.

  • Compared to sampling-based paradigms, which directly performs generation, search-based paradigms involves both generation and explicit evaluation phases. Thus, the more tokens are mainly consumed on explicit evaluations.
  • Different strategies also consumes different amounts of tokens. For example, in voting, we directly feed all candidates into LLMs for evaluation, while, in scoring, we have to feed candidates one by one for evaluations. The repeated instruction prompts and answer outputs further result in more tokens.

To compare different strategies under the same token budget, we should carefully control the budget of generation, in case that more tokens are consumed in evaluation. Thus, for simplicity, we propose to conduct experiments to compare DeepConf-ToT and DeepConf-EGART. In such a case, since intermediate thoughts are evaluated and selected via confidence-based scores, the consumption of evaluations is zero. We can then evaluate the capabilities of the two selection strategies.

According to the results in the table below, we can observe that DeepConf-ToT fails to outperform DeepConf-EGART in almost all cases, even with the comparable computational budgets. These results demonstrate the advantage of EGART to those simple strategies.

Hope these results can help address your concerns.

Qwen2.5-3B

MethodsGSM8K Acc. (%) ↑GSM8K kTok ↓MATH Acc. (%) ↑MATH kTok ↓AIME 24 Acc. (%) ↑AIME 24 kTok ↓AIME 25 Acc. (%) ↑AIME 25 kTok ↓
DeepConf-ToT78.0±1.618.7851.1±1.128.315.0±2.440.013.3±0.040.67
DeepConf-ToT (Budget-Matched)78.5±1.124.9251.2±0.938.218.3±2.453.766.7±0.054.88
DeepConf EGART78.4±0.425.6652.8±0.438.5110.0±4.754.485.0±2.455.25

Qwen2.5-7B

MethodsGSM8K Acc. (%) ↑GSM8K kTok ↓MATH Acc. (%) ↑MATH kTok ↓AIME 24 Acc. (%) ↑AIME 24 kTok ↓AIME 25 Acc. (%) ↑AIME 25 kTok ↓
DeepConf-ToT81.9±0.015.2854.9±0.620.966.7±4.730.433.3±0.033.63
DeepConf-ToT (Budget-Matched)83.0±1.119.9156.9±1.527.826.7±0.041.421.7±2.444.10
DeepConf EGART83.0±0.520.1057.5±0.127.868.3±2.441.196.7±0.044.18
Reviewer_e5viOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates why Tree-of-Thoughts (ToT), despite employing explicit step-wise self-evaluation, can underperform Self-Consistency Chain-of-Thought (SC-CoT) on mathematical reasoning tasks. The authors develop theoretical analyses of both paradigms, characterizing SC-CoT through the probability distribution over final answers and ToT through the evaluation margin between correct and incorrect intermediate thoughts. Their analysis highlights the importance of candidate coverage and the preservation budget in determining whether promising reasoning branches survive the search process.

Building on these analyses, the paper empirically examines alternative evaluation and selection strategies, including voting, independent scoring, sampling with and without replacement, and top-k selection. The authors further introduce Entropy-Guided Adaptive Reasoning Tree (EGART), which uses evaluation entropy to adapt the number of intermediate thoughts preserved during search. Experiments on four mathematical reasoning benchmarks with two Qwen2.5-Instruct models investigate the relationships among evaluation behavior, candidate coverage, accuracy, and reasoning-path budgets.

Critical strengths
  1. The paper examines an important question about the role of self-evaluation in search-based reasoning, moving beyond the assumption that stronger evaluators necessarily lead to better reasoning. The distinction between evaluation reliability and intermediate-thought coverage provides a useful perspective on why explicit search may not consistently outperform sampling-based reasoning. This framing is relevant to understanding the limitations of LLM-based evaluation when the correctness of intermediate reasoning steps is difficult to assess.

  2. The theoretical analysis provides a useful framework for interpreting the two reasoning paradigms. The analysis of SC-CoT relates majority-voting success to the probability margin between the correct answer and the most competitive incorrect answer, whereas the single-step ToT analysis relates candidate preservation to the evaluation margin and evaluation budget. These formulations make the role of candidate availability explicit, providing a conceptual basis for the subsequent empirical investigation.

  3. The empirical study examines multiple dimensions of ToT behavior rather than evaluating only the proposed intervention. The comparisons among voting, scoring, sampling, and top-k selection provide informative evidence about how different evaluation and selection strategies influence reasoning performance. The introduction of EGART further connects these observations to adaptive test-time computation, while reporting both accuracy and the average number of reasoning paths offers a useful perspective on the accuracy-efficiency trade-off.

Critical weaknesses
  1. The theoretical characterization of ToT relies on assumptions that may limit its explanatory scope for practical multi-step reasoning. In particular, Theorem 2.4 analyzes the probability of retaining at least one correct intermediate thought under independently conducted evaluations and a margin defined through expected evaluation scores. The relationship between this single-step guarantee and end-to-end reasoning accuracy remains insufficiently established, since retaining a correct intermediate thought does not necessarily ensure a correct continuation or final answer.

  2. The empirical evidence does not fully isolate the claim that coverage, rather than evaluation, is the primary determinant of ToT performance. The comparisons between sampling with and without replacement, as well as between voting and independent scoring, change candidate selection behavior and the distribution of retained thoughts simultaneously. Although the results in Figure 3 are consistent with a coverage effect, they do not independently establish the relative causal importance of coverage and evaluation quality.

  3. The evaluation-entropy analysis raises questions about how uncertainty relates to candidate preservation across evaluation strategies and task difficulties. The reported associations differ between voting and scoring, and the relationships become less stable on challenging mathematical problems. The results therefore do not establish evaluation entropy as a consistently interpretable indicator of reasoning quality across settings, which limits the generality of the uncertainty-based interpretation underlying EGART.

  4. The empirical scope is relatively narrow for conclusions concerning self-evaluation in LLM reasoning more broadly. The experiments use two model sizes from the same Qwen2.5 family, while the AIME benchmarks contain only 30 problems each. The generalizability of the observed patterns across model families, stronger reasoning models, and non-mathematical search tasks remains uncertain, and small accuracy differences on the AIME benchmarks may be difficult to interpret robustly.

  5. The efficiency evaluation does not fully characterize the relationship between reasoning-path reduction and total inference cost. The reported average number of preserved reasoning paths is a useful budget-related metric, but it does not directly measure all generation and evaluation costs incurred during search. Furthermore, some EGART configurations preserve more paths than their corresponding ToT baselines, while their accuracy effects vary across configurations. The efficiency benefits therefore appear dependent on the evaluation strategy, preservation budget, and task setting, rather than being uniform across the reported experiments.

Paper Formatting and Presentation Concerns

The presentation of the main experimental comparison could be refined to improve readability. In particular, Figure 1 contains multiple small panels and dense annotations, which may make performance differences difficult to assess at normal reading scale. The visual presentation of the main comparisons could therefore be clearer for readers who are not already familiar with the experimental setup.

The manuscript uses evaluation entropy, evaluation confidence, ranking, and coverage as closely related but distinct concepts. Their interpretation is central to the proposed analysis, particularly because entropy behaves differently under voting and scoring. The distinction between these concepts is not always immediately apparent from the presentation, which may increase the effort required to follow the argument from the empirical observations to the design of EGART.

Questions for authors

See Weaknesses Above.

9 comments
AuthorsOpenReview ↗

Response to Weakness 1. Limitation of Assumptions Adopted in Theorem 2.4.

Thanks for your time and efforts in reviewing our work. We here provide responses towards your concerns as follows.

Response: According to your comments, we agree that the correctness of a single step does not guarantee the success of the entire reasoning. However, empirically, a successful reasoning is based on a series of correct reasoning parts. For exmaple, given a sequence of completely correct reasoning intermediate thoughts, the final answer will probably be correct. Motivated by such an intuition, for simplicity, we focus on the success of a single reasoning and evaluation step in ToT.

To further address your concern here, we conduct GAME24 experiments with GPT-5.4 and GPT-5.4-mini, respectively. Compared to conventional mathematical tasks (e.g., GSM8K and AIME), the intermediate thoughts of GAME24 are more clear and verifiable. We here examine the relationship between the loss of viable branches and the success of reasoning. Specifically, we examine the first three rounds of each case to check whether viable branches are lost and whether the reasoning is successfully performed.

According to the results in the table, we can observe that

  • the reasoning fails when all viable branches are lost in the previous rounds;
  • the reasoning may also fail even if viable branches are preserved;
  • in most cases, if viable branches exist, the reasoning will probably be successful.
ModelViable paths lost within the first three roundsFinal failures after early lossNo early loss detectedFinal failures without detected early loss
GPT-5.43433/34 (97.06%)661/66 (1.52%)
GPT-5.4-mini3030/30 (100.00%)705/70 (7.14%)

These empirical observations fit our intuition above. We hope these results can help address your concern.

AuthorsOpenReview ↗

Response to Weakness 2. Concern about Empirical Study of Coverage in ToT.

Response: Response: We appreciate your positive attitudes that our empirical results in Fig. 3 support our claim regarding coverage. To address your concern, we first provide detailed explanations about the unreliability of self-evaluation, and then empirically demonstrate the efficacy of coverage in branch preservation.

Limitations of Self-Evaluation

Case 1: Invalid Evaluation Signals. We here investigate self-evaluation via the validity of evaluation signals. Specifically, we count the fraction of the cases where at least one invalid evaluation result is included. The results are based on seeds 41 & 42. According to the table below, for Qwen2.5-3B, 13.7%, 14.25%, 33.33%, and 35.83% of evaluation decisions on GSM8K, MATH, AIME 24 & 25, respectively, contain at least one invalid evaluation result that does not map to a valid candidate.

ModelsDatasetsAt least one Invalid
3BGSM8K13.70%
3BMATH14.25%
3BAIME2433.33%
3BAIME2535.83%
7BGSM8K1.63%
7BMATH3.38%
7BAIME246.67%
7BAIME2514.17%

Case 2. Voting Exists Position Bias. In the paper, as we have demonstrated in Figs. 14-21, there exist position bias when perform self-evaluations with voting. Specifically, when adopting voting for evaluation, the first multiple options are preferred by LLMs. This indicates that the order, instead of the correctness, matters in such a case. Thus, the high-scored branches may benefit from the order.

An Intuition of Coverage

We here leverage the log file of GAME24 and replay the selection process on the same pool. Specifically, we first calculate the average width of EGART, and then perform selection, respectively, with the fixed-width strategy and entropy-based strategy with the given width. The fixed-width strategy preserves a roughly same number (with a deviation of 1) of branches for each node, while entropy-based strategy determines the budget adaptively for each node.

According to the table below. Entropy-based strategy achieves higher preservation rate (e.g., $\sim$2%) than the fix-width strategy, since the correct intermediate thoughts may be out of the coverage budget due to the reliable evaluations. This is why sampling w/o replacement and TopK performs better than sampling with replacement. The former two strategies enforce larger coverage budget than the later which tends to sample duplicated branches due to the probability, thereby limiting the coverage.

ModelCandidate poolsMean preserved widthEntropy-rule preservation rateBudget-matched fixed-width preservation rateDifference
GPT-5.42564.33687.89%85.86%+2.03%
GPT-5.4-mini2464.11096.34%94.94%+1.40%

We hope these explanations can help address your concern.

AuthorsOpenReview ↗

Response to Weakness 3. Inconsistent Interpretations of Evaluation Entropy under Different Evaluation Strategies.

Response: Thanks for your question. According to your comments, your concerns mainly refer to

  1. how uncertainty measure (i.e., entropy) associates with branch preservation when evaluation strategies differ;
  2. how uncertainty measure associates with branch preservation when tasks are challenging.

How uncertainty relates to preservation across different evaluation strategies

We agree that the evaluation entropy measures different aspects when evaluation strategies are different.

  • Voting is performed on multiple options. Thus, the entropy of voting measure the consistency of multi-round self-evaluation, indicating the confidence of LLMs in a specific option.
  • Scoring is performed independently on each candidate. Thus, the entropy of scoring measures the distribution of the quality among a set of candidates.

Thus, intuitively, when the evaluation entropy is high, a inconsistent voting or similar scores on the set of candidates are obtained, which indicate a disagreement in branch selection. In such a case, we tend to preserve more branches in case of missing the promising candidates.

In contrast, when the evaluation entropy is low, a concensus is achieved in voting or a few extremely good/bad options exist in the candidate set, which provide a definite decision. Thus, we tend to preserve fewer branches with a risk of missing potentially promising options.

Thus, although the meaning of the evaluation entropy differs across strategies, the decision directions are consistent.


How uncertainty relates to preservation in difficult tasks

We agree that empirical results in Figs. 4 & 8 fail to establish a stable relation between evaluation entropy and performance. But it does not mean that the association between branch preservation and uncertainty is unstable. Empirically, in Table 1 & 2, we can observe that comparable or even better performance is achieved in most cases that fewer branches are preserved. This phenomenon demonstrates that entropy signals remains effective even in hard tasks. Moreover, our investigation of GAME24 also shows that viable candidates can be preserved via our entropy-based strategy.

Reviewer_VdW8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates why Tree-of-Thoughts (ToT) can underperform Self-Consistency Chain-of-Thought (SC-CoT) on mathematical reasoning tasks. It provides probability bounds for SC-CoT answer selection and single-step ToT candidate preservation, followed by experiments varying evaluation and selection strategies. The authors further introduce Entropy-Guided Adaptive Reasoning Tree (EGART) as an intervention that adjusts the number of retained branches using self-evaluation entropy. The research question is important, and the distinction between evaluation magnitudes, rankings, and candidate coverage is potentially useful. However, the current evidence does not sufficiently isolate the proposed explanations, and several interpretations of answer diversity and evaluation entropy are stronger than the results support.

Critical strengths
  1. Understanding when self-evaluation helps search is an important question. The paper appropriately examines evaluation and selection separately rather than treating ToT as a single indivisible procedure.
  2. Comparing voting against independent scoring, and sampling against deterministic selection, provides informative observations about how search behavior changes.
  3. The analysis highlights the importance of the correct answer’s probability advantage in SC-CoT and the evaluator’s ability to preserve a correct candidate in ToT.
Critical weaknesses
  1. SC-CoT aggregates terminal answers through plurality voting, whereas the formulated ToT selects the terminal state with the highest evaluator score. Thus, their accuracy difference may arise from final-answer aggregation rather than intermediate pruning or branch coverage. The paper should compare ToT with terminal-answer voting and independently sampled CoT trajectories with evaluator-based final selection. These controls would help distinguish candidate-generation quality, intermediate pruning failures, and terminal selection errors. Without them, the observed performance gap does not uniquely support the proposed explanation of ToT’s limitations.
  2. Sampling without replacement prevents duplicate selections, whereas sampling with replacement can spend multiple retention slots on the same state. Better performance without replacement therefore demonstrates the usefulness of distinct retained candidates, but does not by itself demonstrate unreliable evaluation. Even a reliable evaluator may benefit from preserving several different promising states rather than repeatedly selecting one. Conversely, stochastic selection can overlook a highly ranked correct state without implying that the ranking is wrong.
  3. The SC-CoT theorem concerns the probability gap between the correct answer and the strongest incorrect answer. It does not imply that lower overall answer diversity is generally beneficial.

Indeed, the paper itself notes that dispersing a fixed amount of incorrect-answer probability across more alternatives can help plurality voting. Conversely, a concentrated distribution can be confidently wrong. The number of unique sampled answers therefore cannot establish concentration around the correct answer without additional information.

Questions for authors

See Weaknesses.

6 comments
AuthorsOpenReview ↗

Response to Weakness 1. Performance gap derived from different answer extraction.

Thanks for your time and efforts in reviewing our work. We here provide responses towards your concerns as follows.

Response to Weakness 1: We think there is a misunderstanding here. For your concern about answer aggregation/selection, we aggregate the answer of ToT with plurality voting as well in implementation. Thus, evaluators will only affect the intermediate thoughts. We have updated this implementation detail in Appendix C 1.3 of our revised version.

AuthorsOpenReview ↗

Response to Weaknee 2. Sampling w./wo. replacement cannot demonstrate the unreliable self-evaluation.

Response: To address your concern about reliability of self-evalution, we provide explanations from a preliminary to comprehensive empirical results.

Preliminary for Our Empirical Study on Self-Evaluation

  • [A simple intuition.] The empirical study on self-evaluation is built on a simple intuition that sampling with replacement can achieve better performance by repeatedly sampling promising branches with high scores, if self-evaluation is reliable. This intuition is well supported by previous work. For example, Sun et al. [1] demonstrates that correct prefix thoughts facilitates better reasoning performance, while erroneous prefix thoughts degrades the performance.
  • [Main Challenge.] The main challenge of the study is the lack of ground truth for intermediate thoughts. As a surrogate, we explore self-evaluation by measuring the performance of different selection strategies on the given evaluations.

Detailed Explanations about Reliability of Self-Evalaution

We first assume that self-evaluation is reliable. Correspondingly, intermediate thoughts with higher scores are more promising to achieve good performance.

  • Case 1. High-score candidates fail to achieve better performance. With the intuition, if self-evaluation is reliable, candidates with high score will correspond to the most promising ones. Thus, repeatedly sampling of these branches facilitates better performance. As shown in the below tables, sampling with repl. tends to include more candidates with high scores. However, inferior performance is achieved.

Sampling with Replacement

ModelsDatasetsUnselected High / Total High
Qwen2.5-3BAIME2415/94(15.96%)
Qwen2.5-3BAIME2516/95(16.84%)
Qwen2.5-3BGSM8K454/3,637(12.48%)
Qwen2.5-3BMATH778/5,671(13.72%)
Qwen2.5-7BAIME243/71(4.23%)
Qwen2.5-7BAIME255/72(6.94%)
Qwen2.5-7BGSM8K280/3,339(8.39%)
Qwen2.5-7BMATH479/5,136(9.33%)

Sampling without Replacement

ModelsDatasetsUnselected High / Total High
Qwen2.5-3BAIME2423/76(30.26%)
Qwen2.5-3BAIME2517/79(21.52%)
Qwen2.5-3BGSM8K2,740/7,792(35.16%)
Qwen2.5-3BMATH3,069/9,297(33.01%)
Qwen2.5-7BAIME2437/121(30.58%)
Qwen2.5-7BAIME2526/86(30.23%)
Qwen2.5-7BGSM8K4,274/11,668(36.63%)
Qwen2.5-7BMATH6,701/18,243(36.73%)
  • Case 2. Sampling w/o replacement outperforms TopK in the Qwen2.5-3B case. Both sampling w/o replacement and TopK include a same number of distinct branches. The main difference is that the selection of TopK is determinant while that of sampling exists stochasticity. If self-evalution is reliable, TopK should consistently outperform sampling w/o replacement. However, in Fig. 3, we can observe that sampling outperforms TopK in the voting case of Qwen2.5-3B.

  • Case 3. Position bias in voting. In Appendix C.2 of our paper, we demonstrated that there exist position bias when adopting voting as the evaluation strategy. Specifcially, LLMs prefer the first-NN options in a sequence of candidates. Thus, in such a case, the order, instead of the correctness, matters for evaluation.

  • Case 4. Invalid signals are included in self-evaluation. Moreover, we also investigate the validity of self-evaluation. Specifically, we count the fraction of invalid self-evaluation results in log files of seed 41 & 42. According to the table below, for Qwen2.5-3B, 13.7%, 14.25%, 33.33%, and 35.83% of evaluation decisions on GSM8K, MATH, AIME 24 & 25, respectively, contain at least one invalid evaluation result that does not map to a valid candidate.

ModelsDatasetsAt least one Invalid
3BGSM8K13.70%
3BMATH14.25%
3BAIME2433.33%
3BAIME2535.83%
7BGSM8K1.63%
7BMATH3.38%
7BAIME246.67%
7BAIME2514.17%

[Conclusion.] With all these aspects taken into consideration, we summarize that self-evaluation is not sufficiently reliable in our paper.

Essence of Distinct Branches

According to your comments, it is distinct branches that facilitate better performance. Here, we would like to highlight two observations:

  • More Branches. As you mentioned, sampling w/o replacement and TopK, which contain more distinct branches, achieve better performance than sampling with replacement.
  • Stochaticity. In some cases, such as the voting case with Qwen2.5-3B and the cases regarding hard tasks, sampling w/o replacement performs better than TopK.

Thus, the essence of distinct branches is more budget for including correct candidates, and stochasticity, instead of order, can help when evaluations are unreliable.

We hope these discussions can help solve your concern.

[1] Sun et al. Well Begun, Half Done: Reinforcement Learning with Prefix Optimization for LLM Reasoning, AAAI 2026.

AuthorsOpenReview ↗

Response to Weakness 3. Mismatch of Empirical Results (Fig. 2b) and Theoretical Results

Response: Thanks for your insightful and constructive comments. We agree that the empirical results in Fig. 2b do not match the theoretical results. To address your concern, we will update Fig. 2b.

Specifically, in the updated figure, the x-axis will be the gap py−qmaxp_y-q_{\rm max} and the y-axis will be the fraction of cases with such a gap. We use circles with different sizes to describe the accuracy. For your convenience, we also report results in the following tables.

From both the figure and the results in the table, for the simple task (e.g., GSM8K), most cases have the positive gap (i.e., py>qmaxp_y>q_{\rm max}) and the performance with larger gaps is much better. For the hard task (e.g., AIME24), the cases concentrated around a negative gap that is slightly smaller than zero (i.e., py−qmax<0p_y-q_{\rm max}<0). Accordingly, the accuracies are much worse. These phenomena are consistent with our theoretical results.

We hope the results help address your concern.

GSM8K

Series-1.0-0.9-0.8-0.7-0.6-0.5-0.4-0.3-0.2-0.1-0.00.10.20.30.40.50.60.70.80.91.0
Qwen2.5-3b (40)0.10.10.40.20.60.60.81.11.43.02.13.83.18.15.811.913.814.214.114.10.8
accuracy0.00.00.00.00.00.00.00.00.00.028.698.0100.0100.0100.0100.0100.0100.0100.0100.0100.0
Qwen2.5-3b (80)0.00.20.20.30.50.40.91.11.62.42.83.13.26.87.710.812.716.715.511.61.5
accuracy—0.00.00.00.00.00.00.00.00.040.5100.0100.0100.0100.0100.0100.0100.0100.0100.0100.0
Qwen2.5-7b (40)0.00.30.20.20.20.71.01.82.02.23.35.25.28.64.89.110.611.811.115.16.7
accuracy—0.00.00.00.00.00.00.00.00.034.1100.0100.0100.0100.0100.0100.0100.0100.0100.0100.0
Qwen2.5-7b (80)0.10.20.20.20.30.70.62.11.72.53.05.25.28.15.89.211.011.812.112.18.0
accuracy0.00.00.00.00.00.00.00.00.00.057.5100.0100.0100.0100.0100.0100.0100.0100.0100.0100.0

AIME24

Series-1.0-0.9-0.8-0.7-0.6-0.5-0.4-0.3-0.2-0.1-0.00.10.20.30.40.50.60.70.80.91.0
Qwen2.5-3b (40)0.00.00.00.00.06.70.06.723.340.020.00.00.03.30.00.00.00.00.00.00.0
accuracy—————0.0—0.00.00.00.0——100.0———————
Qwen2.5-3b (80)0.00.00.00.00.03.33.310.010.053.316.70.00.03.30.00.00.00.00.00.00.0
accuracy—————0.00.00.00.00.040.0——100.0———————
Qwen2.5-7b (40)0.00.00.00.00.03.33.320.023.326.710.00.03.33.30.03.30.03.30.00.00.0
accuracy—————0.00.00.00.00.00.0—100.0100.0—100.0—100.0———
Qwen2.5-7b (80)0.00.00.00.00.03.30.026.716.730.010.00.03.33.30.03.30.00.03.30.00.0
accuracy—————0.0—0.00.00.00.0—100.0100.0—100.0——100.0——

Comments on the paper

8 comments
AuthorsOpenReview ↗

Please Check Our Revised Manuscript

Dear Reviewers,

Thanks for your time and efforts in reviewing our work. We appreciate your constructive comments and suggestions for our paper. Now, we have posted our responses to your comments and revised our manuscripts based on the comments.

Here is an overview of our revised manuscript:

  1. [Update Fig. 2b and Its Discussion] According to Reviewer VdW8 and PfcF, Fig. 2b failed to reflect our theoretical results. Thus, we have updated it and its discussion in Section 3.1 based on our content in rebuttal.

  2. [Clarification of Empirical Results regarding ToT] According to the feedback of all reviewers, the empirical results should be further polished. Thus, we rewrite Section 3.2 to make it more clear.

  3. [More Experiments]

    • According to the feedback of Reviewer e5vi and PfcF, more powerful models and benchmarks are required. Thus, we perform GAME24 on GPT-5.4 and GPT-5.4-mini in Section 4.3.
    • According to the feedback of Reviewer PfcF, comparison with other methods is required. Thus, we add experiments regarding DeepConf in Section 4.4.

We are open to any further questions. We hope we can address your concerns during the rebuttal phase.

Best regards,

Authors of Submission 21574

AuthorsOpenReview ↗

A Summary of Reviewers' Comments and Our Responses (2/2)

About Motivation Studies on Self-Evalution of ToT (Reviewer VdW8 & e5vi)

Unintuitive empirical analyses about ToT (Reviewer VdW8 & e5vi).

Both Reviewer VdW8 & e5vi mentioned that the empirical results in Section 3.2 are not intuitive enough to obtain the insights regarding self-evaluation and coverage of intermediate thoughts.

The main reason here is that there does not exist ground truth for intermediate thoughts so that we can use these alignable signals to validate our theoretical results by measuring the correctness of intermediate thoughts and the selections.

As a surrogate, we use different selection strategies to evaluate the reliability of self-evaluation based on the evaluation results.

To address the concern of reviewers, we have provide a series of empirical evidence to demonstrate the unreliability of self-evaluation and discuss the efficacy of coverage. Please refer to Weakness 2 of Reviewer VdW8 and Weakness 2 of Reviewer e5vi.


About Generalizability & Robustness (Reviewer e5vi & PfcF)

Limited models and tasks. The experinments are conducted on limited models and tasks; (2) robustness of results and limited improvements on AIME benchmarks.

To address this concern, we further conducted experiments on GAME 24, a combinatorial tasks adopted for search-based tasks, with GPT-5.4-mini and GPT-5.4, respectively. The results show that EGART can achieve much better average accuracy with a comparable any-solution performance to ToT.

Statistically stability on AIME benchmarks.

For the concern regarding the results of AIME benchmarks, we agree that AIME contains limited problems. Thus, in this paper, we ran AIME with two different seeds and report accuracy with standard deviation. The deviation matches the level of CoT-SC and ToT. Meanwhile, we also demonstrate that EGART can achieve comparable performance with fewer reasoning paths by taking advantage of the two signals on AIME benchmarks.

To further address this concern, we also perform AIME tasks with GPT-5.4-mini on three different random seeds (cf. the results above and in Section4.2 in the manuscript). The results show that EGART can steadily outperform ToT with fewer token consumptions.


About Efficiency (Reviewer e5vi & PfcF)

The average number of preserved intermediate thoughts fails to indicate efficiency.

The main goal of the average number of preserved reasoning paths is to demonstrate whether evaluation entropy helps preserve viable intermediate thoughts for a comparable or better performance, instead of efficiency.

Lack computation-budget comparison. (Reviewer PfcF) Our paper did not perform computation-matched study to compared the consumption of tokens.

According to the comments of Reviewer PfcF, we first compare the strategy in EGART with the confidence measure proposed in DeepConf [1]. The results show that confidence-based self-evaluation measure is also unreliable and achieves inferior performance than EGART.

Moreover, since DeepConf does not rely on generation-based self-evaluation, we can avoid the budget balance issue derived from both generation and evaluation. Thus, we perform computation-matched study regarding ToT and EGART. According to the empirical results, EGART achieves better performance in most cases than ToT with a relatively similar token budget.

[1] Fu et al., Deep Think with Confidence. ICLR 2026

AuthorsOpenReview ↗

A Summary of Reviewers' Comments and Our Responses (1/2)

Clarification of the Theme and Scope of the Paper (Reviewer PfcF)

As a response to the novelty concern of Reviewer PfcF, we here make clarifications as follows:

  • The main theme of this paper is to understand the reliability of self-evaluation and discuss potential signals that are meaningful for search-based reasoning paradigms.
  • EGART is a probing tool, instead of a proposed method for SoTA performance, to validate the efficacy of ranking information and evaluation signals.
  • The main contributions of this paper are (1) comprehensive and detailed analyses on different paradigms, and (2) exploration and validation of ranking and evaluation entropy.

Both Reviewer VdW8 and e5vi appreciate the problem (i.e., self-evaluation) studied in our work, and all reviewers recognize our contribution in theoretical analyses on the two reasoning paradigms.


About Theoretical Results (all reviewers)

Preliminary: Due to the lack of ground truth for intermediate thoughts on mathematical tasks (e.g., GSM8K), we perform empirical validations for theoretical results with GAME24 tasks since the intermediate thoughts are verifiable.

A shared concern among all reviewers is the theoretical results. Generally, the concerns mainly involve the following aspects:

Mismatch of theoretical and empirical results about SC-CoT (Reviewer VdW8 & PfcF). Theoretical results discuss the probability gap, while empirical results discuss the uniqueness of answers.

We have updated the Fig. 2(b) and the corresponding discussion in Section 3.1. Now, the discussions are consistent.

Theorem 2.4 only discuss a single evaluation step. Thus, (1) the success of single step does not mean successful reasoning (Reviewer e5vi & PfcF); (2) fail to consider generation-stage failure (Reviewer PfcF).

For concern (1), we agree with the statement. Then, we conduct experiments on GAME24 to demonstrate the reasoning fails when all viable branches are lost in previous rounds, while the reasoning will probably be successful when viable branches are preserved.

For concern (2), on the one hand, if there is not any correct intermediate thought in candidates, the probability of self-evaluation selecting a correct branch is necessarily zero; on the other hand, the empirical results also show that the reasoning fails when all viable branches are lost in previous rounds.

Fail to demonstrate evaluation entropy is an optimal estimator to determine preservation (Reviewer PfcF).

From the statistical perspective, evaluation entropy provides a standard effective number of options under a given probability distribution. Incorrect evaluation of intermediate processes is an important yet unsolved problem in stepwise reasoning. From the theoretical perspective, it is intractable to determine the optimal budget for preservation.

To partially address this concern, we perform fixed-width, relative-threshold (a beam search strategy), and entropy-based strategies with a same preservation budget on a shared pool of GAME24, with GPT-5.4 and GPT-5.4-mini. The empirical results demonstrate that the entropy-based strategy can preserve correct intermediate thoughts in more cases.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.