acceptodds
Under review as a conference paper at ICLR 2027

TsuGO: Probing Search Efficiency in LLM Reasoning via Go Life-and-Death Problems

Abstract

The evaluation of LLM reasoning is moving from final-answer accuracy to process-level assessment, yet existing methods still fail to capture how models plan reasoning paths and allocate reasoning resources—that is, how they organize search. Prior process-level methods focus on the coherence and redundancy of chain-of-thought (CoT), and most benchmark tasks have a single objective solvable by static capabilities such as derivation and tool use, leaving search organization unmeasured. We introduce TsuGO , a process-level reasoning benchmark for evaluating Search Efficiency in LLM reasoning through Go life-and-death problems.These problems provide closed and verifiable solution spaces with an inherent adversarial structure, making candidate generation, response checking, branch comparison, and backtracking necessary parts of reasoning rather than incidental trace patterns. By constraining the solution space, TsuGO disentangles domain knowledge from search organization, parses CoT into a structured search tree, and reports Search Efficiency together with Token Efficiency and other diagnostic metrics and visualizations. Experiments show that current LLMs remain far from stable tsumego solving: stronger models succeed by finding the correct candidate earlier and sustaining effort on productive branches, but most models still behave much closer to unguided search algorithms than to neural-guided KataGo. Longer CoT or higher Token Efficiency does not necessarily imply better search. Our results identify search organization and reasoning-resource allocation as missing dimensions in LLM reasoning evaluation.

Primary area:
datasets and benchmarks
Keywords:
MLLM, Reasoning, LLM, EfficientAI, GO

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_tAspOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TsuGO, which evaluates "search organization ability" of LLMs via Tsumego problems in GO. This benchmark contains 1500 problems and can serves as a process-level judge for LLMs' search ability. The authors use Gemini-2.5-Pro to parse the CoTs into process search trees, and define the evaluation matric SearchE as a proxy of search efficiency metric. Authors claim that: current LLMs are far from solving this tsumego benchmark; stronger LLMs are earlier to reach the correct solution and invest wisely in the correct reasoning branch; and SearchE is a better evaluation signal compared with TokenE, which fits the accuracy more closely.

Critical strengths
  1. Using tsumego as a testbed is novel and interesting. It makes since the solution space is bounded and verifiable. It has a good property of adversarial structure - a move should be able to cope with the best response from the adversary. This makes candidate generation, verification and backtracking necessary and load-bearing in the reasoning traces.

  2. The benchmark is well-formatted: it is transformed into SGF/JSON format, and multiple modalities share the same standard states. The label quality is paid enough attention to. It is also of good transparency: it introduces MCTS/KataGO as non-LLM reference. All the prompts are also released in the paper.

  3. The benchmark result is far from saturation, which leaves room for improvement in future works.

Critical weaknesses
  1. It is unclear whether tsumego is a good proxy for general LLM search ability. There could be alternative explanations for poor search behavior:
  • Weak domain priors. Go life-and-death is sparsely presented in pretraining data, so LLMs have poor generative ability in this domain; LLMs may fail to propose reasonable candidates before search organization comes into play.
  • Generation budget. Kimi-K2.5's mean ITT under K=None is 28.7k–31.8k tokens, against a 32k maximum output length (Table 16). This implies that most of its open-setting responses may be truncated, so its collapse to 4.4% on K=None Hard may reflect the budget rather than search organization. Please report truncation and no-answer rates per model and setting, and re-evaluate with a larger budget.
  • Perception and state-tracking failures. The authors' own failure analysis (Fig. 12) attributes 24% of errors to coordinate/state drift (18%) and visual grounding (6%). More fundamentally, the paper does not test whether search ability measured on TsuGO transfers to other search-heavy tasks. Prior work has characterized related failures in other settings: frequent switching between reasoning paths on challenging math problems (Wang et al., 2025), wandering behaviors in search-heavy computation tasks (Lu et al., 2025) and performance collapse as problem size grows (Shojaee et al., 2025). If model rankings on TsuGO disagree with rankings on other adversarial games (e.g., chess endgames, Othello) or on such computational tasks, TsuGO is more likely measuring Go-specific competence than general search ability. Reporting cross-task rank correlations would directly address this concern.
  1. The contribution is limited to a benchmark plus evaluation, and its claimed utility is untested. The conclusion positions TsuGO as a basis for Search-Efficiency-sensitive training, yet the paper contains no training experiments. Several natural and feasible questions remain unanswered.
  • If TsuGO is used as an RL environment with outcome reward only (first-move correctness), does SearchE improve?
  • Can SearchE serve as a process reward? Will it introduce reward hacking?

I feel that the contribution would be not so narrow if either (i) authors provide a suite of adversarial structure games, e.g. including other games or computation tasks (ii) authors provide sufficient experiment, results and insights on treating this proposed benchmark as a trainable environment and show how training on this dataset / environment alone can or cannot improve the general search capability of LLMs.

  1. The process-tree parsing and extraction relies on a single LLM parser (Gemini-2.5-Pro) which may not be faithful. Readers are not convinced whether such parsing introduces hallucination; or omissions of critical solution steps; or the parsing errors. Human examination cannot be scaled up. The paper should at least cross-validate with multiple parsers or CoT monitors from different model families, and report the agreement or stability of the main metrics across parsers.

Reference:

[1] Wang et al., 2025, Thoughts Are All Over the Place, https://arxiv.org/abs/2501.18585

[2] Lu et al. 2025, Reasoning LLMs are Wandering Solution Explorers, https://arxiv.org/abs/2505.20296

[3] Shojaee et al. 2025, The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity, NeurIPS 2025

Questions for authors
  1. What evidence supports the claim that TsuGO disentangles domain knowledge from search organization? Does performance change when models are given domain hints, such as liberty counts or a few solved tsumego as in-context examples?

  2. What fraction of responses are truncated or contain no parsable answer, per model and setting? How does Kimi-K2.5's K=None accuracy change with a larger output budget?

  3. Do model rankings on TsuGO correlate with rankings on other search-heavy tasks, such as other board games or the computational tasks?

  4. Under outcome-only RL on TsuGO, does SearchE improve together with accuracy? Does any gain transfer to search tasks outside Go?

  5. How stable are the main metrics, and how well do parsers agree, when trees are extracted by parsers from other model families?

  6. Does the parser see the reference answer during extraction?

2 comments
AuthorsOpenReview ↗

Reply_1 to Reviewer tAsp

Thank you for recognizing the novelty, verifiability, and adversarial structure of TsuGO. We first answer your Questions then address the remaining scope and interpretation concerns.

1. Q1: Domain knowledge, search, and hints

The main-table comparison between K=4 and K=None provides a partial operational separation of domain knowledge and search organization. Supplying candidates reduces the burden of proposing plausible first moves, while the open setting requires candidate generation as well as subsequent verification. Table 3 illustrates this distinction: Kimi scores 52.0/27.8% on Easy and 34.0/4.4% on Hard. The comparison helps identify how candidate support changes search behavior; both settings still require domain knowledge. Knowledge injection is a valuable extension. However, life-and-death solving combines board perception, state maintenance, candidate generation, and adversarial verification. The size and source of gains from liberty counts or solved examples must therefore be examined carefully through paired comparisons of both outcomes and process behavior, rather than attributed to a single component.

2. Q2: Output budgets, truncation, and larger-budget evaluation

The 32k baseline budget is a deliberate choice for fair comparison of search efficiency within a common resource allowance. Table 16 specifies 32k output budgets for Kimi and most evaluated open-model families. Our question is how models organize search and complete reasoning within a specified budget; this constraint is central to the evaluation design. We will report actual output lengths, truncation rates, and no-answer rates by model and setting, distinguishing initial attempts, retries, and final records. Truncation is itself a dimension of budget-conditioned capability: reaching a solution within the available resources is part of effective search. Larger budgets can provide supplementary analysis, but change the evaluation condition and do not replace the fair fixed-budget comparison.

3. Q3: Cross-task model rankings

Thank you for the suggestion; we will consider extending the comparison to suitable general tasks. At submission, to our knowledge, we had not identified a sufficiently matched protocol for fairly comparing these process metrics across tasks. A useful extension should compare a common set of models under explicit task interfaces and resource budgets. The relative ordering observed on TsuGO shows qualitative agreement with some general capability rankings. We will investigate this agreement more systematically through supplementary task comparisons and rank-correlation analysis, making the comparison protocol explicit.

4. Q4: Outcome-only RL, SearchE, and transfer

Training with TsuGO-related objectives is a future direction we are actively exploring. Thank you for suggesting outcome-only RL and transfer experiments. Building on this benchmark, we can compare outcome rewards with objectives involving SearchE and related metrics, examining whether training improves both task success and search organization. Transfer to search tasks outside Go is a natural part of that follow-up evaluation. These are ongoing research questions motivated by the benchmark's verifiable outcomes and process diagnostics.

5. Q5: Extractor reliability and agreement across families

The extractor has been evaluated through human checking and adjudication, as reported in Appendix C. Table 10 (lines 982-992) reports pass rates of 98% for candidate sets, 97% for order, 95% for parent-child assignments, 93% for terminal judgments, and 96% for metric-consistent trees. The protocol in lines 1007-1020 checks the extracted structure against the visible trace, targeting properties that determine the search metrics. This evidence supports the reliability of the current extraction procedure. Comparisons across parser families would complement the human validation, and we will consider applying independent parsers to the same stratified traces and examining the stability of metrics and rankings.

AuthorsOpenReview ↗

Reply_2 to Reviewer tAsp

6. Q6: Reference-answer access

The parser has one task: extracting the visible trace; it does not receive the reference answer or metric-related scoring instructions. Its inputs are the model's visible reasoning and a fixed schema, with no gold key, target metric values, or scoring weights. It extracts expressed candidates, continuations, and judgments without being asked to solve the puzzle or optimize SearchE. Reference answers enter only the later metric-computation stage. The model's own proposed answer may occur in its trace, but this is not the gold answer.

7. Perception and state-maintenance costs

The substantial allocation to perception and verification is itself a finding about search-reasoning behavior. In our trace analysis, most of the examined models allocate approximately 10-50% of their visible reasoning tokens to board perception, including both reasoning and non-reasoning models. This exposes how finite reasoning resources are divided between grounding, state checking, and subsequent search. Our measurement focus is this observable CoT behavior on verifiable adversarial tasks, rather than Go-playing strength alone. Perception, verification, and state maintenance are not unique to Go; their resource costs reveal behavioral preferences relevant to search organization. We will document the token segmentation and distinguish these total-text costs from candidate-branch metrics.

8. Why an adversarial Go testbed?

The benchmark is centered on search within a verifiable adversarial solution space, not on games as the contribution. Go life-and-death problems offer candidate moves, best opponent responses, and checkable refutations across diverse difficulty. An unpruned tree grows roughly as b^d with branching factor b and depth d; local puzzles retain this challenge while making solutions verifiable. The availability of diverse, annotated local problems supports evaluation at scale. Certain forcing-line Gomoku puzzles can have narrow useful candidate sets, so other domains need comparable diversity and solution coverage before being added. Go's broad branching and difficult value estimation also help explain its prominence in neural-guided search research, exemplified by AlphaGo. We welcome extensions, while maintaining the independent value of a well-constructed adversarial testbed.

9. The new efficiency dimension

TsuGO introduces an operational definition of LLM search efficiency in verifiable adversarial environments, adding a distinct evaluation dimension. SearchE measures wrong-branch allocation, the first exploration of a correct candidate, and candidate rank, connecting the organization of reasoning to a verifiable solution space. This complements accuracy and token cost. Prior studies of path switching, wandering, and complexity motivate related questions. Our additional contribution is making search allocation measurable against correct candidates and adversarial refutations, rather than characterizing verbosity alone. We will clarify this relationship to the cited work.

Reviewer_4VCuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces TSUGO, a process-level reasoning benchmark for evaluating Search Efficiency in LLM reasoning through Go life-and-death problems. TSUGO disentangles domain knowledge from search organization, parses CoT into a structured search tree, and reports Search Efficiency together with Token Efficiency and other diagnostic metrics and visualizations. The study shows that Longer CoT or higher Token Efficiency does not necessarily imply better search.

Critical strengths
  1. TsuGO is a process-level benchmark for evaluating Search Efficiency of LLMs in adversarial tasks with verifiable solution spaces. It also provides visual representation of the game as well.
Critical weaknesses
  1. The paper claims to evaluate "internal search organization ability" of LLMs, but it just evaluates LLM's CoT. There is no evidence presented in the paper that it evaluates "internals" of LLM searching capabilities.
  2. The paper is difficult to follow. Details of how the dataset is constructed are missing, like why SGF is not defined and why there are 1500 problems in the dataset but only 600 problems are used for evaluation. The paper does not refer to exact sections of the appendix in the main draft, and I have to manually search the appendix to get information.
Questions for authors
  1. How good is the LLM-based extractor (section 4.3) in terms of identifying action, edges and nodes? how is Structural validation enforced for closed source models?
  2. Why pass@k not used for the LLMs evaluation if their temperature is not set to zero?
  3. Why all figures (except figure 6) are not referred in the draft?
  4. Why Eq 5 is scale invariant?
  5. Can TusGO be used in PRM training and fine-tuning?
  6. Does visual representation helps the model as compared to only providing json format to the model to solve the problem?
2 comments
AuthorsOpenReview ↗

Reply_1 to Reviewer 4VCu

Thank you for recognizing the value of TsuGO as a process-level benchmark for verifiable adversarial reasoning. We address the measurement scope, validation, and evaluation protocol below.

1. Visible traces and internal search

TsuGO diagnoses observable search behavior rather than directly measuring hidden computation. Candidate exploration, verification, and backtracking provide an indirect behavioral reference for how search strategies are expressed in model outputs, without establishing a complete reconstruction of internal search. This boundary does not remove the value of process-level behavioral evaluation. Building on this work, future studies could combine accessible internal signals with more direct experiments to investigate the relationship between visible traces and internal search. We will align the wording with the scope of the present measurements.

2. SGF and the evaluation subset

SGF is Smart Game Format, a widely used format for exchanging game records and game trees internationally. It represents board states, the acting side, and variations. We apologize for omitting its definition and specification reference at first use and will cite the SGF FF[4] specification. The 1,500 problems describe the curated collection, whereas the evaluation subset supports comparisons at difficulty levels appropriate for current models. Section 3.3 (lines 213-215) describes the manuscript's 600-problem subset across three tiers and reserves harder remaining subsets for more capable models. Appendix A (lines 856-859) explains that the current tiers remain unsaturated. Releasing a larger collection and evaluating selected difficulty subsets therefore serve different purposes; one evaluation round need not cover all 1,500 problems.

3. Appendix navigation

Thank you for identifying the navigation issue. We will add precise section and table pointers for the key steps: Appendix A for data, C for extraction, D for metrics, E/F for model and search settings, and G for prompts. Definitions needed to interpret the main results will remain in the main text, making the detailed protocols easier to locate.

4. Extractor validation

This validation is already reported in Appendix C. Table 10 (lines 982-992) gives human-check pass rates of 98% for candidate sets, 97% for order, 95% for parent-child assignments, 93% for terminal judgments, and 96% for metric-consistent trees, covering the structural properties relevant to the search metrics. The protocol appears in lines 1007-1020: 300 responses stratified across models, difficulty, modality, candidate interfaces, and correctness are independently checked against the visible text by two experts, assisted where needed by the normalized board and KataGo, followed by joint adjudication. These are property-level pass rates, not node/edge F1 or a claim to reconstruct hidden computation. We will add a direct pointer from the main text to this evidence.

5. Closed-source models and structural validity

Outcome correctness can be checked, while complete fidelity to hidden reasoning cannot be guaranteed. Closed-source models' final answers can be scored against the same reference-solution standard. Structural validation checks the parsed JSON, rather than enforcing or validating hidden computation; outcome correctness and process fidelity must therefore be distinguished. Gemini summaries still provide a reference for observable candidates, judgments, and branch organization, but do not measure complete internal search. This observability limit is unavoidable when evaluating reasoning through closed-source APIs. We will distinguish verifiable outcomes from summary-derived process measurements and interpret each within its scope.

AuthorsOpenReview ↗

Reply_2 to Reviewer 4VCu

6. Temperature and pass@k

Pass@k addresses a different objective from the main process evaluation in this paper. It measures success across multiple attempts, whereas we focus on search behavior within individual reasoning traces; final accuracy is one reference dimension. Nonzero temperature does not require replacing this process objective with pass@k.

In our preliminary repeated-sampling experiments, the Search metrics examined showed relatively small variation. Considering manuscript space and evaluation cost, we use trace comparisons under a common single-attempt budget. We will report the preliminary sampling settings and observed variation, and distinguish independent attempts from failure retries to support this choice.

7. Figure references

Thank you for pointing out that the connection between figures and the text could be clearer. We will check the references to every main and appendix figure and add the corresponding question, reading guidance, and textual pointer, making the visual evidence easier to follow alongside the discussion.

8. Scale invariance of Eq. (5)

SWR measures the fraction of total branch allocation spent on wrong branches, rather than its absolute size. If candidate branches contain 100 nodes in total and wrong branches contain 20, SWR is 20%. Doubling every branch proportionally gives 40/200, still 20%. This is the intended scale invariance: uniform scaling leaves the ratio unchanged. Expanding only selected branches or changing extraction granularity does not necessarily preserve it.

9. PRM training and fine-tuning

This is a future research direction building on the present benchmark. TsuGO's verifiable outcomes and process diagnostics can support investigations of PRMs, fine-tuning, and search-behavior optimization, but the present study does not establish training benefits. Such work requires separate training/test data, checks of continuation validity, and evaluation after optimization; a diagnostic score should not automatically be treated as a validated training reward.

10. Visual versus textual inputs

The relevant results are already reported in Appendix D, Table 13 (lines 1195-1234). The main observation is that modality preferences depend on the model and setting, rather than images uniformly improving accuracy. For example, Kimi's K=4 Easy Symbolic/Image accuracies are 57/51%, whereas Qwen235B's K=4 Hard accuracies are 31/39%, giving opposite directions.

For the four models with complete visual results, equally averaging the 24 paired Symbolic/pure-Image cells across three difficulties and two interfaces gives 20.6% versus 18.6%. Thus, text is modestly better on average under this comparison. We will specify the averaging convention and modality differences, without generalizing this result to every model or arbitrary JSON inputs.

Reviewer_fQUeOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

TsuGO is a benchmark that uses Go life-and-death problems to evaluate how LLMs organize search. The main evaluation has 600 problems in three tiers, each posed with four given options (K=4) or none (K=None). An LLM extractor turns visible chains of thought into search trees, on which the authors define Search Efficiency (SearchE), a weighted sum of wrong-branch waste (SWR), first hit (SFH) and correct-candidate rank (SCR), contrasted with a token-based score (TokenE). Nine LLMs are evaluated, with MCTS and KataGo as references. The paper claims SearchE is a stronger process signal than TokenE and that LLMs lie between unguided and guided search.

Critical strengths

S1. The task design is well motivated. Life-and-death problems have stored solution trees with principal variations and refutations (§3.2), which in principle allows reasoning steps to be checked against ground truth. Why this matters to my assessment: the benchmark idea is worth pursuing, which is why the errors below matter.

Critical weaknesses

W1. The benchmark construction contradicts itself.

  • Algorithm 2 sets A={c⋆}∪EA = \{c^{\star}\} \cup E, removes every accepted answer from the proposal pool (P←P∖AP \leftarrow P \setminus A), and builds the options as {c⋆}∪D\{c^{\star}\} \cup D. Each problem therefore has exactly one accepted option, i.e. 300 in the 300-problem audit subset. Table 5 and l.757-758 and l.786-788 report 319 "verified correct or equivalent-correct entries".
  • The audit subset is "half of the 600-problem main evaluation" (l.750, l.847), but Table 7's caption says it "is separate from the 600-problem main evaluation set" (l.811).
  • The bounded prompt asks the model to "choose the optimal first move for Black" (Appendix G, l.1417-1418), while Table 7 lists 32 of 100 Medium and 15 of 100 Hard problems as White to play.
  • §4.3 selects final trees "from parallel extractors by rule filtering and cross-validation" (l.267-268), but Appendix C names only Gemini-2.5-Pro as "the primary visible-trace extractor" (l.918); no other extractor, filtering rule or selection rule is given, and the domain prior dictionary is not provided.

Why this matters to my assessment: the answer key, the evaluated set and the prompt are the basis of every number in the paper, and the paper describes each of them in two incompatible ways.

W2. Reported numbers contradict the paper's own definitions, tables and claims.

  • By Eq. (8)-(10), TMF is a maximum over all nodes including the root, whose children are the SBC candidates, so TMF cannot be smaller than SBC. In Table 12 every LLM cell has TMF below SBC (e.g., Kimi-K2.5, K=4 Medium: TMF 2.1, SBC 4.1); only MCTS satisfies the inequality.
  • CCER is the rank among all non-root nodes (Eq. 4), so SCR=(CCER−1)/SBC\mathrm{SCR} = (\mathrm{CCER}-1)/\mathrm{SBC} can exceed 1, contradicting "values in [0, 1]" (l.1116).
  • At K=4 Medium, eight of nine LLMs score 22.0-29.0 (Table 3), inside a 95% band of roughly 19-31 around the 25% chance level for 200 problems. Their process metrics coincide with an order-independent null model (SWR 75, SFH 25, SCR 37.5, SearchE 32.5): open models give SearchE 30.1-33.5. No random baseline is reported, yet these cells are ranked and interpreted.
  • The text contradicts the tables. "Kimi-K2.5 and Gemini-3.1-Pro concentrate on fewer candidates" (l.399), but Kimi-K2.5 has the highest K=None SBC of all LLMs (6.4, 6.8, 7.6; Table 12). "Gemini models lead on easy settings" (l.368), but Gemini-2.5-Flash at K=4 Easy (32.0) is below six of seven open models. Contribution 3 states that "even frontier LLMs lag far behind neural-guided KataGo" (l.109-110), but at K=None Easy Gemini-3.1-Pro scores 65.0 and KataGo 49.0 (Table 3).
  • Nine references are wrong or out of date: authors added, dropped or misnamed, one venue wrong, one DOI mismatched, one entry matching no record, and three preprints cited although published versions exist. They are listed in full under Questions For Authors.

Why this matters to my assessment: the central metrics do not satisfy their own definitions, the K=4 Medium results are indistinguishable from chance, and several headline statements are contradicted by the paper's tables.

Questions for authors
  1. Where do the 319 accepted entries in Table 5 come from, given that Algorithm 2 leaves exactly one per problem? Is the audit subset part of the 600 evaluated problems or not?
  2. What prompt was actually sent for White-to-play problems? Please report K=4 accuracy split by side to play.
  3. How are TMF and SCR actually computed? Please correct Eq. (4) and Eq. (8)-(10) or Table 12 so that the reported values satisfy the definitions.
  4. Please report a random baseline and confidence intervals for every K=4 cell, together with the null-model values of SWR, SFH, SCR and SearchE.
  5. Which extractors run in parallel, and what are the filtering and selection rules? Please provide the domain prior dictionary.

Also could you explain the problems listed below:

  1. "Kanishk Gandhi, Denise Lee, Gabriel Grand, Muxin Liu, Winson Cheng, Alane Suhr, and Noah D Goodman. Stream of search (SoS): Learning to search in language. arXiv preprint arXiv:2404.03683, 2024." The sixth author is Archit Sharma, not Alane Suhr.
  2. "Olga Golovneva, Moya Peng Chen, Spencer Poff, Martin Corredor, Luke Zettlemoyer, Maryam Fazel-Zarandi, and Asli Celikyilmaz. ROSCOE: A suite of metrics for scoring step-by-step reasoning. In Proceedings of the 2023 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 6556–6576, 2023." The paper was published at ICLR 2023, not NAACL-HLT 2023, and the second author is Moya Chen.
  3. "Xidong Feng, Ziyu Wan, Muning Wen, Stephen M McAleer, Ying Wen, Weinan Zhang, and Jun Wang. Alphazero-like tree-search can guide large language model decoding and training. ICML, 2024." The ICML 2024 version lists Ziyu Wan as first author and Xidong Feng second.
  4. "Yuan Sui et al. Stop overthinking: A survey on efficient reasoning for large language models. arXiv preprint arXiv:2503.16419, 2025." The first author is Yang Sui.
  5. "Yichuan Ma, Linyang Li, Yongkang Chen, Peiji Li, Jiasheng Ye, Qipeng Guo, Dahua Lin, and Kai Chen. Mixing expert knowledge: Bring human thoughts back to the game of Go. In Advances in Neural Information Processing Systems, 2025. doi: 10.48550/arXiv.2601.16447." The NeurIPS 2025 version has its own DOI (10.52202/085713-3164); the arXiv DOI is given instead.

These are examples, not a complete list: other references have similar problems, and I ask the authors to check the whole bibliography.

3 comments
AuthorsOpenReview ↗

Replay_1 to Reviewer fQUe

Thank you for recognizing the value of TsuGO's verifiable adversarial task design. We clarify the counting units, evaluation interfaces, implemented metrics, and scope of the conclusions below.

1. Counting units: 319 versus 300

This issue concerns different counting units. The engineering workflow distinguishes an initial pool of 319 candidate problems from a selected 300-problem subset for consistent stratified evaluation and reporting. We will make the sampling step explicit, distinguish problem counts from option-entry counts, and reconcile the corresponding text and tables.

2. Equivalent answers under the two interfaces

The distinction between equivalent solutions and multiple correct options is already illustrated in Appendix A. Under the benchmark's answer policy, K=4 includes only one accepted first move among the four options, whereas K=None accepts every verified equivalent first move. A problem having equivalent solutions does not imply that its four-option interface contains multiple correct choices.

Specifically, Table 4 (lines 709–710) gives the accepted set {P15, N14}, while only P15 appears among the options. Algorithm 2 excludes accepted moves from distractor proposals, and Table 6 (lines 775–782) explicitly removes N14 as an equivalent correct move. Appendix A (lines 850–853) also describes checking equivalent solutions and accepted first moves. These explanations are present in the manuscript; we will consolidate them and make their relationship to the reported option counts explicit.

3. Audit size versus evaluation scope

An audit sample and the evaluation scope are different quantities. Audit means a quality check using samples balanced across difficulty tiers. The 300 refers to the sampled audit, while 600 denotes the evaluation scope described in the manuscript. The checks examine reference solutions, equivalent first moves, and distractor correctness, providing quality evidence for the audited items.

We will extend verification to every item in that 600-problem scope, checking solutions, equivalent first moves, option correctness, and omitted equivalents. We will also reconcile the text and Table 7 caption to make the set relationship and verification coverage explicit, distinguishing sampled checks from completed full-set verification.

4. Side to play and subsequent search

The intended task starts from an explicitly specified acting side. The prompt supplies side-to-play so that the model evaluates candidates and continuations from a fixed starting state, rather than spending additional reasoning on turn inference.

Under color-symmetric local rules and objectives, exchanging stones, the acting side, objectives, and relevant history gives corresponding states with T(σ x, σ a) = σ T(x, a) and V(σ x) = V(x), measured from the acting player's perspective. This explains why one framework handles both colors; it does not prove conflicting wording harmless. We will provide the actual side fields, accuracy split by side, and a side-neutral prompt comparison to assess that separate issue.

5. TMF and SBC measure different tree levels

The proposed inequality does not apply to the implemented within-branch fan-out. SBC counts root candidates; the reported fan-out counts move children of non-root move nodes. Four root candidates with at most two opponent replies per branch can yield SBC = 4 and fan-out = 2 simultaneously.

The inequality holds for a maximum including the root, as in the written equation. We will align that equation with the reported computation and state the node scope explicitly, distinguishing root breadth from continuation branching.

AuthorsOpenReview ↗

Replay_2 to Reviewer fQUe

6. CCER and SCR use candidate ranks

The implementation uses candidate ranks, not reasoning-node ranks. Exploring A for 20 steps, B for 10 steps, and the correct candidate C third gives CCER = 3, regardless of the preceding branch lengths.

For m candidates and finite rank r, SCR = (r - 1)/m; the third of four gives 0.5, and an absent correct candidate receives 1. This computation is bounded. We will define the ordered candidate sequence explicitly, align the written equations with the implementation, and specify SFH/SCR boundary rules.

7. Near-chance Medium results and the broader evaluation

Near-chance model performance does not establish that the benchmark is invalid. Medium should be interpreted alongside Easy, Hard, and open conditions. Table 3 reports MCTS/UCT and KataGo; Section 5.4 (lines 457-462, Figure 6) discusses MCTS approaching random allocation under K=4.

These search references are distinct from a purely random null. We will add the corresponding null and K=4 confidence intervals, preserving dependence between modalities on the same problem. Broader trends will not substitute for uncertainty estimates when interpreting small differences within near-chance cells.

8. The random expectation of SearchE

The reviewer's value of 32.5 is consistent with the SearchE aggregation under a specified null and the candidate-rank convention clarified above. Assume all four candidates are explored, exactly one is correct, branch allocation is equal, and candidate order is uniformly random. Then E[SWR] = 0.75, E[SFH] = 0.25, and E[SCR] = 0.375. Using the fractional inputs of Eq. (2), E[SearchE] = 100 × [0.5(1 − 0.75) + 0.3(0.25) + 0.2(1 − 0.375)] = 32.5.

Thus, 32.5 is an expected SearchE score, not accuracy or a universal baseline for every tree. Different candidate coverage or allocation requires a corresponding null. We will align the written rank definition with the candidate-order computation and qualify near-null rankings; the calculation does not by itself invalidate the broader diagnostic.

9. Domain priors and the extraction pipeline

The domain prior is preprocessing terminology, not reference-answer supervision. It consists of cues for candidate exploration, variation reading, position evaluation, and backtracking, without supplying the problem's answer. We will release the vocabulary, role mappings, sources, and prompt usage.

The pipeline we can substantiate uses Gemini-2.5-Pro followed by structural sanitation and retries. Concurrent requests and retries are not independent-parser agreement; we will describe the actual operations and validation evidence separately.

10. Observability of Gemini reasoning

The observability restriction is already explicit in both the main text and appendix. Section 4.1 (lines 239–243), Section 4.4 (lines 304–307), and Appendix E (lines 1278–1280) identify summaries as observable artifacts, not complete internal computation. This agrees with Google's official documentation. Interpreting Gemini's candidate counts as internal search breadth would disregard that stated measurement boundary.

This limitation applies to Gemini and does not replace a separate analysis of Kimi. Candidate count and concentration of branch resources are different quantities and should be reported separately. As emphasized in the experimental discussion, summary-derived and more extensive visible-trace measurements should not be treated as directly comparable measures of hidden computation.

11. KataGo under a fixed search budget

This is a budget-conditional comparison, not an unrestricted ranking across all settings. KataGo is evaluated at 200 visits. Open accuracies of 49/46/43% vary less across difficulty tiers than K=4 accuracies of 97/75/58%; these conditions require separate interpretation.

For a fixed network, A_d(N) depends on difficulty d and visit budget N; the table provides N = 200. More visits provide additional search opportunities, but improvement requires a budget sweep rather than an assumption of monotonic accuracy or difficulty independence. Visits are not equivalent to LLM token budgets, so conclusions remain specific to the interface and budget.

12. References

These concerns are best addressed by checking the specific citation versions against official records. We appreciate the bibliographic details raised. Preprint records and publisher metadata may be updated as manuscripts are revised or publication details are finalized. We will check the cited versions against current official records, standardize authors, venues, DOIs, and publication versions, and review the remaining bibliography for consistency.

Dear authors

Thank you for the detailed rebuttal and additional experimental results. I appreciate the authors' efforts to systematically address my concerns.

I increase the score from 1 to 2

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.