acceptodds
Under review as a conference paper at ICLR 2027

When AlphaGo Thinks: Mastering Go with an LLM-based Reasoning System

Abstract

Benefiting from the rapid development of reinforcement learning with verifiable rewards (RLVR), reasoning LLMs have shown remarkable capabilities on a wide range of reasoning tasks, especially in mathematics and code. However, when the focus shifts to reasoning tasks with scarce textual corpora, such as Go, their performance remains far from satisfactory. Although recent studies have trained LLMs that approach the level of professional Go players, these models still struggle to produce precise professional Go analysis. In this work, we investigate the possibility of building a Go-domain reasoning system based on LLMs. We first examine the capability boundaries of LLMs of different sizes when they serve as policy models and reward models for Go tasks. Based on these findings, we build an agent system consisting of a sampler and a reward model. We further propose Term-DPO, an algorithm that substantially improves the model's accuracy in expressing Go-related professional expressions. Through extensive experiments, we build the first LLM-based reasoning system that reaches the capability level of specialized Go models and surpasses top human players. At the same time, the system provides interpretable analyses with accurate Go terminology.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Reasoning, Reinforcement Learning with Verifiable Rewards, Preference Optimization, Game-Playing Agents

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_qD5eOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how to improve the decision-making ability and the accuracy of natural-language analysis of reasoning LLMs in the specialized domain of Go. The authors identify two main challenges faced by existing LLMs in Go tasks. First, candidate move generation and final move selection require distinct capabilities, which are difficult for a single model to balance effectively. Second, existing training methods primarily focus on move quality, providing insufficient supervision for the accurate use of professional Go terminology and board-state analysis.

To address these challenges, the authors propose a dual-model reasoning framework consisting of a Sampler and a Text Reward Model, which are responsible for candidate move generation and evaluation-based selection, respectively. Both components are trained using task-specific SFT and RL objectives. In addition, the authors introduce Term-DPO, which expands approximately 5,000 human-corrected Go analyses into 500,000 preference pairs through progressive terminology replacement and text rewriting. These preference pairs are then used for DPO training to improve the accuracy of professional Go terminology. The authors also construct KataGo-Eval-Levelwise, a benchmark designed to evaluate Go decision-making capabilities across different difficulty levels.

The experiments cover Qwen3 models of different sizes and evaluate the proposed approach through Go benchmarks, full-game matches, and human evaluations. The resulting system achieves a 92.1% move hit rate on KataGo-Eval-Levelwise, substantially outperforming the Single LLM baseline (82.8%). It also wins 73 out of 81 games against nine professional Go players and achieves near-even results against strong KataGo models under the specified search budget. Furthermore, Term-DPO improves professional terminology precision to 87.4% while maintaining comparable move prediction performance. The paper also includes model scaling studies, reward model comparisons, and ablation experiments to further examine the contributions of different components.

Critical strengths
  • This paper investigates the decision-making and analytical capabilities of LLMs in the specialized domain of Go. It proposes a dual-model framework to address the distinction between candidate move generation and final move selection. The overall approach is straightforward, and the research problem is practically meaningful.

  • The authors construct large-scale preference data from human-corrected professional Go analyses, reducing the need for extensive manual annotation while significantly improving the accuracy of professional Go terminology and largely preserving the model's move prediction performance.

  • The paper conducts experiments across Qwen3 of different sizes and introduces a difficulty-level benchmark for Go evaluation. The system is further evaluated through full-game matches against KataGo, matches against professional Go players, and human evaluations. In addition, model scaling studies, reward model comparisons, and ablation experiments provide further evidence supporting the effectiveness of the proposed approach.

Critical weaknesses
  • The proposed Sampler + Text RM framework largely follows the existing Generate–Verify–Select paradigm, while Term-DPO adopts the standard DPO objective. Although these two components improve move selection and professional terminology usage, respectively, they address relatively independent problems, and the paper does not sufficiently demonstrate their synergy at the methodological level.

  • The authors construct Chosen–Rejected pairs through progressive terminology replacement, assuming that responses with fewer replaced terms are of higher overall quality. However, the number of replaced terms does not necessarily have a strictly monotonic relationship with the actual quality of Go analysis, and rewriting with GPT-5.1 may alter the original semantics. While the human-annotated dataset is a valuable contribution to the community, the reliability of the large-scale synthetic preference labels remains questionable.

  • Term-DPO significantly improves the accuracy of professional Go terminology, but correct terminology usage does not necessarily imply accurate board-state analysis or reasoning. The current evaluation primarily focuses on terminology-related metrics and relatively limited human evaluations, lacking systematic verification of the correctness of generated move sequences and the reliability of win-rate estimates.

  • The proposed approach requires deploying two separate models, the Sampler and Text RM, which introduces additional inference overhead. Although the authors provide comparisons with repeated sampling, evaluations against other candidate verification and selection strategies under comparable computational budgets remain relatively limited.

Questions for authors

I'm open to raise my score if the authors could address the following concerns. Given the substantial engineering effort involved in the experiments, I believe that a few lightweight additional experiments and clarifications would be sufficient.

  • Could the authors clarify whether there is meaningful synergy between the Sampler + Text RM framework and Term-DPO? For example, could a more explicit component-wise ablation study demonstrate whether combining these two components yields benefits beyond their individual contributions?

  • Have the authors examined whether the preference relationships constructed through progressive terminology replacement are consistently valid? Although the paper provides some human validation of the preference labels, further analysis of how the number of replaced terms and GPT-5.1 rewriting affect label accuracy would help establish the reliability of the large-scale synthetic preference dataset.

  • Beyond terminology accuracy and the existing human evaluations, could the authors further investigate whether Term-DPO genuinely improves the model's Go analysis and reasoning capabilities? For example, a more fine-grained evaluation of generated move sequences, positional assessments, and win-rate estimates using KataGo or professional Go players would provide stronger evidence.

  • Could the authors provide more comprehensive comparisons between the dual-model framework and alternative Generate–Verify–Select strategies under matched inference compute budgets? For example, increasing the number of candidates sampled by a single model or employing alternative candidate reranking strategies would help clarify the necessity of separately training the Sampler and Text RM, as well as the trade-off between performance gains and computational cost.

1 comment
AuthorsOpenReview ↗

Response to Reviewer qD5e(1/2)

First, thank you for the time and effort you devoted to reviewing our paper, and for your insightful questions and suggestions! We respond below to each of the weaknesses and questions you raised. We hope our responses address your concerns!

Weakness 1 / Q1: Complementary roles of the dual-model framework and Term-DPO

Thank you for this valuable question! In our view, the dual-model framework and Term-DPO serve primarily complementary roles. The Sampler and Text RM improve candidate generation and move selection, while Term-DPO improves the reliability of the accompanying professional analysis. To examine this directly, we compared the dual-model system before and after Term-DPO.

Specifically, we invited ten strong amateur and professional Go players to evaluate the analyses and move predictions on 30 positions. Terminology accuracy increased from 27% to 95%. In comparisons between the two systems’ responses, the version with Term-DPO achieved a 100% helpfulness preference rate over the version without it. Meanwhile, move accuracy remained largely unchanged, at 92.0% before Term-DPO and 92.1% after it. These results show that combining the two components allows the system to make strong move predictions while providing analyses that players find more helpful. We will clarify this complementary relationship in the appendix and include these comparative results.

Dual-model systemMove accuracy (%)Terminology accuracy (%)Helpfulness preference over the system without Term-DPO (%)
Without Term-DPO92.027.0Reference
With Term-DPO92.195.0100.0

Weakness 2 / Q2: Expert agreement before and after rewriting

Thank you for this interesting question! To investigate it, we collected 100 preference pairs and assessed expert agreement both before and after rewriting. Each pair was judged by three Go players, yielding 300 judgments in each condition. Before rewriting, 281 of the 300 judgments favored the response labeled chosen, giving an agreement rate of 93.67%. After rewriting, 274 of the 300 judgments favored chosen, giving an agreement rate of 91.33%. Thus, even after rewriting, experts still preferred the intended positive response in most judgments.

This supports the reliability of the constructed preferences and indicates that rewriting generally preserves the quality differences experts recognize. Some labels nevertheless remain inconsistent with expert judgments. We will include these results to clarify how rewriting affects preference-label reliability. From a Go player’s perspective, the terminology and positional descriptions applicable to most positions are unambiguous, making it relatively straightforward to judge whether a term or positional analysis is correct.

Preference pairsUnique pairs evaluatedTotal expert judgmentsJudgments favoring chosenAgreement (%)
Before rewriting10030028193.67
After rewriting10030027491.33

Weakness 3 / Q3: Accuracy of positional analysis beyond terminology

Thank you for this insightful question. First, variation prediction and win-rate estimation are mainly learned during the earlier training stages. Term-DPO mainly focuses on improving professional analysis. Our existing experiments show that move accuracy is preserved after Term-DPO.

In addition, the expert corrections used to construct the training data include corrections to inaccurate positional analysis. To evaluate this more directly, we selected 50 positions for a focused human evaluation of the outputs before and after Term-DPO. Players judged whether the model correctly assessed which side had the advantage and whether its analysis of the position was accurate. After Term-DPO, advantage-assessment accuracy increased from 28.0% to 74.0%, and positional-analysis accuracy increased from 36.0% to 78.0%. These results provide clear evidence that the trained model also improves its analysis and understanding of specific Go positions.

Training stagePositionsAccuracy of advantage assessment (%)Accuracy of positional analysis (%)
Before Term-DPO5028.036.0
After Term-DPO5074.078.0
Reviewer_CbNYOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper builds an LLM reasoning system which can play the game GO. The approach uses two components, a sampler and a text reward model. Each component is trained separately in a supervised fine-tuning stage, followed by RL and then term-DPO. The model is trained using a mixed of reasoning data from other LLMs and 50 million positions collected from GO. Evaluation shows that their two model system outperforms Single-LLM, defeating the three smaller Kata-Go nets and is near-even with the two larger ones.

Critical strengths
  • Thorough evaluation. The authors validate their approach in a range of settings including against Kata-Go engines of varying sizes, games against professional players and comparisons against prior LLM based lines, demonstrating the benefit of their approach.
Critical weaknesses
  • Weak method. The entire training approach is reliant on a GO engine to label millions of positions and assign scores to rollouts. It does not make sense from an efficiency perspective to distill learning into an LLM from an inference engine which is orders of magnitude smaller. The reward function is problematic for the same reason, since it learns to optimise the value assigned by the KataGo engine, rather than optimise using a verifiable reward signal observable from the environment. This is distillation instead of reinforcement learning with verifiable rewards (RLVR), and caps the system at the teacher’s level. This is inherently limited compared to previous AI systems trained to play GO.
  • Single-turn training. The model is trained using single moves based on pre-annotated positions where there is no interaction with the environment. The model never trains on positions from its own rollouts. For a reasoning system, a multi-turn setup that interacts with its own environment would be a more natural fit, for example playing out its predicted variations against a board simulator.
Questions for authors
  • How does the system perform against the teacher model it was trained on in tournament settings?
  • What is the per-move compute of the reasoning system compared to KataGo opponents?
1 comment

Dear Reviewer CbNY,

Thank you very much for your valuable feedback on our work. As the author–reviewer discussion period draws to a close, we would greatly appreciate your thoughts on whether the additional experiments and clarifications in our rebuttal have addressed your concerns. Even a brief response would be very helpful. Thank you again for your time and consideration!

Best regards,

The authors of Submission 31948

Reviewer_2WT2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper builds a Go reasoning system from Qwen3-8B models on top of the cold-start SFT + GRPO framework of Ma et al. (2026). A sampler trained with a coverage reward proposes candidate moves; a text-based reward model trained with a pairwise ordering reward selects among them. Term-DPO expands ~5,000 expert-annotated analyses into ~500K preference pairs via progressive terminology replacement from a ~300-term vocabulary plus GPT rewriting, raising terminology precision to 87.4% without hurting move accuracy. A new benchmark, KataGo-Eval-Levelwise (774 positions, difficulty-stratified via 32 KataGo/HumanSL reference models), is introduced. The system reaches 92.1% top-eight hit rate vs 82.8% for a single LLM, wins 65.7% of 300 games against the single LLM, is near-even with KataGo-256c/384c at 4,096 visits, and wins 73/81 games against nine professionals (2-9 dan). Ablations cover coverage-vs-selection scaling, Text RM vs a bucket-token RM, reasoning budgets, and Term-DPO components, with three-seed intervals for scaling and Wilson/bootstrap intervals for games and human evaluation.

Critical strengths
  1. The evaluation has unusual breadth for this line of work: 168,300 tournament games against 32 engines, 81 timed games against ranked professionals, blinded human evaluation of analysis quality by 12 professionals, and a difficulty-stratified benchmark with top-1 and score-loss metrics. These evidence streams converge, and the benchmark and protocol are likely to be reused.
  2. Term-DPO is well isolated. The ablations — no rewriting, original-only pairing, update-matched DPO-Repeat, SFT-5K — attribute the gain to the progressive-replacement construction rather than annotation volume or update count, and move accuracy is preserved.
Critical weaknesses
  1. The contribution is incremental over Ma et al. (2026), and the framing overclaims. The training pipeline is inherited; sampler+selector is the standard generator/verifier pattern (Cobbe et al., 2021); and "Mastering Go" is not supported — the system is near-even with mid-size KataGo networks at a fixed, modest search budget and sits below KataGo-384c in Elo. More importantly, Go is an atypical "domain with scarce textual corpora" because it has a perfect oracle for annotation, rewards, and evaluation; nothing here demonstrates transfer to domains without one, which is where the stated motivation actually bites.
  2. The work depends on heavy, partly non-reproducible infrastructure: 50M KataGo-annotated positions, reasoning data distilled from LoGos/GLM-5.1/Kimi-K2.5, GPT-5.1 rewriting, and 32xH800-scale training. The 774-position benchmark is sampled from hit-rate intervals of the same reference models used for evaluation, which may embed selection effects. Independent replication would be difficult.
Questions for authors
  1. How does the system do against KataGo at larger visit counts (e.g., 32k)? The near-even claim is specific to 4,096 visits; the trend would clarify the strength ceiling.
  2. Benchmark, RL rewards, and coverage rewards all use KataGo-family references. Did you evaluate hit rates against a different engine (a held-out KataGo generation or another engine) to quantify circularity?
  3. How sensitive is Term-DPO to the replacement order and to the coverage of the ~300-term vocabulary? Even a small sweep would help.
  4. In the professional matches, what was the system's per-move time usage relative to the humans' 30 minutes + byo-yomi? Equal-time conditions matter for interpreting the 73-8 record.

Comments on the paper

4 comments

Dear Reviewers,

Thank you again for your time and constructive feedback. We have provided additional experiments and clarifications in our rebuttal and would greatly appreciate your thoughts on whether these responses address your concerns. Even a brief follow-up would be very helpful, and we would be happy to clarify any remaining questions.

Best regards,

The authors of Submission 31948

Dear Reviewers,

As the discussion period is coming to an end, I kindly encourage you to review and respond to the authors' rebuttal as soon as possible.

Thank you very much for your time and valuable contributions to the review process.

Best regards,
Area Chair

AuthorsOpenReview ↗

Summary of Revisions and Additional Experiments

We sincerel thank all reviewers for their careful reading and helpful suggestions. Following your feedback, we conducted additional experiments and clarified the following points in our responses:

  • The dual-model architecture. To examine the need for the Sampler + Selector design, we added two comparisons: Single LLM generation followed by Bucket RM selection, and an RL-trained single model that generates four candidates and chooses among them. Our system achieved a 9.5 percentage-point higher hit rate than the four-candidate model, with similar output length and slightly lower latency. These comparisons further demonstrate the benefits of training the Sampler and Text RM separately.

  • The contribution of Term-DPO. We invited professional Go players to assess the training preference pairs and the model’s analyses. Agreement with the intended preferences exceeded 91% both before and after rewriting. We also added an ablation of the replacement procedure. The results showed that constructing preference pairs by progressively replacing correct terms in a fixed order led to better terminology precision. In the dual-model system, Term-DPO increased terminology accuracy from 27% to 95% while preserving move accuracy. Evaluations involving professional players also showed improvements in the accuracy of positional analysis and the helpfulness of the model’s responses.

  • The robustness of our evaluation. To examine whether the benchmark reflects human playing strength and whether the results depend on the choice of KataGo judge, we evaluated six models with a different judge and invited ten players of different ranks to complete 200 benchmark positions. Our system retained its advantage over Single LLM under the new judge, and human scores on KataGo-Eval-Levelwise generally followed playing strength. To examine how KataGo’s search budget affects match outcomes, we also added matches against five KataGo opponents at 32k visits and against the training teacher at 8k visits. We acknowledge that our system remains weaker than the teacher. Our aim is to combine strong Go play with useful natural-language analysis, building a reasoning system based on a general-purpose LLM that can play at a high level and offer practical help to human players.

We sincerely appreciate your suggestions and the time you have devoted to reviewing our work. We will include these additional experiments and clarifications in the revised paper. Thank you again for helping us improve the work!

Dear Reviewers 2WT2 and CbNY,

Thank you for your detailed feedback on our work. As the discussion period will close in a few hours, we would greatly appreciate your thoughts on whether our responses have addressed your concerns. If anything remains unclear, we would be happy to provide further clarification.

Thank you once again for your time and consideration.

Best regards,

The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.