acceptodds
Under review as a conference paper at ICLR 2027

Go-MLLM-Bench: Perception, Decisions, and State Updates in 1919 Go

Abstract

Multimodal models are increasingly expected to act on what they see, but reading a scene, choosing an action from it and updating it after the action are different skills that one success rate mixes together. We introduce Go-MLLM-Bench, which asks about the same 1919 Go positions through an image, an exact board matrix or the move history, and scores board reading, move prediction and board updates separately. We will release nested training cohorts from disjoint games with board, engine and mixed targets. The main finding is simple. Models can learn to read the board almost perfectly, but this skill does not carry over to choosing moves or to updating the board after a move. A fine-tuned model that reads nearly every tracking image exactly never produces an exact update, even from the true previous board with a decoder that forces valid output. Handing the update to a rule engine, starting from the model's own reading, recovers every state. The benchmark also shows that locating a salient mark is not reading the board, that cell accuracy rewards copying a stale board, and that legal-move decoding lengthens games without finishing any. Benchmarks should measure what a model does with a state, not only whether it can report it.

Primary area:
datasets and benchmarks
Keywords:
Go Game, Multimodal LLM, LLM Agent, Dataset and Benchmark

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_XTKfOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This is careful, well-controlled benchmark work with genuinely useful negative results and a methodology I would like other benchmark papers to imitate strict scoring, token-matched controls, syntax vs legality separation, separate counting of "what the model does with the state." The idea of measuring perception, decision, and update separately on one state is a real contribution.

Critical strengths
  1. The decomposition is the contribution, and it is well chosen. Reading / choosing / updating are routinely conflated in multimodal-agent evaluation, and this benchmark separates them on identical states with identical accounted denominators. As evaluation-design work this is clean, motivated, and portable in spirit. 2.The findings are sharp, quotable, and consequential. "Recognition can coexist with failed state update, and a rule engine given the same readings fixes it entirely" is a crisp localisation of what is missing. "Legal decoding lengthens games without finishing any, and no arm ever passes" is an important caution against reading constrained-decoding improvements as capability gains. 3.Strict scoring done properly. Invalid is failure, no repairs, no answer ever dropped from a denominator. This is exactly the discipline the field needs and it is what makes several of the results legible.
Critical weaknesses
  1. There are no error bars anywhere in the training study. Statistical power is the paper's central weakness, and it bears directly on its headline claims. 2.The closed-loop probe nominally has 120 episodes, but 92 end on an invalid action within ~4 plies on average and 26 fail initialisation; even with a legal-move trie the episodes run to a 600 s cap with no pass. 3.Teacher strength and metric leniency. Labels come from 64-visit KataGo. That is a comparatively weak/noisy teacher, and Hit@5 is scored against candidate lists holding five moves on 182/300 positions. Random-legal is quoted at 1.95%, which is a useful anchor, but the effective difficulty is much closer to "top-5 over the engine's shortlist" than to "best move of 361."
Questions for authors

1.Since only 3/233 changed points are captures, do you plan a capture/ko-heavy tracking split? Without it, "updates fail" may not generalise to the interesting part of Go dynamics. 2.Please re-verify Hit@5 rankings on a subset with substantially higher visit counts, and report whether adapter orderings are stable.

1 comment
AuthorsOpenReview ↗

Response to Reviewer XTKf

W1: No error bars or seeds. Every arm now has three seeds, with the per-seed values, seed SD and paired source-game intervals reported separately (App. E.1). The headline contrast holds in every seed: exact board +69 to +73 points, and image Hit@5 versus the token control +0.00, −0.67 and +2.33 points, each with a 95% CI containing zero. Clusters are source games, never repeated views or windows.

W2: Closed-loop episodes are short and censored. We agree that they are not 120 independent estimates of playing strength. The reanalysis accounts for all 280 historical episodes and replays all 4,665 accepted plies. It separates invalid initialization (26), invalid actions (92), ledger and wall-time stops, and late responses.

  • With the legal trie, all 40 episodes have 0 invalid proposals among 2,074 proposals and reach the time cap after 100–107 plies.
  • No model ever passes; KataGo passes 6 times.

We read this as "valid execution did not yield completion within the budget", not as inevitable non-completion or as losses. Intervals resample the five starts. The new Gemini subset (App. E.4) is reported with its own censoring categories.

W3/Q2: Weak teacher and lenient Hit@5. We relabeled a frozen, phase/side-balanced subset of 120 games at 64 and 4,096 visits, and 24 of them additionally at 1,024 and 16,384 visits, reusing the same model answers (App. E.6–E.8, Tables 27–31).

  • At 4,096 visits, the mixed adapter and token control tie (20/120 each; paired difference 0.0 [−7.5, +7.5]). Gemini leads with 45/120.
  • The 64- and 4,096-visit best moves agree on 85/120 positions.
  • Full five-move shortlists rise from 65/120 to 108/120 positions. We report this because a larger shortlist can raise Hit@5 without any change in the model.
  • On the 24-game ladder, Hit@5 counts for the four main arms are unchanged from 4,096 to 16,384 visits, while Hit@1 changes slightly.

Fine orderings of nearby adapters can change, so we no longer claim a strict ranking. The mixed-versus-control comparison is stable.

Q1: A capture/ko-heavy tracking split. Added: 240 windows from 102 games (80 single-capture, 80 multi-capture, 40 ko, 40 controls; 312 captured stones).

  • All frozen and directly trained models score ≈0 exact (best: 0.14% for U-mixed).
  • By stratum, changed-cell accuracy is 0.6–1.5% for trained models, versus 8.8/5.2/3.8% for the frozen mixed control on single/multi/ko windows. Direct update training reduced local capture handling while improving preservation of unchanged cells.
  • Endpoint removal recall is ≤0.43%.

So update failure does extend to capture dynamics.

Reviewer_27gVOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Go-MLLM-Bench, a benchmark to evaluate MLLMs on Go games. It includes three tasks: perception, decision, and state update. Results show that models can recognize the board after fine-tuning, but still have poor decision and state update abilities.

Critical strengths
  1. Clear design. The benchmark separates different abilities, which is useful for understanding model failures.

  2. Good experiments. The paper compares different inputs and training methods.

  3. Interesting findings. Good perception does not mean good reasoning or state update.

  4. Easy to evaluate. Go has clear rules and results can be checked automatically.

Critical weaknesses

1.Small benchmark. Only 300 board states and 60 state transitions.​

  1. Poor generalization. The performance drops a lot when using different board styles.​

  2. Missing experiments. The authors do not train models directly on state update tasks.​

  3. Decision metric is limited. KataGo Top-5 agreement may not show the real decision quality.

​5. Limited experiments. Fine-tuning only uses one random seed.

Questions for authors

1.Why not train models directly on state update? Will the problem still exist?​

  1. How to know whether failures come from reasoning or different data distributions?​

  2. Can the authors test more board states and different Go rules?

​4. Why use Top-5 agreement instead of move quality?

​5. Can we really say perception does not help decision, since the decision model itself is weak?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 27gV

W1/Q3: Small benchmark; more boards and rules. We added the following panels, all from games disjoint from earlier training and evaluation (also under D4 symmetry):

  • 1,000 positions from 1,000 games (three seeds). The 20k mixed adapter reads 72.2 ± 0.9% of boards exactly (69.3% on the original 300). Its Hit@5 is 10.0 ± 0.3%, versus 12.4 ± 0.3% for the token control, which reads 0 boards. The paired difference is −2.3, −1.9 and −2.9 points for seeds 42/43/44, so on the larger panel the mixed adapter is, if anything, slightly worse.
  • 600 update windows (120 games) and 240 capture/ko-enriched windows (102 games), plus the original 60.
  • A 120-case history/rule panel. It contains 30 natural ko parents and, reported separately, 30 constructed variants of one self-capture motif under Tromp–Taylor, Chinese and Japanese settings. No model gets both the legal and the illegal case right for any parent.

W2: Poor generalization to other board styles. We agree, and the new panel makes this explicit (App. E.3). It renders the same 120 positions in the training palette, a line-art palette and a high-contrast palette.

  • Over three seeds, the mixed adapter reads 89/86/85 of 120 boards in the training palette, but 0/120 in line-art in every seed and 1–2/120 in high-contrast.
  • Hit@5 hardly changes across the three palettes (≤2.5 points in every seed).

So learned board reading is style-specific, and move choice does not depend on it. This is consistent with the main claim that reading does not transfer to decisions. The paper now says this.

W3/Q1: Train directly on state updates. Done (Sec. 6, Fig. 10; App. E.2). We trained update adapters on 8,000 transition windows, from the base model (U-base) and from the mixed adapter (U-mixed), with 3 seeds each. Results (mean ± seed SD, 600 representative windows, syntax-only decoding):

  • Exact update: 0.61 ± 0.25% (U-base) and 1.11 ± 0.35% (U-mixed), versus 0% for both frozen controls.
  • Changed cells: 1.4% and 1.2%, versus 3.2% for the frozen mixed control.
  • Capture/ko-enriched windows: 0.00% and 0.14 ± 0.24% exact.

Training mostly teaches the model to keep unchanged cells (99.3–99.7%) and to copy the previous board (15–38% of windows), not to apply the move. So the problem remains with direct supervision at this budget. We do not claim that no training could succeed.

Q2: Reasoning failure or data distribution? Three controls address this:

  • On 800 held-out windows from the training distribution, exactness is still 0.25–1.13%, so a train/test shift does not explain the failure.
  • Syntax-only decoding changes exactness by ≤0.06 points, so formatting does not explain it either.
  • A rule engine given the same readings updates every state correctly.

What is missing is applying the transition. We describe this behaviorally and do not claim a specific internal mechanism.

W4/Q4: Top-5 agreement instead of move quality. We added a quality metric (App. E.7, Tables 29–30):

  • Every legal selected move receives an equal-budget 4,096-visit root-restricted evaluation against the engine's preferred move.
  • Moves within 2 points of the reference: 28/120 for the mixed adapter, 32/120 for the token control and 54/120 for Gemini.
  • Mixed minus control: −3.3 pp [−10.8, +4.2] on this rate, and +0.48 points [−0.59, +1.57] in score gap on the 53 positions where both moves are legal.

The conclusion is unchanged.

W5: Only one seed. All 14 adapters now have three seeds (Sec. 6, Fig. 9; App. E.1, Table 23). The board-reading gain (≈+70 points) and the null Hit@5 difference (−0.67 to +2.33 points, all CIs include 0) hold in every seed. Seed SDs of Hit@5 are 0.0–1.7 points.

Q5: Can we conclude perception does not help decisions, if the decision model is weak? Fair point. Our claim is narrower: this training did not produce a measurable gain on the decision endpoint. To test sensitivity to a weak policy, we hold a strong policy fixed (KataGo, 4,096 visits) and give it either the true board or the board reconstructed by the B2/B3 adapters on all 300 games (App. E.5, Table 26). Invalid reconstructions stay in the denominator.

Board given to KataGoAcceptable move (legal, ≤2 pts)Hit@5
True board100.098.7
Mixed B3 reading82.7 [79.0, 86.3]79.3 [75.3, 83.3]
Board-only B2 reading71.0 [66.3, 75.7]66.3 [61.7, 71.0]

With a strong policy, better perception does help: B3 minus B2 is +11.7 pp [7.3, 16.0] in acceptable moves, and every exactly read board yields an acceptable move. So your concern is right. The board reading the adapters learn is useful as input; what fails is the adapter's own use of it. We revised the wording accordingly: perception does not transfer within the model to its decisions, rather than "perception does not help decisions".

Reviewer_27gVOpenReview ↗

Response to authors

Dear authors, thank you for response. I have updated my score.

AuthorsOpenReview ↗

Reply to Reviewer 27gV

Thank you for revisiting the paper and updating your assessment. We especially appreciated your question about whether a weak move selector could hide the value of better board reading. It prompted us to give the reconstructed boards to the same strong Go program. The program chose better moves from the more accurate boards. That result helped us say more precisely what our experiments show: the improvement in reading was useful, even though the trained model showed no corresponding improvement in its own move choices. Thank you for helping us make that distinction clear.

Reviewer_XL7xOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper propose a benchmark called Go-MLLM-Bench to evaluate the multimodal model's ability on Go. The paper asks the same question about the same Go position in three ways (image, matrix, SGF) and evaluate across diverse models.

The paper breaks down "understanding a board state" into three tasks: reading board, selecting chess, and updating state, and then demonstrates that they're separate abilities, imroving one ability doesn’t necessarily improve one another. Through experiments on Qwen-VL-4B, the paper found that the model is able to learn board reading (69%), but this ability does not transfer to move selection (13.7%, on par with the control group), and certainly does not transfer to updating (0/60).

Critical strengths
  1. The paper gives a clear task formulation for Go. Instead of collapsing performance into a single accuracy number, it evaluates board reading, move selection, and state updating as separate abilities, which yields a substantially finer-grained assessment.

  2. The paper uses LoRA for training. It constructs targeted data for these tasks, trains the models, and observes their performance improvements on those tasks.

Critical weaknesses
  1. Closed-loop play and all decoding controls are run only on the local Qwen3-VL-4B arms (base, mixed, token control); the gateway models are never played and never receive the coordinate or legal-move trie. The conclusion that "no episode finishes" is therefore a statement about a 4B LoRA model, not about MLLMs generally. Running the strongest baseline (Gemini 3.8 Flash) through even a subset of the closed-loop protocol, with the same trie conditions, would substantially broaden the claim.

  2. The article doesn't explain new terms when they first appear, which makes it quite difficult to read for anyone unfamiliar with Go. (For instance, engine agreement, separate probes, controlled adapters in the intro.)

Questions for authors
  1. This paper considers three views for a state: image, matrix, and SGF. Only the first view involves multimodal image-reading capabilities, while the other two pass solely through the LLM module. Is that correct? If so, could this difference partly explain the difference in capabilities?

  2. What is the value of the task itself? Given that systems such as AlphaGo already play Go far better than any human, what does measuring a MLLM's Go ability tell us that we do not already know?

5 comments
AuthorsOpenReview ↗

Response to Reviewer XL7x

W1: Closed-loop results only cover the 4B model. We agree, and we have narrowed the claim. The introduction now states that closed-loop non-completion is shown for the local checkpoints, not for MLLMs in general.

We also ran Gemini 3.8 Flash through the closed-loop protocol (App. E.4): 1,200 s, 128 calls and 256 plies per episode, as in our matched-decoding panel. Two caveats apply:

  • Hosted APIs do not expose logits, so a token trie cannot be reproduced. A disjoint probe showed that the provider's JSON-schema enum is not enforced (2/5 compliant). The coordinate and legal-move conditions therefore list allowed moves in the prompt. We label this text-list assistance, not equivalent to the trie.
  • For cost, we ran the first two of the five frozen starts: 24 episodes covering 2 colors × {vision, oracle state} × {raw, coordinate list, legal list}.

Results:

  • No episode finished.
  • 7 episodes ended on an illegal move. 6 ended on an output-format violation, mostly a correct move wrapped in a markdown code fence, which strict scoring rejects for every model.
  • 11 ended on 120-s relay timeouts. We report these as censored operational stops, not model failures.
  • The longest runs were oracle state with the legal list (22 and 47 plies).
  • In static evaluation, Gemini is clearly stronger (45/120 vs 20/120 Hit@5 at 4,096 visits).

So the closed-loop failure is not unique to the 4B model, but we do not claim that stronger models cannot finish games under other interfaces or budgets.

W2: Terms are not explained. These are now defined at first use:

  • Engine agreement: the model's move is among the engine's top candidates.
  • Probes: tests outside the 300-position benchmark.
  • Controlled adapters: LoRA updates from the same base that differ only in training targets and matched budget.
  • Komi, SGF, capture, ko and pass are defined in the task section.

Q1: Only the image view uses vision. Correct. Images pass through the vision encoder; the matrix and SGF views are text. This difference can contribute to the gap. The views also differ in information (the matrix gives occupancy directly; SGF requires replay but carries history), serialization and length. We therefore present them as paired access conditions, not an isolated vision ablation. The paper now states this.

Q2: Why evaluate Go when AlphaGo exists? We do not aim to measure Go strength. Go gives exact state labels, executable transitions and checkable legality on the same examples. This lets us ask whether a general-purpose model that can report a state can also use and update it. A specialist engine's strength does not answer that question for MLLMs, which are increasingly deployed as agents that must track state. The new results sharpen the point: a model can read 71% of boards exactly yet gain nothing in move choice, and direct update training still fails.

Thank you for the additional experiments and revisions. The closed-loop experiment is helpful. However, my concerns about the experiment settings and readability remain only partially addressed, and I would appreciate further clarification.

  1. Could you provide the original prompts and model responses by an anonymous link?

    Please provide a reviewer-accessible link to the exact prompts and raw responses, ideally including complete logs for models' output. Given the timeouts and formatting-related stops, these logs would help assess whether clearer prompts or output handling could improve completion, and be helpful for understanding models' behavior via case study.

  2. Readability may require more than short definitions.

    Adding brief definitions has not fully resolved the issue. "Engine agreement" still assumes readers know what "engine" means, while "controlled adapters" should explicitly identify what is controlled. Defining "separate probes" simply as "tests outside the 300-position benchmark" does not explain what these tests involve or why they are separate.

    These examples reflect a broader presentation issue: several compact technical labels appear before readers have enough context to interpret them. Please revise the introduction and Figure 1 together so that readers can understand the evaluation structure without reconstructing it from later sections or the appendix. (Also some wording, such as “arm,” “protocol,” “cohort,” and “controlled,” feels AI-generated to me, and I find its meaning unclear in context. Maybe it's better to explain some terms in your own words.)

AuthorsOpenReview ↗

Reply to Reviewer XL7x

Thank you for taking the time to read the revision and for letting us know that the Gemini experiment was helpful. Your follow-up made us see where our first revision fell short. We added definitions, but still left readers to work out how the experiments fit together. We also agree that you should be able to inspect what the model was asked and what it actually returned.

You can now browse all 24 Gemini episodes or download the full records from the anonymous transcript archive. For each request, it shows the exact prompt, the board input, the reply as received, and how we scored it. Across 92 requests, the relay returned 81 replies. The remaining 11 are marked as timeouts.

One example shows why your request matters. In episode 03, Gemini played Black, saw the board image, and received a list of legal moves. The prompt said, “Return strict JSON with the sole key move.” Its full reply was:

```json
{"move": "O19"}
```

O19 was on the supplied legal list. The Markdown fence caused the parser to reject the reply, ending the episode before the move was played. A parser that accepted fenced JSON would have allowed this turn to continue. The log cannot tell us how the rest of that game would have gone.

We will revise the introduction and Figure 1 together around one board position and three questions: Which stones are on the board? Which move would you play next? After a specified move, what should the board look like? The figure will show what the model sees and what counts as a correct answer for each question. In the text, we will introduce KataGo as a Go-playing program before describing how we compare its candidate moves with the model’s choice. We will also say what we held fixed in the main training comparison: the two models start from the same checkpoint and receive a matched budget of training tokens, while one learns from both board descriptions and moves and the other learns from moves alone. Finally, we will explain why we test repeated updates and play through games separately: a model can answer a question about a single position correctly and still lose track of the board after several turns. Your examples helped us pinpoint what a reader needs on the first page.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all reviewers. In the revised PDF, all new or changed text, figures and tables are marked in blue. A new main-text Section 6 summarizes the requested experiments, and Appendix E gives full details. We also narrowed the scope statement and defined the terms on first use.

  1. Three training seeds for all 14 adapters (28 new trainings, 2,400 answers each). Exact board reading for the 20k mixed adapter is 69.3/71.0/73.3%. Its image Hit@5 minus the token-matched control is +0.00, −0.67 and +2.33 points for seeds 42/43/44, and every interval contains zero.
  2. Direct training on state updates. We trained 6 update adapters (2 initializations × 3 seeds, 8,000 windows) and tested them on 600 representative and 240 capture/ko-enriched windows from disjoint games. Exact updates reach only ≈1%.
  3. 1,000 additional games and a rendering-style panel. The main pattern replicates. Learned board reading does not survive a palette change.
  4. Higher-search engine relabeling and move quality (App. E.6–E.8). We use 4,096 and 16,384 visits and equal-budget forced-move evaluation.
  5. A strong fixed policy (KataGo) on true vs. predicted boards. Better reading gives better moves when a strong policy uses it, so the failure lies in the model's own use of its reading.
  6. Closed-loop play with Gemini 3.8 Flash. We ran 24 episodes on two starts, accessed through a third-party relay. No episode finishes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.