- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This is careful, well-controlled benchmark work with genuinely useful negative results and a methodology I would like other benchmark papers to imitate strict scoring, token-matched controls, syntax vs legality separation, separate counting of "what the model does with the state." The idea of measuring perception, decision, and update separately on one state is a real contribution.
Critical strengths
- The decomposition is the contribution, and it is well chosen. Reading / choosing / updating are routinely conflated in multimodal-agent evaluation, and this benchmark separates them on identical states with identical accounted denominators. As evaluation-design work this is clean, motivated, and portable in spirit. 2.The findings are sharp, quotable, and consequential. "Recognition can coexist with failed state update, and a rule engine given the same readings fixes it entirely" is a crisp localisation of what is missing. "Legal decoding lengthens games without finishing any, and no arm ever passes" is an important caution against reading constrained-decoding improvements as capability gains. 3.Strict scoring done properly. Invalid is failure, no repairs, no answer ever dropped from a denominator. This is exactly the discipline the field needs and it is what makes several of the results legible.
Critical weaknesses
- There are no error bars anywhere in the training study. Statistical power is the paper's central weakness, and it bears directly on its headline claims. 2.The closed-loop probe nominally has 120 episodes, but 92 end on an invalid action within ~4 plies on average and 26 fail initialisation; even with a legal-move trie the episodes run to a 600 s cap with no pass. 3.Teacher strength and metric leniency. Labels come from 64-visit KataGo. That is a comparatively weak/noisy teacher, and Hit@5 is scored against candidate lists holding five moves on 182/300 positions. Random-legal is quoted at 1.95%, which is a useful anchor, but the effective difficulty is much closer to "top-5 over the engine's shortlist" than to "best move of 361."
Questions for authors
1.Since only 3/233 changed points are captures, do you plan a capture/ko-heavy tracking split? Without it, "updates fail" may not generalise to the interesting part of Go dynamics. 2.Please re-verify Hit@5 rankings on a subset with substantially higher visit counts, and report whether adapter orderings are stable.
Response to Reviewer XTKf
W1: No error bars or seeds. Every arm now has three seeds, with the per-seed values, seed SD and paired source-game intervals reported separately (App. E.1). The headline contrast holds in every seed: exact board +69 to +73 points, and image Hit@5 versus the token control +0.00, −0.67 and +2.33 points, each with a 95% CI containing zero. Clusters are source games, never repeated views or windows.
W2: Closed-loop episodes are short and censored. We agree that they are not 120 independent estimates of playing strength. The reanalysis accounts for all 280 historical episodes and replays all 4,665 accepted plies. It separates invalid initialization (26), invalid actions (92), ledger and wall-time stops, and late responses.
We read this as "valid execution did not yield completion within the budget", not as inevitable non-completion or as losses. Intervals resample the five starts. The new Gemini subset (App. E.4) is reported with its own censoring categories.
W3/Q2: Weak teacher and lenient Hit@5. We relabeled a frozen, phase/side-balanced subset of 120 games at 64 and 4,096 visits, and 24 of them additionally at 1,024 and 16,384 visits, reusing the same model answers (App. E.6–E.8, Tables 27–31).
Fine orderings of nearby adapters can change, so we no longer claim a strict ranking. The mixed-versus-control comparison is stable.
Q1: A capture/ko-heavy tracking split. Added: 240 windows from 102 games (80 single-capture, 80 multi-capture, 40 ko, 40 controls; 312 captured stones).
So update failure does extend to capture dynamics.