acceptodds
Under review as a conference paper at ICLR 2027

Finding the Move Is Not Winning the Game: XiangqiBench for Closed-Loop Evaluation of LLM Agents

Abstract

Static evaluations credit a language model for naming the right move, but an agent must carry a plan through to a verified outcome while an opponent responds. We introduce XiangqiBench, an executable benchmark that measures this difference in Chinese chess: starting from 119 tactical endgames with forced mates supported by engine or checks-only search, an LLM agent must deliver checkmate against an engine defender. An interactive REPL interface separates real moves, state queries, and forward simulation, and we record 8,568 multi-turn trajectories from 12 frontier LLMs under two observation protocols. Three signals that look like competence each overstate closed-loop success. (i) The Conversion Gap: models play the stored reference first move in 26.1% of Sighted trials, yet only 13.9% of these trials end in a win. (ii) The Consistency Gap: the leading model reaches 38.7% pass@3 but only 5.9% pass^3, winning all three trials on 7 of the 46 positions it ever wins. (iii) The Simulation Gap: 32.3% of accepted simulation calls stop on an illegal move, and in 49.3% of comparable cases the real defender replies differently from the line the agent simulated; self-authored rollouts check legality but cannot anticipate the opponent. Finding the move is not winning the game: agent evaluations should score closed-loop outcomes and report reliability alongside coverage.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
agent eval

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_TG7AOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces XiangqiBench, a benchmark that evaluates whether LLM agents can convert winning Chinese chess positions into actual checkmates against an engine. Experiments on 12 LLMs show that identifying the correct first move does not guarantee eventual success, and that performance is often inconsistent across repeated attempts. The results highlight the importance of closed-loop evaluation rather than static move prediction.

Critical strengths

The paper evaluates a broad set of recent LLMs and provides detailed trajectory level analyses of their failures.

Critical weaknesses

Here are several weaknesses:

  1. The paper provides limited new insight into agent reasoning. Prior work has already shown that strong single-turn performance does not guarantee successful multi-turn execution [1], that occasional success can differ substantially from reliable success across repeated trials [2], and that LLM agents struggle with planning and acting in dynamic game environments [3]. Xiangqi provides another evaluation domain, but the paper does not demonstrate why this domain leads to sufficiently new conclusions beyond these established findings.
  1. The benchmark is narrow and potentially biased. It contains only 119 forced mate positions, 116 of which come from a single public collection, while 106 belong to the same threat race category. The lack of newly generated or contamination-controlled cases and broader game situations limits its coverage and generalizability as a benchmark of agent reasoning.
  1. Several findings are partly induced by the evaluation design. The Sighted and Restricted settings change the prompt and multiple available tools simultaneously. The simulation tool also requires the model to invent the opponent’s responses rather than interact with an actual opponent. Consequently, the reported observation and simulation gaps provide limited causal insight into the models’ underlying reasoning abilities.

[1] MINT: Evaluating LLMs in Multi Turn Interaction with Tools and Language Feedback

[2] τ bench: A Benchmark for Tool Agent User Interaction in Real World Domains

[3] BALROG: Benchmarking Agentic LLM and VLM Reasoning on Games

Questions for authors

Is there an anonymized code repository available for reproducing the benchmark and experimental results?

4 comments
AuthorsOpenReview ↗

Response to Reviewer TG7A [1/2]

Thank you for asking what the benchmark contributes beyond established findings about agent performance. We agree that the general knowing–doing gap, game-based evaluation and repeated-trial reliability have clear precedents, and the revision cites MINT and BALROG in Related Work. The case for XiangqiBench rests on a more specific contribution: linking proposed continuations and committed actions to adversarial outcomes from a controlled starting position. This measurement yields findings that the cited benchmarks do not report.

W1. The contribution is a concrete diagnostic protocol

XiangqiBench makes the following measurements within the same interaction:

  1. Which move does the agent actually commit from a search-supported mating position?
  2. Which two-sided continuation does it propose and test in an executable copy of that position?
  3. Does it adopt the tested move, how does the defender actually respond, and does the game end successfully?

The starting position supplies a concrete conversion target, the rules environment makes proposed transitions checkable, and the defender responds independently after commitment. Relative to the cited works:

  • MINT compares success across interaction budgets; we condition on an action within one trajectory. Of the Sighted trials that begin with the stored reference move, GPT-6.1-sol wins 59/156, Gemini 3.1 Pro 62/157 and GPT-6.1-terra 37/172, so most such games are still lost.
  • τ-bench introduced pass^k, which we adopt; here it shows where reliability fails. GPT-6.1-sol reaches pass@3 35.3% but pass^3 5.0%, against 45.8% and 0.6% if its 18.5% per-trial rate held independently on every position, so wins concentrate on particular positions.
  • BALROG agents act from the interaction history; ours can also test a two-sided line on a copy of the board, logged next to the real game. This separates whether the line was legal, whether it was adopted and whether the defender followed it.

This joint measurement supports three findings beyond the cited works:

  • Failures come late, from counterplay. In the 81 positions where the defender threatens mate in one, the four strongest models are checkmated at ply 2 in only 5–10 of their 243 trials each, yet their median failed trial lasts 1.1 times the mate distance. They see the immediate threat but lose the race afterward (W2).
  • A verified hypothetical mate is not a win, although the tool helps. GPT-6.1-sol has 117 Sighted games in which some simulation reports a mate, but only 54 end in a real win. This is a game-level association; the mating line need not have been played. Removing simulate nonetheless lowers wins for both GPT-6.1 models (W3).
  • On-demand board access does not raise conversion. Across six models, Sighted minus Restricted pass@1 ranges from −4.5 to +2.0 points, and no interval lies above zero (W3).

W2. Corpus scope and unresolved contamination

We agree that the corpus is narrow. It contains 119 classical tactical compositions, with 116 from one source and 106 classified as threat races. The threat-race concentration is deliberate. In a threat race, a separate search with Black to move also finds a mate for Black, so Red must keep forcing, and a slow move lets the defender win first. This counterplay is what makes conversion depend on the defender's actual replies: for the four strongest models, 1,033 of 1,089 failed Sighted trials in threat races are lost on the board, against 63 of 137 in the unopposed positions. The category therefore describes counterplay rather than a single tactical motif. Even so, the concentration in one source and one category limits generalization, and category and mate-distance breakdowns do not increase the number of distinct starting positions.

Historical solution text is withheld from prompts and scoring, but prior exposure to published compositions cannot be ruled out. Recalling the first move is nevertheless not enough to win: as shown in W1, 61–80% of the four strongest models' Sighted trials that begin with the stored reference move are still lost. Memorization could still affect later moves, and the reference move need not be the only winning move, so we report reference agreement as observable behavior rather than as evidence of recall.

Our claim is about tactical conversion on this fixed corpus. The harness accepts any FEN position and admission is automated, so such sets can be added without changing the interaction interface.

AuthorsOpenReview ↗

Response to Reviewer TG7A [2/2]

W3. Corrected controls separate tool availability from stronger causal claims

We agree that the original setting comparison changed prompts, tools and feedback together and could not isolate an observation effect. The revision replaces it with two single-factor comparisons, both on newly collected interactions with corrected check feedback and accurate budget instructions:

  • Board access. On all 119 positions, Sighted and Restricted share the same system prompt apart from the view_board entry. Both receive the actual opening board and subsequent move reports, and both retain legal_moves and simulate. Sighted additionally permits view_board; neither automatically refreshes the board afterward. No consistent benefit of board access appears.
  • Simulation. On 60 positions, a separate ablation removes only simulate and its prompt description. Wins fall from 45/180 to 30/180 for GPT-6.1-sol and from 26/180 to 13/180 for GPT-6.1-terra.

The details and the limits of each result follow.

Optional board access

Sol and terra invoke view_board in only 11 and 2 Sighted games, so their small setting differences say little about consulting the board. GPT-5.5 and Gemini invoke it in 200/357 and 146/357 games, yet win 31/357 and 66/357 in Sighted versus 44/357 and 82/357 in Restricted. Query use is self-selected, so we do not compare users with non-users. The result concerns on-demand access under this protocol: Restricted agents still receive the opening board and every move, so it does not show that state tracking is unimportant.

What self-authored simulation tests

The agent supplies both sides of a candidate line, and simulate executes it on a copy of the board; after a real move, the defender chooses its own reply. This is deliberate. Filling in the engine's reply would add an opponent-search oracle and test a different, assisted agent. The design measures two things: whether the agent keeps its imagined position legal (for GPT-6.1-sol, 83% of lines are fully legal at 1–2 plies and 46% at 5–8), and what its own hypothesis is worth against the real defender (W1). A successful line shows success only along the assumed replies, so we report divergence from the defender's actual reply descriptively, conditional on the number of legal replies.

What the simulation intervention establishes

On the 60 positions, pass@1 drops by 8.3 percentage points [1.1, 15.6] for GPT-6.1-sol and 7.2 [0.6, 13.9] for GPT-6.1-terra (95% stratified case-bootstrap intervals); Mistral wins no trial in either arm. The tool is therefore useful even though it does not supply the opponent's reply. The subset was stratified by sol's Restricted outcomes (30 solved, 30 unsolved), not by the Sighted baseline. Reweighted to the 117 eligible positions, the drops are 7.1 and 4.1 points, with intervals that include zero, so the result does not establish a benefit across all positions.

Q. Is an anonymized repository available?

No anonymized repository is currently available during review. Upon acceptance, we will release the repaired harness, the case records, corrected trajectories, defender configurations, and scoring and verification scripts.

Thank you for your thoughtful response and the effort you put into addressing my concerns. My main reservation about XiangqiBench remains. A benchmark should help the community understand where current methods fall short and identify promising directions for follow-up work. I still find it difficult to draw such insights from the reported results, so I will maintain my original score. I appreciate the discussion.

Thank you for following up. Your criterion is that a benchmark should identify limitations of current methods and suggest promising follow-up work. Our revised evidence addresses both: a reference first move often fails to convert, and a simulated mating line can fail against the actual defender even though simulation access improves success on the sampled subset. These results motivate testing defensive-reply coverage and replanning after unexpected responses.

Could you clarify whether you consider these measurements unsupported, already established by the cited work, or valid but insufficiently useful? Your final comment reiterates the overall conclusion without specifying which of these objections applies. We acknowledge the corpus limitations and are not claiming that every concern has been resolved; we would appreciate a specific assessment of the revised evidence.

Reviewer_E6KtOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper evaluates whether LLM agents can convert tactical endgame positions into wins, extending evaluation beyond selecting a reference first move. The benchmark contains 119 cases, twelve models, two observation settings, and three retained attempts per case, for 8,568 trajectories. The paper relates first-move correctness to eventual success and analyzes simulation-tool use, illegal actions, and repeated attempts. Appendices document implementation issues, rule replays, and a corrective ablation on selected cases.

Critical strengths
  1. The evaluation targets a meaningful gap in agent behavior. Measuring terminal success after interaction distinguishes recognizing a promising move from carrying out the required sequence. This makes the endgame setting useful for studying failures that a static move-selection score can miss.

  2. The trajectory analysis goes beyond counting tool calls. The paper distinguishes simulation requests, legal simulated paths, and adoption of simulated moves. This is a useful basis for examining whether access to a tool translates into effective planning. The disclosed protocol problems and corrective analyses also help readers identify the limits of those observations.

Critical weaknesses
  1. The main results were collected with incorrect feedback that can affect decisions. Appendix A.5 states that all 57,000 Check fields shown to agents read No, including the corresponding per-step fields in simulation outputs. The defect also prevents the collection-time consecutive-check rule from firing. Board transitions and move legality remain intact, so the trajectories still describe behavior under the implemented environment. However, agents received incorrect information during interaction. Replay can repair some terminal labels; it cannot recover the decisions that would have been made with correct feedback. The effect may differ between Sighted agents, which can inspect the board, and Restricted agents, which lack continuous board observations. This leaves the main performance gaps and planning diagnoses insufficiently validated for the intended benchmark protocol.

  2. The corrective ablation does not resolve the scope of the primary conclusions. The original settings differ in prompts as well as observations: the Restricted prompt describes a standard starting position, while the Sighted prompt promises a forced move at budget exhaustion although the implementation declares a forfeit. Appendix D corrects feedback and prompt issues, but evaluates two models on model-specific sets of 20 or 21 cases previously solved at least once. These comparisons are informative within the selected cases. They do not establish how correction affects previously unsolved cases or the full benchmark, and changes to prompts, feedback, and the API gateway for one model complicate comparison with the original experiment. The paper needs to separate findings supported within the corrected subset from explanations of the original full-set gap.

Questions for authors
  1. Are corrected-protocol results available for cases chosen independently of prior success? Which main findings remain supported when correct Check feedback and accurate budget instructions are used? If such evidence is unavailable, how would you restrict the main claims to the protocol actually evaluated?
  2. Which conclusions about observations and tools follow from comparisons within the corrected ablation alone? Please make explicit where interpreting the original between-setting gap still depends on the prompt and feedback differences.
3 comments
AuthorsOpenReview ↗

Response to Reviewer E6Kt [1/2]

Thank you for distinguishing the integrity of a recorded trajectory from the validity of the information available when the agent acted. Your central concern is correct: repairing a terminal label cannot recover the decisions an agent would have made with accurate feedback. We have therefore replaced the original main evaluation with new interactions under the corrected protocol, and separated the conclusions supported by each comparison.

W1 / Q1. Corrected results are available for all 119 positions

The revised manuscript includes corrected-protocol runs for six models, with three trials per setting on every one of the 119 positions. The full position set includes previously unsolved cases and was not selected on prior model success. This extends beyond the original selected-case corrective ablation, although it is not a rerun of all twelve original models.

During the new collection, check is computed for the correct side after real moves and throughout simulation. The perpetual-check rule is applied during play. Both settings begin with the actual endgame position, use the same command syntax, and receive instructions that state the implemented forfeit rules. Both retain legal_moves and simulate; Sighted additionally permits view_board. After the opening report, neither receives automatic full-board refreshes. This comparison changes optional board-query availability within the corrected protocol; it does not reconstruct the original observation settings.

The corrected runs give:

ModelSighted winsRestricted wins
GPT-6.1-sol66/35759/357
Gemini 3.1 Pro66/35782/357
GPT-6.1-terra39/35738/357
GPT-5.531/35744/357
GPT-OSS 120B2/3572/357
Mistral Large 30/3573/357

Which findings remain supported?

A reference first move is insufficient for conversion. In Sighted, GPT-6.1-sol wins 59/156 trials that start with the stored reference move, GPT-6.1-terra 37/172, and Gemini 62/157. These counts follow the same trajectories from the first accepted action to the terminal outcome. They support the value of evaluating the continuation rather than ending evaluation at a promising first action. They do not establish that a matching action reflects knowledge of the whole solution.

Occasional and repeated success measure different aspects of performance. GPT-6.1-sol wins at least once on 42/119 positions (pass@3 35.3%), but all three times on only 6/119 (pass^3 5.0%). If its 18.5% per-trial rate held independently on every position, these would be 45.8% and 0.6%, so wins concentrate on particular positions. This reflects differences between positions, not dependence between attempts on the same position.

Simulation access is useful on the sampled ablation subset. The separate intervention described below supports this narrower finding, with population-level uncertainty retained.

Long simulated lines remain error-prone. For GPT-6.1-sol, 83% of simulated lines are fully legal at 1–2 plies but 46% at 5–8. Divergence from the defender's actual reply is now reported descriptively, conditional on the number of legal replies.

The original observation-gap interpretation is withdrawn. The corrected comparisons do not show a consistent positive advantage from optional board queries. This is not evidence of equivalence, and neither condition tests automatic refresh after every move. Restricted agents can reconstruct the position from the opening report and move history. Furthermore, a game ending in checkmate or stalemate does not reveal whether the earlier mistake arose from state tracking, rule application or move selection. We consequently limit the observation claim to optional query availability under the stated protocol.

AuthorsOpenReview ↗

Response to Reviewer E6Kt [2/2]

W2 / Q2. What the original corrected subset supports

In short, the original corrected subset supports only conditional contrasts on the selected cases. Any interpretation of the original between-setting gap still depends on the prompt and feedback differences, so the revision withdraws it rather than attributing it to observation.

The original Appendix D used model-specific sets of 20–21 positions that had already been solved at least once. With automatic refresh retained, removing the query-command bundle reduced pass@1 by 15.0 percentage points for Gemini and 14.3 for GPT-5.5. With the query commands retained, removing automatic refreshes produced drops of 13.3 and 4.8 points, respectively. Without automatic refreshes, removing the query bundle did not lower pass@1 further. These contrasts describe the selected cases and do not establish an unconditional benefit of the query bundle.

These comparisons address different interventions. Removing a query bundle cannot identify the contribution of simulate alone; removing refreshes while allowing queries measures the effect of automatic presentation when information remains obtainable on demand. Neither establishes the result on previously unsolved positions or the full corpus. Nor do they decompose the original between-setting gap into effects of prompts, erroneous feedback, gateway changes and observations. We therefore do not use them to explain how much of the original gap was caused by any one defect.

The new experiments separate two questions:

QuestionPositionsWhat differs between arms
Optional board-query availabilityAll 119view_board availability and its prompt entry; legal_moves and simulate remain available
Simulation availabilityA fixed set of 60simulate availability and its prompt description; view_board and legal_moves remain available

The new board-query result cannot replace the old refresh contrast, because no new arm automatically refreshes the board after each move. Its advantage is a cleaner, full-position comparison for the distinct question of optional query access.

The new simulation ablation and its sampling boundary

The 60 positions were sampled with a fixed seed: 30 previously solved and 30 unsolved by GPT-6.1-sol in Restricted. Selection did not use the Sighted baseline outcomes. It is nevertheless success-stratified rather than an unselected population sample. The two positions requiring the rule-based fallback defender are excluded, leaving 117 eligible positions.

On the selected positions, the main-run Sighted trials provide the baseline:

ModelWith simulationWithout simulationDrop in pass@1, percentage points (95% interval)
GPT-6.1-sol45/18030/1808.3 [1.1, 15.6]
GPT-6.1-terra26/18013/1807.2 [0.6, 13.9]
Mistral Large 30/1800/1800.0 (no wins in either arm)

The intervals resample positions within the two sampling groups, with each position's trials kept together. Reweighting to the eligible population gives drops of 7.1 points [−0.2, 14.5] for sol and 4.1 [−1.3, 9.9] for terra. Those intervals include zero. We therefore distinguish evidence for a benefit on the sampled 60 positions from the less certain population-level effect. The zero-versus-zero Mistral result does not establish an effect for that model.

Simulation also provides factual transition and legality feedback. The intervention therefore evaluates access to the complete tool under this protocol; it does not isolate a particular internal planning mechanism.

The revision contains 4,284 main trials and 540 no-simulation trials. Replaying every record with the rules engine reproduces every message the agent received and every outcome.

Thank you again for your review. Our revised main evaluation covers all 119 positions, including previously unsolved cases. We also distinguish it from the simulation ablation and have withdrawn the original causal interpretation of the observation gap. Could you let us know whether this addresses the scope and validity concerns in your review, or which parts still need clarification?

Reviewer_iEJNOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces XiangqiBench, which evaluates multi-turn tactical conversion in Chinese chess (Xiangqi) across 119 classical endgame compositions. The models are not graded by whether they merely get the first move in the sequence correct, but whether they are able to convert the winning position through a sequence of moves executed through a terminal REPL. The LLM plays Red for up to 40 plies against Pikafish, an engine derived from Stockfish. Twelve models are evaluated over three trials per case under two protocols: "sighted" and "restricted observation".

The paper highlights three gaps between apparent and closed-loop competence:

  1. Consistency Gap: Gemini 3.1 Pro leads "sighted" at 22.1% pass@1 and 38.7% pass@3 but achieves only 5.9% pass^3.
  2. Conversion Gap: across "sighted" trials, 26.1% match the stored engine root move, yet only 13.9% of those trials end in a win.
  3. Simulation Gap: 32.3% of accepted simulations stop on an illegal move, and among 1,677 turns where an agent adopts the root move of a fully legal simulated line, the defender plays a different reply in 49.3%.
Critical strengths
  1. Game puzzles continue to challenge frontier LLMs, and playing puzzles out to the end offers additional signal that separates models that look tied on Move 1. On static first-move agreement in sighted play, Gemini 3.1 Pro and GPT-5.5 are basically tied (45.1% vs. 42.9% in Section 5.2), yet Gemini finishes the win 22.1% of the time while GPT-5.5 wins just 9.0%. That 2.5x split is a clean, concrete example of why one-move puzzle accuracy overstates multi-turn tactical play. Making the model supply both sides' moves inside "simulate" is also well thought out because it tests internal board/legality tracking. Indeed, legality drops from 96.3% at 1 ply to 37.4% at 5–8 plies (Section 6), yet having "simulate" still boosts win rates by 14–15 percentage points on the Appendix D subset (Table 2).

  2. Detailed metrics and breakdowns: metrics/tables/figures look good and are informative, offering insights into various aspects of model performance.

  3. Bug disclosures in the appendices. Appendices A–D are upfront about what went wrong during data collection. They detail the broken Check: No flag, the mismatched Restricted blindfold prompt, and unlogged API parameters. This reviewer appreciates the authors being forthcoming.

Critical weaknesses
  1. While I appreciate the honesty about the issues while running the benchmark, it does mean that the 8k main games come from an un-rerun buggy harness, the decoding settings weren't logged, and no code or dataset is linked.
  • Appendix A.5 acknowledges that is_checking() checked the wrong side after each move, every single real turn and simulate step printed Check: No, even on checks and checkmates. In mating puzzles built around forcing checks, false Check: No feedback inside simulate (and on every move in blindfold Restricted play) is a huge confounding factor and in my opinion invalidates the results as currently presented. Appendix D fixed this only for two models on ~20 puzzles they had already solved at least once in the buggy run, but this is far from a sufficient overall fix.
  • Prompt and command mismatches in both modes: "main restricted" accidentally used a 32-piece starting-board blindfold prompt and dropped the move-1 legal moves (appendix B.2); fixing that in Appendix D shrinks the sighted vs restricted gap to just 6.7 and 1.6 points (Table 2), meaning Section 5.3 mostly measures a prompt mistake (Mistral Large 3 and MiniMax M2.5 forfeited 310/357 and 313/357 Restricted games before Move 1). Going the other way, the sighted prompt mixes function(args) names with bash commands and Qwen 3.7 Max forfeits 25.5% (91/357) of its sighted games for invalid/missing commands vs. just 3.4% in restricted (Tables 8 and 19).
  • No code release and unlogged decoding budgets: temperature, thinking/reasoning budgets, and max tokens were never saved (Appendices A.3, J), and while Appendices A–B refer to "the released environment," there is no repo link and the reproducibility statement makes no commitment to open-sourcing the code, puzzles, or logs. Prompts alone aren't enough to reproduce an interactive engine harness.
  1. Disagreeing with Pikafish on a losing reply isn't necessarily failing to anticipate the opponent. Every puzzle is a forced win for Red, so every Black reply loses. On checks with only 1 legal Black move, any legal simulated reply matches automatically (0% divergence). When Black has several losing defenses, Pikafish tie-breaking counts as "divergence" even if the model tested a reasonable defense. In standard chess, Stockfish often does not choose the maximally tricky defense in losing positions. This is why people have created engines like LeelaOdds bots. All this is to say that the model may be simulating very reasonable or even better defenses even when it is not Pikafish's chosen move. It's certainly fine to track that as a stat, but it should be framed in that context.
Questions for authors
  1. Will you rerun the benchmark with the fixes in place?

  2. Will you open source the full code? Appendices A–B mention "the released environment," yet no anonymous repo is linked and the reproducibility statement doesn't mention releasing code or data. Will you open-source the repaired harness code, the 119 puzzle records, and the 8,568 game logs?

  3. While it can certainly be valuable to have a hand-selected set of 119 positions, why not support larger datasets and partitions?

4 comments
AuthorsOpenReview ↗

Response to Reviewer iEJN [1/2]

Thank you for the careful review. You identified problems that affect what the agents were told while choosing their actions, and therefore the interpretation of the original results. We agree that disclosing these problems and replaying the trajectories are insufficient. The revised evaluation is based on newly collected interactions with the repaired harness.

W1 / Q1. We have rerun the complete position set with corrected feedback

The revised manuscript reports new runs on all 119 positions, with three trials per setting for six models. These runs include positions that the models had never solved previously; selection is not restricted to successful cases from the original experiment. We have not rerun all twelve original models.

The repaired harness computes check for the correct side after real moves and at each simulated ply. The stated perpetual-check rule is applied during play. Both settings begin with the actual endgame position, and the command syntax and budget instructions reflect the implemented behavior. Thus the new agents made their decisions using the corrected feedback. The original 8,568 trajectories remain records of the original protocol and do not supply the empirical evidence for these corrected-protocol conclusions.

Each model–setting combination contains three trials on each of the 119 positions, or 357 trials in total.

ModelSighted pass@1pass@3pass^3Restricted pass@1pass@3pass^3
GPT-6.1-sol18.535.35.016.529.45.9
Gemini 3.1 Pro18.534.55.923.037.010.1
GPT-6.1-terra10.918.55.010.621.04.2
GPT-5.58.716.04.212.324.42.5
GPT-OSS 120B0.60.80.00.60.80.0
Mistral Large 30.00.00.00.81.70.0

All values are percentages.

The two distinctions you found useful remain visible in the corrected data. In Sighted, GPT-6.1-sol wins only 59 of the 156 trials that begin with the stored reference move; Gemini wins 62 of 157. A first-action score would therefore credit many trajectories that subsequently fail. Repeated trials provide another distinction: GPT-6.1-sol wins at least once on 42/119 positions, but wins all three trials on only 6/119. These measurements establish a difference between selecting a promising first move, completing the game, and doing so repeatedly. They do not establish that the model knows a complete solution when it selects the reference move.

W2. The corrected protocol changes the observation question

The incorrect standard-starting-board instruction has been removed. Both settings receive one opening position report containing the actual board, piece list and FEN. Thereafter they receive move reports and can request legal_moves from the first turn. Both use one command in a bash block, and the prompts state the implemented forfeit rules rather than promising a forced move at budget exhaustion. These fixes remove the early forfeits you identified: in Restricted, every model now makes a first move in at least 351 of its 357 trials, including Mistral Large 3 (351/357).

In the corrected setting comparison, both agents retain legal_moves and simulate; Sighted additionally permits view_board. Neither receives automatic board refreshes after the opening. We make this distinction explicit because the new Sighted condition differs from the original automatically refreshed condition. Comparing old and new scores cannot isolate the effect of correcting check feedback, the prompt, or the available tools.

Under the corrected protocol, we do not detect a consistent positive benefit from optional board-query availability. GPT-6.1-sol and GPT-6.1-terra request the board in only 11 and 2 Sighted trials, respectively. GPT-5.5 and Gemini use it in 200/357 and 146/357 trials, yet win 31/357 and 66/357 in Sighted versus 44/357 and 82/357 in Restricted. These results qualify the original observation-gap interpretation; they do not show that board information is unnecessary. Restricted agents retain enough external information to reconstruct the position, and actual query use is self-selected.

The simulation intervention is now separate. On 60 fixed positions, removing only simulate lowers wins from 45/180 to 30/180 for GPT-6.1-sol and from 26/180 to 13/180 for GPT-6.1-terra. This supports a benefit of tool access on the sampled subset. The positions were stratified by sol's Restricted outcomes, and the population-weighted intervals include zero, so we do not generalize the subset result to every position or model. The earlier 14–15-point losses from removing a query bundle while retaining automatic refresh should not be described as an isolated simulation effect.

AuthorsOpenReview ↗

Response to Reviewer iEJN [2/2]

W3 / Q2. Request settings, verification and release

The new records preserve the configured request settings, returned model identities, interactions and terminal outcomes. Temperature, top-p and reasoning effort were left unspecified; provider defaults are therefore an explicit reproducibility limitation, rather than values we can reconstruct retrospectively. The revision comprises 4,284 main trials and 540 no-simulation trials, and all 4,824 records pass offline replay checks. These checks establish fidelity of the recorded interactions; the new collection, rather than replay, addresses the incorrect decision-time feedback.

There is currently no anonymized repository available during review. Upon acceptance, we will release the repaired harness, the 119 case records, corrected trajectories, configurations, and scoring and verification scripts.

W4. Defender disagreement is a descriptive statistic

We agree with your interpretation. A simulated reply can be a reasonable defense even when Pikafish selects another losing continuation. Conversely, when only one reply is legal, any legal prediction agrees automatically. We therefore condition the descriptive disagreement statistic on the number of legal replies and make no claim that a different reply establishes inferior opponent modeling.

For example, GPT-6.1-sol's adopted simulated lines diverge from the actual reply in 173/311 cases where the defender has at least two legal moves. This describes how often the recorded branch differs from subsequent play. It does not rank the quality of the two defenses or identify the reason a game was lost.

Q3. Larger datasets and partitions

As W1 shows, the main gaps are already clear at 119 positions. The harness accepts additional FEN positions and supports admission checks, so extending the dataset does not require a new interaction interface. The present evidence nevertheless concerns 119 classical tactical compositions. The revision adds category and mate-distance breakdowns, but repeated trials and finer breakdowns do not increase the number of distinct starting positions. These descriptive slices do not test generalization to held-out source collections. Generated positions, game-derived endgames and separately held-out sources would be useful tests of generalization. We retain that limitation rather than treating the present sample size as sufficient to establish broader agent-reasoning claims.

Thank you again for your review. We have now completed the corrected evaluation on all 119 positions with six models, directly addressing your request for a full rerun. We would appreciate your assessment of whether this addresses your concerns about the original feedback and protocol errors, and which concerns remain. We are happy to clarify any point in the revised manuscript or our responses.

Thank you for running the full 119-position evaluation under the repaired harness and for transparently scoping back the observation-gap, simulation, and defender-disagreement claims. That addresses my primary validity concern about the original un-rerun logs. I am still reviewing the revised manuscript to weigh the narrowed scope. I appreciate the substantial effort and will factor all of this into my final assessment during the Reviewer–AC discussion.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Overall response: revised manuscript and corrected evaluation (Part 1/2)

We thank all three reviewers for their careful reading and constructive feedback. Reviewer iEJN highlighted the value of following a promising first move through to the game outcome, the detailed diagnostic analyses, and the disclosure of implementation problems. Reviewer E6Kt recognized the importance of evaluating tactical conversion and the usefulness of distinguishing simulation requests, legal candidate lines, and their adoption. Reviewer TG7A noted the model coverage and detailed trajectory-level analysis. We appreciate these assessments, together with the concerns about experimental validity, the interpretation of the findings, and the benchmark's contribution and scope.

The revised manuscript and our point-by-point responses address these concerns through new experiments and a narrower interpretation of what they establish. The principal changes are:

1. New full-set experiments replace the original empirical evidence. We repaired the real and simulated check feedback, applied the stated perpetual-check rule during play, and corrected the initial-position, command-syntax and budget instructions. We then collected three trials per setting on all 119 positions for six models, including previously unsolved positions. This gives 4,284 new main trials, supplemented by 540 no-simulation trials. All 4,824 records pass offline replay checks. The new collection addresses the information available when agents made decisions; replay checks establish the fidelity of the resulting records. We do not treat relabelled original trajectories as corrected-protocol evidence.

2. Board-query availability and simulation access are now examined separately. In the full-set comparison, both settings receive the opening position and subsequent move reports, and retain legal_moves and simulate; Sighted additionally permits view_board. Neither automatically refreshes the board after the opening. We withdraw the original causal interpretation of the observation gap: the corrected comparisons do not show a consistent positive benefit from optional board queries, and they do not establish that board information or state tracking is unimportant. GPT-5.5's mixed-budget totals are descriptive; they do not supply a fixed-budget controlled comparison.

The separate simulation ablation removes only simulate and its prompt description on 60 fixed positions. Wins fall from 45/180 to 30/180 for GPT-6.1-sol and from 26/180 to 13/180 for GPT-6.1-terra. These results support a benefit on the sampled subset. We also report its success-stratified selection and the uncertainty of the estimates reweighted to the 117 eligible positions, whose intervals include zero.

3. The retained findings are tied to observable behavior. In Sighted, GPT-6.1-sol wins 59/156 trials beginning with the stored reference move, and Gemini wins 62/157. Sol wins at least once on 42/119 positions but all three times on only 6/119. These measurements distinguish first-action agreement, terminal conversion and repeated success. We now treat disagreement with the defender's reply descriptively, rather than equating it with poor opponent modeling. Relative to MINT, τ-bench and BALROG, our contribution is the specific protocol linking executable candidate lines, committed actions, independent defender replies and terminal outcomes; we acknowledge the established precedents for the broader findings.

4. The corpus provides a compact, verifiable challenge that remains far from saturated. Our choice of tactical endgames is motivated by the ability to follow a sequence of decisions through to an objectively scored outcome. Each trial starts from a search-supported winning position, and the agent must sustain a successful continuation while the configured defender responds. Scoring depends on a terminal win under the stated rules, with the moves and feedback available for replay.

The task is already demanding within a short tactical horizon: stored mate distances span 3–11 plies, while evaluation allows up to 40. In Sighted, Gemini 3.1 Pro and GPT-6.1-sol each win only 66/357 trials (18.5%). With three attempts per position, they solve 41/119 (34.5%) and 42/119 (35.3%) positions, respectively. Even the highest observed Restricted result, Gemini's, is 23.0% pass@1 and 37.0% pass@3. Thus, for the model with the highest pass@3 in each setting, roughly two-thirds of the positions remain unsolved within three attempts.

AuthorsOpenReview ↗

Overall response: revised manuscript and corrected evaluation (Part 2/2)

For this diagnostic goal, raw instance count alone is not the measure of benchmark value. The current corpus already exposes substantial failures of conversion under checkable rules. Expansion is most useful when it broadens source and tactical coverage or improves the precision of the conclusions. We agree that difficulty does not remove the coverage limitation: the corpus remains concentrated in classical compositions and threat races, and prior exposure is uncontrolled. We therefore scope the present findings to these positions while supporting targeted extensions, rather than treating a larger count by itself as evidence of a stronger benchmark.

5. Reproducibility and scope are stated explicitly. Category and mate-distance breakdowns characterize the existing corpus; repeated trajectories do not substitute for additional distinct positions. There is no anonymized repository available during review. Upon acceptance, we will release the repaired harness, case records, corrected trajectories, configurations, and scoring and verification scripts.

We would be grateful if the reviewers could reassess the revised manuscript in light of the corrected experiments and the clarified scope of the claims. Please let us know which concerns remain unresolved; we are available to respond during the remaining discussion period. Thank you again for the feedback that prompted these substantive revisions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.