- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper introduces XiangqiBench, a benchmark that evaluates whether LLM agents can convert winning Chinese chess positions into actual checkmates against an engine. Experiments on 12 LLMs show that identifying the correct first move does not guarantee eventual success, and that performance is often inconsistent across repeated attempts. The results highlight the importance of closed-loop evaluation rather than static move prediction.
Critical strengths
The paper evaluates a broad set of recent LLMs and provides detailed trajectory level analyses of their failures.
Critical weaknesses
Here are several weaknesses:
- The paper provides limited new insight into agent reasoning. Prior work has already shown that strong single-turn performance does not guarantee successful multi-turn execution [1], that occasional success can differ substantially from reliable success across repeated trials [2], and that LLM agents struggle with planning and acting in dynamic game environments [3]. Xiangqi provides another evaluation domain, but the paper does not demonstrate why this domain leads to sufficiently new conclusions beyond these established findings.
- The benchmark is narrow and potentially biased. It contains only 119 forced mate positions, 116 of which come from a single public collection, while 106 belong to the same threat race category. The lack of newly generated or contamination-controlled cases and broader game situations limits its coverage and generalizability as a benchmark of agent reasoning.
- Several findings are partly induced by the evaluation design. The Sighted and Restricted settings change the prompt and multiple available tools simultaneously. The simulation tool also requires the model to invent the opponent’s responses rather than interact with an actual opponent. Consequently, the reported observation and simulation gaps provide limited causal insight into the models’ underlying reasoning abilities.
[1] MINT: Evaluating LLMs in Multi Turn Interaction with Tools and Language Feedback
[2] τ bench: A Benchmark for Tool Agent User Interaction in Real World Domains
[3] BALROG: Benchmarking Agentic LLM and VLM Reasoning on Games
Questions for authors
Is there an anonymized code repository available for reproducing the benchmark and experimental results?
4 comments
Response to Reviewer TG7A [2/2]
W3. Corrected controls separate tool availability from stronger causal claims
We agree that the original setting comparison changed prompts, tools and feedback together and could not isolate an observation effect. The revision replaces it with two single-factor comparisons, both on newly collected interactions with corrected check feedback and accurate budget instructions:
- Board access. On all 119 positions, Sighted and Restricted share the same system prompt apart from the
view_boardentry. Both receive the actual opening board and subsequent move reports, and both retainlegal_movesandsimulate. Sighted additionally permitsview_board; neither automatically refreshes the board afterward. No consistent benefit of board access appears. - Simulation. On 60 positions, a separate ablation removes only
simulateand its prompt description. Wins fall from 45/180 to 30/180 for GPT-6.1-sol and from 26/180 to 13/180 for GPT-6.1-terra.
The details and the limits of each result follow.
Optional board access
Sol and terra invoke view_board in only 11 and 2 Sighted games, so their small setting differences say little about consulting the board. GPT-5.5 and Gemini invoke it in 200/357 and 146/357 games, yet win 31/357 and 66/357 in Sighted versus 44/357 and 82/357 in Restricted. Query use is self-selected, so we do not compare users with non-users. The result concerns on-demand access under this protocol: Restricted agents still receive the opening board and every move, so it does not show that state tracking is unimportant.
What self-authored simulation tests
The agent supplies both sides of a candidate line, and simulate executes it on a copy of the board; after a real move, the defender chooses its own reply. This is deliberate. Filling in the engine's reply would add an opponent-search oracle and test a different, assisted agent. The design measures two things: whether the agent keeps its imagined position legal (for GPT-6.1-sol, 83% of lines are fully legal at 1–2 plies and 46% at 5–8), and what its own hypothesis is worth against the real defender (W1). A successful line shows success only along the assumed replies, so we report divergence from the defender's actual reply descriptively, conditional on the number of legal replies.
What the simulation intervention establishes
On the 60 positions, pass@1 drops by 8.3 percentage points [1.1, 15.6] for GPT-6.1-sol and 7.2 [0.6, 13.9] for GPT-6.1-terra (95% stratified case-bootstrap intervals); Mistral wins no trial in either arm. The tool is therefore useful even though it does not supply the opponent's reply. The subset was stratified by sol's Restricted outcomes (30 solved, 30 unsolved), not by the Sighted baseline. Reweighted to the 117 eligible positions, the drops are 7.1 and 4.1 points, with intervals that include zero, so the result does not establish a benefit across all positions.
Q. Is an anonymized repository available?
No anonymized repository is currently available during review. Upon acceptance, we will release the repaired harness, the case records, corrected trajectories, defender configurations, and scoring and verification scripts.
Thank you for your thoughtful response and the effort you put into addressing my concerns. My main reservation about XiangqiBench remains. A benchmark should help the community understand where current methods fall short and identify promising directions for follow-up work. I still find it difficult to draw such insights from the reported results, so I will maintain my original score. I appreciate the discussion.
Thank you for following up. Your criterion is that a benchmark should identify limitations of current methods and suggest promising follow-up work. Our revised evidence addresses both: a reference first move often fails to convert, and a simulated mating line can fail against the actual defender even though simulation access improves success on the sampled subset. These results motivate testing defensive-reply coverage and replanning after unexpected responses.
Could you clarify whether you consider these measurements unsupported, already established by the cited work, or valid but insufficiently useful? Your final comment reiterates the overall conclusion without specifying which of these objections applies. We acknowledge the corpus limitations and are not claiming that every concern has been resolved; we would appreciate a specific assessment of the revised evidence.
Response to Reviewer TG7A [1/2]
Thank you for asking what the benchmark contributes beyond established findings about agent performance. We agree that the general knowing–doing gap, game-based evaluation and repeated-trial reliability have clear precedents, and the revision cites MINT and BALROG in Related Work. The case for XiangqiBench rests on a more specific contribution: linking proposed continuations and committed actions to adversarial outcomes from a controlled starting position. This measurement yields findings that the cited benchmarks do not report.
W1. The contribution is a concrete diagnostic protocol
XiangqiBench makes the following measurements within the same interaction:
The starting position supplies a concrete conversion target, the rules environment makes proposed transitions checkable, and the defender responds independently after commitment. Relative to the cited works:
This joint measurement supports three findings beyond the cited works:
simulatenonetheless lowers wins for both GPT-6.1 models (W3).W2. Corpus scope and unresolved contamination
We agree that the corpus is narrow. It contains 119 classical tactical compositions, with 116 from one source and 106 classified as threat races. The threat-race concentration is deliberate. In a threat race, a separate search with Black to move also finds a mate for Black, so Red must keep forcing, and a slow move lets the defender win first. This counterplay is what makes conversion depend on the defender's actual replies: for the four strongest models, 1,033 of 1,089 failed Sighted trials in threat races are lost on the board, against 63 of 137 in the unopposed positions. The category therefore describes counterplay rather than a single tactical motif. Even so, the concentration in one source and one category limits generalization, and category and mate-distance breakdowns do not increase the number of distinct starting positions.
Historical solution text is withheld from prompts and scoring, but prior exposure to published compositions cannot be ruled out. Recalling the first move is nevertheless not enough to win: as shown in W1, 61–80% of the four strongest models' Sighted trials that begin with the stored reference move are still lost. Memorization could still affect later moves, and the reference move need not be the only winning move, so we report reference agreement as observable behavior rather than as evidence of recall.
Our claim is about tactical conversion on this fixed corpus. The harness accepts any FEN position and admission is automated, so such sets can be added without changing the interaction interface.