- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper studies whether LLMs adapt when game objectives or rules change. It evaluates Qwen3-8B and Qwen3-14B on classic, misère, and novel board games using a shared GGP referee. Results suggest that reasoning improves action validity and recovery from errors, but does not consistently improve competitive performance.
Critical strengths
The common GDL referee provides a controlled evaluation environment with consistent legality checks and game semantics. Separating competitive performance from invalid-action recovery and inference cost is also sensible.
Critical weaknesses
Here are several weaknesses:
- The evaluation is limited to two models from the same family, neither of which represents the current state of the art. It covers only two sizes of Qwen3, two classic–misère game pairs, and one original game. As a result, the paper is neither sufficiently comprehensive as a benchmark nor sufficiently novel as a methodological contribution.
- The claimed “objective-adaptation gap” is not convincingly established. It is inferred mainly from lower win rates on the misère variants, but Table 1 shows that opponent strength also changes substantially across games.
- The comparison between Instruct and Reasoning is confounded by the generation budget: the former is limited to 128 tokens, while the latter receives 1,024 tokens. This is particularly problematic because many failures of Qwen3-14B Instruct are explicitly attributed to truncation.
Questions for authors
- Do the robustness differences remain when Instruct and Reasoning use the same generation budget?
- Can the authors provide direct evidence that the models choose moves appropriate for the classic objective but inappropriate for the inverted objective?
- Do the main findings hold for stronger models or models from other families?
2 comments
Thank you for your response and for the effort you put into addressing my concerns. I believe a benchmark should offer insight beyond reporting which model achieves which score on each task. The results presented here give the community little guidance on why certain tasks remain difficult or where follow-up work would be most useful. The evaluation also covers only two models, which makes it difficult to tell whether the findings generalize across approaches. For these reasons, I will maintain my original score.
Response to Reviewer sjTD: Robustness, Move-Level Evidence, and Model Generalization
We thank the reviewer for highlighting important aspects regarding the interpretation and scope of the study. The questions help distinguish more clearly between the results already supported by the current protocol and those that could be further examined through additional controls or a broader set of evaluated models.
Q1. Do the robustness differences remain when Instruct and Reasoning use the same generation budget?
The study evaluates Instruct and Reasoning as complete inference configurations. Instruct is intended to produce direct responses without an explicit reasoning phase and therefore uses fewer tokens, whereas Reasoning is given a larger budget to support that phase. This setup allows both modes to be compared under conditions consistent with their intended use. An additional analysis with matched generation budgets could complement these results by specifically controlling the maximum generation length. Similarly, the experiments could be repeated using the provider-recommended configuration for each mode while keeping the rest of the protocol unchanged. In addition, Qwen3-8B Instruct and Qwen3-14B Instruct use exactly the same configuration and still show clearly different robustness profiles. This indicates that the observed behavior does not depend only on the generation budget.
Q2. Can the authors provide direct evidence that the models choose moves appropriate for the classic objective but inappropriate for the inverted objective?
The current results mainly capture behavioral changes at the match level. The classic–misère pairs preserve the board, action space, legality rules, and state transitions while changing the success criterion, but a reduction in win rate alone does not identify which specific decisions produce that difference. The logs allow us to examine this point directly. We can analyze missed immediate winning opportunities in TicTacToe/Connect4, avoidable completions of the model’s own lines in Suicide/NotConnect4, and the proportion of actions consistent with the active objective. This would provide move-level evidence of a possible carry-over from the classic objective and complement the aggregate results without attributing the observed behavior to a specific causal mechanism.
Q3. Do the main findings hold for stronger models or models from other families?
The current study is not intended to establish broad generalization across model families. We used two Qwen3 sizes to maintain a controlled experimental setting and compare configurations under the same protocol. However, we have conducted preliminary tests with additional models. In the misère games, we also observed a decrease in performance relative to their corresponding classic variants. In Centralis, at least one of the frontier models tested achieves a high win rate, showing that a model outside the main experimental configurations can understand and play the new rule system effectively. These results are preliminary and are not used to support the current conclusions. Including at least one frontier model in an extended version would allow us to examine more directly which patterns persist beyond the Qwen3 family and which may depend on model capability or inference configuration.