acceptodds
Withdrawn from ICLR 2027

From Intuition to Deliberation: Self-Evolving Reasoning for Strategic Card Games

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Strategic card games, characterized by partial observability and dynamic adversarial interactions, serve as critical benchmarks for evaluating the decision-making intelligence of Large Language Models (LLMs). Most LLM-based approaches prioritize policy performance while overlooking explicit reasoning generation. Achieving transparent decision-making typically relies on high-quality Chain-of-Thought (CoT) data; however, such annotations are scarce and costly in strategic card games, whereas prompt-generated rationales are often ungrounded or unreliable. Moreover, incorporating explicit reasoning into discrete decision-making tasks may incur generative overhead and risk performance degradation. To address these challenges, we propose Metis, a self-evolving framework designed to harmonize competitive proficiency with explicit strategic reasoning. Metis comprises three synergistic components: mixed Supervised Fine-Tuning (SFT), structured-reward Reinforcement Learning (RL), and an iterative self-distillation loop. Specifically, mixed SFT alongside structured-reward RL bootstraps initial policy-reasoning alignment and mitigates sparse supervision in discrete action spaces, while the iterative self-distillation loop anchors reasoning logic to high-confidence strategic actions, transforming autonomous expert-level decisions into reliable supervision signals. Extensive experiments on GuanDan and DouDiZhu demonstrate that Metis achieves a synergistic enhancement of competitive card-playing proficiency and explicit strategic reasoning. Our code is available at https://anonymous.4open.science/r/Metis.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Self-Distillation, Strategic Card Games

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_DHKPOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

本文提出了 Metis 框架,旨在提升卡牌游戏中的战略决策能力,并生成明确的推理过程。该框架结合了混合监督微调(SFT)、带有结构化奖励的组相对策略优化(GRPO),以及利用经行动验证的推理轨迹所进行的迭代自蒸馏。

在“掼蛋”和“斗地主”游戏中的实验表明,该框架在对抗基于规则的对手时,胜率分别达到了 78% 和 76%,优于所评估的其他大语言模型(LLM)及训练基线。消融实验考察了奖励组成部分、推理格式以及蒸馏周期等要素。然而,目前的验证主要集中在游戏表现和行动有效性上,对于推理质量或推理忠实度,尚缺乏充分的直接证据。

Critical strengths
  1. The soft-matching reward provides partial credit for matching card types and attributes, offering a practical way to alleviate reward sparsity in large discrete action spaces. The ablation results support its effectiveness.

  2. The self-distillation pipeline filters generated reasoning traces through action consistency and an LLM judge, providing a practical approach to constructing CoT supervision without human annotations.

Critical weaknesses
  1. The reward function focuses on action matching and formatting rather than reasoning correctness. The generated CoT may simply rationalize decisions instead of reflecting genuine strategic reasoning.

  2. The evaluation relies heavily on weak opponents. Metis achieves only a 21% win rate against DouZero (Table 8), questioning its actual strategic competence.

  3. The contribution of self-distillation remains unclear. Without compute-matched baselines, the improvements could simply result from additional training iterations and supervision.

  4. Only 100 matches are evaluated per configuration, without confidence intervals or multi-seed experiments.

  5. The length reward lacks convincing justification. Removing it slightly improves win rates against rule-based opponents in both games (Table 3). More importantly, longer reasoning traces are not necessarily better reasoning.

Questions for authors
  1. Without direct evaluation of CoT correctness, what evidence supports the claim of improved strategic reasoning rather than post-hoc rationalization?

  2. Why is Metis considered competitive when it wins only 21% of games against DouZero (Table 8)?

  3. How can the gains be attributed to self-distillation without controlling for additional training steps and data?

  4. Are the reported gains statistically significant across multiple seeds, given only 100 matches per configuration?

  5. Why is the length reward necessary when removing it improves win rates against rule-based opponents in both games (Table 3)?

Reviewer_4gLEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary
  1. The paper trains Qwen2.5-7B-Instruct to play GuanDan and DouDizhu and write a short rationale before each move. The method, Metis, combines three things: mixed SFT on action-only and CoT samples, GRPO with a reward that soft-matches the teacher's action, and a self-distillation loop. The loop keeps only rationales whose action matches the DanZero/DouZero teacher and that pass an LLM judge.

  2. After 3 cycles, Metis gets 78%/97% on GuanDan and 76%/97% on DouDizhu against rule/random opponents, with about 99% action validity. That beats prompted GPT-4o/Claude and single-pass SFT/GRPO baselines run in reasoning mode.

  3. There are also ablations on CoT format, reward terms, number of cycles and teacher source.

Critical strengths
  1. Table 2 is a useful negative result. With the same data, forcing CoT drops win rate a lot (69% to 38% on GuanDan, 83% to 27% on DouDizhu vs rule), and mixed training only gets part of it back. That's good to know for anyone who assumes CoT comes for free in decision tasks.

  2. The pipeline clearly beats the reasoning-mode baselines (78 vs 44, 76 vs 41 against SFT+GRPO). Prompts, rewards and code are all released, so it should be easy to reproduce.

Critical weaknesses
  1. The transparency claim is never actually evaluated. Only win rate, validity and CoT length are measured. The two case studies in Appendix D have obvious mistakes: saying a straight can beat a bomb, reading Aces as Jokers, and getting wrong who played the red joker. The moves are right but the reasons are wrong, which looks like post-hoc rationalization. Since transparency is the paper's main selling point, this is the main reason for my score.

  2. Metis doesn't really beat action-only models. In Table 7, action-only SFT+GRPO matches it on GuanDan (79% vs 78%), and action-only SFT beats it on DouDizhu (83% vs 76%), even though Metis gets about 3x the training with no matched control. On top of that, the "RL" reward only imitates the teacher's action, the loop is basically STaR/ReST (neither is cited), and every result is a single 100-game run with no CIs.

  3. Some of the data doesn't add up. The "w/o soft-matching" curve is identical in Fig. 2 and Fig. 3, even though those are different games. Table 3 and Table 1 report the exact same validity numbers but different win rates, and DouZero is 88% in Table 7 but 78% in Table 8. These are probably copy errors, but as it stands I can't trust the reward ablation.

Questions for authors
  1. What happened with the duplicated curve in Figs. 2/3 and the Table 1/Table 3 mismatch? What setup was Table 3 run with?

  2. Is there any quantitative check of CoT quality? For example, whether the CoT is consistent with the game state, or whether the action changes when you remove or corrupt the CoT.

  3. How does Metis compare with an action-only version of the same 3-cycle loop, or with SFT+GRPO at matched compute?

  4. Which LLM is the judge? How many samples pass in each cycle? Which checkpoint produced the Appendix D examples?

  5. Report multiple seeds with CIs, and compare against STaR/ReST-style baselines and Mastermind.

Reviewer_bCT6OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Metis, a self-evolving framework that aims to improve strategic decision-making while enabling explicit reasoning in complex card games. Metis combines mixed supervised fine-tuning (SFT), structured-reward GRPO, and iterative self-distillation, where generated reasoning traces are filtered using expert actions and an LLM judge. Experiments on GuanDan and DouDizhu demonstrate substantial improvements over standard SFT and GRPO baselines, with performance peaking after three self-distillation iterations. The results suggest that Metis can maintain strong decision-making performance while generating natural-language reasoning traces.

Critical strengths
  1. Multi-faceted experimental analysis. The paper evaluates the proposed framework from multiple perspectives, including baseline comparisons, ablation studies, and analyses of self-distillation iterations and teacher models, providing useful insights into its effectiveness and behavior.

  2. Interesting empirical observations. The paper presents several interesting findings, such as the negative impact of explicit CoT on decision-making performance and the performance saturation observed during iterative self-distillation.

Critical weaknesses
  1. Limited evaluation and insufficiently controlled comparisons. The main experiments primarily rely on random and rule-based opponents, with limited evaluations against stronger and more diverse strategies. Additionally, the lack of compute-matched baselines makes it difficult to determine whether the improvements across self-distillation iterations come from the proposed mechanism or simply additional training.

  2. Insufficient analysis of empirical observations. Although the paper identifies several interesting phenomena, such as CoT-induced performance degradation and performance saturation during self-distillation, the proposed explanations remain largely speculative and lack direct experimental validation.

Questions for authors
  1. Evaluation against stronger opponents. The main experiments primarily evaluate performance against random and rule-based opponents. Although additional comparisons with DouZero are provided in the appendix, could the authors further evaluate Metis against a more diverse set of strong opponents to better assess its strategic robustness and generalization?

  2. Effectiveness of iterative self-distillation. Since each self-distillation iteration involves additional SFT and GRPO training, could the authors provide comparisons under matched training budgets to clarify whether the observed improvements come from iterative data refinement rather than simply additional training?

  3. Explanation of empirical observations. The paper reports several interesting phenomena, including performance degradation with explicit CoT and performance saturation after three self-distillation iterations. Could the authors provide further analysis or experimental evidence to better explain the underlying mechanisms behind these observations?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.