- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper aims to answer whether equipping LLM players with a structured strategic assistant can improve their team's performance in games. To answer this question, the authors propose an asymmetric cognitive graph (ACG) framework. The proposed framework can track observed game events, players' private beliefs about other players' hidden roles, and public suspicion and trust. Then, a weight scoring rule that utilizes the recording information can evaluate candidate strategies and provide action recommendations to the LLMs. Experimental studies in Avalon and Werewolf show the impact of the proposed methods on game-playing outcomes.
Critical strengths
This paper studies an important and interesting question: whether providing structured strategic assistance to LLM players can improve their team's performance in game playing. By designing a framework that integrates observed outcomes, private beliefs, and public perceptions and conducting experiments across different games, this paper examines the effects of this assistance and the conditions under which it is beneficial.
Critical weaknesses
-
Lacking an ablation test makes it difficult to tell which components (observed outcomes, private beliefs, and public perceptions) are necessary or whether a simpler assistant would work just as well.
-
The experimental results only show the changes in win rates without sufficiently explaining why the proposed structure only helps in some settings and hurts in others. I would expect a clearer explanation to answer the question of "whether providing structured strategic assistance to LLM players can improve their team's performance in game playing".
-
I would expect more comparisons regarding the existing methods, such as MultiMind. Without sufficient comparison with the existing methods, it is hard for me to evaluate the benefits of ACG or the technical contributions of this work.
Questions for authors
-
Can you provide ablation results to show that the proposed structure provides benefits over a simple assistant? Specifically, which components are responsible for the performance improvements in Avalon and the degradation in Werewolf?
-
Can you provide a detailed discussion regarding when the proposed structure improves team performance instead of just listing the change in win rates? Are the negative results caused by incorrect beliefs, ineffective rules, or action overrides?
-
I would expect more clarification regarding the technical difference between the proposed method and methods such as MultiMind. More discussions and comparisons can better support innovation in ACG design techniques.
3 comments
Thank you for your response. However, I think my questions have not yet been fully addressed. Regarding my first comment, I appreciate the authors' addition of comparisons. However, such comparisons do not directly answer my questions, whether a full assistant is better than a simple assistant. Moreover, I would appreciate it if the authors could clarify the evaluation metrics, as I have no direct research experience on this topic. The current results are really hard to interpret. As per my second comment, I expect a detailed explanation rather than a table listing why those changes occur. As for my last question, I want to help understand why this paper's method/structure is beneficial. Listing the differences does not help me to appreciate the technical innovation and contribution. Given my broad familiarity with this field and no direct research experience on this topic, I retain my current score.
Thank you for reading our response and clarifying your concerns. We respect your assessment and recognize that our reply should have explained the findings more clearly, rather than mainly adding comparisons.
Briefly, ACG aims to distinguish what a player privately believes from what others publicly support, and connect both to action. Our positive evidence is specific: graph-formatted inputs improved factual-reference accuracy in our test, while better quest progress did not translate into more final wins. The current controls do not establish that the full assistant outperforms a simpler one.
Your feedback highlights the need for concrete examples and accessible metrics. Our broader goal is to understand when structured assistance can turn individual evidence into successful team decisions. Your comments will help us communicate that goal and the supporting evidence more clearly.
Thank you again for your time and constructive discussion, also your academic insights and the time and effort you have devoted to our work.
Thank you so much for your careful review and helpful suggestions. We added component controls, phase-level outcomes, and a direct technical comparison with MultiMind.
1. What improves in Avalon, and why can final wins differ?
The separate Avalon cohort has 50 selected valid completions per arm and distinguishes three outcome stages:
Exploratory Full-minus-baseline differences are +15.1pp for quest-success fraction, +27.2pp for rejecting Evil-containing teams, and -16.7pp for all-Good approval. Figure 6 reports all seven endpoints and uncertainty. These counts locate where progress is lost; they do not prove that ACG exposes Merlin. Full has no established terminal advantage.
A same-state test improves factual-citation matching by 4.61pp (111 pairs, 29 seeds; Holm-3 p=0.00018), but not Brier or vote safety, with longer prompts. Factual binding, progress, and winning are distinct outcomes.
2. Do beliefs, rules, or overrides explain the Werewolf decline?
We tested 20 midgame checkpoints, one per seed, from previously inspected, valid newer Good-only games. Five prompt conditions vary belief updates, public perception, and advice; no-controller reuses Full's response. All 100 generations are protocol-valid. Other votes remain fixed; only the current ballot is resolved. This is not independent confirmation or new full games.
One checkpoint traces a positive mechanism: the LLM proposes a Good target; belief updates and control redirect the vote to a wolf, turning a 3-3 tie into a 4-2 wolf exile. Removing updating or control removes this effect; hiding advice or removing public perception does not.
This is a local execution benefit. All four planned comparisons have raw and Holm-adjusted p=1.0; raw votes match across prompts in 19/20 states. Belief-update Brier differences are inconclusive. A census of 25 valid source games finds 19 vote overrides: nine change Good targets to wolves, eight do the reverse, and two preserve alignment.
Appendix H.4 reports all diagnostics, including the incomplete 160-terminal campaign. They test local pathways but do not identify a single cause of historical terminal decline.
3. What is technically different from MultiMind?
MultiMind uses multimodal cues, learned theory-of-mind, and 500-iteration MCTS in five-player One Night Ultimate Werewolf with role swaps. ACG uses explicit evidence updates, separate private beliefs and public support, local role scores, and optional execution control in fixed-role games, without base-LLM fine-tuning.
Related Work and Appendix A.3 compare these designs. Our five-player exploration changed ACG's role-transition model and voting objective, preventing a fair ranking. Appendix H documents historical E2 input risks and the separate Avalon cohort's recovery limits; those results are not pooled.