- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
MAFIA 1K, a large-scale social deduction game benchmark comprising 1,000 full-length Mafia game logs across six environments of varying complexity,
Critical strengths
1.Mixture-of-LLMs simulation increases behavioral and lexical diversity, which the authors verify quantitatively.
Critical weaknesses
-
[1] is primarily about KTO-based agentic training, and its accompanying data should not be treated as a benchmark baseline for comparison. Moreover, Table 1 uses only the number of games as the comparison metric. The paper should include more detailed statistics, such as the number of tokens, number of rounds, number of players, and average utterance length per round.
-
Why use LLM-simulated data instead of real-world game logs? LLM-synthesized data inherently suffers from a reality gap. The authors should clarify the data synthesis funnel and disclose the filtering rules and the detailed pipeline. They should also collect a small set of real game logs as a calibration reference.
-
The environment design is flawed. As shown in Table 2, each game still contains only one additional special role, and there are only four special roles in total across all environments. Such a limited setting can hardly support more general conclusions. If roles such as Fool and Jester appeared in the same game, the benchmark would be more challenging.
-
The related work survey is insufficient. The authors claim that “existing benchmarks ... typically restricted to basic two-faction setups without special roles.” This motivation does not hold, as some existing works [2] already include special-role settings in their data.
-
All simulation backbones are 2024 70B-scale non-reasoning models, while the evaluated models include strong reasoning models such as Gemini-2.5-Pro and GPT-5-mini. Because the data generators are weaker than the evaluated models, the Mafia players’ deception may be unsophisticated or follow fixed patterns. This limits the benchmark’s difficulty, so it may not fully measure the ability to detect high-level deception. The authors should discuss this issue, or run a small-scale comparison using a stronger simulation model.
[1] NeurIPS 2025, Multi-agent KTO: Enhancing Strategic Interactions of Large Language Model in Language Game [2 ]EMNLP 2026, Beyond Survival: Evaluating LLMs in Social Deduction Games with Human-Aligned Strategies
Questions for authors
I suggest that the authors evaluate at least one more frontier model, such as Claude Opus or Claude Sonnet. A meaningful benchmark should guide model development, and this work is more likely to become a mainstream evaluation leaderboard if it reveals the weaknesses of frontier models.
3 comments
Response to Reviewer LZxX (2/2)
W3. Environment design
Thank you for raising this point. We realize the role composition may not have been clear from Table 2, and we will make it more prominent: MAFIA 1K contains 6 special roles in total (Doctor, Detective, Vigilante, Fool, Serial Killer, and Jester), and each Expanded game contains 3 special roles, namely the Doctor and Detective of the Standard setup plus one additional role. Each Expanded environment extends the Standard setup along a different axis (Appendix D), so that the effect of each added role on the observer can be examined separately. This led to the role-specific findings in Section 5.2: misinformation (Fool) and a pseudo-Mafia (Serial Killer) make inference harder, while an extra Villager ability (Vigilante) or an elimination-seeking Neutral (Jester) does not. We agree that combining roles such as Fool and Jester in one game would make the benchmark more challenging, and we will include this as an important direction for future work.
W4. Related work and [2]
Thank you very much for pointing us to [2]. We will cite and discuss it, and add it to Table 1. As we understand it, [2] curates televised human Werewolf games and evaluates whether models align with the strategies of the winning faction, while MAFIA 1K provides text-native games with asynchronous dialogue across 6 controlled environments and evaluates observer-style faction inference against fixed ground truth; we see the two as complementary. We also agree that the sentence you quoted was overstated, and we will revise it to state precisely that existing Mafia full-game log datasets are limited to two-faction setups without special roles.
W5. Generator strength and difficulty
We agree that this is an important concern and appreciate you raising it. We hope the following observations are helpful. First, in our data the ability to play and the ability to observe appear to be dissociated (Appendix H): among the simulation backbones, player win rate and observer accuracy are not correlated, and the strongest player is in fact the weakest observer. Second, the logs remain challenging even for strong observers: the frontier models above stay at or below 51.8%, and human participants reach 38.1% on the matched 120 segments. Third, the generators were deliberately excluded from the primary evaluation set (Section 4.1) and are reported separately in Table 3. We will add a discussion of this point to the Limitations.
Once again, we are grateful for your thoughtful review, which has made the paper clearer and more complete. We hope these revisions address your concerns, and if so, we would be truly grateful if you would kindly consider raising your rating. We would be very happy to provide any further analysis during the discussion period.
Response to Reviewer LZxX - Follow-up: Completed and Corrected Table 1 Statistics
As promised, we have completed the extended Table 1 by computing the statistics directly from each publicly released dataset. In doing so, we found that several figures in our earlier table were taken from the original papers and did not follow a common counting convention; for example, the figures for Lai et al. (2023) included its 8 Avalon games. We now compute every statistic from the released data with a single definition, which also changes the figures for Eckhaus et al. (2025) and Ibraheem et al. (2022), and we replace our earlier estimate for MAFIA 1K with exact counts. The table below supersedes the earlier one: utterances count player messages only (system or moderator messages are excluded), cycles are day–night cycles per game where defined, and words are whitespace-delimited.
| Dataset | Game | Games | Players per game | Utterances | Cycles per game | Words per utterance |
|---|---|---|---|---|---|---|
| Stepputtis et al. (2023) | Avalon | 20 | 6 | 2,384 | n/a | 16.8 |
| Eckhaus et al. (2025) | Mafia | 33 | 7–12 (avg. 7.7) | 2,508 | 2.7 | 4.4 |
| Ibraheem et al. (2022)‡ | Mafia | 44 | 5–12 (avg. 9.6) | 1,825 | 3.2 | 4.7 |
| Lai et al. (2023)† | One Night Ultimate Werewolf | 191 | 3–6 (avg. 4.5) | 24,070 | n/a | 7.1 |
| Ye et al. (2025) [1] | Werewolf | 331 | 7–9 (avg. 8.7) | 3,759 | 2.2 | 144.9 |
| MAFIA 1K (ours) | Mafia | 1,000 | 8–10 (avg. 9.1) | 36,814 | 3.2 | 35.2 |
† The One Night Ultimate Werewolf portion of the dataset; the 8 Avalon games are excluded. ‡ Computed from Basic-Human*, which is the data of Ibraheem et al. (2022) converted into our format.
MAFIA 1K contains roughly three times as many games as the largest prior dataset and about 1.5 times as many utterances as the largest prior corpus. This table, together with [2], will replace Table 1 in the revision.
With this addition, we have addressed every point raised in your review: the frontier-model evaluation, the extended dataset statistics, the synthesis pipeline and its calibration against real logs, the environment design, the related work, and the generator strength. We are sincerely grateful for your suggestions, which have made the paper more complete, and we would be truly grateful if you would kindly consider updating your rating in light of these revisions.
Response to Reviewer LZxX (1/2)
We sincerely thank the reviewer for the careful and constructive review, which has helped us improve the paper substantially.
Q. Evaluation of Claude models
Thank you for this valuable suggestion. Following it, we evaluated Claude Opus 5.5 and Claude Sonnet 5.5 on all 7 environments with the same protocol (MafiaAcc, %):
Claude Opus 5.5 sets a new best score in 6 of the 7 environments, yet stays at or below 51.8% everywhere. Moreover, since the final segment of each game contains the publicly announced outcome, we also evaluated the remaining segments, where the Mafia must be inferred from the dialogue and votes alone; there, Claude Opus 5.5 reaches at most 40.9% (further details in our response to Reviewer mqPQ). We fully agree that a meaningful benchmark should reveal the weaknesses of frontier models. We will add both models to Table 3.
W1. Table 1: the role of [1] and detailed statistics
We agree that [1] is primarily a training resource rather than a benchmark, and we apologize for the ambiguity in Table 1. In the revision, we will add a column stating each dataset's original purpose, so that [1] is clearly marked as training data. We have also extended Table 1 as you suggested. Since a "round" is defined differently across games (day–night cycles in Mafia and Werewolf, quests in Avalon, a single night in One Night Ultimate Werewolf), we report cycles where they are defined, and we report words rather than subword tokens so that the counts do not depend on a particular tokenizer.
For the other datasets, the table reports the statistics published in each paper, except the cycles and utterance length for Ibraheem et al. (2022), which we computed from its released data (Basic-Human*). For the remaining publicly released datasets, we are computing the missing statistics directly from the original data and will add them to this table during the discussion period; the complete table will replace Table 1 in the revision.
W2. Simulated logs, the synthesis pipeline, and calibration against real logs
We fully agree that real game logs are invaluable and that LLM-generated data carries a reality gap. We have tried to address this concern in 3 ways.
Why simulation. Released human game logs are relatively small (at most a few hundred games in Table 1) and are mostly limited to basic setups. Our aim of 6 controlled rule configurations, including three-faction games with Neutral roles and 100–400 games each, was unfortunately not attainable through human data collection, and simulation allowed us to vary one rule component at a time while keeping everything else fixed.
Pipeline and filtering. The full pipeline is described in Section 3.2, with the orchestration details and pseudocode in Appendix M, and the code is released. Once the simulation pipeline was finalized, every generated game was included in MAFIA 1K; no game was filtered by outcome, length, or content. We will state this clearly in the revision.
Calibration against real logs. We should have made this much clearer in the paper. MAFIA 1K includes Basic-Human*, which converts the human-played games of Ibraheem et al. (2022) into the same format, and all 24 models and our human participants were evaluated on it. The model rankings agree closely between simulated and real logs: across the 24 models in Table 3, the Spearman correlation between scores on Basic-MoL and on Basic-Human* is 0.93. In addition, we built 9 simple non-LLM observers that use only surface features of the public log (the most-mentioned and most-voted players, a filter that removes players ruled out by public game events, and their combinations). Even the best of them falls 6.2–14.8 pp below GPT-5-mini and GPT-OSS-120B, and the gap is largest on the human-generated logs (full results in our response to Reviewer mqPQ). We will make this calibration much more prominent in the revision.