acceptodds
Under review as a conference paper at ICLR 2027

MAFIA 1K: A Large-Scale Benchmark for Social Deductive Reasoning from Simulated Game Logs

Abstract

Large language models (LLMs) have demonstrated strong reasoning capabilities across diverse domains, yet their ability to reason about social contexts - where information is asymmetrically distributed and some participants are deliberately deceptive - remains understudied and difficult to evaluate. Social deduction games provide a well-controlled setting for this challenge, but existing benchmarks based on full-length game logs are limited in scale and environmental complexity. We introduce MAFIA 1K, a large-scale social deduction game benchmark comprising 1,000 full-length Mafia game logs across six environments of varying complexity, generated via asynchronous LLM-based social simulations using a Mixture-of-LLMs (MoL) scheme that increases behavioral diversity while maintaining faction balance. The benchmark adopts an observer-style evaluation in which models predict Mafia faction membership from public game logs, isolating social deductive reasoning from confounding factors inherent to interactive gameplay. Our evaluation of 24 LLMs shows that the best model reaches an overall assessment of 43.5%, with top models performing on par with human observers on the same task, and that reasoning models consistently outperform non-reasoning models in every environment. A same-task human evaluation shows that humans achieve meaningful accuracy on both LLM-generated and human-generated logs, with longer response times associated with higher accuracy. An information-source ablation further shows that models rely on both dialogue and voting channels, with different models weighing these channels differently. We release the dataset, evaluation code, and human evaluation data at: https://anonymous.4open.science/r/Mafia-1K-C9B5

Primary area:
datasets and benchmarks
Keywords:
LLM Evaluation, Benchmark, Social Deductive Reasoning, Deception Detection, Social Deduction Games, LLM-Based Social Simulation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_LZxXOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

MAFIA 1K, a large-scale social deduction game benchmark comprising 1,000 full-length Mafia game logs across six environments of varying complexity,

Critical strengths

1.Mixture-of-LLMs simulation increases behavioral and lexical diversity, which the authors verify quantitatively.

Critical weaknesses
  1. [1] is primarily about KTO-based agentic training, and its accompanying data should not be treated as a benchmark baseline for comparison. Moreover, Table 1 uses only the number of games as the comparison metric. The paper should include more detailed statistics, such as the number of tokens, number of rounds, number of players, and average utterance length per round.

  2. Why use LLM-simulated data instead of real-world game logs? LLM-synthesized data inherently suffers from a reality gap. The authors should clarify the data synthesis funnel and disclose the filtering rules and the detailed pipeline. They should also collect a small set of real game logs as a calibration reference.

  3. The environment design is flawed. As shown in Table 2, each game still contains only one additional special role, and there are only four special roles in total across all environments. Such a limited setting can hardly support more general conclusions. If roles such as Fool and Jester appeared in the same game, the benchmark would be more challenging.

  4. The related work survey is insufficient. The authors claim that “existing benchmarks ... typically restricted to basic two-faction setups without special roles.” This motivation does not hold, as some existing works [2] already include special-role settings in their data.

  5. All simulation backbones are 2024 70B-scale non-reasoning models, while the evaluated models include strong reasoning models such as Gemini-2.5-Pro and GPT-5-mini. Because the data generators are weaker than the evaluated models, the Mafia players’ deception may be unsophisticated or follow fixed patterns. This limits the benchmark’s difficulty, so it may not fully measure the ability to detect high-level deception. The authors should discuss this issue, or run a small-scale comparison using a stronger simulation model.

[1] NeurIPS 2025, Multi-agent KTO: Enhancing Strategic Interactions of Large Language Model in Language Game [2 ]EMNLP 2026, Beyond Survival: Evaluating LLMs in Social Deduction Games with Human-Aligned Strategies

Questions for authors

I suggest that the authors evaluate at least one more frontier model, such as Claude Opus or Claude Sonnet. A meaningful benchmark should guide model development, and this work is more likely to become a mainstream evaluation leaderboard if it reveals the weaknesses of frontier models.

3 comments
AuthorsOpenReview ↗

Response to Reviewer LZxX (1/2)

We sincerely thank the reviewer for the careful and constructive review, which has helped us improve the paper substantially.


Q. Evaluation of Claude models

Thank you for this valuable suggestion. Following it, we evaluated Claude Opus 5.5 and Claude Sonnet 5.5 on all 7 environments with the same protocol (MafiaAcc, %):

ModelBasic-Human*Basic-MoLStandard-MoLVigilanteFoolSerial KillerJester
Claude Opus 5.538.049.949.551.846.539.947.1
Claude Sonnet 5.535.249.041.843.843.835.442.0
Best in Table 340.949.044.446.639.935.843.4

Claude Opus 5.5 sets a new best score in 6 of the 7 environments, yet stays at or below 51.8% everywhere. Moreover, since the final segment of each game contains the publicly announced outcome, we also evaluated the remaining segments, where the Mafia must be inferred from the dialogue and votes alone; there, Claude Opus 5.5 reaches at most 40.9% (further details in our response to Reviewer mqPQ). We fully agree that a meaningful benchmark should reveal the weaknesses of frontier models. We will add both models to Table 3.


W1. Table 1: the role of [1] and detailed statistics

We agree that [1] is primarily a training resource rather than a benchmark, and we apologize for the ambiguity in Table 1. In the revision, we will add a column stating each dataset's original purpose, so that [1] is clearly marked as training data. We have also extended Table 1 as you suggested. Since a "round" is defined differently across games (day–night cycles in Mafia and Werewolf, quests in Avalon, a single night in One Night Ultimate Werewolf), we report cycles where they are defined, and we report words rather than subword tokens so that the counts do not depend on a particular tokenizer.

DatasetGameGamesPlayers per gameUtterancesCycles per gameWords per utterance
Stepputtis et al. (2023)Avalon2062,384n/a–
Eckhaus et al. (2025)Mafia337–123,593––
Ibraheem et al. (2022)Mafia449.6 (avg.)2,1623.24.7
Lai et al. (2023)Werewolf / Avalon1994–626,647n/a–
Ye et al. (2025) [1]Werewolf3317 or 9–––
MAFIA 1K (ours)Mafia1,0008–10~36.8K2.5–3.434.3–35.8

For the other datasets, the table reports the statistics published in each paper, except the cycles and utterance length for Ibraheem et al. (2022), which we computed from its released data (Basic-Human*). For the remaining publicly released datasets, we are computing the missing statistics directly from the original data and will add them to this table during the discussion period; the complete table will replace Table 1 in the revision.


W2. Simulated logs, the synthesis pipeline, and calibration against real logs

We fully agree that real game logs are invaluable and that LLM-generated data carries a reality gap. We have tried to address this concern in 3 ways.

Why simulation. Released human game logs are relatively small (at most a few hundred games in Table 1) and are mostly limited to basic setups. Our aim of 6 controlled rule configurations, including three-faction games with Neutral roles and 100–400 games each, was unfortunately not attainable through human data collection, and simulation allowed us to vary one rule component at a time while keeping everything else fixed.

Pipeline and filtering. The full pipeline is described in Section 3.2, with the orchestration details and pseudocode in Appendix M, and the code is released. Once the simulation pipeline was finalized, every generated game was included in MAFIA 1K; no game was filtered by outcome, length, or content. We will state this clearly in the revision.

Calibration against real logs. We should have made this much clearer in the paper. MAFIA 1K includes Basic-Human*, which converts the human-played games of Ibraheem et al. (2022) into the same format, and all 24 models and our human participants were evaluated on it. The model rankings agree closely between simulated and real logs: across the 24 models in Table 3, the Spearman correlation between scores on Basic-MoL and on Basic-Human* is 0.93. In addition, we built 9 simple non-LLM observers that use only surface features of the public log (the most-mentioned and most-voted players, a filter that removes players ruled out by public game events, and their combinations). Even the best of them falls 6.2–14.8 pp below GPT-5-mini and GPT-OSS-120B, and the gap is largest on the human-generated logs (full results in our response to Reviewer mqPQ). We will make this calibration much more prominent in the revision.

AuthorsOpenReview ↗

Response to Reviewer LZxX (2/2)

W3. Environment design

Thank you for raising this point. We realize the role composition may not have been clear from Table 2, and we will make it more prominent: MAFIA 1K contains 6 special roles in total (Doctor, Detective, Vigilante, Fool, Serial Killer, and Jester), and each Expanded game contains 3 special roles, namely the Doctor and Detective of the Standard setup plus one additional role. Each Expanded environment extends the Standard setup along a different axis (Appendix D), so that the effect of each added role on the observer can be examined separately. This led to the role-specific findings in Section 5.2: misinformation (Fool) and a pseudo-Mafia (Serial Killer) make inference harder, while an extra Villager ability (Vigilante) or an elimination-seeking Neutral (Jester) does not. We agree that combining roles such as Fool and Jester in one game would make the benchmark more challenging, and we will include this as an important direction for future work.


W4. Related work and [2]

Thank you very much for pointing us to [2]. We will cite and discuss it, and add it to Table 1. As we understand it, [2] curates televised human Werewolf games and evaluates whether models align with the strategies of the winning faction, while MAFIA 1K provides text-native games with asynchronous dialogue across 6 controlled environments and evaluates observer-style faction inference against fixed ground truth; we see the two as complementary. We also agree that the sentence you quoted was overstated, and we will revise it to state precisely that existing Mafia full-game log datasets are limited to two-faction setups without special roles.


W5. Generator strength and difficulty

We agree that this is an important concern and appreciate you raising it. We hope the following observations are helpful. First, in our data the ability to play and the ability to observe appear to be dissociated (Appendix H): among the simulation backbones, player win rate and observer accuracy are not correlated, and the strongest player is in fact the weakest observer. Second, the logs remain challenging even for strong observers: the frontier models above stay at or below 51.8%, and human participants reach 38.1% on the matched 120 segments. Third, the generators were deliberately excluded from the primary evaluation set (Section 4.1) and are reported separately in Table 3. We will add a discussion of this point to the Limitations.


Once again, we are grateful for your thoughtful review, which has made the paper clearer and more complete. We hope these revisions address your concerns, and if so, we would be truly grateful if you would kindly consider raising your rating. We would be very happy to provide any further analysis during the discussion period.

AuthorsOpenReview ↗

Response to Reviewer LZxX - Follow-up: Completed and Corrected Table 1 Statistics

As promised, we have completed the extended Table 1 by computing the statistics directly from each publicly released dataset. In doing so, we found that several figures in our earlier table were taken from the original papers and did not follow a common counting convention; for example, the figures for Lai et al. (2023) included its 8 Avalon games. We now compute every statistic from the released data with a single definition, which also changes the figures for Eckhaus et al. (2025) and Ibraheem et al. (2022), and we replace our earlier estimate for MAFIA 1K with exact counts. The table below supersedes the earlier one: utterances count player messages only (system or moderator messages are excluded), cycles are day–night cycles per game where defined, and words are whitespace-delimited.

DatasetGameGamesPlayers per gameUtterancesCycles per gameWords per utterance
Stepputtis et al. (2023)Avalon2062,384n/a16.8
Eckhaus et al. (2025)Mafia337–12 (avg. 7.7)2,5082.74.4
Ibraheem et al. (2022)‡Mafia445–12 (avg. 9.6)1,8253.24.7
Lai et al. (2023)†One Night Ultimate Werewolf1913–6 (avg. 4.5)24,070n/a7.1
Ye et al. (2025) [1]Werewolf3317–9 (avg. 8.7)3,7592.2144.9
MAFIA 1K (ours)Mafia1,0008–10 (avg. 9.1)36,8143.235.2

† The One Night Ultimate Werewolf portion of the dataset; the 8 Avalon games are excluded. ‡ Computed from Basic-Human*, which is the data of Ibraheem et al. (2022) converted into our format.

MAFIA 1K contains roughly three times as many games as the largest prior dataset and about 1.5 times as many utterances as the largest prior corpus. This table, together with [2], will replace Table 1 in the revision.

With this addition, we have addressed every point raised in your review: the frontier-model evaluation, the extended dataset statistics, the synthesis pipeline and its calibration against real logs, the environment design, the related work, and the generator strength. We are sincerely grateful for your suggestions, which have made the paper more complete, and we would be truly grateful if you would kindly consider updating your rating in light of these revisions.

Reviewer_nwMjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces MAFIA 1K, a benchmark for social deductive reasoning consisting of 1,000 LLM-simulated Mafia games across six environments. Models act as third-party observers and predict Mafia membership from public dialogue and voting logs. The paper evaluates 24 LLMs and additionally studies human performance, reasoning vs. non-reasoning models, information-source ablations, and simulation diversity.

Critical strengths
  1. The benchmark is substantially larger and more diverse than prior social-deduction datasets, covering asynchronous interactions and multiple special-role environments.

  2. The experimental evaluation is broad, including 24 LLMs, human evaluation, dialogue/voting ablations, and several robustness analyses.

  3. The paper is generally well organized and provides detailed benchmark construction and evaluation protocols.

Critical weaknesses
  1. Human evaluation uses only 120 sampled logs, while the LLM scores in Table 4 come from the full benchmark rather than the same instances. This weakens the “on par with humans” claim, especially since slower human participants are competitive with or better than the strongest LLMs.

  2. All 1,000 games are generated by LLM agents with structured role prompts and reasoning guidelines. The current checks do not fully rule out simulator-specific behavioral or semantic shortcuts.

  3. The reasoning vs. non-reasoning comparison is not fully controlled. Reasoning mode may also improve format compliance and use more inference compute, so the gains cannot be attributed solely to better social reasoning.

Questions for authors

Please address weakness 1-3.

3 comments
AuthorsOpenReview ↗

Response to Reviewer nwMj (1/2)

We thank the reviewer for the positive assessment of the benchmark's scale, the breadth of the evaluation, and the presentation. We address the three weaknesses in order, with the key result stated first in each.


W1. Human evaluation on matched items

Result: on the identical 120 segments, both GPT-OSS-120B and GPT-5-mini outperform the human mean (+21.9 and +18.2 pp; paired 95% CIs exclude zero).

The LLM numbers in Table 4 were the environment-wide scores from Table 3. We recomputed the comparison on exactly the 120 segments shown to participants (identical inputs), with a paired item-level bootstrap (10,000 resamples; MafiaAcc, %):

ModelEnvironmentItemsHumanModelDifference95% CI
GPT-OSS-120BBasic-Human*3040.355.0+14.7[+3.0, +26.7]
Basic-MoL3043.773.3+29.7[+16.7, +41.3]
Standard-MoL6034.255.8+21.7[+13.2, +30.2]
All12038.160.0+21.9[+15.8, +28.1]
GPT-5-miniBasic-Human*3040.360.0+19.7[+8.0, +32.0]
Basic-MoL3043.748.3+4.7[−9.3, +18.3]
Standard-MoL6034.258.3+24.2[+16.2, +32.2]
All12038.156.3+18.2[+11.8, +24.4]

Regarding the slower participants: on these same segments, the slower half of human responses (Table 4: 48.6 / 48.6 / 40.3 on Basic-Human*, Basic-MoL, Standard-MoL) is below both models in every environment, except GPT-5-mini on Basic-MoL, where the two are tied (48.3 vs. 48.6). We will replace Table 4 with this matched comparison and revise the "on par" claim to the matched statement that GPT-OSS-120B and GPT-5-mini outperform human observers on the 120 sampled segments.


W2. Simulator-specific shortcuts

Result: simple observers that exploit surface regularities fall well below the models, the margin is largest on human-generated logs, and the strongest models stay well ahead even on segments without terminal information.

(a) Simple baselines. We implemented 9 non-LLM observers that use only the public log, counting either over the latest cycle in the segment or cumulatively from the start of the game:

  • Most-mentioned: the 2 players mentioned most often in the dialogue.
  • Most-voted: the 2 players who received the most votes.
  • Game rules: exclude players announced as killed by the Mafia or revealed as the Jester, then choose uniformly among the rest.
  • Game rules + heuristics: apply the game rules first, then the most-mentioned or most-voted heuristic among the remaining candidates.

Even taking the best of the 9 baselines in each environment, a post hoc choice that favors the baselines (MafiaAcc, %):

EnvironmentBest baselineGPT-5-miniGPT-OSS-120B
Basic-Human*26.140.935.6
Basic-MoL32.444.038.8
Standard-MoL29.142.535.3
Vigilante33.444.742.7
Fool30.838.637.4
Serial Killer27.335.834.2
Jester33.441.140.2

The largest margin appears on Basic-Human*, which consists of human-generated games and therefore contains no simulator-specific cues of any kind. For the results of each of the 9 baselines, please also see our response to Reviewer mqPQ.

(b) Segments without terminal information. The final segment of each game contains the announced outcome, so we also compared the observers on non-terminal segments only, where any advantage must come from the dialogue, votes, and in-game events. In addition to the two models above, we include Claude Opus 5.5, which we newly evaluated on the full benchmark during the discussion period (also in response to Reviewer LZxX) (MafiaAcc, %):

EnvironmentRandomBest of nine baselinesClaude Opus 5.5GPT-5-miniGPT-OSS-120B
Basic-Human*20.723.529.127.620.9
Basic-MoL20.027.640.934.328.7
Standard-MoL22.226.640.233.726.0
Vigilante25.031.140.334.033.4
Fool22.230.139.630.329.9
Serial Killer22.226.932.029.627.9
Jester22.229.738.632.531.7

Claude Opus 5.5 exceeds the best baseline by 5.1–13.6 pp in all seven environments, and GPT-5-mini is above all nine baselines in all seven environments. The margin over simple observers grows with the observer's capability, which is what one expects if the scores reflect inference over the dialogue rather than shortcuts in the simulated logs.

(c) Surface style and generators. Paraphrasing every utterance into short, informal text, keeping game flow and votes fixed, largely preserves scores (Appendix G), so the verbosity of the simulated dialogue does not drive the results. Generator-specific variation (Table 30) is what the MoL design mitigates: each player's backbone is drawn at random within every game, so no single generator's patterns dominate a game, and all environment comparisons use MoL logs.

AuthorsOpenReview ↗

Response to Reviewer nwMj (2/2)

W3. Controlling the reasoning versus non-reasoning comparison

Result: the reasoning advantage persists among format-compliant outputs in all 21 model–environment pairs, holds on identical weights, and accuracy rises monotonically with reasoning effort on a fixed checkpoint.

(a) Format compliance. For Qwen3-235B, the two variants are separately released checkpoints (Thinking-2507 and Instruct-2507); for GLM-4.6 and DeepSeek-v3.2, the same checkpoint is used and only the inference mode is switched. Over all seven environments (%):

ModelModeMafiaAccCompliance rateAccuracy among compliant outputs
Qwen3-235BReasoning (Thinking)38.393.840.8
Non-reasoning (Instruct)17.898.218.1
GLM-4.6Reasoning37.895.639.5
Non-reasoning20.175.126.7
DeepSeek-v3.2Reasoning38.695.940.2
Non-reasoning25.387.928.8

Among compliant outputs only, reasoning mode remains 22.7, 12.7, and 11.4 pp more accurate, and it is more accurate in all 21 model–environment pairs: in every one of the 7 environments, the reasoning variant's accuracy among compliant outputs exceeds the non-reasoning variant's by 18.5–26.2 pp for Qwen3-235B, 3.4–16.6 pp for GLM-4.6, and 5.6–14.4 pp for DeepSeek-v3.2. For Qwen3-235B, the non-reasoning variant is in fact more compliant (98.2% vs. 93.8%). The reasoning advantage is therefore not explained by format compliance. For the full per-environment table, please also see our response to Reviewer mqPQ.

(b) Inference compute. To vary the amount of deliberation on a fixed checkpoint, we evaluated GPT-OSS-120B at three reasoning-effort levels on five environments (medium is the setting used in the paper; MafiaAcc, %):

EnvironmentLowMedium (paper)High
Basic-Human*22.235.638.7
Basic-MoL35.838.839.7
Standard-MoL34.435.342.7
Vigilante36.642.746.6
Serial Killer30.834.239.7
All five33.837.441.6

On identical weights, accuracy rises from low to medium to high in every environment (33.8 → 37.4 → 41.6 overall), while format violations remain negligible at every level (at most one response). We will add the average number of reasoning tokens per effort level during the discussion period. Since additional deliberation is precisely what consumes additional compute, the two are one intervention rather than competing explanations; our claim is that intermediate deliberation improves performance on this task, and the analyses above show that this improvement is neither a format-compliance artifact nor a difference between checkpoints. We will state this scope in Section 5.2 and describe the Qwen3-235B comparison as one between checkpoints.


We are grateful for your careful review, which pointed directly to the analyses above. We hope they resolve the three concerns, and if so, we would be truly grateful if you would raise your rating accordingly. We would be glad to provide any further analysis during the discussion period.

AuthorsOpenReview ↗

Response to Reviewer nwMj - Follow-up: Reasoning Tokens per Effort Level

As promised, we measured the amount of reasoning that GPT-OSS-120B performs at each reasoning-effort level. We sampled 100 segments in proportion to the sizes of the five environments and ran the same segments at all three levels, recording the reasoning tokens reported for each response.

Reasoning effortMean reasoning tokensMedian reasoning tokens
Low495466
Medium (paper)1,9831,829
High14,90115,184

On identical weights, raising the effort level increases the amount of reasoning by roughly 4× from low to medium and a further 7.5× from medium to high, and, as reported in our earlier response, accuracy rises with it in every environment, while format compliance stays at 98–100% at every level. This confirms that the gain comes from additional deliberation itself rather than from format compliance or differences between checkpoints, which is the claim we will state in Section 5.2.

Together with our earlier response, this completes our answer to all three of your concerns. We are grateful for your careful review, and we would be truly grateful if you would kindly consider updating your rating in light of these results.

Comments on the paper

4 comments
AuthorsOpenReview ↗

Response to Reviewer mqPQ (1/3)

We thank the reviewer for the careful reading, including cross-checking Table 4 against Table 3. We ran analyses for all four questions; we summarize the results first and then address each question.

Summary

  • Q1. Simple observers that use only public system events, game rules, and dialogue/vote counts fall 7.7-14.8 pp below GPT-5-mini and 6.2-9.5 pp below GPT-OSS-120B in every environment, even when the best of nine heuristic baselines is chosen post hoc for each environment.
  • Q2. The LLM numbers in Table 4 were environment-wide scores. On the identical 120 segments, both GPT-OSS-120B and GPT-5-mini exceed the human mean (+21.9 and +18.2 pp; paired 95% CIs exclude zero). We will revise the "on par" claim accordingly.
  • Q3. Among format-compliant outputs only, reasoning mode still improves accuracy by 11.4-22.7 pp. For the two models where only the inference mode is switched on the same checkpoint (GLM-4.6, DeepSeek-v3.2), the gains are 12.7 and 11.4 pp.
  • Q4. After subtracting survivor-based chance for every segment, the environment comparisons of §5.2 are largely preserved. Generator sensitivity (Table 30) is the risk that MoL was designed to mitigate.

Q1 / W1. Rule-based & event-only baselines; terminal announcements

Terminal announcements. Yes, they are included. Each game log ends with two system lines: a termination reason (e.g., "All mafia eliminated.") followed by the winner declaration (e.g., "Villagers win."). Because evaluation segments are cumulative, these lines appear only in the final segment of each game. We included them by design, since this information is publicly announced at the end of an actual Mafia game, and the LLM and human evaluations used identical inputs. We will state this explicitly in §3.3.

Fixed-size random choice. For a candidate set of n players containing k Mafia members, a uniformly random pair has expected MafiaAcc k/n. We computed this exactly for every segment, over all players (k = 2) and over the players still alive at the end of the segment (MafiaAcc, %):

EnvironmentRandom (all players)Random (alive players)
Basic-Human*20.927.5
Basic-MoL20.023.4
Standard-MoL22.228.1
Vigilante25.028.1
Fool22.228.9
Serial Killer22.227.4
Jester22.228.0

Simple baselines. Following your suggestion, we implemented 9 non-LLM observers that use only the public log. The suffix l denotes counting over the latest cycle in the segment (i.e., the dialogue and votes immediately preceding the prediction), and c denotes counting cumulatively from the start of the game.

  • Most-mentioned (al, ac): the two players mentioned most often in the dialogue.
  • Most-voted (vl, vc): the two players who received the most votes.
  • Game rules (g): remove the players that the public system events rule out (players announced as killed by the Mafia, and a publicly revealed Jester), then choose uniformly among the remaining candidates.
  • Game rules + heuristics (gal, gac, gvl, gvc): apply the game rules first, then the corresponding heuristic among the remaining candidates.
EnvironmentalacvlvcggalgacgvlgvcGPT-5-miniGPT-OSS-120B
Basic-Human*22.522.225.420.423.924.324.326.121.140.935.6
Basic-MoL24.319.327.725.624.631.032.426.826.044.038.8
Standard-MoL23.517.627.127.929.029.128.028.428.142.535.3
Vigilante23.321.432.231.832.931.132.033.432.044.742.7
Fool22.419.229.630.628.429.629.330.830.238.637.4
Serial Killer22.918.426.527.325.727.125.125.927.135.834.2
Jester21.119.627.330.427.129.131.432.033.441.140.2

Even taking the best of the 9 baselines in each environment, a post hoc choice that favors the baselines, the strongest simple observer remains 7.7–14.8 pp below GPT-5-mini and 6.2–9.5 pp below GPT-OSS-120B in all 7 environments. It is also below every open-weight reasoning model above 100B parameters in Table 3 (GPT-OSS-120B, Qwen3-235B, GLM-4.6, DeepSeek-R1, DeepSeek-v3.2) in every environment. The gap is largest on the human-generated logs (Basic-Human*: 26.1 vs. 40.9 / 35.6), indicating that the models' advantage is not specific to simulated dialogue. Counting who is accused or voted most, and ruling out players through public events, therefore recovers only part of what the models achieve.

We will add the baselines to §5 and the appendix. We will also clarify in §1 that isolating refers to separating inference from gameplay ability (an observer cannot fail by playing poorly), and describe the measured construct as faction inference under deception and information asymmetry.

AuthorsOpenReview ↗

Response to Reviewer mqPQ (2/3)

Q2 / W2. Matched human comparison

Thank you for catching this. The LLM numbers in Table 4 were the environment-wide scores from Table 3, not scores on the 120 segments. We recomputed the comparison on exactly the 120 segments shown to the participants (identical inputs), with uncertainty from a paired item-level bootstrap (10,000 resamples; MafiaAcc, %):

ModelEnvironmentItemsHumanModelDifference95% CI
GPT-OSS-120BBasic-Human*3040.355.0+14.7[+3.0, +26.7]
Basic-MoL3043.773.3+29.7[+16.7, +41.3]
Standard-MoL6034.255.8+21.7[+13.2, +30.2]
All12038.160.0+21.9[+15.8, +28.1]
GPT-5-miniBasic-Human*3040.360.0+19.7[+8.0, +32.0]
Basic-MoL3043.748.3+4.7[−9.3, +18.3]
Standard-MoL6034.258.3+24.2[+16.2, +32.2]
All12038.156.3+18.2[+11.8, +24.4]

On the identical items, both models exceed the human mean overall, and the only environment-level interval that includes zero is GPT-5-mini on Basic-MoL. We will replace Table 4 with this matched comparison and revise "on par with human observers" in the Abstract, Introduction, §5.3, and Conclusion to the matched statement that the top models outperform human observers on the same items. The human study retains its role as a calibration anchor: it shows that the logs carry deductive signal that people can extract, and, as reported in §5.3, LLM errors concentrate on the logs that humans also find difficult.


Q3 / W2. Reasoning mode versus format compliance

Checkpoints. For Qwen3-235B, the reasoning and non-reasoning variants are separately released checkpoints (Thinking-2507 and Instruct-2507), as Table 6 indicates. For GLM-4.6 and DeepSeek-v3.2, the same checkpoint is used and only the inference mode is switched.

Decomposition. Over all seven environments (%):

ModelModeMafiaAccCompliance rateAccuracy among compliant outputs
Qwen3-235BReasoning (Thinking)38.393.840.8
Non-reasoning (Instruct)17.898.218.1
GLM-4.6Reasoning37.895.639.5
Non-reasoning20.175.126.7
DeepSeek-v3.2Reasoning38.695.940.2
Non-reasoning25.387.928.8

Accuracy among compliant outputs, by environment (%; R = reasoning, NR = non-reasoning):

EnvironmentQwen3-235B R / NRGLM-4.6 R / NRDeepSeek-v3.2 R / NR
Basic-Human*37.7 / 17.334.6 / 31.239.9 / 26.0
Basic-MoL41.7 / 19.443.0 / 27.640.4 / 29.1
Standard-MoL41.8 / 17.440.0 / 27.040.6 / 27.9
Vigilante45.9 / 22.045.7 / 29.146.3 / 32.8
Fool39.9 / 13.736.3 / 22.138.1 / 23.7
Serial Killer35.2 / 15.831.6 / 20.734.3 / 28.7
Jester38.8 / 20.336.9 / 30.740.3 / 33.5

Restricted to compliant outputs, reasoning mode remains 22.7 pp (Qwen3-235B), 12.7 pp (GLM-4.6), and 11.4 pp (DeepSeek-v3.2) more accurate, and it is more accurate in all 21 model–environment pairs. For GLM-4.6 and DeepSeek-v3.2, where the weights are identical, this separates the effect of the inference mode from training differences. Format compliance accounts for part of the official gap where the non-reasoning mode violates the format more often (GLM-4.6: 17.7 → 12.7 pp; DeepSeek-v3.2: 13.2 → 11.4 pp), whereas for Qwen3-235B the non-reasoning variant is in fact more compliant (98.2% vs. 93.8%). The reasoning advantage is therefore not explained by format compliance. We will add this decomposition to Appendix E alongside the official metric, which we keep unchanged as you suggested, and qualify the Qwen3-235B comparison in §5.2 as a comparison between checkpoints.

AuthorsOpenReview ↗

Response to Reviewer mqPQ (3/3)

Q4 / W1. Environment and generator effects

Chance-adjusted environment comparison. We subtracted the survivor-based chance level from each model's score. Because chance is computed for every segment from the players alive at that point, this also accounts for longer games contributing more late-cycle segments, where chance is higher (MafiaAcc, %):

EnvironmentRandom (alive)GPT-5-miniMargin over randomGPT-OSS-120BMargin over random
Basic-Human*27.540.9+13.435.6+8.1
Basic-MoL23.444.0+20.738.8+15.5
Standard-MoL28.142.5+14.435.3+7.2
Vigilante28.144.7+16.742.7+14.7
Fool28.938.6+9.737.4+8.5
Serial Killer27.435.8+8.434.2+6.8
Jester28.041.1+13.140.2+12.2

The conclusions of §5.2 are largely preserved. Basic-MoL has the largest margin for both models; Vigilante remains above Standard-MoL for both models; Jester is comparable to (GPT-5-mini, −1.3 pp) or above (GPT-OSS-120B) Standard-MoL; and Serial Killer has the smallest margin for both models, with Fool the second smallest for GPT-5-mini. One refinement is that, after adjustment, Fool is not harder than Standard-MoL for GPT-OSS-120B (8.5 vs. 7.2 pp). We will report chance-adjusted margins alongside the raw scores and revise the corresponding sentence.

Generator effects (Table 30). We agree that generating games with a single backbone can leave model-specific artifacts, and Table 30 shows exactly this sensitivity. This risk is what MoL was designed to mitigate (§3.2). MoL does not claim to remove every generator-specific regularity; rather, by assigning a randomly chosen backbone to each player within every game, it prevents any single generator's utterance patterns or outcome biases from dominating a game, similar in spirit to an ensemble. §5.1 reflects this: each single-backbone simulation is skewed in at least one respect (e.g., Dolphin games are strongly Mafia-biased, with a 72.5% Mafia win rate), whereas only MoL combines voting diversity, lexical diversity, and faction balance. All environment comparisons in Table 3, except the Standard-Single column, use MoL logs, so generator-specific variation is spread evenly across environments rather than confounded with them. Moreover, the simple baselines in Q1 are computed on these MoL logs, and surface regularities such as who is mentioned or voted most fall well short of the models, which addresses which regularities account for the predictions.

The same baselines also explain the spread in Table 30, as the differences across single-backbone simulations reflect how much signal each generator places in its logs (Standard-Single, MafiaAcc, %):

GeneratorGPT-5-miniGPT-OSS-120BMost-mentioned (al)Most-voted (vc)
Dolphin35.833.918.233.6
Nemotron49.645.724.228.5
Tulu-336.737.925.825.8
LLaMA-3.352.950.432.130.1
Qwen2.538.638.926.431.4

The two extremes of Table 30 line up with the strength of the dialogue cues. LLaMA-3.3 logs, on which observers score highest, are those in which the most-mentioned players are most often the actual Mafia (32.1%). Dolphin logs, on which observers score lowest, have the weakest dialogue cue (18.2%, below the 22.2% chance level) and are strongly Mafia-dominated, meaning that the in-game Villagers themselves rarely identified the Mafia. Generator sensitivity thus reflects how informative each generator's logs are, and MoL averages over it within each game. We will add this analysis next to Table 30.


In Q1, you noted that you would revise your view of the benchmark's construct validity if simple baselines were shown to fall substantially below the language-based models. The results above show that the strongest of nine baselines remains 6.2–14.8 pp below both models in every environment, including human-generated logs. If these results, together with the matched human comparison and the compliance decomposition, address your concerns, we would be grateful if you would consider updating your assessment accordingly. We would be glad to provide any further analysis during the discussion period.

AuthorsOpenReview ↗

Response to Reviewer mqPQ: Non-terminal Analysis

Thank you for the prompt and detailed follow-up. We address your remaining concern on terminal segments first, then the two suggestions.

Terminal segments: proportion and non-terminal results

The final segment of each game contains the termination reason and the winner announcement. These terminal segments account for the following proportion of all evaluated segments:

EnvironmentTerminal segments
Basic-Human*31.0%
Basic-MoL29.8%
Standard-MoL29.8%
Vigilante36.4%
Fool29.3%
Serial Killer40.5%
Jester35.7%

On the remaining non-terminal segments, no outcome information is available, so any advantage must come from the dialogue, votes, and in-game events. We report the models and the nine simple baselines on these segments only (MafiaAcc, %). Following your point on the survivor-based reference, the chance level here is computed over all players, which is unaffected by the game outcome. In addition to the two models discussed so far, we include Claude Opus 5.5, which we newly evaluated on the full benchmark during the discussion period (also in response to Reviewer LZxX. Later to be added in the manuscript); its full results will be added to Table 3.

EnvironmentRandomBest of nine baselinesClaude Opus 5.5GPT-5-miniGPT-OSS-120B
Basic-Human*20.723.5 (vl)29.127.620.9
Basic-MoL20.027.6 (gac)40.934.328.7
Standard-MoL22.226.6 (vc, gvc)40.233.726.0
Vigilante25.031.1 (vc, gvc)40.334.033.4
Fool22.230.1 (vc, gvc)39.630.329.9
Serial Killer22.226.9 (vc)32.029.627.9
Jester22.229.7 (gvc)38.632.531.7

Without any terminal information, the strongest observer, Claude Opus 5.5, exceeds the best of the nine baselines, chosen post hoc for each environment, by 5.1–13.6 pp in all seven environments, including the human-generated Basic-Human* logs. GPT-5-mini is above all nine baselines in all seven environments. GPT-OSS-120B is close to the best baseline on these segments. The advantage over simple observers therefore persists without terminal information and grows with the observer's capability, which is what one expects if the gains come from inference over the dialogue rather than from information directly available from the game rules. It also shows that the non-terminal segments are where MAFIA 1K separates strong observers from rule-and-frequency heuristics, leaving clear headroom for future models.

In the revision, we will keep the terminal segments in the benchmark, report the proportion of terminal segments, and add non-terminal MafiaAcc as a supplementary metric next to MafiaAcc in the main results, so that the contribution of terminal information is visible for every model.

Q2. We agree, and will limit the revised claim to GPT-OSS-120B and GPT-5-mini on the 120 sampled segments.

Generator effects. We agree that MoL mitigates rather than removes generator-specific cues, and will phrase it as such.


We are sincerely grateful for your engagement across both rounds. Your questions led directly to the analyses that now strengthen the paper: the random, rule-based, and heuristic baselines; the matched human comparison with paired uncertainty; the compliance decomposition with the checkpoint clarification; the chance-adjusted environment comparison; and the non-terminal comparison above. You mentioned that clarifying this last point would help you reassess your rating, and we hope the results above do so. If you feel your concerns have now been resolved, we would be truly grateful if you would raise your rating accordingly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.