- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
One LLM seat plays a five-round Mafia game on a communication graph against scripted players, then ranks all players by suspicion. Its ranking is compared with an offline "shadow" accusation-minus-defense tally computed on the messages it received. The model trails the shadow in 34 of 44 cells. For non-neighbor adversaries, a two-way swap decomposition attributes 76-100% of the reciprocal-rank gap to "placement": the model puts neighbors on top. A premise-correction prompt slightly improves within-group ordering. The authors state the results identify no mechanism, repair, or transfer.
Critical strengths
Scoring the model against a reference on its own received stream is a sound way to separate delivery from use of information.
The authors are careful about scope and preregistration.
Critical weaknesses
The shortfall is produced by the environment and the prompt. The prompt says investigators always tell the truth, yet about 80% of honest accusations after round one are false. Under that premise, ranking a neighbor who accuses you first is the correct deduction. The shadow rule is the same tally the honest scripts use to generate accusations, i.e., the environment's own generative rule. The "adversary" accuses a random neighbor on a coin flip, so this is noise detection, not adversary detection. When the premise is corrected, the change in the gap is not significant.
The contribution is a post-hoc bookkeeping device, poorly presented. The placement/ordering split is a two-factor Shapley decomposition of reciprocal rank, a metric dominated by the top slot. Neighbor-first rankings attribute the gap to placement almost by construction. The authors decline to draw any recommendation from it. Of the preregistered hypotheses (N1-N5), only N5 survived, so the headline analysis is post-hoc. The main text is hedged sentence by sentence and depends on roughly fifty appendix sections over 64 pages.
The shadow sees a 2-round window while the model sees the full history. Parser-filled rankings (as low as 77% named). No pinned model checkpoints. One scripted environment with no learning component.
Questions for authors
With a truthful prompt (honest players may err; the adversary acts at random), does a gap remain, and is it placement-dominated?
How does the model compare with the Bayes posterior under the known scripts?
What does the decomposition predict that the MRR gap does not?