acceptodds
Under review as a conference paper at ICLR 2027

Neighbors as Top Suspects: Diagnosing Ranking Shortfalls in LLM Adversary Detection

Abstract

LLM agents increasingly make decisions from other agents' claims. Even when evaluators observe the received messages, a scalar outcome does not reveal where the LLM's final ranking differs from a reference derived from that stream. We record the messages delivered to one LLM seat in a five-round Mafia game with fixed-policy surroundings and derive an offline accusation-minus-defense shadow ranking. We separate non-neighbor placement (the full-ranking slots occupied by players not heard directly) from ordering within those slots. Across four models and eleven configurations, model mean reciprocal rank (MRR) is below the shadow in 34 of 44 cells. On complete raw rankings with a non-neighbor adversary, placement's signed contribution accounts for 76–100% of the mean shadow-minus-model reciprocal-rank gap and remains the larger contribution in connected configurations. A separate delivered-accusation analysis shows that the adversary can rank highly among non-neighbors yet below neighbors overall. On the same 224 out-of-view GPT-5.6 games across seven relay configurations, with complete raw rankings in every arm, premise correction accompanies higher model MRR and improved mean within-group adversary rank. The ordering contribution decreases without a detectable placement-contribution change; structure disclosure shows no detectable change in either contribution. Corrected-premise profiles vary across models, with unequal complete-output coverage. These post-hoc, reference-relative diagnostics describe one scripted environment without identifying mechanisms, demonstrating repair, or establishing transfer.

Primary area:
foundation or frontier models, including LLMs
Keywords:
multi-agent evaluation, LLM agents, adversary detection, deception detection, hidden-role games, communication graphs

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_1QNXOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

One LLM seat plays a five-round Mafia game on a communication graph against scripted players, then ranks all players by suspicion. Its ranking is compared with an offline "shadow" accusation-minus-defense tally computed on the messages it received. The model trails the shadow in 34 of 44 cells. For non-neighbor adversaries, a two-way swap decomposition attributes 76-100% of the reciprocal-rank gap to "placement": the model puts neighbors on top. A premise-correction prompt slightly improves within-group ordering. The authors state the results identify no mechanism, repair, or transfer.

Critical strengths

Scoring the model against a reference on its own received stream is a sound way to separate delivery from use of information.

The authors are careful about scope and preregistration.

Critical weaknesses

The shortfall is produced by the environment and the prompt. The prompt says investigators always tell the truth, yet about 80% of honest accusations after round one are false. Under that premise, ranking a neighbor who accuses you first is the correct deduction. The shadow rule is the same tally the honest scripts use to generate accusations, i.e., the environment's own generative rule. The "adversary" accuses a random neighbor on a coin flip, so this is noise detection, not adversary detection. When the premise is corrected, the change in the gap is not significant.

The contribution is a post-hoc bookkeeping device, poorly presented. The placement/ordering split is a two-factor Shapley decomposition of reciprocal rank, a metric dominated by the top slot. Neighbor-first rankings attribute the gap to placement almost by construction. The authors decline to draw any recommendation from it. Of the preregistered hypotheses (N1-N5), only N5 survived, so the headline analysis is post-hoc. The main text is hedged sentence by sentence and depends on roughly fifty appendix sections over 64 pages.

The shadow sees a 2-round window while the model sees the full history. Parser-filled rankings (as low as 77% named). No pinned model checkpoints. One scripted environment with no learning component.

Questions for authors

With a truthful prompt (honest players may err; the adversary acts at random), does a gap remain, and is it placement-dominated?

How does the model compare with the Bayes posterior under the known scripts?

What does the decomposition predict that the MRR gap does not?

Reviewer_Tb49OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

Evaluating an LLM within a fixed-policy Mafia game against a same-stream rule reference, this paper decomposes the performance gap into placement and ordering components to isolate the respective contributions of non-neighbors' overall rank positions and their within-group ordering.

Critical strengths
  1. The proposed offline shadow rule holds the evaluated model's actually received message stream fixed, and introduces a two-component Shapley transplant decomposition to decouple the scalar ranking gap into the slots occupied by non-neighbors and the ordering within those slots.

  2. While replacing the occupant in a co-run alters subsequent messages, the shadow comparator evaluates the model's actually received stream offline. This distinction prevents confounding trajectory-induced differences with the model's decision-making.

Critical weaknesses
  1. First, the paper lacks a clear Threat Model mapping its toy setup, a five-round Mafia game with a coin-flipping adversary and deterministic tally-based bots, to real-world multi-agent security or coordination scenarios, leaving the practical relevance of the findings unclear. Second, the manuscript omits both mathematical formalization and intuitive illustration: it provides zero formal equations for the graph communication dynamics, the shadow scoring rule, or the core placement-versus-ordering Shapley decomposition , and lacks any concrete case study or multi-round diagram illustrating how neighbor echoes trap the LLM's reasoning. Finally, the presentation reads like an uncurated execution log rather than a cohesive paper. Great obstacles were caused to the readers’ understanding of the core contributions. Most notably, Figure 1 is literally a 3×23 \times 2 plain-text table mislabeled as a figure, and the main text is cluttered with over thirty fragmented cross-references to a 50+ page appendix of script outputs, severely hindering readability.

  2. The headline claim that LLMs rank neighbors above non-neighbors is largely driven by synthetic setup artifacts rather than intrinsic reasoning flaws. Specifically, an incomplete neighbor-only roster, a deterministic tie-breaking rule targeting the LLM, and a false prompt premise jointly trap the model into deducing that its neighbor accusers must be the adversary. Moreover, switching to an oracle-informed reference Appendix B.16 completely reverses the shortfall from out-of-view to in-view, confirming the finding is heavily environment- and comparator-dependent.

Questions for authors
  1. What real-world security or coordination scenario does the scripted game represent?

  2. Which existing experiments show that placement dominance itself, rather than merely the overall performance gap, persists after correcting the roster, prompt premise, and tie-breaking artifacts?

  3. Can you specify the shadow scoring rule and placement–ordering decomposition mathematically, and illustrate their computation using one actual trajectory?

Reviewer_obnhOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates ranking shortfalls in LLM adversary detection within a scripted, partially observable Mafia environment, focusing on how models rank players they cannot hear directly. To examine this problem, the authors introduce a diagnostic framework that compares the model’s final ranking with an offline “shadow” ranking derived by tallying accusations minus defenses in messages received during the same game. The framework separates the reciprocal-rank gap into two components: the positions occupied by non-neighbors and their internal ordering. It quantifies their contributions by exchanging these components between the two rankings and averaging over both exchange orders. The experiments span four language models and eleven network configurations and include prompt interventions to examine changes in each component. The findings suggest that non-neighbor placement accounts for most of the average gap on complete-output samples, while premise correction improves within-group ordering without a detectable change in the placement contribution in the GPT-5.6 analysis.

Critical strengths
  1. Separating group placement from within-group ordering reveals how an adversary can rank highly among non-neighbors while remaining low in the overall ranking. This is more informative than reporting aggregate performance alone and constitutes the paper’s strongest potential contribution.
  2. The paper distinguishes an actively participating rule baseline from an offline shadow comparator. This avoids treating same-seed games as identical-input comparisons when the evaluated agent’s statements affect subsequent communication. The distinction strengthens the interpretation of comparisons within each experimental arm.
Critical weaknesses
  1. The evidence does not sufficiently establish the central diagnosis under a correctly specified and robust evaluation. The original prompt states that investigators always provide truthful information, whereas their later statements frequently target innocent players. Under these instructions, suspecting a neighbor who falsely accuses the evaluated seat can be a reasonable deduction. The premise-correction experiments are relevant, but their heterogeneous results and unequal coverage leave uncertainty about the scope of the headline finding. Furthermore, alternative-comparator checks mainly address aggregate gaps or audibility contrasts rather than directly validating placement dominance. Complete-output filtering also retains different populations across models. Together, these issues limit the interpretation of placement dominance as a reliable diagnosis of LLM ranking shortfalls.

  2. The paper does not present the experimental evidence clearly enough to support its claims. There is no integrated overview connecting the communication environment, model outputs, shadow scoring, and component exchanges. The main text also lacks a consolidated performance table across models and configurations that reports uncertainty and retained sample sizes. Moreover, differences in sample populations and outcome definitions across analyses are not presented systematically, making it difficult to determine which evidence supports each conclusion and whether results are directly comparable. A clearer experimental overview and more structured presentation of results are needed to make the central argument easier to assess.

Questions for authors
  1. The corrected-premise experiments are important for interpreting the main finding. Please report the placement and ordering contributions, total shadow–model gap, and complete-output coverage for each retained model–configuration cell. To what extent do these results support placement dominance under accurate task instructions?
  2. Does placement remain the dominant contribution when the decomposition is computed using the alternative reference rankings examined in the paper, on the same complete-output samples? The reported aggregate-gap and audibility analyses do not directly establish this.
  3. I am concerned that complete-output filtering may select different types of games across models. How do retained and excluded games differ in topology, degree, adversary mentions, and model performance? Whether the headline finding holds on a common complete-output sample across models would materially affect my assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.