- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper investigates identity-dependent conformity in LLMs, examining whether social identity labels alter how a target LLM weighs peer opinions in multi-agent settings. The authors evaluate 12 open-weight LLMs (3B–72B across the Llama, Gemma, and Qwen families) over nine benchmarks. Empirical results reveal group identity biases: relative to an unlabeled baseline (13.3% conformity), models conform more readily to an incorrect in-group majority (16.7%) than to an out-group majority. This paper demonstrated that prompt-level identity cues can bias information aggregation and even invert the corrective effect of dissenting voices.
Critical strengths
-
This paper investigates the vulnerability in multi-agent collaboration. While multi-agent literature predominantly focuses on whether deliberation improves task accuracy, this work focuses on a less-attended failure mode: LLMs alter their judgments based on superficial identity labels assigned to their peers. Framing this problem has clear relevance for multi-agent debate, voting, cross-verification, and human-AI collaboration.
-
The experiment on social identity is well-designed, spanning several groups (i.e., AI vs. human, model family, and Minimal Group). The Minimal Group condition is interesting: despite carrying no inherent priors regarding domain competence or social status, arbitrary "Blue" and "Green" labels still induce marked shifts in conformity.
-
This work examined the interplay between majority and dissenting minority (Ally) identities, uncovering a counterintuitive breakdown in error correction. The authors evaluate four Majority-Ally identity configurations and reveal that introducing a correct out-group ally against an erroneous in-group majority actually increases conformity. This finding suggests that merely injecting a correct agent into a deliberative pool does not guarantee improved decision quality and can actively degrade it if identity dynamics are ignored.
Critical weaknesses
-
The experimental design relies exclusively on an incorrect majority, which confounds epistemic resistance with identity-driven rejection and thus fails to substantiate the claim of systematic "Out-group Divergence." Observing a lower conformity rate towards an erroneous out-group majority merely shows that the model is less likely to abandon its initial correct judgment when confronted with out-group falsehoods; it does not prove that the model actively discounts or rejects out-group information irrespective of its validity. The current results only demonstrate differential susceptibility to false consensus rather than a generalized aversion to out-group perspectives.
-
The statistical rigor of several headline findings is undermined by the absence of multiple-comparison corrections across an extensive array of hypothesis tests. The reported p-values are uncorrected despite testing numerous identity categories, Majority–Ally configurations, reasoning modes, and safety-credential conditions. Because several key claims rest on marginal significance levels hovering near the threshold, it is questionable whether these findings would survive standard family-wise error rate or false discovery rate adjustments.
-
A substantial gap in external validity exists between the paper's static, scripted setup and the dynamic multi-agent systems it purports to inform. Rather than evaluating genuinely interactive agents, the experiments expose a single target model to seven hard-coded, single-turn text responses devoid of authentic reasoning chains, supporting evidence, or multi-round deliberation dynamics. While the authors acknowledge this simplification in the appendix, at the current stage of multi-agent system research, reducing multi-agent interaction to static prompt injection can no longer be treated as a benign limitation.
Questions for authors
See the weaknesses.
3 comments
Thanks for the response. Some of my concerns are addressed. However, I think this paper brings limited new insights, and there is a substantial gap compared to current multi-agent systems. I will keep my original rating.
Thank you for your response. The revision answers each of the three weaknesses in your original review:
- Incorrect-majority confound. Section 3.2 explains why the ally conditions rule out this reading: a correct out-group ally raises conformity to an incorrect in-group majority instead of lowering it.
- Multiple comparisons. The main text and Appendix F now report Benjamini-Hochberg q-values, which all main results survive.
- Static, scripted peers. The new Section 5 replaces them with live LLMs that argue and rebut one another over three rounds on clinical vignettes (MedQA).
If something is left unresolved we would appreciate your feedback on what is still missing. Otherwise, given how much we have improved the paper based on your feedback, we would be grateful if you would consider updating your score.
Dear Reviewer,
We posted our response to your review on 12 November, but it was mistakenly visible only to the chairs. You can now see it. We will upload a revised version of the paper soon.
Thank you, The Authors