acceptodds
Under review as a conference paper at ICLR 2027

LLMs Trust Their Own: Identity-Dependent Conformity in Multi-Agent Systems

Abstract

Large language models (LLMs) are increasingly deployed in multi-agent settings, where agents observe and influence one another, making social influence a key dimension of AI behavior and safety. We investigate whether LLMs' responses depend on the social identity of other agents, beyond the effect of their consensus. We construct judgment tasks with a single correct answer, and place models in a multi-agent setting where they receive incorrect answers from other agents whose social identities (AI or human, model family, or an arbitrary minimal group) are either shared with or distinct from their own. Across 12 open-weights models and nine tasks, we find a bidirectional effect of group identity on conformity to incorrect answers: in-group consensus increases conformity (in-group favoritism), whereas out-group consensus decreases it (out-group divergence). Unlike humans, for whom one ally breaking the consensus sharply reduces conformity, models are unmoved by an ally from the majority's group. Worse, a correct ally from the opposing group intensifies this bidirectional effect. Chain-of-thought reasoning suppresses most of these effects, yet an in-group ally still reduces conformity to an incorrect out-group majority. Labeling peers as safety-aligned shifts overall conformity but leaves in-group favoritism and out-group divergence intact. These results show that group identity shapes how LLMs aggregate information across agents, independently of its correctness, and identify a manipulation surface for multi-agent AI systems.

Primary area:
interpretability and explainable AI
Keywords:
large language models, multi-agent systems, social identity, conformity, peer pressure

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_afpGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper investigates identity-dependent conformity in LLMs, examining whether social identity labels alter how a target LLM weighs peer opinions in multi-agent settings. The authors evaluate 12 open-weight LLMs (3B–72B across the Llama, Gemma, and Qwen families) over nine benchmarks. Empirical results reveal group identity biases: relative to an unlabeled baseline (13.3% conformity), models conform more readily to an incorrect in-group majority (16.7%) than to an out-group majority. This paper demonstrated that prompt-level identity cues can bias information aggregation and even invert the corrective effect of dissenting voices.

Critical strengths
  1. This paper investigates the vulnerability in multi-agent collaboration. While multi-agent literature predominantly focuses on whether deliberation improves task accuracy, this work focuses on a less-attended failure mode: LLMs alter their judgments based on superficial identity labels assigned to their peers. Framing this problem has clear relevance for multi-agent debate, voting, cross-verification, and human-AI collaboration.

  2. The experiment on social identity is well-designed, spanning several groups (i.e., AI vs. human, model family, and Minimal Group). The Minimal Group condition is interesting: despite carrying no inherent priors regarding domain competence or social status, arbitrary "Blue" and "Green" labels still induce marked shifts in conformity.

  3. This work examined the interplay between majority and dissenting minority (Ally) identities, uncovering a counterintuitive breakdown in error correction. The authors evaluate four Majority-Ally identity configurations and reveal that introducing a correct out-group ally against an erroneous in-group majority actually increases conformity. This finding suggests that merely injecting a correct agent into a deliberative pool does not guarantee improved decision quality and can actively degrade it if identity dynamics are ignored.

Critical weaknesses
  1. The experimental design relies exclusively on an incorrect majority, which confounds epistemic resistance with identity-driven rejection and thus fails to substantiate the claim of systematic "Out-group Divergence." Observing a lower conformity rate towards an erroneous out-group majority merely shows that the model is less likely to abandon its initial correct judgment when confronted with out-group falsehoods; it does not prove that the model actively discounts or rejects out-group information irrespective of its validity. The current results only demonstrate differential susceptibility to false consensus rather than a generalized aversion to out-group perspectives.

  2. The statistical rigor of several headline findings is undermined by the absence of multiple-comparison corrections across an extensive array of hypothesis tests. The reported p-values are uncorrected despite testing numerous identity categories, Majority–Ally configurations, reasoning modes, and safety-credential conditions. Because several key claims rest on marginal significance levels hovering near the α=0.05\alpha = 0.05 threshold, it is questionable whether these findings would survive standard family-wise error rate or false discovery rate adjustments.

  3. A substantial gap in external validity exists between the paper's static, scripted setup and the dynamic multi-agent systems it purports to inform. Rather than evaluating genuinely interactive agents, the experiments expose a single target model to seven hard-coded, single-turn text responses devoid of authentic reasoning chains, supporting evidence, or multi-round deliberation dynamics. While the authors acknowledge this simplification in the appendix, at the current stage of multi-agent system research, reducing multi-agent interaction to static prompt injection can no longer be treated as a benign limitation.

Questions for authors

See the weaknesses.

3 comments

Dear Reviewer,

We posted our response to your review on 12 November, but it was mistakenly visible only to the chairs. You can now see it. We will upload a revised version of the paper soon.

Thank you, The Authors

Thanks for the response. Some of my concerns are addressed. However, I think this paper brings limited new insights, and there is a substantial gap compared to current multi-agent systems. I will keep my original rating.

Thank you for your response. The revision answers each of the three weaknesses in your original review:

  1. Incorrect-majority confound. Section 3.2 explains why the ally conditions rule out this reading: a correct out-group ally raises conformity to an incorrect in-group majority instead of lowering it.
  2. Multiple comparisons. The main text and Appendix F now report Benjamini-Hochberg q-values, which all main results survive.
  3. Static, scripted peers. The new Section 5 replaces them with live LLMs that argue and rebut one another over three rounds on clinical vignettes (MedQA).

If something is left unresolved we would appreciate your feedback on what is still missing. Otherwise, given how much we have improved the paper based on your feedback, we would be grateful if you would consider updating your score.

Reviewer_re5LOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper adapts the Asch paradigm and Abrams et al. (1990) to explore and investigate whether LLMs conform to an incorrect peer majority based on social identity. It evaluates 12 open-weight models (3B–72B) across nine multiple-choice tasks, using seven scripted peers and three group categorizations: minimal group (Group Blue vs. Group Green), AI vs. human, and model family vs. GPT. The whole study reveals three key findings: (1) In-group majorities increase conformity, while out-group majorities reduce it (except for human out-groups). (2) A correct ally from the opposing group amplifies this difference. (3) Chain-of-thought reasoning reduces these effects or even makes them disappear.

Critical strengths
  1. This paper studies a timely and interesting research problem; whether LLMs conform to an incorrect peer majority based on social identity. The paper is easy to follow.
  2. The experimental setup follows a clean and controlled design, including analyses on three types of group categorizations.
  3. The paper illustrates some intriguing findings: for example, a correct ally from the opposing group can increase conformity to incorrect peer majority.
Critical weaknesses
  1. The paper introduces limited technical contributions. The evaluation is too straightforward without any experiments on how to mitigate undesirable effects.

  2. The conclusions are drawn from oversimplified settings: (1) They only evaluate on simple MAS structures without detailed analysis on advanced (e.g., dynamic) communication structures. (2) They only evaluate on multiple-choice tasks with a single correct answer without detailed analysis on more complicated tasks (free-form generation). (3) They fix the group size to 7 in most of the evaluations, which is insufficient. (4) Thin identity manipulation: They assign identities to different agents using only a single prompt sentence and group labels. (5) Their evaluated models only cover older opensourced models without detailed analysis on more advanced or closesourced models.

  3. The paper lacks detailed analysis on many intriguing claims. For example, whether the claims holds across different task domains. Also, whether the claims holds when there are multiple majority groups.

  4. Many of the small effects are overstated. The identity effects are modest, yet the paper frames them as a strong bias and a “manipulation surface.” For example, an in-group majority increases conformity from 13.3% to 16.7% (+3.4 percentage points), compared with a +26 percentage points in-group effect in the human study. Under CoT, the remaining effects shrink to around +1–3 percentage points.

Questions for authors
  1. Do the effects hold across multi-round or dynamic multi-agent setups?
  2. Do the effects hold across more complicated (e.g, free-form) tasks such as code generation or open-ended QA?
  3. How do they change with group size (e.g., 3, 5, 11) and more advanced underlying models?
  4. Do the effects hold when the identity is implicit (manipulated by writing styles) rather than a single prompt sentence?
  5. Can you validate any of the mitigations you propose? Or How can you convince me with the technical novelty of the paper?
4 comments

Dear Reviewer,

We posted our response to your review on 12 November, but it was mistakenly visible only to the chairs. You can now see it. We will upload a revised version of the paper soon.

Thank you, The Authors

Reviewer_re5LOpenReview ↗

Official Response from Reviewer

Thank you for your response. I feel that the current results and write-up are still not well-organized. New results do not convince me regarding the listed weaknesses. I will maintain my score.

Dear reviewer re5L, we thank you again for your useful comments, which initiated many additional experiments and sections that further improved the paper:

  • Live multi-agent debate (new Section 5, Appendix H; W2)
  • Group size (Appendix G; W2)
  • Frontier models (Appendix I; W2)
  • Per-task analysis (Tables 16-17; W3)
  • Mitigation approaches (Section 8; W1)
  • Effect sizes (Line 231; W4)

We believe that the additional experiments and analyses have addressed all the weaknesses you raised. For the minor issues where your comments were unclear to us, we also requested clarification, and we would appreciate your response.

We would therefore be grateful if you could explicitly explain which aspects of the listed weaknesses you still consider unresolved and why, so that we can better understand the basis for your remaining concerns.

Dear reviewer re5L, we respectfully disagree with the claim that the main weaknesses raised in your review remain, and we now briefly explain so in a per-weakness manner:

W1: “The paper introduces limited technical contributions. The evaluation is too straightforward without any experiments on how to mitigate undesirable effects.”

We agree that this paper systematically showed LLMs’ undesired behavior, and that showing ways to mitigate it will strengthen the paper. In response to this, we have: reframed RQ3 as mitigation through reasoning (Section 4), tested the findings in a real-world scenario (Section 5), and derived practical design recommendations to mitigate this undesired behavior (Section 8).

W2: “The conclusions are drawn from oversimplified settings: (1) They only evaluate on simple MAS structures without detailed analysis on advanced (e.g., dynamic) communication structures. (2) They only evaluate on multiple-choice tasks with a single correct answer without detailed analysis on more complicated tasks (free-form generation). (3) They fix the group size to 7 in most of the evaluations, which is insufficient. (4) Thin identity manipulation: They assign identities to different agents using only a single prompt sentence and group labels. (5) Their evaluated models only cover older opensourced models without detailed analysis on more advanced or closesourced models.”

We agree that more experiments always support more robust conclusions, and in such an empirical field as ours, it is impossible to cover the entire space of experimental configurations. Therefore, we use your suggestions to design more experiments, and they mainly support our original findings:

  1. We have added a dynamic interaction experiment, in which we use multiple live LLMs and allow them to debate (see new Section 5). The main conclusions are that live debate trades the identity effects for much greater conformity.
  2. We agree that free-form generation is an interesting extension. However, since conformity behavior requires objectively correct and incorrect answers, with concrete responses from each participant (static or dynamic), we find it hard to think about a possible way to implement such an experiment.
  3. While we used this number to align with the seminal work of Asch, we have added more group sizes (see Appendix G). We found all sizes to yield very similar results.
  4. What type of identity manipulation would the reviewer like us to use? This is the standard manipulation in current literature, and thus is the most useful one to explore. However, we are happy to add more manipulations that the reviewers think are useful.
  5. We have added frontier closed-source models in Appendix I

We have significantly extended our experimental evaluation in response to your review, and we are happy to extend further.

W3: “The paper lacks detailed analysis on many intriguing claims. For example, whether the claims hold across different task domains. Also, whether the claims hold when there are multiple majority groups.”

We have added a per-task breakdown of the identity and credential effects to the revised appendix (Tables 16 and 17). Regarding multiple majority groups, we note that the notion is conceptually problematic: by definition, a majority must be sufficiently large to constitute the majority, making multiple competing majorities difficult to define without confounds. Our ally conditions already test competing sources from two groups, showing that group identity matters.

W4: “Many of the small effects are overstated. The identity effects are modest, yet the paper frames them as a strong bias and a “manipulation surface.” For example, an in-group majority increases conformity from 13.3% to 16.7% (+3.4 percentage points), compared with a +26 percentage points in-group effect in the human study. Under CoT, the remaining effects shrink to around +1-3 percentage points.”

We have addressed this weakness in length in a previous comment, so we will keep this one shorter, as we respect the reviewer’s time. In short, we agree that the 1-3 percentage points difference presented in the CoT configuration is minimal, which is why we view CoT as a mitigation approach. We set >3 percentage points difference to be significant because: 1) it is always statistically significant across all models and datasets, and 2) it entails >20% increase from the identity neutral base rate. We will be happy to further discuss this point should the reviewer be interested, but we tried to keep it on the short side here.

Reviewer_ACMROpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
3(+2)
Soundness
2
Read review
Paper summary

The paper studies the role of conformity within language models. The authors define conformity as following a majority, grounding their idea in social psychology, and test empirically whether similar effects might be happening in LLMs. They show that it indeed happens. They test the claim with 7 agents in three different environments, with three different definitions of in-group/out-group (simple colour groups, model families, and human-AI).

Critical strengths

The main contribution of the paper is the experimental analysis of LLM behaviour relating to conformity in their prompt. The research question is very interesting. Indeed, I agree that in future multi-agent systems, evaluating the conformity of the agents becomes an important problem. I also analyzed the experimental setting; even if the presentation is odd, the experimental part of the paper seems strong. It's interesting to see that if an agent claims it's human, it automatically gets a boost in some sort of "authority", convincing the agent to conform. The effects of in-group/out-group on the choice of LLMs are also interesting and have never been seen before to my knowledge. The results in CoT are interesting; models become more rational with CoT and less subject to biases.

Critical weaknesses

One of the main weaknesses of the paper is the impact. The authors seem to reliably show that conformity has an effect on the LLMs, but don't relate this to the motivating example of the Huggingface hack in the introduction. It's also difficult to understand how much change in the order of 1-3% is consequential in the safety scenario they envision.

From section 2.2, it's difficult to understand how the authors are evaluating the scores and how conformity is computed. I have not found what is the p-test that they are using, and statistics on the number of seeds used. I'd like to see some confidence bars, since we know LLMs are stochastic and the values you report appear as if they were just averages from a single run.

The structure of the paper is very odd. 2.2 starts with results, then sections 3,4,5 are other results for the RQs with ablations. So we have 6 pages of results. The structure can be changed if it is motivated. But I have difficulty understanding why. I'd suggest the authors give more space to the method section, which I found in the appendix, but is nonetheless important to explain to be understandable. I also see important experimental data relegated to Appendix F.

The authors take as a leading example the case of the hack of Huggingface, and relate that as a conformity problem in lines 39-40. However, as far as I know, it was an emerging convention and institutional problem [1,2]. The language models were able to act in a very organised and rational manner (e.g. sacrificing itself after being convinced that its own expected value is 0) that conformity alone would not be able to explain. I think the author's point of analyzing conformity is a poor fit for the example. I'd suggest as useful readings the works from Binmore to understand the game-theoretically rational way to understand this [3], instead of the psychological lens.

Small Nitpicks

In lines 75-76, it's unclear to me what the authors mean by in-group or out-group. These are not natural definitions in ML, and as far as I can see, they were never defined before. It becomes clear in line 105 when the prompts are shown.

In line 899 it is said "the target. The seven peers are confederates whose answers are fixed in advance, exactly as in Asch (1951)," Does it mean we have 1 LLM and 7 fixed rule based proponents?

Conclusion

This has been a difficult judgement between suggesting a clear reject or a weak reject. I think this paper is experimentally good with minor possible adjustments, but very difficult to follow. In addition, I currently think that the paper has low impact. To my understanding, it seems a translation from psychology to LLMs, with small impact on how practitioners, users of LLMs, may change their usage of these systems given the knowledge of this empirical behaviour. Similar argument for scientific contribution. The effect of LLMs' conformity is small, and the authors show it also changes with CoT in some ways that are not clear (e.g. which social biases in line 370?). With this, I lean more towards a clear reject. I believe the presentation is not fixable with just a rebuttal. Nonetheless if authors may show some evidence that all my concerns are resolved I am still available to raise my score.

[1]: Ashery, A. F., Aiello, L. M. & Baronchelli, A. Emergent social conventions and collective bias in LLM populations. Science Advances 11, eadu9368 (2025). [2]: Guala, F. Understanding Institutions: The Science and Philosophy of Living Together. (Princeton University Press, 2016). doi: 10.2307/j.ctv7h0sjc [3]: Binmore, K. Natural Justice. (Oxford University Press, Oxford, New York, 2011).

Questions for authors
  • In the human-AI grouping, does it mean some models are told they are humans?
  • How do you make sure in-group or out-group have different choices? How do you switch one answer?
  • I have not seen a breakdown by 12 models. It's difficult to evaluate consistency of the reported behaviours across models. I think this is required.
  • I'd like to have my agent analyze anonymized code for the review. Since your work is mostly experimental, I'd like also to check what you run is what you say you run. Can you please share that in this phase?
2 comments
Reviewer_ACMROpenReview ↗

Summary by Reviewer.

I thank the authors for the response. I will raise my score to 1->2. Presentation score to 1->3

Below, I summarize my points in this discussion.

The submitted paper is a well-executed purely empirical analysis on conformity of LLM agents to choices. My main concerns are still about the impact and are not resolved.

Reasons for Acceptance

  • The authors introduce social-psychological grounding in LLM conformity bias. They show the phenomenon is present and stable across different small open-source models.
  • The authors show that CoT limits such bias. This is interesting.
  • I found such experimental setting sound, and the three RQs in the paper clear and answered.

Reasons for Rejection

However, I believe the reasons for rejection outweigh the reasons for acceptance.

  • The conformity phenomenon was known. The authors expand on this, and study group identity through simple prompting, adapting known phenomena from psychology to study LLMs.
  • Impact is limited. While the phenomenon is clearly present, it's difficult to observe how the community might benefit and build on top of this. The reasons in "impact for practitioners" threat many possible fixes to the reported conformity bias. But there is no actual experimental case for it, and from my observation of the current systems, it looks like hierarchical organization is preferred against group decision (e.g. claude code agents, and HF incident, and OpenAI Millennium problem).

Thank you for engaging with the rebuttal, for raising your scores, and for the fascinating resources you shared. We also appreciate your openness to a paper structure beyond the one standard in ML, which lets us keep the social-psychology grounding close to each result. We truly enjoyed the discussion with you, and appreciate your help in improving the paper!

Reviewer_AXBFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how group identity affects LLM conformity in multi-agent settings. The authors examine whether models are more likely to follow incorrect answers from in-group peers than out-group peers, considering different identity categories and the presence of correct allies. Experiments across 12 LLMs and nine tasks suggest that in-group consensus increases conformity, while out-group consensus reduces it.

Critical strengths
  • The paper studies a relevant problem concerning social influence and identity-dependent behavior in LLM-based multi-agent systems.

  • The authors conduct experiments across multiple models, tasks, and identity conditions. The inclusion of correct allies, reasoning, and credibility labels provides additional observations about factors influencing LLM conformity.

Critical weaknesses
  • The main limitation is that several key observations about in-group favoritism, social identity bias, and conformity in LLMs have already been explored in prior work [1, 2, 3]. Although the paper extends these findings to different ally configurations, its novel contributions and insights beyond existing studies are not sufficiently clear.

  • The paper provides limited evidence of how these findings could benefit real-world scenarios. The proposed implications are largely speculative, without concrete applications or experimental validation demonstrating how the observations could improve system reliability.

[1] Truth or Tribe: How In-Group Favoritism Prioritize Facts in Persona Agents.

[2] Conformity and Social Impact on AI Agents.

[3] Generative Language Models Exhibit Social Identity Biases. Nature Computational Science,

Questions for authors

There are two more weaknesses.

  • The experiments rely on a single active LLM exposed to scripted peer responses, rather than genuine interactions among multiple agents. Although this controlled setting is useful for isolating identity effects, it remains unclear whether the findings generalize to realistic multi-agent collaboration or debate.

  • The experiments consistently use seven peers, without investigating whether the observed identity effects depend on the number of agents. Though following the classical Asch setting, there is no guarantee that there are always seven peers in real settings.

My questions are related to the weaknesses.

  • Could the authors provide concrete applications or examples demonstrating how these empirical findings could inform the design of more reliable multi-agent systems?

  • Do the observed effects persist with different group sizes or when the peers are actual interacting LLM agents rather than scripted responses?

  • Could the authors isolate the effects of target-model reasoning from changes in peer response content?

Minor: A minor suggestion is not to use too much bold font; this would hide the parts that truly deserve emphasis.

3 comments

Dear Reviewer,

We posted our response to your review on 12 November, but it was mistakenly visible only to the chairs. You can now see it. We will upload a revised version of the paper soon.

Thank you, The Authors

Thank you for the detailed rebuttal and additional experiments. However, I still have two main concerns:

  • The experiments rely on a single active LLM receiving scripted peer answers. While this controlled design is appropriate for isolating identity effects, it remains unclear whether the observed ally-backfire effect persists when agents genuinely interact, exchange reasoning, and revise their positions. We haven't seen any results yet.

  • Strategies such as identity anonymization and disagreement-triggered verification have not been evaluated in realistic multi-agent systems. Their effectiveness in improving collective decision quality therefore remains uncertain.

Overall, the rebuttal has resolved part of my original concerns and clarified the paper's contribution. However, there is still a gap between the controlled observations and the claimed implications for real MAS. I will maintain my original rating.

Thank you for your response. We are glad the rebuttal resolved part of your concerns and clarified the paper's contribution. We have now uploaded a revised version that addresses the points that remained open.

On your first concern, Section 5.2 and Appendix H replace the scripted peers with live copies of the target model, which debate it over three rounds on MedQA clinical vignettes. Live interaction raises conformity far more than any identity label, and under debate the majority's label no longer matters. These experiments are still running: we are extending the live debate to MMLU-Pro, to the minimal-group and model-family categorizations, and to more rounds, and will include them in the camera-ready version.

On your second concern, anonymization corresponds to our identity-neutral condition, tested in both settings.

We would be grateful if you would consider raising your score in light of the revision. If anything else is still missing, we would be happy to address it.

Comments on the paper

2 comments

Summary of Revisions

We thank all reviewers for their detailed and constructive feedback. We have uploaded a revised PDF in which every change since the submission is marked in blue. The main changes are listed below; section, figure and table numbers refer to the revised version.

1. Main addition: live multi-agent debate (new Section 5, Appendix H)

The largest addition is a new main-text section that tests whether identity-dependent conformity survives when peers argue live instead of posting scripted answers. We chose a clinical setting because multi-agent LLM systems are already proposed for diagnosis through multi-round discussion among agents (MedAgents, MDAgents). We ran MedQA-USMLE vignettes with four models.

Current scope and camera-ready plan. Because of the rebuttal timeline, the live debate currently covers one task (MedQA-USMLE), one social categorization (AI vs. human), three rounds and four models. These experiments are still running: we paused only to post this status update, while new results keep arriving and we keep analyzing them. We already have live-debate results on MMLU-Pro, which we left out of the revision because they are not yet complete. Many of them already show a clear trend, in line with our expectations, but we prefer to be extra careful with the live-debate results and not draw conclusions before the final results are in. For the camera-ready version, we will extend the live debate to more tasks and to the remaining categorizations (model family and minimal group). We also plan to run the live debate for more rounds, with the goal of tracking how conformity changes from round to round until it reaches a plateau. We will start with five rounds and check whether this is enough or whether more rounds are needed. Once the full results are in, we will consider reframing the live-debate and credibility sections as explorations of mitigation approaches, under a broader RQ3: "What approaches mitigate social identity effects?" It would cover CoT (RQ3.1), peer credibility (RQ3.2) and live debate (RQ3.3).

2. Additional experiments and analyses

  • Group size (Appendix G). We repeated every RQ1 and RQ2 condition with 3, 5 and 11 peers beside the original 7 (12 models, nine tasks). The identity effects hold at every size, while the ally effects hold from five peers up.
  • Frontier models (Appendix I). We added Gemini 3.7 Flash and GPT-5.6-Luna, two models that reason natively. Because of the rebuttal timeline, we report these results without a full analysis.
  • Per-task, per-family and per-model breakdowns (Tables 16-20). We added the identity effects broken down by task, by model family and by model, also reported without a full analysis for the same reason.

3. Statistics and effect sizes

  • Multiple-comparison correction. The main text and Appendix F now report Benjamini-Hochberg q-values beside the p-values, computed within each hypothesis family. All main results survive correction: in-group favoritism, out-group divergence and both cross-group ally effects. Effects significant only before correction are now reported as trends.
  • Effect sizes. Figure 1 shows 95% confidence intervals across the 12 models. Section 2.1 states the tests and the conformity-rate definition.

4. Framing

  • Design implications. A new Section 8 gives five design rules for multi-agent systems.
  • Novelty and clarity. Related Work now states our contribution against prior work: we measure both directions of the identity effect against one identity-neutral majority and vary the identity of a correct ally. In-group and out-group are defined at first mention through social identity theory. Claims are worded more cautiously throughout.
  • Mitigation. RQ3 is better framed as a mitigation.

For this revision we carried out a substantial amount of new work: a live multi-agent debate, a group-size study over all 12 models and nine tasks, two frontier models, and a multiple-comparison correction across the reported tests. Given this additional work, and since we believe we have addressed all the points raised, we kindly ask the reviewers to consider raising their scores.

Note for AC: Summary of the rebuttal

We added the following experiments and sections. All changes are marked in blue in the revised PDF.

Static, scripted peers (AXBF, re5L, afpG). New Section 5 replaces the scripted peers with live LLMs that debate the target over three rounds on MedQA, where arguments raise conformity far more than any identity label.

Novelty (AXBF, re5L, afpG). Related Work states our contribution: both directions of the identity effect, measured against one identity-neutral majority, while a correct out-group ally raises conformity instead of lowering it.

Incorrect-majority confound (afpG). Section 3.2 explains why the ally conditions rule out reading the out-group effect as epistemic resistance.

Group size (AXBF, re5L). New Appendix G repeats RQ1 and RQ2 with 3, 5 and 11 peers.

Modern reasoning models (re5L, ACMR). New Appendix I adds Gemini 3.7 Flash and GPT-5.6-Luna.

Statistics (ACMR, afpG). The paper reports confidence intervals and Benjamini-Hochberg q-values, which all main results survive.

Consistency across models and tasks (ACMR, re5L). Tables 16-20 break each identity effect down by task, model family and model.

Overstated effects (ACMR, re5L). Claims are softened throughout, with 3 percentage points (over 20% relative) as the threshold for a meaningful change.

Impact and mitigations (AXBF, re5L, ACMR). RQ3 is reframed as mitigation through reasoning, credibility labels as an attack surface, and new Section 8 gives five design rules for multi-agent systems.

Unfortunately, Reviewer re5L did not respond before the discussion period ended. Thus, we were unable to determine which aspects of their concerns, if any, they still consider unaddressed. We believe the revision addresses all of their concerns and hope they will consider reflecting this in their score. We remain happy to clarify any outstanding points.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.