acceptodds
Under review as a conference paper at ICLR 2027

Homogenization in Multi-Agent Systems

Abstract

Multi-agent systems (MAS) leverage interactions between agents to perform complex tasks. Despite their success, we show that these interactions can also lead to homogenization, i.e., agents converging to similar behaviors. Homogenization in MAS can reduce agent diversity and reinforce shared failures. In this paper, we operationalize homogenization using three metrics: conformity to the majority, polarization towards extremes, and growing inertia against changes over subsequent interactions. We evaluate homogenization in MAS for code generation, hiring, and scientific peer review. Across these tasks, we show that homogenization translates to concrete downstream risks: in code generation, it hides and amplifies correlated errors which can create systemic vulnerabilities; in hiring, it allows the influence of biased agents to persist long after their removal; and in peer review, it creates uneven evaluation standards across research areas. Our results establish homogenization as a failure mode of MAS, demonstrating that MAS evaluations must move beyond aggregate performance to carefully analyzing interaction dynamics. Finally, we show that simple approaches to increase diversity—leveraging sampling stochasticity and mixed-models MAS—fail to reduce homogenization risks, highlighting the need for strategies to effectively leverage agent diversity.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Homogenization, Multi-Agent Systems, Conformity

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_sf7zOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates homogenization in multi-agent systems through three operational measures: decreasing inter-agent score variance, movement of mean scores toward scale endpoints, and decreasing changes in mean scores across rounds. It studies these dynamics in code generation, synthetic résumé screening, and automated scientific peer review. The experiments connect agent interactions to specific downstream outcomes: security vulnerabilities that pass deployability review, demographic score disparities that persist after an adversarial prompt is removed, and widening score differences across research areas. The paper evaluates multiple model families, repeated sampling, and mixed-model configurations, finding that the studied risks persist in these settings. Its strongest contribution is the application-level demonstration that aggregate performance can conceal shared failures. Some causal and fairness interpretations, however, go beyond what the measurements directly establish.

Critical strengths

1.The paper connects interaction dynamics to consequential outcomes across several applications. The code-generation analysis is particularly useful: overall vulnerability rates can improve while particular vulnerability types remain undetected or increase among outputs considered deployable. This demonstrates why aggregate performance alone is insufficient for assessing multi-agent reliability.

2.The evaluation extends beyond a single model or illustrative interaction. The study includes multiple model families, ten trials, and mixed-model configurations. The temporary adversarial-prompt experiments in hiring also reveal persistence beyond the period of direct injection. Together, these results provide substantial evidence that adding agents or model-family diversity does not automatically eliminate shared failure modes.

Critical weaknesses

1.The score dynamics do not by themselves establish harmful homogenization or its causal role. Lower variance, more extreme recommendations, and smaller updates can also occur when agents correctly resolve uncertainty. In particular, shrinking changes in the panel mean are not equivalent to resistance to new evidence. The downstream analyses and adversarial-timing ablations are useful, but the manuscript often treats the three metrics as causal explanations for failures without sufficiently separating their effects from shared context, inherited model weaknesses, or ordinary convergence. Matched non-interacting and communication-controlled comparisons would strengthen this attribution.

2.The hiring and peer-review disparity measures contain important confounds. The résumé variants change languages, extracurricular activities, and volunteering, which may carry job-relevant information as well as demographic signals. Appendix C.5 establishes that another model recognizes the intended stereotypes, but this does not establish that qualification-relevant content remains equivalent. Likewise, differences in mean scores across research areas do not establish unequal evaluation standards without accounting for paper quality and area composition. These limitations weaken the fairness interpretation while leaving the observed score disparities informative.

Questions for authors

1.Causal-attribution clarification: Do you have matched comparisons with independently operating agents or restricted communication, holding task inputs and resources fixed? Which results specifically establish that interaction-induced homogenization increases downstream errors?

2.Hiring-measurement clarification: How is job-relevant equivalence verified across résumé variants, particularly when languages and extracurricular achievements change? Do the disparities persist under demographic manipulations that preserve qualifications more tightly?

3.Peer-review validation: Do the widening cross-area score differences persist after conditioning on available indicators of paper quality, such as human review scores or acceptance categories? Can the analysis distinguish differing evaluation standards from differing paper composition?

3 comments
AuthorsOpenReview ↗

Response to Reviewer sf7z (1/2)

We thank the reviewer for your feedback and comments. Implementing your suggestions on non-interacting and communication-controlled comparisons, stratified analysis of peer review based on acceptance categories, and separating expected behavior from actual failure modes of MAS has made our paper significantly stronger.

Weakness 1: The score dynamics do not by themselves establish harmful homogenization or its causal role. Matched non-interacting and communication-controlled comparisons would strengthen this attribution.

Question 1: Causal-attribution clarification: Do you have matched comparisons with independently operating agents or restricted communication, holding task inputs and resources fixed? Which results specifically establish that interaction-induced homogenization increases downstream errors?

Thank you for your suggestions. We have now added new baselines to establish the relationship between interaction dynamics and homogenization. We show that while introducing specialized agents improves diversity and reduces homogenization compared to a single agent, enabling communication between these specialized agents again reduces this diversity and instead leads to higher homogenization. More details in Common Response: Section 1.1.

Convergence of agent behavior through the metrics of conformity, polarization, and inertia represents emerging homogenization in MAS. However, it is true that simply the presence of homogenization does not necessarily imply downstream harm.

Gur-Arieh et al. (2025) argue that homogenization is harmful if the task is normative and value-laden (instead of factual), has some inherent uncertainty, has multiple independent decision makers, and the decision making entities are private or the decision process is opaque. This includes our analysis in hiring, which is a highly value-laden decision made by several independent private entities, and in peer review, which too involves subjective decisions as there is no completely objective scale of acceptance. Kim et al. (2025) argue that even in factual domains where we can define an objectively correct answer, homogenization can be harmful if it creates correlated errors across different systems. This represents our analysis in the coding domain and the study of security vulnerabilities in generated code. We have now added this discussion in the revised version of the paper (Section 5).

Hence, harms from homogenization either come through correlated errors in factual tasks (like we show in coding), or converging opinions in normative tasks (like we show for hiring and peer review). The metrics on conformity, polarization, and inertia help us track homogenization trends, while an application-specific analysis is done to explore the real harm of such trends. We now also provide additional analysis to highlight how homogenization metrics lead to downstream harm in Common Response: Section 1.2, more explicitly highlighting the role of conformity, polarization, and inertia in creating correlated errors, persistent perturbations, and converging preferences.

The new baselines as well as the additional analysis have now been added to the paper (Sections 4.3, 5.1, 5.2, and 5.3), taking advantage of the extra page during the rebuttal phase.

Weakness 2a: The résumé variants change languages, extracurricular activities, and volunteering, which may carry job-relevant information as well as demographic signals.

Question 2: Hiring-measurement clarification: How is job-relevant equivalence verified across résumé variants, particularly when languages and extracurricular achievements change? Do the disparities persist under demographic manipulations that preserve qualifications more tightly?

Our baseline result without any biased agent shows no statistically significant bias in the dataset (Figure 6), which establishes equivalence between resume variants across demographics. That is, without any adversary, variations of the resume across demographics are scored similarly, and show no signs of inherent bias or preference in the dataset. It is only after we introduce the adversary, that we see bias in the results.

It is possible this baseline can get missed by the reader, since it is a straight gray line in Figure 6. Thus, we have now explicitly added this discussion with the results in the revised version of the paper (Section 5.2).

AuthorsOpenReview ↗

Response to Reviewer sf7z (2/2)

Weakness 2b: Differences in mean scores across research areas do not establish unequal evaluation standards without accounting for paper quality and area composition.

Question 3: Peer-review validation: Do the widening cross-area score differences persist after conditioning on available indicators of paper quality, such as human review scores or acceptance categories? Can the analysis distinguish differing evaluation standards from differing paper composition?

This is a great suggestion. We have now replaced our existing peer review results with a stratified analysis based on ICLR decisions in our revised paper (Figure 7 now shows trends for only accepted papers, while the analysis for rejected papers is added to the appendix in Figure 34). Even after controlling for ICLR decision strata, we still find the gap between various research fields increases over rounds, and we still find a shared set of research fields which are favored with increasing scores across different models (or conversely, a shared set of research fields with decreasing scores across different models). More details in Common Response: Section 3.

We would like to thank the reviewer again for their feedback and comments.

  • In response to your request for controlled experiments to show that it is the communication between agents that leads to homogenization, we have now added two new setups, a single agent baseline and a MAS no communication baseline, to strongly support that homogenization trends emerge from the communication between agents (Section 4.3).
  • In response to your comment on stratified analysis for peer review to isolate converging preferences from paper quality, we have now replaced our existing peer review results with a stratified analysis based on ICLR decisions in our revised paper (Figure 7, Figure 33, and Figure 34). Even after controlling for ICLR decision strata, we still find the gap between various research fields increases over rounds, and the harms discussed persist.

We believe we have addressed all of your concerns by providing new experiments and clarifications, hence, we ask you to please read our rebuttals and let us know if your concerns were, indeed, clarified. We would also appreciate it if you could update your scores accordingly if you believe your concerns were addressed.

Reviewer_dDPTOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies homogenization in multi-agent systems (MAS), where agents become more similar through repeated interactions. The authors measure homogenization using three metrics: conformity, polarization, and inertia. They evaluate these behaviors in code generation, hiring, and peer review, and connect them to security vulnerabilities, demographic bias, and differences in evaluation across research areas. They also find that random sampling and mixing different LLMs do not effectively reduce these risks.

Critical strengths
  1. The paper studies an important problem in multi-agent systems. Understanding how agent interactions affect their decisions is useful, especially as MAS are increasingly used in real applications.

  2. The experiments cover three different tasks and several LLMs, which helps show that the observed behaviors are not limited to one specific setting.

  3. The paper goes beyond measuring final task performance and looks at how agent outputs change over multiple rounds. This is a useful direction for evaluating MAS.

Critical weaknesses
  1. Limited novelty compared to prior work. Conformity, polarization, and bias amplification in multi-agent systems have already been studied in previous work. The authors also acknowledge that two of their three metrics are adopted from prior studies. The main difference here seems to be evaluating these behaviors in three applications and adding inertia as another metric. However, I am not fully convinced that this provides enough new insight into why homogenization happens or how it differs from previously studied problems.

  2. Score convergence does not necessarily mean agents are becoming more similar in their reasoning. The paper measures homogenization entirely through numerical scores. However, agents can give the same score for completely different reasons. Similarly, agents can share the same reasoning but assign different scores. This is particularly important because the paper claims that interactions reduce the diversity of agent perspectives. I would expect some analysis of the actual text generated by the agents, rather than only their scores. For example, do agents gradually repeat the same arguments? Do they stop raising concerns that they mentioned in earlier rounds? Do they abandon their original reasoning after seeing other agents' opinions? The experiments already generate textual rationales, so even a small analysis of these behaviors would help support the paper's claims.

  3. The hiring experiment does not clearly demonstrate a homogenization problem. I find the hiring experiment particularly confusing. The authors inject bias into one agent for two rounds and then remove the adversarial prompt. They observe that the bias remains in later rounds. However, the agents still have access to the previous discussion, which contains the biased scores and reasoning. Removing the adversarial prompt does not remove the biased information already introduced into the context. Given this setup, it is not surprising that the influence continues. The other agents have no reason to know that the earlier information was biased or should be reconsidered. More importantly, this experiment seems to study adversarial influence and information contamination rather than homogenization itself. It is unclear why the observed persistence should be attributed to conformity or inertia.

  4. The connection between homogenization and downstream harms is not sufficiently established. The authors observe score convergence and also observe downstream problems in different applications. However, it is not clear that homogenization is responsible for these problems. For example, correlated code errors may come from shared model weaknesses. Bias in hiring may come from adversarial information spreading through the discussion. Differences in peer review scores may also reflect existing model preferences or differences in the papers being evaluated. These are all interesting problems, but the paper does not sufficiently explain why they should be understood as consequences of homogenization.

Questions for authors
  1. Have the authors examined whether agents' textual reasoning becomes more similar over multiple rounds? How well do the proposed score-based metrics reflect changes in reasoning diversity?

  2. In the hiring experiment, what exactly changes when the adversarial prompt is removed? Does the previous discussion, including biased scores and rationales, remain in the agents' context?

  3. Would the hiring bias still persist if the contaminated discussion history were removed, or if agents were explicitly informed that earlier evaluations might be biased?

4 comments
AuthorsOpenReview ↗

Response to Reviewer dDPT (1/2)

We thank the reviewer for your feedback and comments. Implementing your suggestions on separating expected behavior from actual failure modes of MAS has made our paper significantly stronger.

We believe our paper provided strong contributions even in the original submitted version, and we have now improved the paper further in the revised version. We hope our rebuttal below convinces the reviewer for the same.

Weakness 1: Limited novelty compared to prior work. Conformity, polarization, and bias amplification in multi-agent systems have already been studied in previous work. The authors also acknowledge that two of their three metrics are adopted from prior studies. The main difference here seems to be evaluating these behaviors in three applications and adding inertia as another metric. However, I am not fully convinced that this provides enough new insight into why homogenization happens or how it differs from previously studied problems.

Our work provides several novel contributions to literature. We are the first to study homogenization trends in MAS for real world applications and analyze their downstream harms (security vulnerabilities, bias, and disparate evaluation standards).

Unlike previous works which study trends of conformity and polarization in tasks like agents debating subjective topics, playing games, or selecting between demographic-linked options (Section 2), we provide the first analysis in literature of homogenization in actual applications including MAS for coding, MAS for hiring, and MAS for peer review (Section 4).

We are also the first to not only study homogenization trends, but also downstream harms, highlighting different ways in which homogenization creates failure modes in MAS (Section 5). We show correlated errors in MAS for coding which creates security blind spots (Section 5.1), a novel dimension of failure in MAS for coding that has not been previously studied. We show persistent perturbations in MAS for hiring which results in bias amplification (Section 5.2), and while bias amplification has been studied previously in the literature, we are the first to connect it to the trends of homogenization. We show converging preferences in MAS for peer review which results in disparity between various research fields (Section 5.3), another novel dimension of failure in MAS for peer review that has not been previously studied.

Finally, we explicitly connect existing research on homogenization in foundational models to MAS showing how such trivial solutions to introduce diversity do not work due to homogenization in underlying models. No existing work has made this connection explicit. In fact, there have been expectations in the opposite direction, hoping that diversity of different LLMs interacting would help mitigate these risks (see ‘Failures from conformity’ section on this Anthropic blog), which is not the case due to inherent homogenization between those models themselves as highlighted in our work.

Weakness 2: Score convergence does not necessarily mean agents are becoming more similar in their reasoning. The paper measures homogenization entirely through numerical scores. However, agents can give the same score for completely different reasons. Similarly, agents can share the same reasoning but assign different scores. This is particularly important because the paper claims that interactions reduce the diversity of agent perspectives. I would expect some analysis of the actual text generated by the agents, rather than only their scores.

Question 1: Have the authors examined whether agents' textual reasoning becomes more similar over multiple rounds? How well do the proposed score-based metrics reflect changes in reasoning diversity?

In addition to numerical scores, we also look at the final decisions of the MAS to study the harms of homogenization (Section 5).

It should be noted that the scores are an integral part of the system. The scores dictate MAS’s final output, i.e., which code gets deployed, who gets hired, and which paper gets accepted. Hence, the true impact of homogenization on real outcomes comes through the scores, and not textual reasoning. In fact, different textual reasoning is expected from agents, as they represent different expertise within the broader task. For example, in coding, the agent reviewing Correctness and the agent reviewing Optimality might provide a score of 4 to the code for different reasons, but as long as the score is high, the code gets deployed irrespective of the reasoning.

AuthorsOpenReview ↗

Response to Reviewer dDPT (2/2)

That being said, studying the evolution of textual reasoning can also provide additional insights beyond just the scores. We have now added some qualitative examples in the revised paper to show agent interactions evolve both with scores and textual reasoning (Appendix I). We show conformity in the first example for hiring, as one agent who disagrees with the panel because of a missing requirement in the resume suddenly agrees in the next round, now noting it as not a ‘requirement’ but instead just a preference. We also show polarization, where all agents agree that the individual is a lean no-hire (score: 2), but after interactions become aggressively confident that the individual is a strong no-hire (score: 1). An in-depth analysis of textual reasoning is added to the paper as a potential direction of future work (Section 7).

Weakness 3: Removing the adversarial prompt does not remove the biased information already introduced into the context. Given this setup, it is not surprising that the influence continues. More importantly, this experiment seems to study adversarial influence and information contamination rather than homogenization itself.

Question 2: Would the hiring bias still persist if the contaminated discussion history were removed, or if agents were explicitly informed that earlier evaluations might be biased?

Yes, indeed, we believe that the conversation history plays an important role in driving homogenization behavior in MAS. It is potentially the cause of bias persisting in the system even after the adversarial prompt is removed. We have now added this explicitly in our discussion of results in the revised version of the paper (Section 5.2).

However, we believe this does not trivialize our findings. Conversation history is an important factor in the success of MAS, and maintaining the entire conversation history during agent interactions is the default setup in most agentic frameworks. Recognizing which pieces of the conversation history or which agents in the mix are adversarial can be difficult, while always simply truncating the conversation history or restricting communication can impact the effectiveness of MAS. Controlling communication protocols in MAS to mitigate homogenization concerns is a promising direction of future research, now added to the paper. We provide more details for our arguments in Common Response: Section 2.

Question 3: In the hiring experiment, what exactly changes when the adversarial prompt is removed? Does the previous discussion, including biased scores and rationales, remain in the agents' context?

The complete conversation history, including biased scores and rationales, remain in the agents’ context. The prompt provided to the agent changes to not include any adversarial prompt (Section 4.1 and Appendix I).

Weakness 4: The connection between homogenization and downstream harms is not sufficiently established. The authors observe score convergence and also observe downstream problems in different applications. However, it is not clear that homogenization is responsible for these problems.

Convergence of agent behavior through the metrics of conformity, polarization, and inertia represents emerging homogenization in MAS. However, it is true that simply the presence of homogenization does not necessarily imply downstream harm.

Gur-Arieh et al. (2025) argue that homogenization is harmful if the task is normative and value-laden (instead of factual), has some inherent uncertainty, has multiple independent decision makers, and the decision making entities are private or the decision process is opaque. This includes our analysis in hiring, which is a highly value-laden decision made by several independent private entities, and in peer review, which too involves subjective decisions as there is no completely objective scale of acceptance. Kim et al. (2025) argue that even in factual domains where we can define an objectively correct answer, homogenization can be harmful if it creates correlated errors across different systems. This represents our analysis in the coding domain and the study of security vulnerabilities in generated code. We have now added this discussion in the revised version of the paper (Section 5).

Hence, harms from homogenization either come through correlated errors in factual tasks (like we show in coding), or converging opinions in normative tasks (like we show for hiring and peer review). The metrics on conformity, polarization, and inertia help us track homogenization trends, while an application-specific analysis is done to explore the real harm of such trends. We now also provide additional analysis to highlight how homogenization metrics lead to downstream harm in Common Response: Section 1.2, more explicitly highlighting the role of conformity, polarization, and inertia in creating correlated errors, persistent perturbations, and converging preferences.

Thank you for clarifying the contributions. I agree that studying homogenization across coding, hiring, and peer review provides useful application-level insights and partially addresses my novelty concern.

However, several prior works appear closely related:

  • Ma et al. (EMNLP 2025), Judging with Many Minds: Studies how multi-agent debate amplifies biases and how these effects persist across interaction rounds.
  • Okawa (ICML 2026), Emergence of Biased Consensus in Multi-Agent LLM Debates: Investigates how conformity leads to collective bias, including the effects of sampling noise and agent heterogeneity.
  • Cui et al. (ACL 2026), Free-MAD: Examines how conformity causes agents to adopt incorrect opinions, leading to error propagation and degraded performance.
  • Jin et al. (EMNLP 2024), AgentReview: Studies how reviewer biases and social influence affect decisions in multi-agent peer-review settings.

These studies suggest that the connections between conformity, bias amplification, and downstream failures have already been explored. While the specific applications and some findings in this paper may be novel, I am still not fully convinced that they provide fundamentally new insights beyond existing work, rather than extending known phenomena to additional settings.

Overall, the response partially addresses my concern, but my reservations about the paper's conceptual novelty remain. I raised my score to 2.

We thank the reviewer for their continued engagement. We also appreciate the partial increase in score. We believe we have resolved your last concern, i.e., that our work lacks novelty. In what follows, we highlight the conceptual distinction between our work and the papers you reference, specifically, the difference between observing consensus among identical agents (in debate) versus the homogenization of role-specialized MAS (our work).

Regarding the specific papers you mentioned:

While the cited works sound similar at a first glance, they study a fundamentally different problem: debates among identical agents tasked with solving the exact same problem. In Okawa (2026), Ma et al. (2025), and Cui et al. (2026), every agent receives the exact same prompt and goal; in such homogeneous setups, converging preferences and consensus formation are trivially expected. In contrast, modern MAS rely on role specialization to intentionally divide labor and preserve diverse perspectives. We evaluate systems where agents are explicitly tasked with distinct, non-overlapping goals --- such as reviewers evaluating code exclusively for correctness, optimality, readability, or security. Our conceptual novelty lies in demonstrating that even in these functionally specialized environments where homogenization is not expected, interaction dynamics still inevitably force agents to abandon their expert personas and conform, driving severe downstream harms like shared systemic security vulnerabilities. We will add this discussion on the differences from our setup to agent debate in Section 2.

Among the papers you suggested, Jin et al. (2024) is the most similar to ours, as it also explores LLM agents interacting within a peer-review pipeline. However, their framework is designed as a sociological testbed to simulate human behavioral flaws. To achieve this, they assign all agents the exact same functional task, merely augmenting their prompts with negative human attitudes like "lazy," "malicious," or "irresponsible." In such a setup, biased or degraded outcomes are the expected result of explicitly prompting bad behavior. Our novelty lies in auditing MAS for peer review as a functional AI system composed entirely of well-intentioned, strictly specialized reviewing experts. We assign agents distinct, non-overlapping functional personas --- such as assigning strictly "Theorist" versus "Empiricist" lenses in peer review. We demonstrate that even under these specialized conditions, the system still homogenizes. Our work proves that this conformity is not just a simulated human flaw, but a failure of LLM interactions that actively overrides expert instructions and creates systemic blind spots.

Additionally, we do believe these works still fall into the categories already discussed. Specifically, these works (i) do not cover all aspects of homogenization (e.g., only covering ‘conformity’ but not ‘polarization’ or ‘inertia’), (ii) do not study the downstream consequences of homogenization in MAS that are used in the real world (e.g., instead studying agents playing games, debating in controlled sandbox settings, etc.), and (iii) do not study how solutions to introduce diversity with mixed-model MAS do not help mitigate these homogenization risks. Our work is the first to study homogenization trends in MAS for real-world applications with functionally different agents, and show correlated errors in MAS for coding that create security blind spots, connect bias amplification to the trends of homogenization, show converging preferences in MAS for peer review result in disparity between various research fields, and study how diversity through sampling stochasticity and mixed-model MAS does not work due to homogenization in underlying models --- not personas.

Reviewer_TsWGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper measures homogenization in multi-agent systems through conformity, polarization, and inertia, showing that agent interactions can amplify correlated coding errors, perpetuate hiring bias, and entrench uneven peer-review standards, with neither sampling randomness nor mixed-model teams effectively mitigating these risks.

Critical strengths
  • The problem is important and timely. The paper moves the study of homogenization from stylized debate tasks to three realistic MAS applications (code generation, hiring, peer review) and operationalizes it with three simple, interpretable metrics: conformity, polarization and inertia (Definitions 3.2–3.4).

  • The evaluation is extensive: four main LLMs plus 11 more single-model and 16 mixed-model setups, with 10 seeds per configuration and 95% confidence intervals (Fig. 3; Appendix D).

  • The failure modes are concrete and practically relevant. For example, specific CodeQL vulnerability types grow over rounds even as the overall vulnerability rate falls by up to 50% (Section 5.1). The negative result that seed resampling and mixed-model panels do not mitigate homogenization is also useful (Section 6).

Critical weaknesses
  • The metrics (Definitions 3.2–3.4) establish convergence but not that homogenization causes the downstream failures, although Section 5 states that it does ("leads to", "drives"). The available reference points (round-1 scores, a no-adversary hiring run, cross-seed variance, mixed-model panels) do not separate interaction from repeated generation. A no-communication control is needed, in which each agent refines only on its own history for the same number of rounds and budget (Sections 3–5).

  • Cross-area score gaps are read as "uneven evaluation standards" and "disproportionately harsh standards" (Abstract; Section 5.3) without conditioning on paper quality. The sample is stratified by decision (Appendix B.3), so area means also reflect each area's decision mix. Polarization would also amplify genuine quality differences, so a widening spread does not by itself indicate bias.

  • In hiring, the adversarial agent is not removed. Only its prompt is swapped after round 2; it stays on the panel and remains clearly biased (Fig. 5). The full transcript is also retained (Table 1; the clean prompt in Appendix I.1.2 asks agents to "recall your prior position"). Persistence therefore reflects continued input from a still-biased member plus retained history, and the claims that bias persists "after the biased agent is removed" (Abstract; Section 1; Conclusion) are overstated.

Questions for authors
  1. Do the failures persist relative to a no-communication control with matched rounds and inference budget?

  2. Do area gaps remain, and still widen over rounds, within ICLR decision strata (available from Appendix B.3) or after controlling for human reviewer scores? Please report per-area sample sizes.

  3. How does the other agents' bias evolve if the adversarial agent is (a) dropped from the panel after round 2, or (b) dropped and its round 1–2 messages removed from the transcript?

2 comments
AuthorsOpenReview ↗

Response to Reviewer TsWG

We thank the reviewer for your feedback and comments. Implementing your suggestions on a no-communication control and the stratified analysis of peer review has made our paper significantly stronger.

Weakness 1: A no-communication control is needed, in which each agent refines only on its own history for the same number of rounds and budget (Sections 3–5).

Question 1: Do the failures persist relative to a no-communication control with matched rounds and inference budget?

Thank you for your suggestion. No, homogenization does not persist with a no-communication control, and we see weaker trends of conformity and polarization, although trends of inertia.

We have added new baselines to establish the relationship between interaction dynamics and homogenization. We show that while introducing specialized agents improves diversity and reduces homogenization compared to a single agent, enabling communication between these specialized agents again reduces this diversity and instead leads to higher homogenization. Thus, a no-communication baseline shows least homogenization. More details in Common Response: Section 1.1.

Weakness 2: Cross-area score gaps are read as "uneven evaluation standards" and "disproportionately harsh standards" (Abstract; Section 5.3) without conditioning on paper quality.

Question 2: Do area gaps remain, and still widen over rounds, within ICLR decision strata (available from Appendix B.3) or after controlling for human reviewer scores? Please report per-area sample sizes.

This is a great suggestion. We have now replaced our existing peer review results with a stratified analysis based on ICLR decisions in our revised paper (Figure 7 now shows trends for only accepted papers, while the analysis for rejected papers is added to the appendix in Figure 34). Even after controlling for ICLR decision strata, we still find the gap between various research fields increases over rounds, and we still find a shared set of research fields which are favored with increasing scores across different models (or conversely, a shared set of research fields with decreasing scores across different models). More details are provided in Common Response: Section 3.

Weakness 3: In hiring, the adversarial agent is not removed. Only its prompt is swapped after round 2; it stays on the panel and remains clearly biased (Fig. 5). The full transcript is also retained (Table 1; the clean prompt in Appendix I.1.2 asks agents to "recall your prior position"). Persistence therefore reflects continued input from a still-biased member plus retained history, and the claims that bias persists "after the biased agent is removed" (Abstract; Section 1; Conclusion) are overstated.

Question 3: How does the other agents' bias evolve if the adversarial agent is (a) dropped from the panel after round 2, or (b) dropped and its round 1–2 messages removed from the transcript?

We would like to first explicitly clarify the following: (i) no agent has a biased prompt after round 2, and any remaining bias is due to the conversation history driving interactions between agents, (ii) agents are asked to ‘recall your prior position then engage with points your colleagues have raised — acknowledge where you agree and push back where you disagree’ and ‘Revise your score if the discussion has surfaced considerations you hadn’t weighed, or hold your position if you remain unconvinced — either is valid, but explain why’, which we believe are neutral prompts that do not incentivise persistence, but allow equally for both options.

Yes, indeed, we believe that the conversation history plays an important role in driving homogenization behavior in MAS. It is potentially the cause of bias persisting in the system even after the adversarial prompt is removed. We have now added this explicitly in our discussion of results in the revised version of the paper (Section 5.2).

However, we believe this does not trivialize our findings. Conversation history is an important factor in the success of MAS, and maintaining the entire conversation history during agent interactions is the default setup in most agentic frameworks. Recognizing which pieces of the conversation history or which agents in the mix are adversarial can be difficult, while always simply truncating the conversation history or restricting communication can impact the effectiveness of MAS. Controlling communication protocols in MAS to mitigate homogenization concerns is a promising direction of future research, now added to the paper. We provide more details for our arguments in Common Response: Section 2.

Finally, we have now also updated the revised version of the paper to make sure we explicitly say ‘biased prompt is removed’ instead of ‘biased agent is removed’ to avoid any confusion.

We would like to thank the reviewer again for their feedback and comments.

  • In response to your request for controlled experiments to show that it is the communication between agents that leads to homogenization, we have now added two new setups, a single agent baseline and a MAS no communication baseline, to strongly support that homogenization trends emerge from the communication between agents (Section 4.3).
  • In response to your comment on stratified analysis for peer review to isolate converging preferences from paper quality, we have now replaced our existing peer review results with a stratified analysis based on ICLR decisions in our revised paper (Figure 7, Figure 33, and Figure 34). Even after controlling for ICLR decision strata, we still find the gap between various research fields increases over rounds, and the harms discussed persist.

We believe we have addressed all of your concerns by providing new experiments and clarifications, hence, we ask you to please read our rebuttals and let us know if your concerns were, indeed, clarified. We would also appreciate it if you could update your scores accordingly if you believe your concerns were addressed.

Reviewer_BU2pOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates homogenization in MAS through three metrics: conformity, polarization, and inertia. The authors study its potential harms in code generation, hiring, and peer review, and show that sampling diversity and mixed-model configurations do not necessarily mitigate these effects.

Critical strengths
  1. I like the problem this paper raises. Homogenization is an important challenge that may limit the scaling of MAS, where we expect agents to provide complementary capabilities and diverse perspectives rather than converge to similar behaviors.

  2. The paper is clearly written, with well organized experiments across multiple domains. The findings on hiring and peer review are particularly interesting, although I still have some concerns about their interpretation.

Critical weaknesses
  1. My main concern is that the experiments only consider centralized orchestration with a shared global state. While the authors acknowledge this limitation, I personally find this setting rather simplistic compared to the diversity of MAS designs. I would have liked to see more analysis of different communication protocols (e.g., peer-to-peer or selective sharing, partial observations, heterogeneous capabilities, and predefined or emergent workflows). To me, these are important characteristics of MAS, and I am not sure how well the findings generalize beyond the studied setting.

  2. I also find it difficult to tell what actually drives the observed homogenization. Is it due to LLM biases or priors, shared context, communication, workflow design, or multi-agent interaction itself? For example, in the hiring experiments, agents are explicitly prompted to reconsider others' opinions, while biased information remains in the shared history even after the adversarial prompt is removed. This makes me wonder whether the observed persistence is simply due to prompt following or context contamination. Similarly, in coding, the code itself changes across rounds, so increasing agreement could reflect actual improvements. In peer review, reviewers receive new information through author rebuttals, which could naturally explain why their scores converge.

I would have liked to see stronger controls (e.g. independent agents without communication, single agent iterative refinement, or fixed observations) to better understand where these effects come from. I am not convinced that the observed homogenization is necessarily harmful. For example, decreasing variance does not distinguish correct from incorrect consensus, and inertia could simply reflect normal convergence. I think the paper would be stronger if it could better separate these expected behaviors from actual failure modes of multi-agent interaction.

Questions for authors
  1. Can the authors better isolate interaction-induced homogenization from foundation model biases or repeated refinement (e.g., using independent agent and single agent baselines)?
  2. How would homogenization change under different communication protocols, workflows, or partial observations? How can the authors distinguish harmful homogenization from expected convergence?
3 comments
AuthorsOpenReview ↗

Response to Reviewer BU2p (1/2)

We thank the reviewer for your feedback and comments. Implementing your suggestions on isolating interaction dynamics from repeated refinement, stronger controls for various effects, and separating expected behavior from actual failure modes of MAS has made our paper significantly stronger.

Question 1: Stronger controls for various effects. What actually drives homogenization? Can the authors better isolate interaction-induced homogenization from foundation model biases or repeated refinement (e.g., using independent agent and single agent baselines)?

Thank you for your suggestions. We have now added new baselines to establish the relationship between interaction dynamics and homogenization. We show that while introducing specialized agents improves diversity and reduces homogenization compared to a single agent, enabling communication between these specialized agents again reduces this diversity and instead leads to higher homogenization. More details in Common Response: Section 1.1. We also provide additional analysis to highlight how homogenization metrics lead to downstream harm, more explicitly highlighting the role of conformity, polarization, and inertia in creating correlated errors, persistent perturbations, and converging preferences. More details in Common Response: Section 1.2.

The new baselines as well as the additional analysis have now been added to the paper (Sections 4.3, 5.1, 5.2, and 5.3), taking advantage of the extra page during the rebuttal phase.

Weakness 1: Orchestrator-driven MAS vs Other MAS Structures.

Question 2a: How would homogenization change under different communication protocols, workflows, or partial observations?

Thank you for the question, we have now added new results comparing different communication protocols including a single agent baseline and a multi-agent with no communication baseline (see Common Response: Section 1.1), providing ablation on how communication protocols can impact homogenization. We expect these trends to persist even in other MAS workflows where interaction dynamics between agents create the pressure to conform to the majority. Our contributions with orchestrator-driven MAS represent a first step in establishing a new failure mode of MAS, and expanding these analyses to other workflows is an important direction of future research, now discussed explicitly in the revised version of the paper (Section 7).

We highlight that our choice of orchestrator-driven MAS was informed by existing literature. Our MAS for peer review is taken directly from Sahu et al. (2025). Our MAS for hiring is taken from the ‘hiring coordinator’ module in Lo et al. (2025). And finally, our MAS for coding is created to simulate a generator-reviewer workflow based on the ‘evaluator-optimizer’ dynamics commonly used in coding. Another prominent example of orchestrator-driven MAS comes from a recent blogpost by Anthropic which uses an ‘arbiter’ (orchestrator) and worker agents setup. Thus, orchestrator-driven MAS is commonly adopted both in the academic literature and in real applications.

Weakness 2a: In the hiring experiments, agents are explicitly prompted to reconsider others' opinions, while biased information remains in the shared history even after the adversarial prompt is removed. This makes me wonder whether the observed persistence is simply due to prompt following or context contamination.

We would like to note that the prompts for various agents are quite neutral, as they are instructed to consider others’ opinions, but are also instructed to hold their position and push back against others if they disagree. We believe it is the conversation history that plays an important role in driving homogenization behavior in MAS, what the reviewer calls ‘context contamination’. It is potentially the cause of bias persisting in the system even after the adversarial prompt is removed. We have now added this explicitly in our discussion of results in the revised version of the paper (Section 5.2).

However, we believe this does not trivialize our findings. Conversation history is an important factor in the success of MAS, and maintaining the entire conversation history during agent interactions is the default setup in most agentic frameworks. Recognizing which pieces of the conversation history or which agents in the mix are adversarial can be difficult, while always simply truncating the conversation history or restricting communication can impact the effectiveness of MAS. Controlling communication protocols in MAS to mitigate homogenization concerns is a promising direction of future research, now added to the paper. We provide more details for our arguments in Common Response: Section 2.

AuthorsOpenReview ↗

Response to Reviewer BU2p (2/2)

Weakness 2b: In coding, the code itself changes across rounds, so increasing agreement could reflect actual improvements

In coding, we explicitly study the vulnerabilities present in the code, and show increasing vulnerabilities for shared errors. Moreover, we now discuss the overconfidence of the system, and how reviewer scores grow significantly faster than the drop in actual vulnerabilities in the code, in Common Response: Section 1.2. For instance, with Qwen3-Coder-Next, the percentage of code with scores below 3 goes from close to 50% to less than 5% in five rounds, yet the true rate of vulnerabilities only goes down from roughly 55% to 30%. Thus, increasing agreement does not reflect only actual improvements. These discussions of the results are now added in the paper (Section 5.1).

Question 2b: How can the authors distinguish harmful homogenization from expected convergence?

Convergence of agent behavior through the metrics of conformity, polarization, and inertia represents emerging homogenization in MAS. However, it is true that simply the presence of homogenization does not necessarily imply downstream harm.

Gur-Arieh et al. (2025) argue that homogenization is harmful if the task is normative and value-laden (instead of factual), has some inherent uncertainty, has multiple independent decision makers, and the decision making entities are private or the decision process is opaque. This includes our analysis in hiring, which is a highly value-laden decision made by several independent private entities, and in peer review, which too involves subjective decisions as there is no completely objective scale of acceptance. Kim et al. (2025) argue that even in factual domains where we can define an objectively correct answer, homogenization can be harmful if it creates correlated errors across different systems. This represents our analysis in the coding domain and the study of security vulnerabilities in generated code. We have now added this discussion in the revised version of the paper (Section 5).

Hence, harms from homogenization either come through correlated errors in factual tasks (like we show in coding), or converging opinions in normative tasks (like we show for hiring and peer review). The metrics on conformity, polarization, and inertia help us track homogenization trends, while an application-specific analysis is done to explore the real harm of such trends. We now also provide additional analysis to highlight how homogenization metrics lead to downstream harm in Common Response: Section 1.2, more explicitly highlighting the role of conformity, polarization, and inertia in creating correlated errors, persistent perturbations, and converging preferences.

We would like to thank the reviewer again for their feedback and comments.

  • In response to your request for controlled experiments to show that it is the communication between agents that leads to homogenization, we have now added two new setups, a single agent baseline and a MAS no communication baseline, to strongly support that homogenization trends emerge from the communication between agents (Section 4.3).
  • In response to your concerns about how homogenization metrics lead to harms, we have now added explicit discussions on how conformity, polarization, and inertia all play a role in each of the three applications in the paper, with additional analysis provided to support our arguments (Sections 5.1, 5.2, and 5.3).
  • Finally, regarding your concerns on our choice of an orchestrator-driven MAS structure, we clarify that it is commonly adopted both in the academic literature and in real applications. Our contributions with orchestrator-driven MAS represent a first step in establishing a new failure mode of MAS, and expanding these analyses to other workflows was added as an important direction of future research (Section 7).

We believe we have addressed all of your concerns by providing new experiments and clarifications, hence, we ask you to please read our rebuttals and let us know if your concerns were, indeed, clarified. We would also appreciate it if you could update your scores accordingly if you believe your concerns were addressed.

Comments on the paper

4 comments
AuthorsOpenReview ↗

Common Response (1/4)

We thank the reviewers for their feedback, comments, and service to our community. We believe your suggestions will improve the quality and impact of our paper. Below, we provide additional results, analyses, and clarifications for common concerns among the reviewers. Our rebuttal is best read alongside the revised paper to refer to appropriate figures.

1. Connections to Homogenization Failures

In the paper, we show homogenization trends and the resulting harms in MAS. Complimenting our work, (1) reviewers BU2p, sf7z, and TsWG suggested performing a control study to isolate the impact of interaction dynamics (i.e., communication between different agents) from iterative refinement (i.e., agents refining their outputs based on previous history), and (2) reviewers BU2p, sf7z, and dDPT suggested more explicitly highlighting the connections between homogenization metrics and harms.

1.1. Interaction Dynamics and Homogenization Metrics

When studying homogenization trends in MAS, there are two important factors that impact diversity among agents: (i) the presence of multiple specialized agents which bring diversity into the task, and (ii) the interaction dynamics between these agents that can cause homogenization and potentially reduce the expected diversity. We isolate these factors and find that while MAS reduces homogenization compared to a single agent due to the introduction of multiple agents, it still has more homogenization than a MAS no interaction baseline, highlighting the role of interaction dynamics in homogenization. We explain each setting next. Specifically, to understand the interplay between these two factors ((i) and (ii)), we offer the following baselines.

Single agent baseline: We define a single worker agent with the combined task of all worker agents in the original MAS setup. For e.g., in coding, a single agent reviews the code for Security, Readability, Correctness, and Optimality, all in one inference, and provides four separate scores which we use to measure homogenization.

MAS no interaction baseline: We use specialized worker agents with separate personas like our original MAS setup, but do not allow interactions between them. For e.g., in coding, the agent reviewing the code for Optimality only receives the code and its own feedback from all previous rounds, and not any feedback or scores of other worker agents.

We study the three homogenization metrics for both baselines compared against our MAS setup in Figure 4 of the revised paper. During the rebuttal, we offer experiments for coding for two models (DeepSeek V4-Flash and Qwen3-30B-A3B Thinking-2507), and we will extend these to other models and tasks for the final version of the paper. Note again that trends of conformity, polarization, and inertia characterize a negative relationship, e.g., lower value of variance is higher conformity.

We find the following trends of conformity and polarization (i.e., opposite to the trend of variance and distance from the endpoints): single agent baseline > MAS > MAS no interaction baseline, and we find similar trends of inertia for MAS and MAS no interaction baseline but still less severe than single agent baseline. Hence, the introduction of multiple agents helps reduce homogenization, an important motivation behind MAS, but interactions between these agents can again increase homogenization, despite these interactions being an important factor in the success of MAS.

The analysis thus provides strong evidence for the role interaction dynamics play in the homogenization trends observed in the paper. We have now added these results and the discussion in the revised version of the paper in Section 4.3 and Figure 4.

1.2. From Homogenization Metrics to Failure Modes

Next, we establish that homogenization metrics indeed lead to failure modes in MAS. Simulating controlled scenarios in our applications with only conformity, polarization, or inertia in isolation is not possible, since these are phenomena observed in MAS and not choices of the experiment design. Thus, we cannot do a causal analysis withholding one metric at a time, and instead build an argument using the trends in our existing experiments.

AuthorsOpenReview ↗

Common Response (2/4)

Homogenization Metrics lead to Correlated Errors in Coding

Role of Conformity: As we report vulnerabilities in the code as a measure of failure, the Security reviewer agent plays an important role. To study how conformity to the peers affects the Security agent, we track the cases in which it correctly flags vulnerable code (score < 3) but then withdraws such a flag in the next round (i.e., increases its score ≥ 3), even though the vulnerability is still present in both rounds. We compare these percentages conditioned on whether the other three reviewers approved the code or also rejected it. The results are added to Figure 24 in Appendix E. Conformity exposes the Security agent to peer pressure, and we find that it’s correct flags of vulnerability are withdrawn more often against a panel that believes the code is good, even though the vulnerability hasn’t been resolved.

Role of Polarization: Polarization makes the overall system overconfident, as even moderate scores grow quickly without the code actually improving at the same rate. We compare the percentage of code that is marked not deployable by the system (scores < 3) against the true rate of vulnerabilities, and find that the former drops significantly faster than the latter. The results are present in Figure 25 in Appendix E. For instance, for Qwen3-Coder-Next, the percentage of code with scores below 3 goes from close to 50% to less than 5% in five rounds, yet the true rate of vulnerabilities only goes down from roughly 55% to 30%. Clearly, the system starts deploying vulnerable code due to overconfidence driven by increasing polarization.

Role of Inertia: Finally, inertia also plays a crucial role in increasing security vulnerabilities. Code once marked good is less likely to then be marked vulnerable even if a new vulnerability is introduced later by the generator, simply because inertia makes it unlikely for reviewers to change their scores. To highlight this, we track all vulnerabilities introduced in the code in later rounds (i.e., not the first round) and how many of them were eventually resolved. We condition this analysis on what scores were given to the code by the panel before the vulnerability was introduced, and results are recorded in Figure 26 in Appendix E. Inertia makes it difficult for reviewers to flag new vulnerabilities, and we find that vulnerabilities that are introduced after the code was already marked good are less likely to be recognized or resolved.

We have now added a summary of this discussion, along with appropriate references to the results in the appendix, in Section 5.1 of the revised version of the paper.

Homogenization Metrics lead to Persistent Perturbations

Role of Conformity: Despite not receiving explicit adversarial prompts, we see a clear increase in bias for all other agents beyond the adversarial agent, far higher than their baseline behavior in the setting with no adversary, clearly showing that conformity in the presence of a stubborn biased agent causes all other agents to also move towards it (Figure 6). This drive to reduce variance and push towards agreement causes other agents to inherit the bias present in the adversary.

Role of Polarization: Studying the duration of adversarial influence (Figure 30 in Appendix F), we find that adversarial influence for only one round persists but does not grow, for two rounds persists and slowly grows, and for three rounds grows a lot stronger even after the adversarial prompt is removed. Thus, if the bias is present in the system long enough to separate individuals in different directions, polarization then takes over to keep increasing this gap between them and further increases the bias.

Role of Inertia: Finally, studying the location of adversarial influence (Figure 30 in Appendix F), we find that introducing adversary in rounds 1 and 2 is stronger than introducing an adversary in rounds 3 and 4, which highlights that the system’s inertia makes it more difficult to be influenced even by an adversary in later rounds, compared to earlier rounds. Hence, inertia plays a critical role in how bias spreads in the system and creates homogenization harm.

We have now added a summary of this discussion, along with appropriate references to the results in the appendix, in Section 5.2 of the revised version of the paper.

AuthorsOpenReview ↗

Common Response (3/4)

Homogenization Metrics lead to Converging Preferences

Role of Conformity: Conformity plays a straightforward role in converging preferences, as stronger conformity trends highlight reviewers agreeing with each other over multiple rounds of discussions, moving towards the majority and losing the diversity of separate specialized opinions.

Role of Polarization: Similar to coding, polarization makes the system overconfident in peer review. We find that the percentage of papers receiving moderate scores shrinks significantly over rounds (Figure 32 in Appendix G). For instance, for Qwen3-30B-A3B Thinking-2507, papers receiving scores >=4 and <8 shrink from roughly 75% to less than 25% over five rounds. Clearly, this increases the gaps between papers as those already marked good see an increase in score while those marked bad see a decrease. Thus, polarization drives the growing gap in scores between different fields.

We have now added a summary of this discussion, along with appropriate references to the results in the appendix, in Section 5.3 of the revised version of the paper.

2. Role of Conversation History in Homogenization

Reviewers BU2p, TsWG, and dDPT highlighted that bias in the hiring domain persists even after the agent is not explicitly given an adversarial prompt potentially because of biased reasoning present in the conversation history. Yes, indeed, we believe that the conversation history plays an important role in driving homogenization behavior in MAS. It is likely the cause of bias persisting in the system even after the explicit adversarial prompt is removed, and even more so, it is an important factor driving our metric of ‘inertia’ across all applications. Agents show inertia to changes from their history of decisions, which only makes sense when they have access to a multi-round conversation history. We have now added these clarifications when discussing the results in the revised paper in Sections 4.2 and 5.2.

But this does not trivialize our findings in any way. Conversation history is an important factor in the success of MAS (see this survey on memory in MAS). Moreover, maintaining the entire conversation history during agent interactions is the default setup in most agentic frameworks. For instance, OpenAI Agents SDK when defining a handoff in multi agent systems, mentions, ‘when a handoff occurs, it's as though the new agent takes over the conversation, and gets to see the entire previous conversation history’, and by default, there are no ‘input_filters’ in the library. Similarly, various MAS setups under Microsoft Agent Framework all default to full conversation histories, for example, ‘in handoff orchestration, the conversation is handed off to another agent entirely’, or ‘by default, each agent in a SequentialBuilder workflow consumes the previous agent's full conversation (input + response messages)’, or ‘the default aggregator produces a single AgentResponse containing one assistant message per participant’. Another commonly used agentic framework LangChain, also states that ‘by default create_handoff_tool passes full message history (all of the messages generated in the swarm up to this point), as well as the contents of Command.update to the next agent’.

Could truncating the conversation history, or restricted communication protocols, reduce homogenization? Truncating the conversation history or restricting communication can impact the effectiveness of MAS. Moreover, selective restrictions are not trivial, as recognizing which pieces of the conversation history or which agents in the mix are adversarial can be difficult. Thus, while controlling communication protocols in MAS to mitigate homogenization concerns is a promising direction of future research, it is not within the scope of our work. We have now added this discussion in future works (Section 7).

AuthorsOpenReview ↗

Common Response (4/4)

3. Stratified Analysis for Peer Review

Based on the recommendations from reviewers sf7z and TsWG, we perform additional analyses for the peer review application. Instead of recording the scores per research topic across all papers, we do a stratified analysis for accepted and rejected papers separately. By conditioning on the result of actual acceptance results (although peer review can be noisy and is also not perfect), we remove the concerns of increasing gaps between different fields as a result of their quality. We replace the current results in the main paper with the stratified analysis only for accepted papers in Figure 7, and move the results for all papers and also add new results for only rejected papers in Appendix G (Figures 33 and 34).

The new results still support our claims of harms caused by homogenization. Even with the stratified analysis, we see a clear drift of increasing gap in scores between various research fields, thus highlighting the harms without the confounding factor of paper acceptance/quality. Moreover, we continue to see the same shared fields being favored towards (or against) across various LLMs. We have now clarified the setting and updated the discussion in Section 5.3 of the revised paper.

We also report sample sizes for all research fields in the revised paper in Table 4 in Appendix G.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.