- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates homogenization in multi-agent systems through three operational measures: decreasing inter-agent score variance, movement of mean scores toward scale endpoints, and decreasing changes in mean scores across rounds. It studies these dynamics in code generation, synthetic résumé screening, and automated scientific peer review. The experiments connect agent interactions to specific downstream outcomes: security vulnerabilities that pass deployability review, demographic score disparities that persist after an adversarial prompt is removed, and widening score differences across research areas. The paper evaluates multiple model families, repeated sampling, and mixed-model configurations, finding that the studied risks persist in these settings. Its strongest contribution is the application-level demonstration that aggregate performance can conceal shared failures. Some causal and fairness interpretations, however, go beyond what the measurements directly establish.
Critical strengths
1.The paper connects interaction dynamics to consequential outcomes across several applications. The code-generation analysis is particularly useful: overall vulnerability rates can improve while particular vulnerability types remain undetected or increase among outputs considered deployable. This demonstrates why aggregate performance alone is insufficient for assessing multi-agent reliability.
2.The evaluation extends beyond a single model or illustrative interaction. The study includes multiple model families, ten trials, and mixed-model configurations. The temporary adversarial-prompt experiments in hiring also reveal persistence beyond the period of direct injection. Together, these results provide substantial evidence that adding agents or model-family diversity does not automatically eliminate shared failure modes.
Critical weaknesses
1.The score dynamics do not by themselves establish harmful homogenization or its causal role. Lower variance, more extreme recommendations, and smaller updates can also occur when agents correctly resolve uncertainty. In particular, shrinking changes in the panel mean are not equivalent to resistance to new evidence. The downstream analyses and adversarial-timing ablations are useful, but the manuscript often treats the three metrics as causal explanations for failures without sufficiently separating their effects from shared context, inherited model weaknesses, or ordinary convergence. Matched non-interacting and communication-controlled comparisons would strengthen this attribution.
2.The hiring and peer-review disparity measures contain important confounds. The résumé variants change languages, extracurricular activities, and volunteering, which may carry job-relevant information as well as demographic signals. Appendix C.5 establishes that another model recognizes the intended stereotypes, but this does not establish that qualification-relevant content remains equivalent. Likewise, differences in mean scores across research areas do not establish unequal evaluation standards without accounting for paper quality and area composition. These limitations weaken the fairness interpretation while leaving the observed score disparities informative.
Questions for authors
1.Causal-attribution clarification: Do you have matched comparisons with independently operating agents or restricted communication, holding task inputs and resources fixed? Which results specifically establish that interaction-induced homogenization increases downstream errors?
2.Hiring-measurement clarification: How is job-relevant equivalence verified across résumé variants, particularly when languages and extracurricular achievements change? Do the disparities persist under demographic manipulations that preserve qualifications more tightly?
3.Peer-review validation: Do the widening cross-area score differences persist after conditioning on available indicators of paper quality, such as human review scores or acceptance categories? Can the analysis distinguish differing evaluation standards from differing paper composition?
3 comments
Response to Reviewer sf7z (2/2)
Weakness 2b: Differences in mean scores across research areas do not establish unequal evaluation standards without accounting for paper quality and area composition.
Question 3: Peer-review validation: Do the widening cross-area score differences persist after conditioning on available indicators of paper quality, such as human review scores or acceptance categories? Can the analysis distinguish differing evaluation standards from differing paper composition?
This is a great suggestion. We have now replaced our existing peer review results with a stratified analysis based on ICLR decisions in our revised paper (Figure 7 now shows trends for only accepted papers, while the analysis for rejected papers is added to the appendix in Figure 34). Even after controlling for ICLR decision strata, we still find the gap between various research fields increases over rounds, and we still find a shared set of research fields which are favored with increasing scores across different models (or conversely, a shared set of research fields with decreasing scores across different models). More details in Common Response: Section 3.
We would like to thank the reviewer again for their feedback and comments.
- In response to your request for controlled experiments to show that it is the communication between agents that leads to homogenization, we have now added two new setups, a single agent baseline and a MAS no communication baseline, to strongly support that homogenization trends emerge from the communication between agents (Section 4.3).
- In response to your comment on stratified analysis for peer review to isolate converging preferences from paper quality, we have now replaced our existing peer review results with a stratified analysis based on ICLR decisions in our revised paper (Figure 7, Figure 33, and Figure 34). Even after controlling for ICLR decision strata, we still find the gap between various research fields increases over rounds, and the harms discussed persist.
We believe we have addressed all of your concerns by providing new experiments and clarifications, hence, we ask you to please read our rebuttals and let us know if your concerns were, indeed, clarified. We would also appreciate it if you could update your scores accordingly if you believe your concerns were addressed.
Response to Reviewer sf7z (1/2)
We thank the reviewer for your feedback and comments. Implementing your suggestions on non-interacting and communication-controlled comparisons, stratified analysis of peer review based on acceptance categories, and separating expected behavior from actual failure modes of MAS has made our paper significantly stronger.
Weakness 1: The score dynamics do not by themselves establish harmful homogenization or its causal role. Matched non-interacting and communication-controlled comparisons would strengthen this attribution.
Question 1: Causal-attribution clarification: Do you have matched comparisons with independently operating agents or restricted communication, holding task inputs and resources fixed? Which results specifically establish that interaction-induced homogenization increases downstream errors?
Thank you for your suggestions. We have now added new baselines to establish the relationship between interaction dynamics and homogenization. We show that while introducing specialized agents improves diversity and reduces homogenization compared to a single agent, enabling communication between these specialized agents again reduces this diversity and instead leads to higher homogenization. More details in Common Response: Section 1.1.
Convergence of agent behavior through the metrics of conformity, polarization, and inertia represents emerging homogenization in MAS. However, it is true that simply the presence of homogenization does not necessarily imply downstream harm.
Gur-Arieh et al. (2025) argue that homogenization is harmful if the task is normative and value-laden (instead of factual), has some inherent uncertainty, has multiple independent decision makers, and the decision making entities are private or the decision process is opaque. This includes our analysis in hiring, which is a highly value-laden decision made by several independent private entities, and in peer review, which too involves subjective decisions as there is no completely objective scale of acceptance. Kim et al. (2025) argue that even in factual domains where we can define an objectively correct answer, homogenization can be harmful if it creates correlated errors across different systems. This represents our analysis in the coding domain and the study of security vulnerabilities in generated code. We have now added this discussion in the revised version of the paper (Section 5).
Hence, harms from homogenization either come through correlated errors in factual tasks (like we show in coding), or converging opinions in normative tasks (like we show for hiring and peer review). The metrics on conformity, polarization, and inertia help us track homogenization trends, while an application-specific analysis is done to explore the real harm of such trends. We now also provide additional analysis to highlight how homogenization metrics lead to downstream harm in Common Response: Section 1.2, more explicitly highlighting the role of conformity, polarization, and inertia in creating correlated errors, persistent perturbations, and converging preferences.
The new baselines as well as the additional analysis have now been added to the paper (Sections 4.3, 5.1, 5.2, and 5.3), taking advantage of the extra page during the rebuttal phase.
Weakness 2a: The résumé variants change languages, extracurricular activities, and volunteering, which may carry job-relevant information as well as demographic signals.
Question 2: Hiring-measurement clarification: How is job-relevant equivalence verified across résumé variants, particularly when languages and extracurricular achievements change? Do the disparities persist under demographic manipulations that preserve qualifications more tightly?
Our baseline result without any biased agent shows no statistically significant bias in the dataset (Figure 6), which establishes equivalence between resume variants across demographics. That is, without any adversary, variations of the resume across demographics are scored similarly, and show no signs of inherent bias or preference in the dataset. It is only after we introduce the adversary, that we see bias in the results.
It is possible this baseline can get missed by the reader, since it is a straight gray line in Figure 6. Thus, we have now explicitly added this discussion with the results in the revised version of the paper (Section 5.2).