- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
In this work, the authors investigate whether the agentic collaboration format affects outputs of a multi-agent decision-making system in a context where safety and fairness are prioritized (all else being equal). In particular, they draw on prior work in auditing and algorithmic fairness and conduct experiments based on a credit lending scenario with several language models followed by rigorous statistical analysis. Their work illustrates that collaboration methods like parallel decision synthesis with majority voting and decentralized deliberation are relatively robust, but other methods with centralized judges tend to be less fair (in terms of adverse impact rate (AIR)), and downstream agents appear to be easily swayed by earlier agents' outputs in the sequential deliberation method. In light of these findings, the authors argue that fairness or alignment of individual models used in a system is insufficient to guarantee overall fairness or alignment of the entire system, and as a result, the interaction topology of agents must be audited directly to say anything conclusive.
Critical strengths
The authors highlight an important result and support it with rigorous experimentation and statistical analysis that efficiently yet effectively explores the space of possible experimental configurations and decisions. In doing so, they also draw upon appropriate research literature and industry practices.
Critical weaknesses
Though the authors' overall argument about the importance of agentic interaction topology is clear, the paper itself was dense and hard to read in a number of sections due to factors including but not limited to prose, figures, and mathematical notation. For example, from the Introduction, the sentence "This boundary-sensitivity effect...also emerges on Qwen-2.5-7B...indicating the effect is not specific to a single model family" confused me given that "Qwen-2.5-7B" is a single model (not a family) and the usage of Llama models was only mentioned in the abstract (not anywhere near this sentence in this section). Moreover, though Figure 1 summarizes the paper overall, the numbers and presentation format diverge from the content in the ordered list describing the "three distinct pathologies" found via experiments. Regarding math and notation, I found Section 3.1 confusing at first (took some time to parse that refers to an ordering of agents while is a decision made by agent based on the permutation) and the symmetry of expressions in Equation 1 in Section 3.5 was not mirrored in Equation 2 (I needed to examine the definitions to verify that they were similar, and referring to identity was also unclear to me at first).
Aside from clarity, I also have some reservations about the selected models and reported results. I was surprised to see that Llama-3.1 and Qwen-2.5 were used as backbones for agents, in addition to the fact that the largest models of those families were referred to as "frontier models" throughout the work, given the releases of newer and more advanced models (e.g., GLM-4.7-Flash, GLM-5.3-Flash, Qwen-3.8, Qwen-3.5). Furthermore, the authors explicitly state that "We report 3.272 [based on results from Llama-3.1-8B] as our headline figure because it draws on the complete decision-level sample," but results from models with at least 70B parameters in the same table (Table 2) appear to contradict this "headline figure" given that CSV outweighs OSV in these borderline cases as well. I also cannot find the ANOVA tables stated to be in Appendix D to study these and similar results in more detail.
Lastly, one could also potentially object to both the setup of the audit study and novelty of the result. Regarding the former, prior work has made arguments objecting to the notion that biased outcomes can be caused by protected attributes when such attributes are socially constructive and constitutive (e.g., Hu & Kohler-Hausmann (2020) "What's Sex Got To Do With Fair Machine Learning?"), in turn questioning the validity of audit experiments and simulations making corresopnding assumptions. As for the latter, in light of anchoring and sycophantic tendencies of LLMs and given that humans are also biased by others' judgments when reviewing candidate information (e.g., in a job hiring scenario or in double-blind research paper review), this result may not be wholly surprising. Even so, one could conversely argue to both points that the authors' conclusion is important to verify and highlight.
Questions for authors
I have a couple of follow-up questions which primarily focus on the choice of models and weight configurations for the experiments:
- Why was INT4 quantization used throughout even for small models?
- I saw the experimental support given mostly equivalent results with one of the models in supplementary materials, but could 3B or 4B models be run at full precision? Alternatively, was INT4 used throughout for parity with large models?
- Why were these models and parameter sizes used in particular?
- I understand the desire for a diversity of sizes and have already raised points about newer models and other families, but even so, other model families like Mistral and sizes like ~15B and ~30B exist as well and were not explored here.