acceptodds
Desk rejected at ICLR 2027

FairWatch: Interaction Topology as a First-Order Variable Shaping Safety and Fairness in Multi-Agent Deliberation

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

AI safety practice assumes that aligning individual models ensures safe collective systems. We show this compositionality assumption fails in deliberative agent networks. System-level safety and fairness depend directly on interaction topology: the communication graph, execution order, and aggregation rule coordinating agent behavior. We audit synthetic credit-underwriting decisions across six open-weight models spanning 3B to 72B parameters, holding prompts, profiles, and weights fixed while systematically varying topology across four standard archetypes. Three structural failure modes emerge. First, in sequential chains, agent execution order generates severe decision variance on ambiguous borderline files, producing outcome reversals for nearly one in five affected applicants, concentrated in a small fraction of boundary contexts. Second, downstream agents abandon private domain signals in nearly all conflicting cases across both model families (Meta LLaMA and Alibaba Qwen 2.5), collapsing epistemic diversity into herd consensus regardless of model scale. Third, under hierarchical judge synthesis, a large-scale model exhibits an adverse impact shortfall for protected applicants that does not survive multiplicity correction, while a matched majority-vote readout remains compliant on the same profiles. These findings show that component-level alignment does not guarantee system-level safety, establishing interaction topology as an essential first-order audit variable for multi-agent governance.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
AI Safety, Algorithmic Fairness, Disparate Impact Auditing, Informational Cascades, Interaction Topology, Multi-Agent Systems

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_xEfwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this work, the authors investigate whether the agentic collaboration format affects outputs of a multi-agent decision-making system in a context where safety and fairness are prioritized (all else being equal). In particular, they draw on prior work in auditing and algorithmic fairness and conduct experiments based on a credit lending scenario with several language models followed by rigorous statistical analysis. Their work illustrates that collaboration methods like parallel decision synthesis with majority voting and decentralized deliberation are relatively robust, but other methods with centralized judges tend to be less fair (in terms of adverse impact rate (AIR)), and downstream agents appear to be easily swayed by earlier agents' outputs in the sequential deliberation method. In light of these findings, the authors argue that fairness or alignment of individual models used in a system is insufficient to guarantee overall fairness or alignment of the entire system, and as a result, the interaction topology of agents must be audited directly to say anything conclusive.

Critical strengths

The authors highlight an important result and support it with rigorous experimentation and statistical analysis that efficiently yet effectively explores the space of possible experimental configurations and decisions. In doing so, they also draw upon appropriate research literature and industry practices.

Critical weaknesses

Though the authors' overall argument about the importance of agentic interaction topology is clear, the paper itself was dense and hard to read in a number of sections due to factors including but not limited to prose, figures, and mathematical notation. For example, from the Introduction, the sentence "This boundary-sensitivity effect...also emerges on Qwen-2.5-7B...indicating the effect is not specific to a single model family" confused me given that "Qwen-2.5-7B" is a single model (not a family) and the usage of Llama models was only mentioned in the abstract (not anywhere near this sentence in this section). Moreover, though Figure 1 summarizes the paper overall, the numbers and presentation format diverge from the content in the ordered list describing the "three distinct pathologies" found via experiments. Regarding math and notation, I found Section 3.1 confusing at first (took some time to parse that π\pi refers to an ordering of nn agents while yπ(i)y_{\pi(i)} is a decision made by agent ii based on the permutation) and the symmetry of expressions in Equation 1 in Section 3.5 was not mirrored in Equation 2 (I needed to examine the yˉ\bar{y} definitions to verify that they were similar, and d∈Dd \in \cal{D} referring to identity was also unclear to me at first).

Aside from clarity, I also have some reservations about the selected models and reported results. I was surprised to see that Llama-3.1 and Qwen-2.5 were used as backbones for agents, in addition to the fact that the largest models of those families were referred to as "frontier models" throughout the work, given the releases of newer and more advanced models (e.g., GLM-4.7-Flash, GLM-5.3-Flash, Qwen-3.8, Qwen-3.5). Furthermore, the authors explicitly state that "We report 3.272 [based on results from Llama-3.1-8B] as our headline figure because it draws on the complete decision-level sample," but results from models with at least 70B parameters in the same table (Table 2) appear to contradict this "headline figure" given that CSV outweighs OSV in these borderline cases as well. I also cannot find the ANOVA tables stated to be in Appendix D to study these and similar results in more detail.

Lastly, one could also potentially object to both the setup of the audit study and novelty of the result. Regarding the former, prior work has made arguments objecting to the notion that biased outcomes can be caused by protected attributes when such attributes are socially constructive and constitutive (e.g., Hu & Kohler-Hausmann (2020) "What's Sex Got To Do With Fair Machine Learning?"), in turn questioning the validity of audit experiments and simulations making corresopnding assumptions. As for the latter, in light of anchoring and sycophantic tendencies of LLMs and given that humans are also biased by others' judgments when reviewing candidate information (e.g., in a job hiring scenario or in double-blind research paper review), this result may not be wholly surprising. Even so, one could conversely argue to both points that the authors' conclusion is important to verify and highlight.

Questions for authors

I have a couple of follow-up questions which primarily focus on the choice of models and weight configurations for the experiments:

  • Why was INT4 quantization used throughout even for small models?
    • I saw the experimental support given mostly equivalent results with one of the models in supplementary materials, but could 3B or 4B models be run at full precision? Alternatively, was INT4 used throughout for parity with large models?
  • Why were these models and parameter sizes used in particular?
    • I understand the desire for a diversity of sizes and have already raised points about newer models and other families, but even so, other model families like Mistral and sizes like ~15B and ~30B exist as well and were not explored here.
Reviewer_ihCPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

AI alignment should focus not only on whether individual models are aligned, but what happens when multiple models interact. The authors here argue that the "interaction networks" through which models engage with each other should be treated as a core object in evaluation. The authors identify several concerning phenomena around (lack of) fairness that arise when models are linked together.

Critical strengths

I think the topic that the authors take on is excellent. The idea of studying fairness not just at the level of one model but networks of models is deeply important and I think the authors articulate the problem generally well.

I was also impressed with the breadth of the authors' literature review. I think they did a good job connecting across fields and situating their work.

I also think that the phenomena the authors identify is important.

Critical weaknesses

Unfortunately, the biggest weakness for me was that the paper writing and structure was very hard to follow. As great as the background connections were (noted above), I felt much of the main text was quite dense and somewhat convoluted (I'm not sure if this was partly from AI-assisted writing or otherwise?)

Some of the results were then also quite confusing --- the error bars around the 95% CIs are very high? (e.g., around OSVindiv/CSVindiv). Do the authors know why that is? And in Table 2, what is driving the bolding? Does that mean statistically significantly better?

The work is also only with 2 LLMs -- it would be great to see the generality of these results to a wider range of LLMs (also across other sophistications).

I do think the ideas of the work are very good, so am hopeful that with some of the above changes, the work could be in a better position to make the case for more assessments of fairness at the level of model interactions.

Questions for authors
  • What does " it does not model any human underwriting desk" mean?
  • Have the authors tried any other (e.g., more powerful) models?
  • Minor: the paper has a lot of stats that are presented at quite granular levels of precision -- is this necessary?
Reviewer_3mr5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how interaction topology affects the stability and fairness of multi-agent LLM decisions. The authors introduce a counterfactual auditing framework for synthetic credit underwriting, comparing four architectures: parallel voting, sequential chains, hierarchical synthesis, and peer deliberation. The authors claim the main findings are that agent ordering can substantially affect decisions on borderline applications, downstream agents frequently follow upstream judgments, and centralized judge synthesis exhibits a nominal fairness disparity in one setting, although this result does not remain significant after multiple-testing correction.

Critical strengths

The paper provides a useful controlled evaluation of multi-agent decision architectures, keeping model weights, agent roles, and underlying applicant profiles fixed within matched comparisons. The 24-permutation analysis is particularly useful for characterizing ordering sensitivity in sequential deliberation.

Critical weaknesses

I have the following major concerns, especially related to novelty.

  • The central claim that individual-agent alignment does not guarantee safe collective behavior has already been established in prior studies, including Misalignment Contagion (COLM 2026). The effects of interaction topology, agent ordering, information propagation, and group conformity have also been investigated in previous work (e.g., Shen et al., EMNLP 2025; Choi et al., ACL 2025). More importantly, other research has already studied topology-dependent discrimination in multi-agent credit decisions, while a May 2026 position paper presented closely related claims about ordering instability, information cascades, and topology-dependent safety. Although FAIRWATCH extends the evaluation to additional architectures and provides permutation-based auditing, its substantive novelty beyond these studies remains unclear.
  • The other point is that the paper interprets ordering instability and downstream conformity as system-level safety failures. However, changes in decisions do not necessarily indicate unsafe outcomes, and downstream agents may legitimately revise their judgments after receiving additional reasoning. I think the current evaluation does not clearly distinguish such justified updates from harmful conformity.
Questions for authors
  • Could the authors clarify the main methodological or empirical contribution beyond existing studies of topology-dependent collective behavior, particularly Bajaj et al. (2026), Shen et al. (2025), and the May 2026 position paper?
  • How do the authors distinguish harmful conformity from justified belief updating? What evidence establishes that the observed decision changes constitute system-level safety failures rather than decision instability?
  • Given that both PAR R2 and HIER R2 fall below the screening threshold on the matched subset, and the reported disparity does not survive multiple-testing correction, what evidence supports a significant topology-specific effect on demographic fairness, rather than a general change in judge decisions?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.