acceptodds
Under review as a conference paper at ICLR 2027

Fair Agents Can Form Unfair Systems: Sharp Limits of Fairness Composition

Abstract

Demographic parity of individual agents does not imply demographic parity of their joint decision. We characterize the largest system disparity compatible with local rate audits and the utility cost of controlling it. By marginal duality, this disparity equals a regularized affine-approximation error, which yields sharp, attained bounds for every voting threshold: under exact local parity, every threshold rule over at least two agents admits a gap of at least . Conversely, any monotone coordinator with worst-case gap at most lies within of an agent–constant mixture that meets the same budget. This bounds the utility advantage of nonlinear coordination, and under an explicit tight-budget condition a mixture is exactly optimal. Matching all pairwise moments does not reduce odd majority's worst-case gap. Experiments separate fairness on the current distribution from robustness to dependence. In a three-LLM committee, the observed majority gap is and the largest observed individual gap is . At larger budgets, observed accuracy differences between full coordination and fitted mixtures range from to percentage points in explicit-cue LLM comparisons. Public-data differences are about – percentage points, while observed and worst-case disparities differ substantially. Together, the results quantify what local fairness audits leave uncontrolled and how much predictive flexibility must be given up to control it.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Fairness Composition, Multi-Agent Systems, Robust Optimization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_QgZVOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

In this work, the authors argue and show that unfairness can arise from an ensemble of fair prediction agents. They begin with a toy example illustrating such a result in theory before introducing notation and theorems to support this result in the general case. In light of this, the authors propose an approach, JointRepair, to reduce the disparity gap introduced by coordinating predictions from multiple agents. They follow this with experiments both via traditional predictive agents (in this case, logistic regression models) and small-scale open-weight LLMs and demonstrate that JointRepair produces competitive accuracy with smaller observed and/or worst-case disparity gaps compared to other coordination methods. Lastly, they conclude with a short discussion and limitations of their work, including a note about optimization being exponential in the size of the set of coordinating agents.

Critical strengths

This paper uses both theoretical and empirical rigor to prove an interesting and important result pertaining to algorithmic fairness and demographic parity in the generative AI era. Results are thus supported with proofs and derivations in appendices, and experimental settings are similarly detailed in supplementary materials. Figure 1 and corresponding analysis in the Introduction are additional helpful in motivating the problem at hand.

Critical weaknesses

In my opinion, the primary weakness of this paper is that it is dense in that the prose is hard to read and the math is hard to parse. Bluntly, it could very well be the case that I do not have the expertise required to fully grasp the theoretical components of this work (my own contributions tend to be more empirical, and my background is not in LP optimization though I am vaguely aware of various related concepts), but regardless, Sections 3 and 4 were difficult to understand at a deep level.

Outside of this, the experiments were structured well but could potentially be improved in a few ways. First, I would have liked to see more about the synthetic data setup in the main body of the work. Moreover, contemporary research has noted issues with the Adult dataset (see Ding et al. (2021) "Retiring Adult") that the authors may want to consult for improved evaluations. Lastly, experiments with at most 5 agents and few public datasets were described in the main body of the work, and the LLM agent experiments used smaller and older agents compared to what is currently available via open-source providers (e.g., Qwen2.5-7B-Instruct compared to GLM-4.7-Flash). To the extent possible (as the exponential runtime constraint may make things difficult) running experiments with more logistic regression agents, more public datasets (e.g., datasets described here or in the folktables package described in the link above) and more and/or larger LLMs could possibly be beneficial in illustrating the theoretical result.

Questions for authors

Strengths and weaknesses noted above, I have the following remaining questions for the authors:

  • Does this (dis)parity result hold for other definitions of fairness (e.g., equal opportunity)?
  • Have you considered settings beyond the binary attribute setup (e.g., ones that reflect multiple races or the consideration of race and gender simultaneously)?
  • I may have missed this, but are the LLM agents calibrated like the logistic regression models on producing fair outputs at the individual level (before coordination)? The crux of this paper is that fairness at the individual level can still result in unfairness after coordination, and both the main body and appendices describing LLM experiments refer to fit and evaluation examples, but particular fine-tuning and calibration details are missing, so I am wondering whether the LLMs are similarly calibrated. Elucidation on this would be appreciated.
Reviewer_6og3OpenReview ↗
Rating
1
Confidence
1
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper deals with an issue in fairness under composition: If composing the approval probabilities of agent approvals who in big batches agree in their approvals, then the aggregate probability is much lower than when the agent approvals are disjoint. The authors characterize to which extent this can happen and do empirical evaluation with three LLM agents.

Critical strengths

The authors address a critical issue of fairness under composition; this is very relevant and worth studying.

Critical weaknesses

First of all, the paper does not set the scope nicely. When starting to read the paper, it was hard to even identify in which field the paper should be placed. The paper starts almost straight away with the problem without describing the setting at all. This makes it almost impossible for anyone outside of the community to understand what the paper is about. Moreover, the authors do not mention where exactly the described setting occurs. Figure 1 is also rather confusing in that the probability of the agents' actions is somewhat mixed with the number of approving agents.

Even to a non-expert, the related work section seems to miss a lot. A simple LLM prompt shows that there is a lot of related literature around, and also mentions two very closely related works that break the novelty claims of the authors:

  • Grgić-Hlača, Zafar, Gummadi & Weller, “On Fairness, Diversity and Randomness in Algorithmic Decision Making” (2017)
  • Worst-case fairness of the majority vote over fair classifiers (PHD thesis, Tübingen).

Finally, the empirical evaluation seems rather limited with only three agents.

Questions for authors

Please tell how the two above-mentioned works differ from your work.

Reviewer_Rr6mOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies fairness in multi-agent decision making showing that individual agents satisfying demographic parity need not produce a fair composed decision. They characterize the worst case system DP gap under constraints on individual approval rate disparities, establish tight bounds for threshold voting rules and show that strong robustness requirements restrict coordinators toward affine rules and mixtures of individual agents and constants. Experiments with LLM committees and traditional classifiers examine fairness amplification.

Critical strengths

The paper addresses an interesting problem of whether individual fairness guarantees are preserved when decisions are composed. The theoretical analysis provides exact, achieveable worst-case bounds for threshold voting, along with a characterization based on affine approximation. The observation that pairwise moment matching does not improve the worst case fairness guarantee for odd-majority voting is interesting. Experiments include both LLM commitees and conventional classification benchmarks, and the authors attempt to distinguish observed disparity from worst case robustness.

Critical weaknesses

The theoretical model reduces agents to stochastic binary decision makers with a fixed coordinator. It is unclear what changes for classical ensemble classification or how the results capture agent interactions. The analysis allows arbitrary cross-agent dependencies, including extreme constructions such as figure 1. It is unclear how frequently such dependencies occur in practice. The increasing worst-case disparity with committee size does not imply that fairness typically deteriorates as more agents are added and shown result with worst case disparity might be misunderstood.

The guarantees assume known bounds on individual-agent DP gaps. Estimating these reliably may require substantial group-wise data, especially given LLM sensitivity to prompts, task distributions, and other contextual factors. The reported V.05V_{.05} values are analytically computed rather than measured under realistic distirbutions. While the experiments demonstrate observed fairness amplification, they do not show whether the worst case vulnerabilities arise naturally. The empirical comparisons lack confidence intervals.

Could the authors comment on technical novelty of the theorems and results, as they compare with prior related results/techniques?

Questions for authors
  • What is the justification for considering DP as the measure? What about other measures of fairness?
  • Could the authors clarify what is specific to AI agents in the formulation? The model is assuming some stochastic binary decision system, not necessarily an LLM agent ensemble.
  • Is it easy to characterize a group wise marginal distribution for an AI agent? What group size is needed to characterize? There could be several confounding factors. LLMs may be sensitive to things like differences in prompts for instance
  • what is the practical significance of worst case disparity? Are those joint distributions achieved typically? For example, the figure 1 example was handcrafted, and seems pathological. Related, theorem 2 might at first glance appear to say that fairness becomes worse with increasing number of agents. However, it is really saying that the worst case possibility gets worse. How to comment on the significance of this result in practice? 
  • can the authors extend the analysis to a plausible cross-agent relationship, instead of allowing arbitrary distributions? What might such models look like?
  • It seems that fixing the approval rate estimate can constrain how large the fairness gap can be in the worst case. Proposition 7 already supports fixed marginal reports and confidence intervals, why do the experiments focus primarily on the global worst-case bounds?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.