acceptodds
Under review as a conference paper at ICLR 2027

Human–AI Disparity Distortion: A Component-Referenced Diagnostic for Hybrid Decision Systems

Abstract

Fairness audits of hybrid human–AI systems often examine disparity in human judgments, algorithmic predictions, or final hybrid outcomes. Taken alone, these quantities do not distinguish the disparity implied by the component systems from non-additive disparity associated with their integration rule. We study this distinction through a component-referenced diagnostic. For an integration rule combining human probability and algorithmic probability , additive pooling yields group disparity exactly by linearity of expectation. We define the Human–AI Disparity Distortion Index (HDDI) as the difference between the disparity produced by and this additive reference. HDDI is identically zero under linear pooling; more generally, nonzero HDDI reflects group-differential departure from additive integration rather than nonlinearity alone. We evaluate HDDI across three complementary settings. On ACSIncome, controlled simulated human–AI pairings establish its behavior across models, groups, and integration weights. On Civil Comments, HDDI extends to identity-sensitive text moderation and is often more pronounced along a non-toxic, false-positive-relevant pathway; finite-sample recovery improves substantially with sample size, with median RMSE decreasing from at to at . On COMPAS, nonzero interior HDDI persists when human probabilities are derived from observed judgments rather than simulation, reaching , although its magnitude depends on how human judgments are aggregated. Across all three settings, HDDI complements rather than replaces conventional disparity and fairness measures.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
human-AI collaboration, algorithmic fairness, hybrid decision systems, fairness auditing, group disparity, non-additive disparity, probability integration, disparity measurement

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_o837OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper primarily investigates how different probability aggregation rules—specifically when combining human and AI predictions within a hybrid decision-making system—affect group disparities in mean predicted probabilities. Using linear fusion (an additive approach) as a reference, the authors examine the Log-odds Pooling method, while noting that the proposed framework theoretically accommodates general aggregation rules. They analyze the difference between the group disparities produced by this method and the linear reference; this difference is defined as HDDI, the paper's core metric, which quantifies the group-level disparity deviation from additive aggregation. The authors further establish the fundamental properties of HDDI through two mathematical propositions: that group disparity under linear fusion is a weighted sum of component disparities, and that HDDI can equivalently be expressed as the difference in mean individual-level fusion deviations between groups.

In the experimental section, the authors first use the ACSIncome dataset to confirm that HDDI behaves as mathematically predicted and responds to controlled variations in simulated human predictions. Next, using the Civil Comments dataset, they examine conditional HDDI in identity-sensitive text moderation scenarios and its statistical recoverability across different sample sizes. Finally, using COMPAS data derived from observed human judgments, the authors demonstrate that nonzero HDDI persists when human probability inputs are based on actual human judgments rather than simulations, while also showing that its magnitude depends on how these judgments are represented probabilistically.

Critical strengths
  1. The research addresses a problem of scientific value: Hybrid decision-making systems involving both humans and AI are increasingly widespread yet entail substantial risks regarding fairness. Since different decision aggregation rules can lead to varying group disparities in mean predicted probabilities, the HDDI metric proposed in this paper offers a method to measure such potentially fairness-relevant deviations relative to an additive reference, providing a valuable perspective on issues in hybrid decision-making systems.

  2. The experimental design is robust and logically sound: The paper provides a clear mathematical definition of HDDI and validates it through a comprehensive series of experiments—ranging from verifying the metric's properties under controlled conditions to analyzing conditional disparity deviations and statistical recoverability in text moderation, and finally assessing its applicability using observed human judgment data. The experimental design demonstrates a clear, progressive logic that offers reasonable empirical support for the proposed diagnostic metric within the scope of the study.

Critical weaknesses
  1. The core definition of HDDI rests on premises that lack robust justification, and it lacks validation regarding generalization across different aggregation methods: HDDI’s calculation uses linear fusion as a reference point; while the authors mathematically demonstrate that linear fusion provides a clear additive component reference, they do not sufficiently justify why this reference is particularly meaningful for fairness diagnosis. Although the authors do not claim that linear pooling is optimal or normatively preferable, I remain unconvinced that deviations from this reference have sufficiently clear practical significance for fairness diagnosis. While subsequent experiments validate HDDI’s computational properties, the fundamental significance of the metric itself remains insufficiently articulated. Furthermore, although the theoretical framework is defined for a broad range of integration operators, the empirical evaluation uses log-odds pooling as its only nonlinear operator, alongside a linear-pooling control; the extent to which the observed patterns hold under other nonlinear rules remains untested.

  2. The experiments do not encompass "human-AI collaborative hybrid decision-making systems" in interactive real-world settings: HDDI is designed for such collaborative systems, yet the paper merely utilizes "actual human judgment data" from the COMPAS dataset, performing a post-hoc mathematical aggregation of these judgments with calibrated COMPAS probabilities rather than investigating the actual decision-making process within a collaborative context. In an interactive human–AI system, humans might accept AI suggestions, adjust their judgments based on the interaction, or selectively adopt AI outputs—or at least remain aware of their role within the system. Consequently, this post-hoc aggregation does not establish HDDI’s applicability to deployed interactive human–AI decision-making, leaving its external validity in such settings open to question.

  3. HDDI’s relationship to fairness is largely indirect, and its practical diagnostic value has not been sufficiently substantiated: Although the paper provides a comprehensive account of HDDI’s proposal, definition, mathematical properties, and validation, it fails to directly demonstrate the added value of this metric for addressing fairness issues in hybrid decision-making systems. For instance, the authors themselves acknowledge that a non-zero HDDI value does not necessarily indicate unfairness, nor does a zero value guarantee fairness. Thus, the paper’s current status is best characterized as proposing a computational metric, HDDI, and demonstrating its ability to measure—as intended—deviations in group disparities (relative to an additive baseline) arising from different aggregation rules. However, while the paper demonstrates that HDDI captures information distinct from final group disparity, further analysis and discussion are still needed regarding how these deviations should be interpreted, under what circumstances they hold practical significance, and what new diagnostic insights they offer compared to traditional fairness metrics. Of course, this does not conflict with the paper's aforementioned strength; HDDI indeed provides a method for measuring these potential fairness-related disparities, yet the paper still needs to further articulate the unique value and role of this measurement capability within the context of practical fairness research.

Questions for authors
  1. Beyond the mathematical arguments already presented in the paper, could the authors further explain the rationale for choosing linear pooling as the reference? In particular, what specific significance does this reference have for fairness diagnosis in hybrid human–AI decision systems?

  2. Could the authors provide a concrete example from the existing experimental results to illustrate what additional fairness-related diagnostic insights HDDI offers beyond directly examining the group disparities of the human, AI, and final hybrid systems? In particular, how should the practical significance of HDDI be interpreted, given that a nonzero value does not necessarily indicate unfairness?

3 comments

Thank you for your thoughtful review and for recognizing the motivation and experimental design of our work. We appreciate your concerns about the choice of reference, the practical value of HDDI, and its applicability beyond the settings originally examined. We have uploaded a revised manuscript, with changes highlighted in blue. We have clarified the rationale for using additive pooling as the reference (Section 3 and Appendix A). Its group disparity is exactly the weighted combination of the component disparities, and it can also be interpreted as the expected prediction under group-independent random source selection. This gives HDDI a clear component-based benchmark without assuming that additive pooling is optimal or inherently fair. To make the diagnostic value more concrete, we added a risk-stratified ACSIncome analysis (Section 5.1 and Appendix D). In the primary setting, aggregate HDDI is −0.000378, while pooled-weight standardized HDDI is +0.000600. Only the lowest AI-risk quartile shows a clearly nonzero local deviation after simultaneous adjustment. This suggests a practical audit follow-up: examining the subgroup where the departure is detectable rather than relying on the aggregate result alone. We also added an audit-prioritization analysis, which shows some setting-dependent screening benefits while making clear that these findings do not establish an effective fairness intervention. We further extended the evaluation to learned hard routing (Sections 4–5 and Appendix C), allowing us to examine an instance-dependent integration rule beyond fixed-weight log-odds pooling. We have also clarified that COMPAS uses observed human judgments combined post hoc, rather than observed human–AI interaction. We believe these additions better explain both the value and the current scope of HDDI. Thank you again for your constructive suggestions; they helped us strengthen the paper and present its contribution more clearly.

Reviewer_o837OpenReview ↗

Response to Authors

I appreciate the authors' clarifications and additional experiments. While I remain somewhat uncertain regarding the specific practical scenarios and the real-world application value of the paper, these responses have addressed most of my concerns; therefore, I have decided to raise my score.

Thank you for taking the time to reconsider our work. We sincerely appreciate your thoughtful feedback and recognition of our revisions. Your suggestions have been very helpful in improving the paper.

Reviewer_YMG4OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes to study how aggregating prediction probabilities between human decision makers and ML models can create group disparities. The authors propose the Human–AI Disparity Distortion Index (HDDI) to capture these disparities. Empirical evidence supports the proposal of the authors.

Critical strengths

The paper's main strengths are:

  1. The motivation for the work is interesting and investigates, to the best of my knowledge, some overlooked areas in fairness;
  2. The mathematical propositions are easy but correct;
  3. The work is generally well positioned in the literature (some references are missing though - see below).
Critical weaknesses

I have a few concerns regarding the work:

  1. As acknowledged by the authors (in lines 121-122):

    This accounting is noncausal: HDDI does not identify differential treatment, selective reliance, or another behavioral mechanism, nor does it determine whether the system is normatively fair

    This is a major limitation in my view, as the question the authors pose is inherently causal. Moreover, I think the paper could better leverage recent causal frameworks; for instance, in [1], the authors propose retrieving the causal effect of learning to defer systems when both a human prediction and an ML model prediction are issued separately. I think the authors are studying a setting that corresponds to something analogous to their Scenario 1 across all instances (as long as human predictions are independent of those issued by the ML model). Thus, I think that under some mild assumptions, the authors could try to extend their work to a causal problem. In particular, since both probabilities are observed for all instances, the contrast between fλf_\lambda and LλL_\lambda might be an identified causal effect of the integration rule. I would encourage the authors to make this explicit and distinguish it from the (unidentified) behavioural mechanisms they disclaim. This would make the paper more solid.

  2. Another main concern I have is the fact that the proposed measure is aggregate. As shown in [2], often aggregate measures for fairness hide local patterns of discrimination due to the Yule effect. I think that some of the results provided by the authors could also be read through this lens.

  3. In the empirical evaluation, I think the authors are mostly simulating humans. However, there are some datasets with real human decision-makers that can be used for the purpose of the current work. See, e.g., [1] and [3] for pointers. For instance, I think that for the ChestXRay dataset one also has access to some sensitive attributes such as gender. Showing some evidence also in that case might be interesting and could further improve the empirical evaluation.

  4. I would also consider adding a simple learning to defer baseline as a way to aggregate between humans and ML models.

  5. Generally speaking, I think the paper could benefit from some extra care in the presentation. For instance, these are some parts I think I would change:

    • first, I would clearly distinguish between the background, the methodology part and the empirical evaluation. Currently, these parts are quite overlapping and make the paper difficult to follow. For instance, in Section 3.3, the authors mix experimental settings and the methodology;
    • None of the propositions references their proofs in the main paper. Adding a pointer after the proposition is a courtesy to the reader that I would appreciate from the authors;
    • regarding empirical evaluation, I also think clearly stating which are the research questions would help the flow of the work. Moreover, I would aggregate the current experimental sections into a single one, and then divide it into a (i)(i) settings subsection, where one describes all the datasets, methods and metrics used and setup of experiments; and $(ii) a results subsection, where one discusses the results, would help improve clarity.

I am willing to improve the score if these concerns are properly addressed in the rebuttal phase.

[1] Palomba et al., A Causal Framework for Evaluating Deferring Systems. AISTATS 2025

[2] Ruggieri et al., Can We Trust Fair-AI? AAAI 2023

[3] Mozannar et al., Who Should Predict? Exact Algorithms For Learning to Defer to Humans. AISTATS 2023

Questions for authors

Please try to address the highlighted weaknesses.

6 comments
AuthorsOpenReview ↗

Response to Reviewer YMG4

Thank you for your careful and constructive review. We appreciate your suggestions regarding causal interpretation, aggregation effects, empirical evaluation, and presentation. We have uploaded a revised manuscript, with changes highlighted in blue. We respond to your concerns below.

Question 1: Causal interpretation of HDDI. Response:We appreciate this important distinction. We agree that when both component probabilities are available for the same instances, the contrast between fλf_\lambda and LλL_\lambda can be directly computed while holding the inputs fixed. We have clarified this interpretation in Section 3 and Appendix A. However, this fixed-input comparison does not necessarily identify the causal effects of deploying a different integration policy, where human judgments or reliance may change. We have also incorporated Palomba et al. (2025) into Related Work to clarify how our diagnostic differs from causal evaluations of deferral systems.

Question 2: Aggregate measures may obscure local disparities. Response: We agree and have added a risk-stratified ACSIncome analysis (Section 5.1 and Appendix D). In the primary HGB setting, aggregate HDDI is −0.000378, while pooled-weight standardized HDDI is +0.000600, with a composition remainder of −0.000978. After simultaneous adjustment across four AI-risk quartiles, only the lowest-risk quartile shows a clearly nonzero local HDDI. These findings illustrate how aggregate results can obscure local deviations and motivate examining subgroup contrasts alongside overall measures. We have also added Ruggieri et al. (2023) to Related Work, while avoiding a causal interpretation of the observed sign reversal.

Question 3: Limited evidence from observed human decision-makers. Response:Thank you for highlighting this limitation and suggesting datasets such as ChestXRay. We have clarified that COMPAS uses repeated observed human judgments combined with algorithmic predictions post hoc, rather than observed human–AI interactions. Although we did not add another interactive dataset in this revision, we now make this distinction explicit in Methods and Discussion. We agree that evaluating HDDI using observed reliance behavior would strengthen future work.

Question 4:Adding a learning-to-defer baseline. Response: Following your suggestion, we added a learned hard-routing experiment on ACSIncome (Sections 4–5 and Appendix C). A validation-trained router selects between frozen human and AI predictions at prespecified budgets, and we compare its performance with matched-rate random hard routing and AI-only prediction. This extends the evaluation beyond fixed-weight log-odds pooling and allows us to examine the contributions of within-group routing–prediction associations and group-level routing differences. We also added Mozannar et al. (2023) to Related Work. The experiment remains a controlled evaluation with simulated human probabilities rather than observed deferral behavior.

Question 5: Presentation and organization. Response:We have reorganized the manuscript to separate the literature review (Section 2), general framework and propositions (Section 3), experimental design (Section 4), and results (Section 5). The Introduction now states the research question and contributions more explicitly. We also moved the log-odds operator into the empirical design, added references to the mathematical derivations in Appendix A, and revised the Discussion to better connect the findings with the scope and limitations of HDDI.

Thank you again for your thoughtful feedback and helpful references. Your comments led us to strengthen the empirical analysis, clarify the interpretation of HDDI, and improve the organization of the paper. We hope the revision addresses your concerns.

Can you please highlight which parts of the manuscript were changed? This would help me assess the implemented changes.

Thanks, I can see them now.

Reviewer_m1LCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

To solve in the current fairness audits, quantities do not distinguish the disparity implied by the component systems from non-additive disparity associated with their integration rule, thus, this paper propose Human–AI Disparity Distortion Index (HDDI) to measure the difference between the group disparity of a hybrid prediction rule and that of a linear rule of the same human and algorithmic probabilities.

Critical strengths
  1. the authors propose a new and clearly defined metric.
  2. the analyzed result may have a informative insights for anyone working on fairness audit, i.e., there is a substantial difference between continuous-rating and binary-response aggregation.
Critical weaknesses
  1. the contribution is incremental with limited novelty and practical value.
  2. All three experiments use the same log-odds integration operator and even COMPAS supplies observed component judgments rather than observed human–AI integration behavior.
  3. The practical insights are limited, lacking a substantive conclusion or intervention for fairness audit through HDDI beyond reporting the component disparities and comparing the two aggregators.
Questions for authors
  1. Can you identify a concrete audit decision changes because HDDI is reported? An example would strengthen the significance of HDDI.
3 comments
AuthorsOpenReview ↗

Response to Reviewer m1LC

Thank you for the thoughtful review. We appreciate your concerns about the novelty and practical value of HDDI. We have uploaded a revised manuscript, with changes highlighted in blue. To address your question about concrete audit decisions, we added an ACSIncome analysis (Section 5.1 and Appendix: Local HDDI and Composition Sensitivity). The aggregate HDDI is −0.000378, while the standardized estimate is +0.000600. After simultaneous adjustment, only the first AI-risk quartile has a clearly nonzero local HDDI. This provides a concrete reason to prioritize that subgroup for further examination rather than relying on the aggregate result alone. We also added a learned hard-routing experiment to examine an integration rule beyond log-odds pooling (Section 5, Learned Hard-Routing Robustness), and an audit-prioritization analysis (Section 5, Audit Prioritization and Decision-Level Boundaries). The latter shows setting-dependent benefits, although none of the evaluated configurations meets our prespecified consequential-opportunity criterion. We have also clarified that the COMPAS analysis combines observed human judgments with algorithmic predictions post hoc, rather than observing human–AI interaction directly. Thank you again for your constructive comments. They helped us better explain the diagnostic value of HDDI and its current limitations.

Thanks to the author; my question has been resolved. My main concern remains the limited novelty. And I’m currently maintaining my score.

Thank you for taking another look at our revision and for confirming that your question has been resolved. We appreciate your candid feedback regarding novelty.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.