- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes Rawls-Shapley learning, a group-fairness objective motivated by the connection between distributive justice and nash bargaining. It maximizes the product of group utility gains, where utility is the average probability assigned to the correct label. the resulting gradients assign larger coefficents to groups with lower utility gains. The paper includes two implementations: RS-Known using observed demographic groups and RS-proxy for supervised soft memberships. The paper comes with theoretical guarantees and a set of experiments on classic benchmark fairness datasets.
Critical strengths
Strength #1: The objective is simple, interpretable, intuitive and principled: it provides a way to give greater marginal weight to groups with lower utility.
Strength #2: The paper gives a careful analysis, showing that Nash welfare does not guarantee the best worst-group outcome, and that its theoretical optimality guarantees do not directly apply to the solutions found by neural network training.
Critical weaknesses
Weakness #1: The experiments lack a matched baseline that maximizes the arithmetic mean of the same soft group utilities, so they do not isolate the benefit of Nash aggregation from the change in training loss. Both RS variants also report lower mean worst-group accuracy and AUC than ERM on both datasets, without uncertainty estimates to assess these differences. Some additional baselines/ablations I'd suggest are group-balanced soft utility, min-max soft utility, and just overall soft utility without group-weightings.
Weakness #2: Incremental contribution relative to existing wealfare + bargaining approaches. Relevant work (not cited in the paper) include [1] which studies welfare aggregation over groups, including geometric-mean welfare and [2] which optimizes a Nash product of group improvements in classification trees.
[1] Cousins, Cyrus. "An axiomatic theory of provably-fair welfare-centric machine learning." Advances in Neural Information Processing Systems 34 (2021): 16610-16621. [2] Liapis, Georgios I., and Lazaros G. Papageorgiou. "Optimisation-based classification tree: a game theoretic approach to group fairness." International Conference on Optimization and Learning. Cham: Springer Nature Switzerland, 2024.
Questions for authors
- Evidence for the Nash aggregation: how much does it outperform simpler baselines mentioned above?
- What is the novelty compared to the two related works mentioned above?