- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors examing exposure fairness in ranking. However, in practice the real attention curve is not known, so it has to by assumed, and fairness under one attention curve, or under a few, is a weak guarantee. Their formula gives the largest disparity compatible with a set of passed checks, for arbitrary targets and overlapping groups. They characterize the blind spot of a finite set of checked curves and they provide upper and lower bounds on the number of curves needed to guarantee a desired blind spot. Moreover, they show that commonly used attention curves are not always able to detect exposure disparities. They design a robust ranking LP that optimizes utility and controls exposure disparity under attention curve uncertainty. Finally, their experimental evaluation shows that policies that are fair under a single logarithmic attention curve are still off target by 36%-44% under some monotone attention curve, and under the worst curve, they deliver only 42%-64% of the target exposure. By contrast, their robust LP keeps the mean deviation below 0.06% of total attention, at a loss of 1% or less in ranking utility relative to the single-curve policies.
Critical strengths
-
The paper is well written and well structured. The scope of the work is clear and it is easy to follow, and the research question provided is well-posed and important. They state that, as prior work showed, the exposure fairness results change based on the assumed attention curve. For that, the provide an exact formula for the largest unfairness a given set of curve checks can miss (the blind spot), and they show that checking every top-r prefix closes this gap entirely, as long as attention doesn’t depend on which items are shown.
-
The paper's evaluation is thorough since they benchmark against several baselines, across three datasets with distinct group structures. Methods are also compared at approximately matched NDCG, so fairness gains are not simply the result of sacrificing more ranking quality. Additionally, they conduct a stress test using attention curves estimated from real click logs, using restricted curve classes as a balanced middle ground, showing that single-curve fairness can still miss 7%-11% of attention, while the robust LP stays near zero.
Critical weaknesses
- The paper lacks in motivation for the audit-size result. Evaluating all k prefix curves is cheap in practice, with k=10-20, and the authors do not provide any real-world scenario where an auditor is restricted to checking a few curves. Moreover, the exact tightness result is quite restrictive since it requires two groups, equal target allocations, and exact checks (holds only for the synthetic German Credit task). The authors must clarify when minimizing checked curves is practically necessary.
Questions for authors
- What is the main contribution of the results in Section 3.3? Checking all k prefixes isn't already cheap? Why do we need to minimize the number of curves?
2 comments
I would like to thank the authors for the clear response.
The motivation and the examples given regarding the number of curves are convincing. I appreciate that the motivation has been added to Section 3.3 and the new experiments to Section 4.5. I will keep my positive score.
Response to Reviewer FuQd
Thank you for the careful review and positive comments. We have uploaded a revised version with changes marked in red and address your concerns below.
Why minimize the number of curves. Section 3.3 gives the smallest number of curves that guarantees a blind spot β, together with curves that attain it. We agree that checking all k prefixes is cheap for a short ranking when the platform holds the policy. But the number of checked curves matters in two other situations.
First, the party that reads a fairness report often has no access to the ranking policy. It sees only the few numbers in the report, each computed under one attention curve chosen in advance. For example, Google Ads reports an advertiser's impression share at three levels, namely the very first ad, the top block of ads, and all ad positions. For hiring tools that score candidates, bias audits under New York City's Local Law 144 also report impact ratios only at the median score. Theorem 4 tells the reader how large a disparity such a report can still miss. For two groups with equal targets, the three impression shares leave a blind spot of at least 0.25, and a single median cutoff leaves 0.5. Theorem 5 tells the designer of a report how many curves, and thus how many numbers, it must contain to guarantee a given blind spot. A blind spot of 0.05, for example, takes 17 numbers at k=100 and 29 at k=1000.
Second, checks cost time and storage on long rankings and large logs. In the experiments added in our response to Reviewer Z4yB, storing each group's exposure under 29 endpoint curves instead of all 1000 prefix curves at k=1000 reduces storage by 97% and the time to compute and resample these values by about 90%. Enforcing fairness is also faster with fewer curves. On German Credit with k=1000 and four or eight groups, the LP constrained only by the endpoint curves takes 63–70% less time to solve than the robust LP, under the same disparity bound and with almost the same utility.
Scope of the tightness result. Theorem 5 holds for arbitrary targets and overlapping groups. Its upper and lower bounds have the same logarithmic growth in the ranking length (Corollary C.11). The two bounds are equal for two groups with equal targets under exact checks. Outside this case they can still stay close. On German Credit at k=1000, the smallest number of curves lies between 22 and 48 for four groups and between 17 and 50 for eight groups, compared with 1000 prefixes. We acknowledge that the gap between the two bounds can be narrowed further, which is an interesting direction for future work. Our main contribution here is a set of curves that is far smaller than the set of all prefixes, together with a proof that its size is optimal up to a constant factor.
Thank you for raising these points. In the revised paper, we have added the motivation of fewer curves to the beginning of Section 3.3 and the new experiments to Section 4.5.