acceptodds
Under review as a conference paper at ICLR 2027

Fair Ranking with Attention Uncertainty

Abstract

Exposure fairness asks that each group of items in a ranked list receive a target share of user attention. Whether a ranking meets this target depends on an attention curve, which specifies how attention is distributed across positions, yet the curve used for evaluation may differ from the one users actually follow. This raises the question of what a fairness score computed under one curve, or a few, guarantees when the true attention curve is unknown. We first characterize the blind spot of a finite set of checked curves, which is the largest exposure disparity that can remain hidden while all checks pass at a given tolerance. For arbitrary targets and multiple, possibly overlapping groups, an exact formula is derived to compute the blind spot from the checked curves within the tolerance. We show that commonly used attention curves can still leave substantial exposure disparities undetected. For exact checks, we also give upper and lower bounds on the number of curves needed to guarantee a desired blind spot. Finally, we design a robust ranking linear program that maximizes utility while controlling exposure disparity under an unknown attention curve. Across Amazon ESCI, TREC Fair Ranking 2022, and German Credit, we show that policies that are fair under a single logarithmic attention curve are still off target on average by 36%–44% of total attention under some monotone attention curve. Under the worst such curve for each query, these policies deliver on average only 42%–64% of the target exposure. They remain off target by 7%–11% of total attention under attention curves estimated from real-world click logs, under which they deliver on average only 80%–91% of the target exposure. By contrast, our robust linear program keeps the mean deviation below 0.06% of total attention, so that it delivers at least 99.8% of the target exposure, with an additional loss of about 1% or less in ranking utility relative to the single-curve policies.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Fair ranking, exposure fairness, robust optimization

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_FuQdOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors examing exposure fairness in ranking. However, in practice the real attention curve is not known, so it has to by assumed, and fairness under one attention curve, or under a few, is a weak guarantee. Their formula gives the largest disparity compatible with a set of passed checks, for arbitrary targets and overlapping groups. They characterize the blind spot of a finite set of checked curves and they provide upper and lower bounds on the number of curves needed to guarantee a desired blind spot. Moreover, they show that commonly used attention curves are not always able to detect exposure disparities. They design a robust ranking LP that optimizes utility and controls exposure disparity under attention curve uncertainty. Finally, their experimental evaluation shows that policies that are fair under a single logarithmic attention curve are still off target by 36%-44% under some monotone attention curve, and under the worst curve, they deliver only 42%-64% of the target exposure. By contrast, their robust LP keeps the mean deviation below 0.06% of total attention, at a loss of 1% or less in ranking utility relative to the single-curve policies.

Critical strengths
  • The paper is well written and well structured. The scope of the work is clear and it is easy to follow, and the research question provided is well-posed and important. They state that, as prior work showed, the exposure fairness results change based on the assumed attention curve. For that, the provide an exact formula for the largest unfairness a given set of curve checks can miss (the blind spot), and they show that checking every top-r prefix closes this gap entirely, as long as attention doesn’t depend on which items are shown.

  • The paper's evaluation is thorough since they benchmark against several baselines, across three datasets with distinct group structures. Methods are also compared at approximately matched NDCG, so fairness gains are not simply the result of sacrificing more ranking quality. Additionally, they conduct a stress test using attention curves estimated from real click logs, using restricted curve classes as a balanced middle ground, showing that single-curve fairness can still miss 7%-11% of attention, while the robust LP stays near zero.

Critical weaknesses
  • The paper lacks in motivation for the audit-size result. Evaluating all k prefix curves is cheap in practice, with k=10-20, and the authors do not provide any real-world scenario where an auditor is restricted to checking a few curves. Moreover, the exact tightness result is quite restrictive since it requires two groups, equal target allocations, and exact checks (holds only for the synthetic German Credit task). The authors must clarify when minimizing checked curves is practically necessary.
Questions for authors
  1. What is the main contribution of the results in Section 3.3? Checking all k prefixes isn't already cheap? Why do we need to minimize the number of curves?
2 comments
AuthorsOpenReview ↗

Response to Reviewer FuQd

Thank you for the careful review and positive comments. We have uploaded a revised version with changes marked in red and address your concerns below.

Why minimize the number of curves. Section 3.3 gives the smallest number of curves that guarantees a blind spot β\beta, together with curves that attain it. We agree that checking all kk prefixes is cheap for a short ranking when the platform holds the policy. But the number of checked curves matters in two other situations.

First, the party that reads a fairness report often has no access to the ranking policy. It sees only the few numbers in the report, each computed under one attention curve chosen in advance. For example, Google Ads reports an advertiser's impression share at three levels, namely the very first ad, the top block of ads, and all ad positions. For hiring tools that score candidates, bias audits under New York City's Local Law 144 also report impact ratios only at the median score. Theorem 4 tells the reader how large a disparity such a report can still miss. For two groups with equal targets, the three impression shares leave a blind spot of at least 0.25, and a single median cutoff leaves 0.5. Theorem 5 tells the designer of a report how many curves, and thus how many numbers, it must contain to guarantee a given blind spot. A blind spot of 0.05, for example, takes 17 numbers at k=100k=100 and 29 at k=1000k=1000.

Second, checks cost time and storage on long rankings and large logs. In the experiments added in our response to Reviewer Z4yB, storing each group's exposure under 29 endpoint curves instead of all 1000 prefix curves at k=1000k=1000 reduces storage by 97% and the time to compute and resample these values by about 90%. Enforcing fairness is also faster with fewer curves. On German Credit with k=1000k=1000 and four or eight groups, the LP constrained only by the endpoint curves takes 63–70% less time to solve than the robust LP, under the same disparity bound and with almost the same utility.

Scope of the tightness result. Theorem 5 holds for arbitrary targets and overlapping groups. Its upper and lower bounds have the same logarithmic growth in the ranking length (Corollary C.11). The two bounds are equal for two groups with equal targets under exact checks. Outside this case they can still stay close. On German Credit at k=1000k=1000, the smallest number of curves lies between 22 and 48 for four groups and between 17 and 50 for eight groups, compared with 1000 prefixes. We acknowledge that the gap between the two bounds can be narrowed further, which is an interesting direction for future work. Our main contribution here is a set of curves that is far smaller than the set of all prefixes, together with a proof that its size is optimal up to a constant factor.

Thank you for raising these points. In the revised paper, we have added the motivation of fewer curves to the beginning of Section 3.3 and the new experiments to Section 4.5.

I would like to thank the authors for the clear response.

The motivation and the examples given regarding the number of curves are convincing. I appreciate that the motivation has been added to Section 3.3 and the new experiments to Section 4.5. I will keep my positive score.

Reviewer_fQ2ROpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies exposure fairness in ranking systems under uncertainty about user attention. Specifically, each group of items has a target share of attention, while the actual attention assigned to different ranking positions is unknown and may differ across users. This uncertainty can lead to violations of group exposure fairness even when a ranking satisfies fairness constraints under a particular attention curve.

The paper first introduces the notion of a blind spot, which characterizes the maximum possible group exposure disparity when fairness constraints are satisfied, up to a tolerance δ\delta, under a finite set of attention curves. The authors derive theoretical characterizations of this blind spot and investigate how many attention curves need to be checked to provide fairness guarantees.

The second part of the paper formulates a robust linear program (LP) that maximizes ranking utility subject to group exposure fairness constraints under all monotone attention curves. The proposed approach is evaluated on several ranking datasets, with results demonstrating improved worst-case exposure fairness at a relatively small cost in ranking utility.

Critical strengths

The paper provides a mathematical framework for quantifying exposure fairness violations when the true attention distribution is unknown. In particular, the blind-spot formulation offers a way to analyze the limitations of checking fairness under only a finite set of attention curves. The theoretical analysis appears technically sound and provides a useful characterization of the problem.

Critical weaknesses
  1. Limited methodological novelty of the proposed robust optimization approach.

The robust LP formulation in the second half of the paper appears to be a relatively straightforward extension of existing fair ranking formulations. It maximizes ranking utility subject to linear group exposure fairness constraints, with robustness enforced by considering a set of attention curves. While the resulting formulation is mathematically valid, I find the underlying optimization approach insufficiently novel. The paper does not clearly establish a substantial methodological advancement over existing robust optimization and fairness-constrained ranking approaches.


  1. Unclear practical implications and limited real-world applicability.

My main concern is the practical usefulness of the proposed framework. Although the authors provide theoretical guarantees on exposure fairness under unknown attention curves, it is unclear how these guarantees should be interpreted or used in real-world ranking systems.

For example, the paper emphasizes the importance of checking fairness under different attention curves, but it does not sufficiently explain how such checks would be performed in practice or how companies operating ranking systems should use the results to make actionable decisions. Furthermore, the worst-case analysis considers all monotone attention curves, but it remains unclear whether this uncertainty model adequately represents realistic user attention behavior.

Consequently, although the experimental results demonstrate improved robustness according to the proposed metrics, I am not convinced that these improvements translate into meaningful practical benefits. This substantially limits the significance of the contribution.

Questions for authors

Please check the weaknesses part.

3 comments
AuthorsOpenReview ↗

Response to Reviewer fQ2R (1/2)

Thank you for the thoughtful review. We have uploaded a revised version with changes marked in red and will address your question below.

Robust optimization. We agree that our LP follows a standard robust optimization framework. The closest prior formulation is the LP of Singh & Joachims (KDD 2018), which enforces exposure constraints under a fixed attention curve. In our setting, the same ranking policy must satisfy the disparity bound under every admissible attention curve. The prefix characterization in Theorem 2 reduces this infinite family of constraints to one disparity bound per prefix and group, making an exact LP formulation possible. For large platforms such as Amazon and Google, evaluating fairness for every logged query may still be very costly. Fewer curve constraints can substantially reduce computational cost while retaining comparable ranking utility within the same exposure disparity bound, as illustrated by the experiments in our response to Reviewer Z4yB. To use fewer curves, we need to know what they guarantee and how to choose them. To this end, Theorem 4 gives the exact disparity bound for a given set of checks, and Theorem 5 bounds how many curves are needed for a target guarantee, with endpoint curves attaining the upper bound.

AuthorsOpenReview ↗

Response to Reviewer fQ2R (2/2)

Practical use. We answer the points of this concern in turn.

How to read the guarantee. For each group, a platform reports a number, which is the largest gap between the group's share of the top rr positions and its target over all depths rr. By Theorem 2, this number equals the group's worst exposure disparity over every monotone attention curve. If it is at most δ\delta, then no population of users, whatever its mixture of reading depths, sees the group's exposure deviate from its target by more than δ\delta.

How the check is performed. The check needs only ranking logs. For each group, count how often it appears in the top rr positions for r=1,…,kr=1,\dots,k, and compare with the target. This takes O(mk)O(mk) time per query and needs no click data. Checks of this form already run in production. For example, the fair ranking deployed in LinkedIn Talent Search measures group representation at every prefix of a result list (Geyik et al., KDD 2019).

What a company does with the result. The check returns the group and the depth with the largest gap, so the platform sees where its ranking departs from the target. To enforce a bound, the platform can keep its relevance model and solve our robust LP. Our experiments on ESCI, TREC Fair Ranking 2022 and German Credit all reveal that the standard single-curve LP is far off target at some reading depth, by 36%–44% of total attention on average. By comparison, the robust LP keeps this deviation below 0.06% with a utility loss of about 1% or less. Such a guarantee over all attention curves matters because the attention model is the part that an outside party can dispute. For instance, in the Google Shopping case, the European Commission used evidence on how clicks fall with position to establish the effect of Google's placement of results, and found strong effects on mobile devices. The Digital Markets Act also requires "transparent, fair and non-discriminatory conditions" for ranking without fixing an attention model. Fairness shown under one chosen attention curve can therefore be challenged by disputing the choice, whereas we proved that a bound on the largest prefix disparity holds under every monotone attention curve.

Whether monotone, worst-case attention is realistic. Monotone attention means only that a lower position never receives more attention than a higher one. The standard position-based models, such as the DCG and RBP discounts, all satisfy it. Diaz et al. (CIKM 2020) also observe that "per existing models, exposure monotonically—often exponentially—decreases with position in a ranking." Eye-tracking studies and click logs show the same pattern. In eye tracking, the share of result pages with a fixation at rank rr falls from about 68% at rank 1 to 6% at rank 10 (Joachims et al., SIGIR 2005). In the Baidu-ULTR logs, the click-through rate falls at every rank, from 0.34 at rank 1 to 0.013 at rank 10. In its Google Shopping decision, the European Commission found that Google's top result receives about 35% of the clicks, and that moving a result from the first to the third position halves its clicks. The direction of the decline is stable and its steepness varies with the device, which is what the monotone class leaves open. The worst case also corresponds to real users. By Lemma 1, every monotone curve is a mixture of users who read from the top and stop at different depths, which is exactly the browsing model studied by Carterette (SIGIR 2011). The largest disparity is therefore attained by the users who stop at some depth. On 88% of ESCI queries, the single-curve LP is furthest off target for the users who read only the top result. When the attention curve depends on the displayed items, which our model excludes, the robust LP still keeps the disparity at 0.020–0.022 for simulated users of the cascade and dynamic Bayesian network (DBN) click models, who stop once a result satisfies them, against 0.25–0.29 for the single-curve LP.

Whether the improvements bring practical benefits. The robust LP also improves fairness under real user attention. Under attention estimated from 325,271 Baidu-ULTR result pages, the single-curve LP delivers on average 80–91% of the target exposure on ESCI, German Credit and TREC, while the robust LP delivers at least 99.8%. The improvement also carries over to decisions made from a ranking. On German Credit, when a decision maker takes the top half of a ranking produced by the single-curve LP, the selection rate of one age group is 0.95 times that of the other. This is also the check used in New York City's bias audits of hiring tools. For the top-ranked applicant alone, this ratio falls to 0.74, below the four-fifths threshold of the US Equal Employment Opportunity Commission. Under the robust LP, the ratio is 1 at every cutoff.

Thank you for raising these points. In the revised paper, we have added evidence for monotone attention to Section 2 and these experiments to Appendices J.7 and J.11.

AuthorsOpenReview ↗

Follow-up on Our Response

Dear Reviewer fQ2R,

Thank you again for your review. As the discussion period ends today, we would like to check whether our response has addressed your concerns. Please let us know if any questions remain or if you have any suggestions for further improving the paper. We appreciate the time and effort you've dedicated to reviewing our work.

Reviewer_Z4yBOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies exposure fairness under unknown, list-independent, monotone attention curves. Using a decomposition into prefix curves, it characterizes the worst-case exposure disparity and the “blind spot” of a finite set of fairness checks, including the effects of nonzero checking tolerances. It further develops endpoint-curve constructions and bounds on the minimum audit size, with exact optimality for two equally targeted groups under zero-tolerance checks. Building on the prefix characterization, the paper formulates a robust linear program that maximizes ranking utility while controlling disparity under every admissible attention curve. Experiments on Amazon ESCI, TREC Fair Ranking 2022, and German Credit demonstrate substantial residual disparities in single-curve fair policies and show that the robust formulation can reduce these disparities with small average additional utility costs relative to the single-curve policies.

Critical strengths

S1. A meaningful distinction between model-specific fairness and robust guarantees

The paper addresses a meaningful limitation of exposure-based fairness: passing a check under an assumed attention model does not establish fairness under other plausible models. It formalizes what a collection of successful checks actually guarantees, providing a useful perspective for interpreting and auditing fairness claims.

S2. Substantive theoretical results on blind spots and audit size

The paper gives an exact blind-spot characterization for arbitrary targets and overlapping groups. For two equally targeted groups, it constructs audits with a matching lower bound under zero-tolerance checks and analyzes how reconstruction coefficients amplify checking tolerances. These results establish the minimum audit size in that setting and explain why multiple checks can still provide weak guarantees.

Critical weaknesses

W1. The practical benefit of minimum-size audits needs clarification.

The minimum-audit-size result is theoretically valuable, but the operational setting in which the number of checked curves is the limiting resource remains unclear. Once the group-by-position occupancy matrix is available, all prefix disparities can be computed in O(mk)O(mk) time using cumulative sums, for mm groups and kk positions. The main robust LP also enforces all prefix constraints. Appendix J.7 uses endpoint curves as LP constraints and evaluates the resulting policies, while Appendix J.11 reports ranking-solver timings; however, neither directly establishes the cost benefit of reducing the checked set. Could the authors clarify the intended information-access or cost model and explain when fewer checks offer a practical advantage over evaluating all prefixes? A corresponding cost analysis or targeted experiment, for example on longer rankings, would strengthen the practical motivation.

Questions for authors

Please see Weaknesses.

2 comments
AuthorsOpenReview ↗

Response to Reviewer Z4yB

Thank you for your positive and constructive feedback. We have uploaded a revised version with changes marked in red and will address your question below.

Checking fewer curves can save computation resources and shorten runtime while still retaining a desirable fairness guarantee. As suggested, we added the following experiments to compare the computational costs of using all prefix curves versus a smaller set of endpoint curves.

Evaluators can estimate exposure disparity from ranking logs. However, evaluating every logged query is often costly, so evaluators tend to sample queries and analyze the corresponding rankings. For example, Biega et al. (TREC 2019) sampled queries from Semantic Scholar logs to evaluate exposure fairness. In the first experiment, we study a similar setting using 20,000/100,000 simulated queries. For each query, we generate ten rankings by randomly assigning a group label to each position. We then compute and store mean exposure over the ten rankings for each query, group, and curve. We compare two methods, one using all kk prefix curves and the other using only the selected endpoint curves. We time both methods on the same CPU core using identical rankings and the same 200 bootstrap resamples of the queries. With fewer curves, each resample aggregates fewer exposure values, and reduces computation.

QueriesGroupskk# Endpoint curvesPrefix storageEndpoint storageStorage savedTime saved
20,00021001716 MB2.7 MB83%72%
20,0002100029160 MB4.6 MB97%91%
20,000161000291.28 GB37 MB97%90%
100,0002100029800 MB23 MB97%90%

Real-world applications can involve many more queries and groups (e.g., Amazon product search with brands as groups), making these savings important at scale.

In the second experiment, we report LP solve time for ranking length k=1000k=1000 on German Credit with groups defined by age and sex. Candidates are sampled with replacement, with targets proportional to population shares. We compare two LPs with the same worst-case disparity bound β\beta:

  • The robust LP maximizes utility with every prefix disparity bounded by β\beta.
  • The endpoint LP maximizes utility with its checked disparities fixed at zero, using a curve set whose blind spot is at most β\beta.

We solve both LPs on the same inputs with the same prescribed disparity bound β=0.05\beta=0.05.

Groups# Endpoint curvesRobust LP median solve time (s)Endpoint LP median solve time (s)Time savedRobust LP utility advantage
448680.85202.0770%0.49%
850585.57219.1463%0.85%

Therefore, checking fewer attention curves lets large platforms such as Amazon trade a small utility loss for a much lower computational cost at the same disparity bound. And we provide a set of curves with a provable fairness guarantee. We have added these experiments and clarified the motivation in the revision. Thank you for raising this point.

Thanks for the response. I will keep my current positive score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.