acceptodds
Under review as a conference paper at ICLR 2027

Coverage Is Not Fairness: Auditing Learned Controls In Participatory Budgeting

Abstract

In participatory budgeting, spending-based group fairness scores can report a perfect result even when nearly all voters receive no funded project they approved. We examine one such measure, cumulative share deficit (CSD), alongside voter coverage, and construct a valid allocation with CSD 0.0000 that excludes 99.0% of voters. Next, we hold the weights and feature definitions of learned project scores fixed and replay them through four allocation rules. At the unpenalized score, iterative payment reduces exclusion from 0.1470 to 0.0530 and increases welfare to 1.74x the direct-score value. Adding approval-count completion brings exclusion to 0.0427 and welfare to 1.99x. Neither payment rule produces a statistically detectable CSD improvement. On 18 held-out Warsaw series, learned endowments reduce CSD from 0.1992 to 0.1736, but a post-lock, four-parameter age table fitted on Warsaw training data has lower mean CSD (0.1624). Two locked external tests, one of learned priority and one of endowments on 32 projected-support series in two new cities, do not confirm transfer (decision: falsified). Finally, a post-hoc replay without refitting finds no detectable series-level difference on 58 native-approval series and unchanged allocations on 36. These results support reporting spending-based fairness alongside voter coverage, and learned interventions alongside the allocation changes they produce.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
algorithmic fairness, participatory budgeting, computational social choice, resource allocation, fairness auditing, cumulative share deficit, voter coverage, Method of Equal Shares, mechanism-informed learning, learned allocation rules, reward hacking, welfare, exclusion, social choice, external generalization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ujo5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks what a learned component does inside a participatory budgeting rule, and what the fairness metric used to judge it actually sees. The metric, cumulative share deficit (CSD), tracks, over repeated elections, how far the spending attributable to a demographic group's approvals falls short of its turnout share. A short decomposition and a toy instance show that the ledger is blind to coverage: CSD can be exactly zero, even as almost no voter receives an approved project.

Three learned interventions are compared. Two keep the Equal Shares payment phase and learn either per-group voter endowments (six features) or the purchase order; the third drops payment and fills the budget greedily from a five-feature learned score. Weights are fitted by CMA-ES on the Warsaw district series through 2022 and tested on the 2023 to 2025 editions of the same districts. A support-floor proposition shows payment-phase purchases need approval share at least cost over kappa times budget, which direct scoring does not.

The findings are mostly cautionary. Learned endowments reduce worst-group CSD relative to plain Equal Shares, but a four-number age lookup fitted afterward performs better. Freezing the unpenalized direct score and reinstating the payment-cuts exclusion raise welfare with no detectable CSD change, though the effect reverses at welfare-regularised scores. Two locked external studies in Katowice and Krakow fail to meet their preregistration criteria, and a post hoc reanalysis of native ballots in Gdynia and Lodz shows no detectable difference. The authors report all of this openly, including a counting bug and a post-lock code amendment, and conclude that coverage and allocation changes should be reported alongside spending-share fairness.

Critical strengths
  1. The reporting is unusually honest and complete, and the numbers check out. I recomputed every statistic reachable from the per-series tables (means, exact sign-flip p-values, bootstrap intervals, welfare ratios, gap closures) and all of them reproduce. The authors report the baseline that beats their method, the two locked studies that failed, a counting bug their own containment check caught, and a post-lock code amendment. This matters to my assessment because it means my objections are about inference and framing, not about whether the results are real; it also means the paper's negative findings (static lookup beats learned context, learned map inert on most external series) can be trusted and are themselves useful to anyone planning to learn parameters inside Equal Shares.

  2. The frozen-score replay is the right experimental idea. Holding score weights and feature definitions fixed while swapping the allocation rule is the only comparison in the paper that comes close to isolating the effect of the payment phase, and it is a methodology other learned-mechanism papers should adopt. Its execution here is compromised (the payment variant orders purchases by learned score rather than minimum price, histories diverge, the effect holds at one score only), but the design is a contribution independent of the results, and it is the main reason I do not view the submission as without value.

Critical weaknesses
  1. The one positive result is in-sample in everything but calendar time, and a trivial static baseline beats it. The "held-out" Warsaw evaluation is based on later editions of the same 18 districts the weights were fitted to, and two of the six endowment features carry deficit history from those same series. On that split, the learned map reaches 0.1736 against 0.1992 for Equal Shares, but a four-parameter age lookup reaches 0.1624, and the paper's own test finds no detectable difference between the two (p = 0.311). The lookup's multipliers put all the weight essentially on one bracket, every disjoint-series fit sits on the coefficient box, and widening the box improves held-out CSD, which together indicate the optimizer is approaching a limit policy (tilt toward the perpetually worst group, which the ethics statement says is 60+ men in 13 of 18 series) rather than learning anything from context. Both external studies fail, and on native ballots the map reproduces Equal Shares exactly on 36 of 58 series, so there is no evidence of transfer. This is consequential because it leaves the paper without a validated learned intervention; what remains is an audit.

  2. The theoretical framing does not deliver what the narrative needs. Proposition 1's construction is not reachable by any rule in the paper (every rule, including both greedy baselines, funds the dummy project on that instance), and the paper's own worst observed exclusion is 0.147, not 0.99. Proposition 2 bounds per-project support, not coverage: a small instance with endowments concentrated on 8% of voters, below the fitted map's own cap of 13.63, has payment fully retained and exclusion 0.92. So "payment retained protects coverage" is an empirical regularity of this corpus, not a property of the mechanism, and the paper's organizing dichotomy between payment-retaining and payment-discarding classes is not supported by its theory. Combined with the first weakness, the paper's title claim rests on one unregularised straw-man policy that the authors themselves decline to recommend.

Questions for authors

CLARIFICATION REQUESTS

  1. Does any rule you implemented, fitted, or hand-designed ever produce an allocation with CSD below 0.01 and exclusion above 0.5 on real or synthetic data? On the Proposition 1 instance, Algorithm 1, greedy-by-count, greedy-by-cost, and the static floor all fund the dummy or nothing. If no rule reaches the construction, in what operational sense can a CSD audit be fooled?

  2. How is the four-bracket age lookup parametrized, and what box was used? The top multiplier 20.0855 equals exp(3) to six figures. If it is a boundary value, is the lookup also a truncated limit policy, and what does the held-out CSD do as its box widens?

  3. What is the limit of the fitted endowment policy as the weight box grows? Table 20 shows the objective still improving at [-40, 40] with four of five fits on the boundary. Is the limit the age lookup, and if so, does the six-feature map add anything beyond the age tilt on any split?

  4. In the projected-support study, how many of the 32 series actuate in each city separately? With roughly six informative pairs per city, could the per-city bootstrap criterion have passed at any effect size short of unanimous sign?

  5. In the frozen-score replay, the payment kernel orders purchases by learned score rather than minimum price. What do the exclusion and welfare numbers look like with standard MES ordering, and with the realized score sequence also held fixed across kernels?

  6. Table 10: The four-feature direct score without cohort concentration reaches 0.1633, better than the reported five-feature model and the learned endowment map. Which model do you intend to present as the direct-score result, and why was the full model kept as the headline?

  7. Table 5, Lodz: greedy approvals-per-cost has CSD 0.0700 and welfare 464,787 against 0.1326 and 447,075 for the learned endowment map. Do you regard this as evidence against the endowment map on the only out-of-city replay with native ballots and temporal holdout?

SUGGESTIONS FOR FUTURE WORK

  1. A coverage-valid replacement for Proposition 2: a lower bound on the size of the union of approver sets of payment-phase winners under capped endowments. My counterexample suggests it must depend on ballot structure, not kappa alone.

  2. A preregistered comparison on a disjoint split in which the learned map must beat the static age lookup, with the box fixed and the optimum shown to be interior.

  3. Actuation-conditional external tests: select series on which the intervention changes the allocation before opening outcomes, or preregister an estimand conditional on actuation, so that a null is informative.

1 comment

Thank you for checking the statistics and identifying the gaps in inference. We have revised the paper around its supported contribution: an audit of learned rules. The results establish neither a benefit from context nor successful transfer.

1. Operational counterexample. We added a checked example using the existing direct-score selector. Among 100 voters in two equal groups, two voters approve a balanced project; the remaining voters approve their group’s alternative. All three projects cost the entire budget. Hand-designed weights v=(0,0,0,0,−1)v=(0,0,0,0,-1) select the balanced project, producing CSD = 0 and exclusion = 0.98. This establishes reachability within the implemented class, not a trained-policy or real-data finding. Appendix O provides the construction.

2–3. Lookup boundaries and contextual value. The lookup uses four log-multipliers in [−3,3][-3,3]; its oldest-bracket coefficient reaches the boundary, giving exp⁡(3)=20.0855\exp(3)=20.0855. We have not evaluated wider lookup boxes. Widening the contextual model’s box improves disjoint-series CSD from 0.1941 to 0.1886, but does not establish convergence. We have demonstrated neither a unique limiting policy nor an advantage over the lookup.

4. Per-city actuation. At seed 42, allocations change in 3/14 Katowice series and 9/18 Krakow series. With only three nonzero pairs, Katowice’s smallest possible two-sided exact sign-flip p-value is .25. Its ideal bootstrap distribution also places 3.42% probability at zero, preventing a negative upper 95% endpoint. We now explain this design limitation while retaining the failed decision.

5. Replay interpretation. The diagnostic uses learned-score ordering, and histories diverge across rules. We have not run the requested matched standard-MES and fixed-history comparisons. The revised paper calls this a fixed-parameter replay and does not attribute its effects to balance depletion alone.

6. Four-feature ablation. Removing concentration improves CSD from 0.1728 to 0.1633 and exclusion from 0.0447 to 0.0413, while welfare falls from 1,303,057 to 1,265,347. We now report this tradeoff prominently. The five-feature model remains the experimental reference, not the best observed model on CSD.

7. Łódź. Yes, greedy approvals-per-cost outperforms the endowment map on CSD, welfare and exclusion in that replay. Section 6 now states this adverse evidence directly.

8. Coverage bound. We added the elementary bound coverage≥s/(κB)\mathrm{coverage}\ge s/(\kappa B), where ss is payment-phase spending. At full spending and κ=13.63\kappa=13.63, this guarantees only 7.34% coverage, consistent with your example. Completion preserves this bound but does not permit replacing ss with final expenditure.

9–10. Further evaluation. We agree that contextual value requires a preregistered disjoint-series comparison against the lookup, and that actuation-conditional effects should be prespecified alongside unconditional effects. Our existing exploratory analyses do not substitute for either study.

Reviewer_87BYOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces fairness in participatory budgeting using cumulative share deficit (CSD), a measure of how spending is distributed across demographic groups defined by voters' age and sex. It shows that CSD can be zero even when 99% of voters receive no approved project. The authors train voting rules on real participatory-budgeting data, using demographic information, past funding deficits, and voting patterns to adjust voters' budgets or project scores, and compare the resulting allocations with existing rules. Although some learned methods improve CSD on Warsaw data, a simple four-parameter age lookup achieves a lower average CSD, but with worse voter coverage and welfare. The experiments on other cities fail to establish that the learned methods generalize or provide consistent improvements.

Critical strengths

I appreciate that the authors report negative experimental results and unsuccessful generalization tests rather than presenting only favorable outcomes.

Critical weaknesses

The paper is really hard to read because it is badly written (e.g. "The corrected post-hoc replay of the first locked study, learned priority inside Equal Shares, remains falsified.") and as a result I struggled to understand the motivation and find any meaningful contribution in there.

The CSD counterexample follow directly on the dependence on demographic proportions and is trivial, and the support bound is also trivial.

Questions for authors

I don't have any questions that could materially affect my assessment, I think the paper simply requires very substantial revision to be intelligible to a human.

1 comment

Thank you for recognizing the reporting of unsuccessful experiments. We agree that the original presentation made the purpose difficult to follow and placed too much weight on elementary propositions.

The sentence you quoted means that, after correcting the evaluation, learned project priorities still failed the study’s predefined success criteria in new cities. We have replaced that wording with a direct statement of the result. We also rewrote the Introduction around the audit’s purpose, explain PB and MES before discussing learned controls, define RES, and clarify the provenance of CSD.

Proposition 1 follows from the spending ledger, and the support floor follows from affordability. We agree on their mathematical simplicity. Their role is to delimit the empirical claims: CSD alone does not certify distinct-voter coverage, and a per-project support floor does not guarantee low exclusion. We do not present them as deep theoretical results or an impossibility theorem for all spending-based measures.

The contribution we ask you to assess is the empirical audit. Holding score parameters fixed while changing the allocation rule reveals substantial coverage differences without a detectable CSD improvement. The static age baseline then challenges whether the contextual model adds value. The external studies examine both performance and how often the learned component changes the funded set.

These findings place limits on the tested approach. The coverage improvement occurs at the unpenalized score and reverses under regularization; the external studies do not establish transfer. The revised Section 3 separates these empirical findings from the elementary theory. It also adds an executable, hand-designed example demonstrating that the implemented score class can reach zero CSD with 98% exclusion, and states the weaker coverage bound that payment accounting actually supports. Both are documented in Appendix O.

Reviewer_qGHzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

The paper studies participatory budgeting (PB) rules with learned components, focusing mainly on the Method of Equal Shares (MES) and its extensions. It introduces the cumulative share deficit (CSD), a group-level fairness measure that compares, over successive elections, the spending attributed to each demographic group (age by sex) with the group’s share of voters, and uses it as the training objective for learned interventions. Three classes of interventions are considered: learned voter endowments within MES, learned project priorities within MES, and learned project scores that replace MES with greedy selection.

The findings are mostly negative or inconclusive. First, the paper shows that CSD can be perfect while almost all voters receive none of the projects they approved (Proposition 1), i.e., CSD does not capture coverage. Second, in a replay that keeps a learned scoring function fixed and varies the allocation rule, MES-like payments substantially improve coverage for one learned policy, while CSD remains essentially unchanged. Third, on held-out elections in Warsaw, learned endowments reduce CSD modestly relative to MES, but a simple age-based rule achieves a lower CSD. Finally, two preregistered external studies in other Polish cities do not meet their decision criteria, and a post-hoc replay on native approval ballots finds no detectable difference from MES. The paper concludes that spending-based fairness measures should be reported together with voter coverage and with how often a learned rule actually changes the allocation.

Critical strengths
  1. The paper works on a timely, interesting, and in-scope general direction. Learned components are gaining ground in PB procedures, and it is important to understand how they should be evaluated and how they relate to the objectives they optimize.

  2. The experimental methodology is unusually transparent and careful, following practices from medicine and psychology. The authors preregistered their external experiments and decision thresholds, reported the outcomes honestly, and included checks that allow the process to be verified. This sets a good precedent for future research. In addition, the ethics discussion is thoughtful and acknowledges the limits of the age-by-sex grouping.

Critical weaknesses
  1. I believe the paper lacks a clear main message, and I see a fundamental issue in its central argument. The paper defines CSD and shows that there is an instance that performs poorly with respect to coverage, i.e., CSD is perfect but nearly all voters are excluded. The paper should explain clearly why this metric is useful, and whether it is used in practice, either in this exact form or in a slightly different one. Otherwise, the negative result shown in Proposition 1 is not very informative. The paper also highlights, based on this measure, that spending-based measures can perform poorly with respect to coverage. This is true, but it is not very informative on its own: there could be another spending-based measure that works well.

  2. The writing and organization of the paper are not yet at the level required for publication. This is, of course, connected to the above issue. In particular, the Introduction does not present the main purpose of the paper: the writing is esoteric, and it is hard for someone who is not an expert in the area to follow (e.g., one must be very familiar with participatory budgeting and the Method of Equal Shares to follow the discussion from the second paragraph on). Many notions appear without explanation, or their explanation is delayed, e.g., MES, which is explained much later, and RES, which appears only as an acronym until the appendix and, even there, is not adequately introduced, despite being an important part of the manuscript. Notation is also introduced late, e.g., the notation in Figure 1. The paper also moves abruptly between topics (e.g., in Section 3, from Proposition 2 to a retrained policy and then to the frozen-score replay). The main contribution is not presented clearly: the relevant paragraph presents multiple points together, without a clear connection between them. Finally, the role of CSD in the paper remains unclear and should be discussed in more depth.

Overall, I find the direction promising, but in its current form I believe the paper is not ready for publication. A clearly stated main message, a justification of CSD (or a coverage-aware alternative), and a thorough revision of the presentation would make it a much stronger submission.

Questions for authors
  1. Could you clarify the main goal of the paper? My understanding is that it analyzes CSD as the objective of learned interventions in participatory budgeting (mostly variants of the Method of Equal Shares), and that its results are largely negative. Is this the intended main message?

  2. CSD is a central notion in the paper, but I believe its discussion is rather brief and unfinished. The paper notes that CSD is not a standard PB axiom (line 114) and that it combines ideas from the literature, yet in many parts of the paper it is treated as an established measure. Could you please clarify (i) which parts of CSD are taken/inspired from prior work and which are your own design choices, (ii) whether CSD or a close variant is used in prior work or in real PB evaluations, and (iii) why it is a meaningful objective for PB?

1 comment

Thank you for recognizing the importance of evaluating learned PB components, the transparent reporting, and the ethics discussion. Your questions identify a problem in our presentation: we introduced the mechanisms before explaining what the audit was meant to establish.

Main goal. Yes, your reading is correct. The results are predominantly negative, and the contribution is an empirical audit of learned allocation rules. We ask what the objective measures, how outcomes change when the same score parameters are used with different allocation rules, and whether the learned intervention improves on simple alternatives or changes allocations elsewhere. The experiments establish neither a benefit from context nor successful transfer. They show why objective improvements alone are insufficient to support those claims.

Where CSD comes from. The exact CSD formula is our design. We adapt the equal-share attribution in Maly et al. (2023, Definition 1), introduced by Lackner, Maly, and Rey (2021), which divides a funded project’s cost equally among its approvers. For our demographic ledger, we restrict this split to approvers with recorded demographics and sum their credits by group. These accounting shares differ from the actual payments made under MES.

Our choices are the demographic partition, entitlement based on a group’s turnout share of actual spending, accumulation across elections, normalization by cumulative entitlement, and evaluation of the worst group before averaging across series. Prior work motivates these choices but does not establish this exact combination. We have no evidence that municipalities use CSD in this form, and the revised paper introduces it as a diagnostic designed for this study.

Why use it? CSD asks whether spending attributed to a group’s approvals keeps pace with its share of turnout over successive elections. This can reveal persistent expenditure imbalances. Using actual spending in the entitlement separates unequal distribution from underspending. However, shorter ballots attract less attributed spending, and the demographic partition affects whose deficit is largest. A lower CSD is therefore insufficient evidence that an allocation is fair or desirable.

What the counterexample establishes. We agree that another spending-based measure could account for coverage. Proposition 1 concerns CSD; it does not establish an impossibility for all spending-based measures. Its purpose is to explain why CSD cannot certify that many voters are served. The empirical relevance appears in Section 3.1: at the unpenalized score, exclusion falls from 0.1470 to 0.0530 with payment, and to 0.0427 with completion, without a statistically detectable CSD improvement. Reporting coverage exposes an outcome the objective does not directly measure.

Presentation changes. We rewrote the Introduction to explain PB and MES before the learned variants and state the audit’s purpose. Section 2 explains CSD’s provenance and defines Rollover Equal Shares and its parameter. Figure 1 defines its symbols in the caption. Section 3 separates the metric limitation, payment accounting, and fixed-parameter replay. It also includes an implemented synthetic example with zero CSD and 98% exclusion, explicitly labeled as hand-designed, together with a spending-conditional coverage bound. Appendix O gives the details and limitations.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.