- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper asks what a learned component does inside a participatory budgeting rule, and what the fairness metric used to judge it actually sees. The metric, cumulative share deficit (CSD), tracks, over repeated elections, how far the spending attributable to a demographic group's approvals falls short of its turnout share. A short decomposition and a toy instance show that the ledger is blind to coverage: CSD can be exactly zero, even as almost no voter receives an approved project.
Three learned interventions are compared. Two keep the Equal Shares payment phase and learn either per-group voter endowments (six features) or the purchase order; the third drops payment and fills the budget greedily from a five-feature learned score. Weights are fitted by CMA-ES on the Warsaw district series through 2022 and tested on the 2023 to 2025 editions of the same districts. A support-floor proposition shows payment-phase purchases need approval share at least cost over kappa times budget, which direct scoring does not.
The findings are mostly cautionary. Learned endowments reduce worst-group CSD relative to plain Equal Shares, but a four-number age lookup fitted afterward performs better. Freezing the unpenalized direct score and reinstating the payment-cuts exclusion raise welfare with no detectable CSD change, though the effect reverses at welfare-regularised scores. Two locked external studies in Katowice and Krakow fail to meet their preregistration criteria, and a post hoc reanalysis of native ballots in Gdynia and Lodz shows no detectable difference. The authors report all of this openly, including a counting bug and a post-lock code amendment, and conclude that coverage and allocation changes should be reported alongside spending-share fairness.
Critical strengths
-
The reporting is unusually honest and complete, and the numbers check out. I recomputed every statistic reachable from the per-series tables (means, exact sign-flip p-values, bootstrap intervals, welfare ratios, gap closures) and all of them reproduce. The authors report the baseline that beats their method, the two locked studies that failed, a counting bug their own containment check caught, and a post-lock code amendment. This matters to my assessment because it means my objections are about inference and framing, not about whether the results are real; it also means the paper's negative findings (static lookup beats learned context, learned map inert on most external series) can be trusted and are themselves useful to anyone planning to learn parameters inside Equal Shares.
-
The frozen-score replay is the right experimental idea. Holding score weights and feature definitions fixed while swapping the allocation rule is the only comparison in the paper that comes close to isolating the effect of the payment phase, and it is a methodology other learned-mechanism papers should adopt. Its execution here is compromised (the payment variant orders purchases by learned score rather than minimum price, histories diverge, the effect holds at one score only), but the design is a contribution independent of the results, and it is the main reason I do not view the submission as without value.
Critical weaknesses
-
The one positive result is in-sample in everything but calendar time, and a trivial static baseline beats it. The "held-out" Warsaw evaluation is based on later editions of the same 18 districts the weights were fitted to, and two of the six endowment features carry deficit history from those same series. On that split, the learned map reaches 0.1736 against 0.1992 for Equal Shares, but a four-parameter age lookup reaches 0.1624, and the paper's own test finds no detectable difference between the two (p = 0.311). The lookup's multipliers put all the weight essentially on one bracket, every disjoint-series fit sits on the coefficient box, and widening the box improves held-out CSD, which together indicate the optimizer is approaching a limit policy (tilt toward the perpetually worst group, which the ethics statement says is 60+ men in 13 of 18 series) rather than learning anything from context. Both external studies fail, and on native ballots the map reproduces Equal Shares exactly on 36 of 58 series, so there is no evidence of transfer. This is consequential because it leaves the paper without a validated learned intervention; what remains is an audit.
-
The theoretical framing does not deliver what the narrative needs. Proposition 1's construction is not reachable by any rule in the paper (every rule, including both greedy baselines, funds the dummy project on that instance), and the paper's own worst observed exclusion is 0.147, not 0.99. Proposition 2 bounds per-project support, not coverage: a small instance with endowments concentrated on 8% of voters, below the fitted map's own cap of 13.63, has payment fully retained and exclusion 0.92. So "payment retained protects coverage" is an empirical regularity of this corpus, not a property of the mechanism, and the paper's organizing dichotomy between payment-retaining and payment-discarding classes is not supported by its theory. Combined with the first weakness, the paper's title claim rests on one unregularised straw-man policy that the authors themselves decline to recommend.
Questions for authors
CLARIFICATION REQUESTS
-
Does any rule you implemented, fitted, or hand-designed ever produce an allocation with CSD below 0.01 and exclusion above 0.5 on real or synthetic data? On the Proposition 1 instance, Algorithm 1, greedy-by-count, greedy-by-cost, and the static floor all fund the dummy or nothing. If no rule reaches the construction, in what operational sense can a CSD audit be fooled?
-
How is the four-bracket age lookup parametrized, and what box was used? The top multiplier 20.0855 equals exp(3) to six figures. If it is a boundary value, is the lookup also a truncated limit policy, and what does the held-out CSD do as its box widens?
-
What is the limit of the fitted endowment policy as the weight box grows? Table 20 shows the objective still improving at [-40, 40] with four of five fits on the boundary. Is the limit the age lookup, and if so, does the six-feature map add anything beyond the age tilt on any split?
-
In the projected-support study, how many of the 32 series actuate in each city separately? With roughly six informative pairs per city, could the per-city bootstrap criterion have passed at any effect size short of unanimous sign?
-
In the frozen-score replay, the payment kernel orders purchases by learned score rather than minimum price. What do the exclusion and welfare numbers look like with standard MES ordering, and with the realized score sequence also held fixed across kernels?
-
Table 10: The four-feature direct score without cohort concentration reaches 0.1633, better than the reported five-feature model and the learned endowment map. Which model do you intend to present as the direct-score result, and why was the full model kept as the headline?
-
Table 5, Lodz: greedy approvals-per-cost has CSD 0.0700 and welfare 464,787 against 0.1326 and 447,075 for the learned endowment map. Do you regard this as evidence against the endowment map on the only out-of-city replay with native ballots and temporal holdout?
SUGGESTIONS FOR FUTURE WORK
-
A coverage-valid replacement for Proposition 2: a lower bound on the size of the union of approver sets of payment-phase winners under capped endowments. My counterexample suggests it must depend on ballot structure, not kappa alone.
-
A preregistered comparison on a disjoint split in which the learned map must beat the static age lookup, with the box fixed and the optimum shown to be interior.
-
Actuation-conditional external tests: select series on which the intervention changes the allocation before opening outcomes, or preregister an estimand conditional on actuation, so that a null is informative.
Thank you for checking the statistics and identifying the gaps in inference. We have revised the paper around its supported contribution: an audit of learned rules. The results establish neither a benefit from context nor successful transfer.
1. Operational counterexample. We added a checked example using the existing direct-score selector. Among 100 voters in two equal groups, two voters approve a balanced project; the remaining voters approve their group’s alternative. All three projects cost the entire budget. Hand-designed weights v=(0,0,0,0,−1) select the balanced project, producing CSD = 0 and exclusion = 0.98. This establishes reachability within the implemented class, not a trained-policy or real-data finding. Appendix O provides the construction.
2–3. Lookup boundaries and contextual value. The lookup uses four log-multipliers in [−3,3]; its oldest-bracket coefficient reaches the boundary, giving exp(3)=20.0855. We have not evaluated wider lookup boxes. Widening the contextual model’s box improves disjoint-series CSD from 0.1941 to 0.1886, but does not establish convergence. We have demonstrated neither a unique limiting policy nor an advantage over the lookup.
4. Per-city actuation. At seed 42, allocations change in 3/14 Katowice series and 9/18 Krakow series. With only three nonzero pairs, Katowice’s smallest possible two-sided exact sign-flip p-value is .25. Its ideal bootstrap distribution also places 3.42% probability at zero, preventing a negative upper 95% endpoint. We now explain this design limitation while retaining the failed decision.
5. Replay interpretation. The diagnostic uses learned-score ordering, and histories diverge across rules. We have not run the requested matched standard-MES and fixed-history comparisons. The revised paper calls this a fixed-parameter replay and does not attribute its effects to balance depletion alone.
6. Four-feature ablation. Removing concentration improves CSD from 0.1728 to 0.1633 and exclusion from 0.0447 to 0.0413, while welfare falls from 1,303,057 to 1,265,347. We now report this tradeoff prominently. The five-feature model remains the experimental reference, not the best observed model on CSD.
7. Łódź. Yes, greedy approvals-per-cost outperforms the endowment map on CSD, welfare and exclusion in that replay. Section 6 now states this adverse evidence directly.
8. Coverage bound. We added the elementary bound coverage≥s/(κB), where s is payment-phase spending. At full spending and κ=13.63, this guarantees only 7.34% coverage, consistent with your example. Completion preserves this bound but does not permit replacing s with final expenditure.
9–10. Further evaluation. We agree that contextual value requires a preregistered disjoint-series comparison against the lookup, and that actuation-conditional effects should be prespecified alongside unconditional effects. Our existing exploratory analyses do not substitute for either study.