- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper investigates the economic costs associated with predictive multiplicity, where multiple classifiers achieve the same or similar population welfare but produce correct predictions for different individuals. They consider a model-selection setting in which each individual has a private value for receiving a correct prediction. The platform selects a classifier using a Gibbs distribution based on reported values and charges critical-value payments to ensure truthful reporting. The authors establish a connection between truthful payments and predictive multiplicity and provide statistical bounds on total payments.
Critical strengths
- The paper considers an interesting problem: classifiers with identical population welfare can affect individuals differently, creating incentives for strategic reporting.
- This paper provides strong theoretical analysis with several nontrivial results. For example, the KL-divergence characterization connects individual payments to the sensitivity of Gibbs selection, and Theorem 2 links payments to disagreement among near-optimal classifiers. Also, Theorem 4 establishes matching upper and lower bounds at all temperatures for a particular population with conflicting beneficiaries.
Critical weaknesses
- The assumptions for theoretical guarantees are restrictive. For example, the beneficiary-consensus condition in Theorem 3 requires all population-optimal classifiers to agree on correctness for individuals with positive values, almost surely.
Questions for authors
See weaknesses.
Response to Reviewer K9f6
Thank you for your supportive evaluation and for highlighting our KL-divergence payment characterization, Theorem 2's localization to near-optimal classifiers, and Theorem 4's matching upper and lower bounds across temperatures.
[W1] Role of beneficiary consensus, distinct optimal classifiers, and general disagreement bounds
Theorem 3's beneficiary consensus condition concerns only binary correctness of population-optimal classifiers on positive-value cases, requiring neither a unique optimal classifier nor identical predictions. In the three-class example in Appendix D.4, two distinct imperfect population-optimal classifiers are correct on the same three quarters of the positive-value cases and predict different wrong labels on the rest, while optimal models may also differ arbitrarily whenever vi=0. Under this condition, Theorem 3 gives vanishing total expected payment on every predetermined positive temperature schedule Tn=o(n).
Universal truthfulness, Theorem 1's individual joint accuracy-payment laws, and our blockwise upper-tail bounds require no consensus. In Theorem 2, any participant whose correctness varies across empirically near-optimal classifiers has expected bill at most bi2/(8T), while the mass outside those near-optima controls other bills. Theorem 4 provides a sharp example under disagreement, exhibiting two population-optimal classifiers benefiting different people in a fair-label population and proving expected total payments of Θ(minVn,,nV2/T) at every positive temperature and even sample size. Whereas recent work such as Haghighat, Anahideh, and Rudin (AAAI 2026) and Hsu, Chen, Zhong, and Semenova (ICML 2026) studies choice among comparably accurate models, we isolate the price of truthful choice under predictive multiplicity, arising in this example with no loss in population welfare.
To address partial agreement, Appendix C.3 Equation (C.7) introduces D2, the population expectation of v2 times the indicator of optimal-model disagreement on correctness. For M>0 nonoptimal classifiers and n≥4V/Δ, this finite-sample upper bound on total expected payment is nD2/(8T) plus two remainders controlled by the welfare gap Δ, whereas if M=0, only nD2/(8T) remains for every n.