- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies fairness in conformal prediction, focusing on differences that may remain within smaller subgroups even when coverage is maintained for larger groups. The authors first propose a way to examine how differences in group composition affect prediction set sizes. They then introduce Tilted Mondrian, which adjusts prediction scores to reduce disparities within smaller subgroups while maintaining group-level coverage guarantees.
Critical strengths
-
The paper addresses an important limitation of group-level fairness in conformal prediction. Even when larger groups receive similar coverage, smaller subgroups may still experience substantial differences. I think this is a meaningful problem, and the paper provides a useful way to study it. I also appreciate the theoretical analysis of how much these differences can be reduced while limiting coverage loss.
-
I appreciate that the proposed method aims to reduce disparities without requiring the model to be trained again. The authors provide theoretical support for maintaining group-level coverage and evaluate the method across different settings and repeated data splits. The experiments show promising improvements in several cases, which supports the practical value of the approach.
Critical weaknesses
-
I understand why the paper uses information from the model's predictions for standardization, and the authors provide some explanation for this choice. However, these features are also closely related to prediction set size. This makes it difficult for me to understand how much of the reported reduction in disagreement comes from differences in subgroup composition and how much comes from comparing samples with similar prediction confidence. I think the paper would benefit from showing whether the results remain similar when different features are used for standardization.
-
I found that the presentation of the experiments could be substantially improved. The main paper presents the primary results, but many useful comparisons and additional experiments are placed in the appendix. I had to spend more time going through different sections of the appendix to understand how consistent the improvements are and how the method compares with other approaches. I think some of these results should be included in the main paper, especially the comparisons and experiments that directly support the main claims. The presentation of the results could also be improved to make it easier to see where the method performs well and where it does not.
Questions for authors
1- Please explain how much the choice of prediction features affects the standardization results? Since these features are closely related to prediction set size, I would like to understand whether the reported reductions remain similar when different features are used.
2- Please provide a more direct comparison with subgroup-level calibration using the same test coverage gap measures? I appreciate the existing comparison of coverage loss and prediction set sizes, but I think comparing the main fairness measure under similar coverage loss limits would help show the additional benefit of the proposed method.
3- Please provide a more explicit summary of how consistently the method improves subgroup fairness across different experiments and data splits? I found the main results promising, but many important results are in the appendix, making it difficult to understand the overall performance of the method and how much improvement it provides.
2 comments
Thanks for your thoughtful answers. Thanks for adding bullet-points to your empirical part in the paper. I will keep my assessment score and will consider your answers when discussing with other reviewers.
Best,
We thank the reviewer for the thoughtful and constructive feedback and for recognizing the importance of the coarse-to-fine fairness problem, our theoretical analysis of mitigation limits, and the practical value of Tilted Mondrian (TM). We address the concerns below with new profile-sensitivity experiments, direct fairness–harm comparisons with Direct Cell Mondrian, and a summary of empirical evidence across datasets and resplits.
W1/Q1. Robustness to the Choice of Standardization Profile
The profile H defines the standardized population QH, allowing us to assess how population composition influences aggregate–stratum fairness comparisons. Prediction confidence, when included in H, represents one component of the composition being standardized.
New sensitivity experiments (see our response to Reviewer
6vX4, Q4). We evaluated three distinct profiles: confidence-only, predicted-label-only, and full profiles, holding the classifier, conformal thresholds, prediction sets, and evaluation splits fixed. See Section 6.1; Appendix D, Table 15. The results show that conflict-strength reductions persist across profiles constructed from different subsets of prediction-derived features. The choice of H determines the standardized population comparison.W2/Q3. Empirical Consistency and Presentation of Results
We thank the reviewer for this constructive suggestion. To make our empirical findings easier to assess, we summarize the key conclusions from the main paper, appendices, and additional experiments; see revised Section 6.2:
Coverage disparity mitigation (Table 2; Appendix F, Table 28). TM achieves positive mean test worst-gap reductions in 10/16 dataset–score configurations, with improvements in 230/320 resplits.
Consistency across datasets (Appendix E.1, Table 24; Appendix F, Table 28). Test worst-gap improvements occur in 71/80 resplits for Bias in Bios, 80/80 for Folktables occupation, 57/80 for Folktables income, and 22/80 for ATUS. Improvements are especially consistent in the first two datasets, while Folktables income exhibits score-dependent gains and ATUS presents smaller initial disparities.
Direct Cell Mondrian comparison (Appendix E.1, Tables 21–23;
c4zs, W3). TM achieves substantially lower mean maximum baseline-relative coverage harm (0.82 vs. 5.55 pp) and lower set-size dispersion in 15/16 configurations.Fairness–harm trade-off (Appendix E.1, Table 24). Under the same test criterion of positive worst-gap reduction and baseline-relative coverage harm ≤ 1 pp, TM satisfies both conditions in 143/320 resplits versus 18/320 for Direct Cell Mondrian.
Propensity standardization (Table 1; Appendix D, Table 15; Appendix F, Table 27;
6vX4, Q4). Conflict strength decreases in 899/900 evaluations using the original standardization profile across repeated data splits. The sensitivity analysis further characterizes how reversal counts vary with the choice of H.Additional fairness interventions (Table 3; Appendix E.2; Appendix G). Set-size-targeted TM reduces test worst set-size gaps in 234/320 resplits, with accompanying coverage–set-size trade-offs. Boosted TM further illustrates the flexibility of the framework through covariate-adaptive score interventions.
Q2. Fine-grained Fairness and Coverage-Harm Trade-offs
Direct Cell Mondrian calibrates explicitly within (G,Z) cells, whereas TM retains G-level calibration and introduces finer-stratum interventions. Their raw coverage gaps therefore reflect different calibration objectives. We compare their results using the same test coverage-gap measure alongside baseline-relative coverage harm.
1. Direct comparison across datasets and scores.
We added direct comparisons in Section 6.2 (Table 4) and Appendix E.1 (Tables 23–24), illustrating their different fairness–harm trade-offs.
2. The additional benefit of Tilted Mondrian.
Our contribution is to separate the granularity of fairness intervention from that of conformal calibration. TM provides an additional finer-stratum intervention layer while retaining the original Group-Mondrian calibration and its finite-sample groupwise validity guarantee. The null intervention θ=0 remains feasible on the tilt split, allowing the procedure to preserve the baseline when further adjustment offers limited benefit.
Thus, Direct Cell Mondrian strengthens cellwise coverage guarantees, while TM provides an approach that seeks to reduce finer-stratum disparities subject to empirical coverage-loss constraints. These comparisons make the practical value and trade-offs of the proposed intervention more explicit (see also our response to
c4zs, W3).