acceptodds
Under review as a conference paper at ICLR 2027

Budgeted Regime Calibration: Allocating Calibration Resolution Across Predictive Regimes

Abstract

Conformal prediction provides marginal coverage guarantees, yet reliability can vary substantially across heterogeneous regions of the input space. Local refinement can reduce this variation, but calibration resolution is finite: assigning more calibration cells to one region leaves fewer cells available elsewhere, while uniform refinement implicitly assumes that all regions benefit equally. We introduce Budgeted Regime Calibration (BRC), which formulates calibration resolution as a fixed-budget allocation problem over preformed regimes. BRC estimates regime-specific responses to candidate refinement levels using out-of-fold calibration on a dedicated STRUCTURE split and allocates the shared resolution budget according to those responses before independent final calibration. BRC selects the allocation that minimizes the largest absolute empirical coverage deviation from the target over fixed STRUCTURE audit groups, thereby controlling an upper bound on worst-group coverage shortfall.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Conformal Prediction, Uncertainty Quantification, Localized Conformal Prediction, Conditional Coverage, Worst-Group Coverage, Calibration, Adaptive Calibration, Calibration Resolution, Budgeted Allocation, Group-Conditional Prediction

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_KtJ7OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies the problem of allocating a given number of calibration cells to diverse, already identified predictive regimes. Proposed Budgeted Regime Calibration (BRC) minimizes worst-case empirical deviations from the target coverage level by estimating regime-specific calibration-error responses using OOF evaluation.

Critical strengths
  1. The framing of the optimization problem for calibration cell budget allocation is straightforward, while the conformal guarantee follows immediately. Hence, it may be practically useful.
Critical weaknesses
  1. In my opinion, the technical novelty of the paper is very limited. The strategy proposed for the budget allocation is an empirical optimization problem without any fundamental technical or methodological contribution. The calibration step, once cells are allocated, is the usual conformal guarantee.

  2. The proposed BRC is not supported by any consequential theoretical guarantee. The main result (Proposition 1) is a rearrangement statement. These results are about empirical minimization only, not at the population level. Empirical minimization is not supported by its uncertainty analysis.

Questions for authors
  1. Could authors identify distinct technical/ methodological novelty in the paper apart from solving the budget allocation problem by known methods?

  2. Can you comment on the data requirements utilized in the overall BRC strategy? How practical is it? Do CAL cell requirements meet in general settings? Isn't it a cyclic problem we might face for regimes with low resolution?

3 comments
AuthorsOpenReview ↗

Clarifying Methodological Novelty, Theoretical Guarantees, and Calibration Feasibility

Response to KtJ7. We distinguish original evidence from rebuttal additions.

Q1. What is novel beyond empirical optimization?

Distinct methodological decision (original Secs. 2–3; App. A.5–A.7). Calibration resolution is a statistical resource: finer cells may improve localization but reduce support. Kiyani et al. [1] study learned partitions; Toccaceli [2] examines bin-count selection; Boström and Johansson [3] discuss limited support in Mondrian categories. Xu et al. [4] allocate confidence levels, while Das et al. [5] allocate training versus calibration data. These differ from allocating a fixed cell budget across preformed regimes, whose support and responses vary. BRC estimates regime-specific responses and jointly allocates cells; neither standard solvers nor conformal quantiles are novel.

Mechanism already presented (original Sec. 3; App. C). Five-fold STRUCTURE OOF estimates resolution-dependent error Ek(b)E_k(b) using identical frozen audit groups across candidates. Uncertainty orders refinement within each regime; the OOF response determines its cell allocation. A joint minimax criterion selects allocation before independent CAL fitting.

Evidence already in the submission (Sec. 4; Apps. F, G, H, I). A CIFAR-100 ViT regime has Ek(1)=0.0247E_k(1)=0.0247, Ek(2)=0.0280E_k(2)=0.0280, Ek(3)=0.0232E_k(3)=0.0232: increasing resolution is not monotonically beneficial. At 2K2K, BRC yields higher mean TEST WPC in 9/10 matched classification settings and smaller mean sets in 8/10, with all comparison factors except allocation fixed. Appendix F compares Random and GlobalU; App. I ablates the objective (Folktables 2K2K WPC: full 0.6927 vs. weighted-only 0.5201); App. H.3 tests non-KMeans regimes. These controls and ablations were already reported in the original submission, not added after review. Larger-budget results are mixed.

New uncertainty analysis (revised App. C.7). Proposition 1 concerns empirical optimization; standard conformal validity does not prove superiority. Under frozen label-independent geometry/folds and conditional independence, we now establish, with probability ≥1−δ\ge1-\delta,

\eta=\max_g\sum_{f:n_{gf}>0}\frac{n_{gf}}{n_g}\sqrt{\frac{\log(2M/\delta)}{2n_{gf}}}.$$ For fitted BRC and Uniform evaluated on independent validation data, we also derive $$\Delta\mathrm{WPC}_{\mathrm{pop}}\ge\widehat\gamma-\widehat\tau-\widehat h-2\epsilon.$$ The first bounds OOF selection uncertainty, not final-CAL WPC; it is conservative and label stratification requires justification. The second is a conditional certificate without demonstrated positivity. **New five-seed FOLD/FROZEN/REFIT tests (App. L.1)** assess transfer. At $2K$, REFIT mean TEST gains were **+0.1592** (Adult), **+0.1290** (Folktables), **+0.0040** (CIFAR-100), and **+0.0160** (ImageNet, separate pipeline); larger-budget effects can be nonpositive. **Q2. What data are required, and does low support cause a cycle?** **Original data requirements (Apps. C, D.7, H.6, H.8, K).** Separate STRUCTURE and CAL cost data. The submitted protocol specified minimum STRUCTURE regime sizes of **100** (classification) and **50** (regression), OOF support **80**, CAL leaf support **30** (tabular), **50** (CIFAR-100), or **20** (ImageNet/regression), and parent support **30**. Unsupported leaves follow a fixed *cell → regime → global* fallback, **not cyclic reallocation**. Fallback reduces localization. Original App. H.6 reported zero fallback in representative CIFAR-100/Folktables sweeps; App. H.8 found positive mean $2K$ WPC differences at **25%, 50%, 100%** CAL fractions; App. K reported small allocation overhead. These tests were already in the submitted paper; feasibility is not universal. **New direct data-cost test (App. L.2).** Uniform-FULL uses **STRUCTURE plus CAL** to fit thresholds; BRC uses STRUCTURE for selection and CAL for calibration. With independently TRAIN-frozen common geometry, BRC had positive five-seed mean $2K$ WPC differences in **all eight** dataset/score comparisons, including CIFAR-100 ViT APS **$+0.0416\pm0.0044$** and Folktables TabNet LAC **$+0.0854\pm0.0080$**. This is evidence under the reference-frozen protocol, not a universal guarantee. **Revised presentation.** Revised Secs. 2–3 clarify notation and audit groups; remaining typos will be corrected. **References.** [1] Kiyani et al., *Conformal Prediction with Learned Features*, ICML 2024. [2] Toccaceli, *CRPS-Optimal Binning for Univariate Conformal Regression*, COPA 2026. [3] Boström and Johansson, *Mondrian Conformal Regressors with Overlapping Categories*, COPA 2026. [4] Xu et al., *Aggregating Conformal Prediction Sets via Alpha-Allocation*, Biometrika 2026. [5] Das et al., *On Optimal Data Splitting for Split Conformal Prediction*, arXiv:2606.31600.

I thank the authors for the revisions and their response. It has improved the clarity and presentation of the paper. Empirical evidence is useful. However my main concerns remain. Firstly, authors identify their optimization as a distinct problem form the existing works. However, it is not clear why the proposed algorithm demonstrates a substantial new methodological insight. Authors also affirm that neither standard solvers nor conformal quantiles are novel. New theoretical analysis is useful but it neither establishes strict improvement over uniform allocation nor OOF bound implies final calibration. I appreciate empirical studies which show favourable performance in limited budgets. My concern about the CAL cell requirements remain unresolved. I understand that whenever there are insufficient number of examples in CAL within a particular regime, there is a fallback strategy. But, this is exactly the regime where we need method to work. In my opinion, the method is off course correct and might be useful in practice , however the concern about limited technical/methodological novelty for ICLR still remains. Hence, I maintain my initial assessment of the paper.

AuthorsOpenReview ↗

Clarifying Methodological Novelty, Theoretical Guarantees, and Calibration Feasibility

Response to KtJ7: Methodological Contribution, Theory, and CAL Feasibility

We thank the reviewer. A distinct optimization formulation alone does not establish methodological significance. BRC's contribution is estimating the reliability benefit of additional calibration resolution, rather than inferring it from predictive uncertainty or sample support.

1. Methodological insight. Uncertainty suggests where refinement may occur; support determines whether it is feasible. Neither measures how local coverage error changes when cells are added. Finer partitions may reduce heterogeneity but weaken quantile estimation, producing nonmonotone responses.

BRC estimates this response for each frozen regime kk and candidate cell count bb using STRUCTURE-side out-of-fold (OOF) calibration. Crucially, all candidate partitions are evaluated on identical frozen audit groups, so their coverage deviations Ek(b)E_k(b) refer to the same reliability target. BRC then chooses B\mathbf B under ∑kBk=C\sum_k B_k=C, lexicographically minimizing maximum local deviation before aggregate deviation. The allocation is frozen before independent CAL fitting. The contribution is comparable response estimation and coupled worst-group allocation, not the solver or quantile.

This differs from learned conformal partitions [1], fixed-count binning [2], and miscoverage-level allocation [3]. BRC allocates cell counts among preformed regimes according to their measured coverage responses, separating refinement placement from refinement utility.

2. New evidence isolating this mechanism (Appendix L.5). We compared BRC with prespecified mean-uncertainty, uncertainty-spread, support, Uniform, and Random allocators across five seeds. Within each seed, predictors, regimes, boundaries, CAL/TEST data, audit groups, and budgets were matched.

At C=2KC=2K on Folktables TabNet, BRC improved paired TEST worst-group coverage (WPC) over mean-uncertainty allocation by +0.2064±0.0511+0.2064\pm0.0511 and over Uniform by +0.1240±0.0459+0.1240\pm0.0459, positive in all five seeds. Against Uniform, marginal coverage was equal at 0.89910.8991 and mean set size was smaller (1.26431.2643 vs. 1.27361.2736). On CIFAR-100 ViT, BRC improved WPC over mean-uncertainty allocation by +0.1414±0.0401+0.1414\pm0.0401, positive across five seeds. This is a test of the information used for allocation, not simply another benchmark: uncertainty magnitude did not substitute for estimated calibration-refinement responses in these comparisons. Appendix L.5 discloses all budgets and its independent predictor-cache protocol.

Earlier mechanism evidence supports this distinction. A CIFAR-100 regime exhibited Ek(1)=0.0247E_k(1)=0.0247, Ek(2)=0.0280E_k(2)=0.0280, Ek(3)=0.0232E_k(3)=0.0232 (App. G), demonstrating a nonmonotone estimated refinement response. On Folktables at 3K3K, the full lexicographic objective achieved WPC 0.73390.7339 versus 0.70320.7032 for maximum-deviation-only and 0.61070.6107 for weighted-only selection (App. I). Both the response signal and allocation criterion therefore matter.

3. Theoretical contribution. Proposition 1 connects the STRUCTURE-side allocation objective to empirical worst-group shortfall and includes matched Uniform as feasible. The conditional 2η2\eta result bounds OOF selection uncertainty under explicit assumptions. Final CAL conformal validity is a separate rank-based property; the OOF bound is not claimed to establish strict final-CAL population superiority. Independent TEST and REFIT analyses evaluate the selected allocations beyond their selection data. The selection analysis and calibration guarantee thus have distinct scopes.

4. CAL support. A constrained total cell count differs from insufficient support in an individual selected cell. The total budget can reflect CAL availability; BRC allocates that budget rather than estimating local thresholds from arbitrarily few observations. The prespecified cell-to-regime-to-global hierarchy is a safeguard, not cyclic reallocation or part of the allocation objective. Appendix H.6 reports zero fallback and substantial local CAL support in representative settings, establishing that their observed gains did not arise from fallback. Appendix H.8 reports positive mean matched 2K2K WPC differences at 25%25\%, 50%50\%, and 100%100\% CAL availability in both settings.

In summary, BRC estimates the reliability consequence of refinement on a common evaluation target and uses it to allocate a shared resolution resource. Appendix L.5 directly examines the significance of this decision signal relative to uncertainty and support proxies. We ask the reviewer to reconsider its methodological significance.

References: [1] Kiyani et al., Conformal Prediction with Learned Features, ICML 2024. [2] Toccaceli, CRPS-Optimal Binning for Univariate Conformal Regression, COPA 2026. [3] Xu et al., Aggregating Conformal Prediction Sets via Alpha-Allocation, Biometrika 2026.

Reviewer_SDqWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies adaptive partition refinement for improving local coverage reliability in conformal prediction under a fixed calibration-cell budget. It proposes Budgeted Regime Calibration, which estimates regime-specific coverage deviations at different refinement levels using out-of-fold calibration and adaptively allocates calibration cells across pre-constructed groups. Experiments on tabular and image classification show that BRC improves worst-group coverage compared to uniform refinement, particularly under tight resolution budgets.

Critical strengths
  1. The paper explicitly formulates the allocation of a fixed budget of calibration cells across predefined predictive regimes as a distinct problem in conformal prediction. As far as I know, this specific problem has been studied for the first time.

  2. The paper conducts comprehensive experiments across multiple tabular and image classification datasets with different backbone models. The results demonstrate consistent improvements in WPC over uniform refinement under tight calibration budgets, supporting the effectiveness of the proposed approach.

Critical weaknesses
  1. The practical significance of this problem needs further clarification. For example, in what scenarios would we need to allocate a fixed number of calibration cells across predefined regimes? As far as I know, the computational of conformal calibration are relatively low, while introducing an additional STRUCTURE split increases the amount of data required.

  2. Although the experimental results show that BRC improves WPC, the reduction in prediction-set size is relatively small. Is it possible that the improvement in WPC comes at the cost of reducing coverage in other groups? More group-wise coverage analysis would be helpful.

Questions for authors
  1. Further clarify the practical significance of the proposed task. In what real-world scenarios would a fixed budget of calibration cells across predefined regimes be necessary?
  2. Provide more analysis of the WPC improvements. Why does the improvement in WPC not lead to a substantial reduction in prediction-set size? Does it come at the cost of reduced coverage in other groups?
  3. Could the authors evaluate BRC with more nonconformity scores, such as RAPS[1] and SAPS[2], to further demonstrate its generalizability?

[1] Angelopoulos A, Bates S, Malik J, et al. Uncertainty sets for image classifiers using conformal prediction[J]. arXiv preprint arXiv:2009.14193, 2020. [2] Huang J, Xi H, Zhang L, et al. Conformal prediction for deep classifier via label ranking[J]. arXiv preprint arXiv:2310.06430, 2023.

5 comments
AuthorsOpenReview ↗

Clarifying Practical Significance, Calibration Data Efficiency, and Robustness Across Conformity Scores

We thank the reviewer for recognizing the fixed-budget problem and for the constructive questions. We distinguish original evidence from targeted additions in revised Appendix L.

1. Practical significance and the additional STRUCTURE split

Already addressed in the original submission. The finite resource in BRC is statistically supported local calibration resolution, not the computation required to obtain conformal quantiles. As formalized in Secs. 2–3, subdividing a regime increases the number of local thresholds but reduces the calibration observations available per cell. For example, a predictor serving heterogeneous operating regimes may have few labels for local calibration. Given a support-feasible budget ∑kBk=C\sum_k B_k=C, Uniform allocates equally; BRC uses regime-specific STRUCTURE-side coverage responses. The budget is a design constraint, not a universal deployment rule. Original Appendix D specifies support and fallback, Appendix H.8 evaluates 25%/50%/100% CAL availability, and Appendix K reports runtime. At 3K3K, the BRC-specific allocation stage added only 0.033 s (CIFAR-100 ViT) and 0.079 s (Folktables TabNet) over matched Uniform. The motivation is statistical support, not computational cost.

New evidence in revised Appendix L.2. We directly tested the data cost of STRUCTURE against Uniform-CAL (same final CAL pool) and Uniform-FULL (STRUCTURE+CAL used for final thresholds). All arms shared the predictor, score, budget, audit groups, TEST, and TRAIN-reference-frozen geometry. BRC used STRUCTURE to select allocations and independent CAL for thresholds; Uniform used equal allocations. At 2K2K, BRC minus Uniform-FULL mean Δ\DeltaWPC on CIFAR-100/Folktables was +0.0416/+0.0392 (APS), +0.0908/+0.0854 (LAC), +0.0268/+0.0256 (RAPS), and +0.0758/+0.0674 (SAPS). All eight five-seed means favored BRC despite Uniform-FULL receiving a larger final calibration pool. Uniform-FULL is an empirical comparator; BRC validity relies on independent CAL.

2. WPC improvement, prediction-set size, and coverage redistribution

Already addressed in the original submission. Sec. 4.2 and Table 1 jointly report MC, WPC, and set size under matched predictors, regimes, scores, data splits, and budgets. At 2K2K, BRC had higher WPC in 9/10 classification settings and smaller mean sets in 8/10. For CIFAR-100 ViT, WPC increased from 0.8016 to 0.8621, while mean set size decreased from 2.5615 to 2.5352. WPC is a minimum over audit groups; mean set size averages all TEST inputs. BRC selects resolution to reduce estimated worst-group coverage deviation, not to minimize average set cardinality. Thus, improving a poorly covered group need not greatly change average size.

New evidence in revised Appendix L.3. We evaluated each supported TEST audit group's paired coverage difference (BRC minus Uniform). "Groups losing coverage" counts negative differences; "worsened undercoverage" counts groups with an increased positive shortfall below 1−α1-\alpha. On CIFAR-100 ViT at 2K2K, 13% of groups lost some coverage, whereas 3% showed worsened undercoverage, under each of APS, LAC, RAPS, and SAPS. These are five-seed mean group-count fractions; affected groups need not coincide across scores. At 4K4K, the loss fraction rose to 23% to 27%, consistent with the weaker matched allocation advantage at larger budgets already shown in Sec. 4.3. We report redistribution without claiming that every group improves.

3. Robustness to RAPS and SAPS

Already addressed in the original submission. Sec. 3 defines a score-modular allocation problem; Appendix H.1 evaluated both APS and LAC on CIFAR-100 ViT, with original 2K2K WPC differences of +0.0606 and +0.1415, respectively. RAPS/SAPS were not included in that original analysis.

New evidence in revised Appendix L.4. As suggested, we evaluated RAPS and SAPS on CIFAR-100 ViT and Folktables TabNet at 2K2K, 3K3K, and 4K4K over five seeds, preserving the original score-construction/calibration protocol. RAPS used λ=0.01\lambda=0.01, kreg=5k_{\mathrm{reg}}=5; SAPS used λ=0.01\lambda=0.01. Each BRC versus Uniform comparison held the score configuration and final CAL observations fixed. At 2K2K, Δ\DeltaWPC (mean ±\pm SD) was +0.0376 ±\pm 0.0030 (RAPS) and +0.0854 ±\pm 0.0076 (SAPS) on CIFAR-100; +0.0350 ±\pm 0.0031 and +0.0772 ±\pm 0.0068 on Folktables. Mean gains remained positive at all budgets, largest at 2K2K. CIFAR-100 uses matched-score evaluations and Folktables uses the Uniform-CAL arm, providing complementary evidence.

Summary.

In summary, the original submission documents the fixed-budget rationale, matched coverage/size results, support, CAL sensitivity, runtime, and APS/LAC. Revised Appendix L adds data efficiency (L.2), groupwise redistribution (L.3), and two-dataset RAPS/SAPS tests (L.4).

Reviewer_SDqWOpenReview ↗

Main confusion

Thank you for the additional experiments. I have no further concerns about the empirical performance of the proposed method.

However, I'm still a bit confused about the motivation for fixing the total number of calibration cells. I understand that having more cells means fewer calibration samples per cell, which may lead to higher variance in the estimated conformal quantiles. But wouldn't this suggest that the number of cells should be adjusted based on the amount of available calibration data?

For example, if there are enough calibration samples to support finer partitions, it seems natural to allow more cells, while a smaller calibration set might require fewer cells. In this case, I'm not sure why the total number of cells needs to be fixed in advance, rather than adapting it to the available calibration data.

I would be willing to raise my score if the authors could further clarify this point.

AuthorsOpenReview ↗

Clarifying Budget Selection and Data-Adaptive Calibration Resolution

Response to SDqW: Why a fixed allocation budget is compatible with data-adaptive calibration resolution

We thank the reviewer for the constructive follow-up and for acknowledging the additional empirical evidence. We agree with the central point: more calibration observations may justify using more calibration cells. BRC does not require a constant number of cells across calibration-set sizes. The distinction is between selecting total resolution and allocating it.

1. Why condition on a fixed budget? Three decisions are involved. First, CAL availability constrains which partition resolutions have adequate statistical support. Second, a practitioner selects an appropriate total cell count CC among feasible options, considering coverage reliability, prediction-set size, and application requirements. Third, BRC determines how those CC cells should be distributed across heterogeneous regimes, solving ∑k=1KBk=C\sum_{k=1}^{K}B_k=C. Thus, CC is an input to each allocation problem, not a universal constant. If more CAL observations become available, a larger CC may be chosen and BRC applied at that resolution.

Conditioning on CC also serves a scientific purpose. Comparing methods with different cell totals would conflate increased localization capacity with the effect of allocation. Holding CC equal for BRC and Uniform isolates the value of distributing the same resolution differently. This does not prevent selecting CC separately for a new CAL configuration.

A related separation appears in partition-based estimation: Toccaceli [1] optimizes bin boundaries for a specified bin count and selects that count separately by cross-validation. BRC addresses a different conditional decision, allocating a chosen number of local calibration cells across preformed regimes according to their estimated coverage responses.

2. Why does allocation matter when CAL is sufficiently large? Increasing CAL size can improve threshold estimation within a fixed partition, but it does not automatically resolve coverage heterogeneity inside its cells. For example, a pooled cell may achieve 90% coverage while two equally prevalent subgroups receive 80% and 100%. More precise estimation of the pooled threshold alone does not distinguish these subgroups. Additional localization may help, but its benefit differs across regimes because refinement can reduce heterogeneity while dividing calibration support. Knowing that more cells are feasible does not establish where those cells are useful. BRC estimates regime-specific resolution responses using common frozen OOF audit groups and allocates the selected budget accordingly.

3. How could CC adapt to CAL availability? Candidate budgets may be screened using planned CAL size, minimum desired direct-cell support, and STRUCTURE-estimated cell masses. Under approximately equal-mass refinement, regime kk with estimated mass p^k\widehat p_k divided into bb cells has expected CAL support approximately NCALp^k/bN_{\mathrm{CAL}}\widehat p_k/b per cell. This is an expectation, not a guarantee of realized support. More CAL observations can therefore expand the set of support-feasible budgets. If several budgets remain feasible, they could be compared on an additional independent validation split using prespecified reliability, set-size, and complexity criteria before final CAL fitting. This is possible deployment guidance, not an automatic budget-selection procedure implemented or validated in BRC.

4. Evidence from existing experiments. We evaluated C∈{2K,3K,4K}C\in\{2K,3K,4K\} rather than assuming one budget is appropriate everywhere. Relative allocation gains were generally most pronounced at 2K2K, consistent with Uniform becoming more competitive as total resolution increases. Appendix H.8 separately varied CAL availability (25%, 50%, and 100%) at matched C=2KC=2K. BRC's mean TEST worst-group coverage advantage over Uniform remained positive on both CIFAR-100 ViT and Folktables TabNet at each fraction. Appendix H.6 also reports substantial local CAL support and zero fallback in representative settings, showing that those gains did not depend on fallback activation. These studies examine allocation under different budgets and CAL sizes, not an optimal rule for choosing CC.

In summary, adapting total calibration resolution to CAL availability and optimizing its allocation are complementary decisions. BRC fixes CC within each allocation problem to identify the effect of distributing a common resolution, not to require the same CC as calibration data increase. We have made this distinction explicit in the revised manuscript and hope it resolves the reviewer's concern.

Reference: [1] Toccaceli, CRPS-Optimal Binning for Univariate Conformal Regression, COPA 2026. https://proceedings.mlr.press/v329/toccaceli26a.html

Reviewer_amFTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Standard conformal prediction guarantees coverage averaged over the full population but may undercover in particular regions of the feature space. Given KK fixed broad groups, an ordering of points within each group, and a budget of CC calibration cells, the authors study how many cells BkB_k to assign to each group, subject to ∑kBk=C\sum_k B_k=C. Each cell receives its own conformal cutoff. Using more cells can make coverage more uniform within a group, but each cutoff must then be estimated from fewer calibration observations, so the data cannot support arbitrarily fine subdivision everywhere. The authors propose BRC, which uses held-out data to estimate the coverage error for each candidate number of cells in each group and selects the allocation with the smallest estimated worst-group error; separate calibration data are then used to estimate the final cutoffs. Across classification tasks, BRC usually improves worst-group coverage over equal allocation under tight budgets, while the gains shrink as the budget grows and the regression results are mixed.

Critical strengths

The paper addresses a gap in work on group-conditional conformal prediction. Existing methods mainly study which groups to condition on or how those groups should be learned, and then calibrate separately within each group. They do not generally study how finely each group should be divided when only a limited number of group-specific cutoffs can be supported. This matters because some groups may contain much more variation than others, while creating more subdivisions leaves fewer calibration observations for each cutoff. Treating the number of cutoffs as a shared resource is therefore a useful problem formulation.

The experiments evaluate the method well by holding the predictor, score, initial groups, data splits, and total number of calibration cutoffs fixed in the main comparison, while also testing alternative baselines and design choices. The method usually improves worst-group coverage over equal allocation and the other classification baselines when the cutoff budget is tight, although the advantage shrinks with larger budgets and performance on regression is mixed.

Critical weaknesses

The paper is harder to read than the method warrants. Core objects are given nonstandard names before being defined. For example, Section 2 introduces “predictive regimes” \mathcal P_k and r(x) without stating that r:\mathcal X\to{1,\ldots,K} and \mathcal P_k={x:r(x)=k}. Terms such as “calibration resolution,” “regime geometry,” and “audit geometry” add more jargon than clarity. Put simply, r assigns a point to a fixed cluster, U orders points within that cluster, and B_k is the number of calibration bins assigned to it. The paper is also repetitive and visibly under-edited, with duplicated passages, “Equation equation” references, and malformed phrases.

The main weakness is that most of the difficult choices sit outside the method. The user must specify the initial groups, an ordering within each group, the candidate subdivisions, and how local coverage is measured. These choices determine which coverage errors the method can detect and where it allocates its limited cutoffs. The allocation is then selected using noisy held-out coverage estimates, with no clear protection against fitting that particular split. The theory shows an advantage over equal allocation for the criterion used to choose the allocation, but does not show that the selected allocation improves group-conditional coverage on new data.

Questions for authors
  1. Proposition 1 compares BRC with equal allocation using the same empirical criterion on the STRUCTURE split. Is there any guarantee or uncertainty analysis showing when the selected allocation should improve worst-group coverage on fresh data rather than fit noise in that split?

  2. How should a practitioner choose the total budget CC and the candidate subdivision levels without using test data? The empirical advantage is strongest at 2K2K and becomes much less consistent at larger budgets, so this choice appears to determine whether BRC helps.

3 comments
AuthorsOpenReview ↗

Addressing Allocation-Selection Uncertainty, Population Generalization, and Budget Specification

We thank the reviewer for recognizing the fixed-budget problem and matched evaluations. We distinguish original evidence from new analyses.

Q1. Does STRUCTURE-selected allocation generalize beyond OOF noise?

Original safeguards (Secs. 3.2–3.4; Apps. C.4, C.6, D). Geometry and audit groups are frozen before OOF evaluation. Uniform matches the predictor, score, regimes, splits, audit groups, and cell budget. Proposition 1 guarantees a no-worse empirical WPC lower bound, not population dominance. CAL fits final cutoffs; TEST is excluded from selection.

New independent evidence (App. L.1). Five-seed FOLD, FROZEN, and REFIT studies test OOF assignments, STRUCTURE perturbations, and reconstructed geometry/allocation. REFIT reconstructs geometry/allocation using 80% of STRUCTURE and evaluates the excluded 20% and independent TEST; CAL fits cutoffs separately. At C=2KC=2K, paired held-out / TEST ΔWPC\Delta\mathrm{WPC} (mean ±\pm sample SD) was: Adult TabNet +0.1334±0.0624+0.1334\pm0.0624 / +0.1592±0.0897+0.1592\pm0.0897; Folktables TabNet +0.1170±0.0391+0.1170\pm0.0391 / +0.1290±0.0476+0.1290\pm0.0476; CIFAR-100 ViT +0.0029±0.0031+0.0029\pm0.0031 / +0.0040±0.0070+0.0040\pm0.0070; ImageNet-1K ResNet-50 +0.0146±0.0030+0.0146\pm0.0030 / +0.0160±0.0016+0.0160\pm0.0016. All four mean differences were positive on held-out and TEST data at 2K2K; CIFAR-100 was weak relative to variability. ImageNet used a separate predictor pipeline. Complete FOLD/FROZEN and 3K/4K3K/4K results include near-zero or negative effects. These results assess fresh-data transfer without implying universal superiority.

New conditional selection-uncertainty result (App. C.7). Let E‾(B)\overline{\mathcal E}(\mathbf B) denote the maximum deviation of fold-fitted conditional mean coverage responses. Under label-independent frozen geometry/folds, exclusion of evaluated-fold labels from threshold fitting, and conditional independence, simultaneous concentration over MM supported group-resolution-fold triples yields, with probability at least 1−δ1-\delta,

E‾(B^)−min⁡B∈FCE‾(B)≤2η,\overline{\mathcal E}(\widehat{\mathbf B})-\min_{\mathbf B\in\mathcal F_C}\overline{\mathcal E}(\mathbf B)\le2\eta,

where

η=max⁡g∑f:ngf>0ngfnglog⁡(2M/δ)2ngf.\eta=\max_g\sum_{f:n_{gf}>0}\frac{n_{gf}}{n_g}\sqrt{\frac{\log(2M/\delta)}{2n_{gf}}}.

The simultaneous bound accounts for data-dependent selection and overlapping training folds. It controls OOF evaluation uncertainty, not final-CAL population WPC; its radius is conservative, and label-stratified sampling requires additional justification.

New population-WPC certificate (App. C.7). For CAL-fitted BRC and Uniform evaluated on independent validation data with common audit groups, we derive

ΔWPCpop≥γ^−τ^−h^−2ϵ.\Delta\mathrm{WPC}_{\mathrm{pop}}\ge\widehat\gamma-\widehat\tau-\widehat h-2\epsilon.

Here γ^\widehat\gamma is the STRUCTURE objective advantage, τ^\widehat\tau the relative validation-transfer discrepancy, h^\widehat h the absolute-error/WPC mismatch, and ϵ\epsilon the validation concentration radius. A positive certificate would establish fitted-pair improvement; positivity was not demonstrated experimentally.

Q2. How should CC and candidate subdivisions be chosen without TEST?

Original protocol (Secs. 2–4; Apps. C, D). CC is a prespecified, support-feasible resource constraint, not a TEST-tuned hyperparameter. BRC optimizes allocation given CC, rather than selecting a globally optimal budget. C=KC=K permits no allocation freedom; 2K,3K,4K2K,3K,4K represent increasingly relaxed budgets. Original matched WPC gains occur in 9/10, 7/10, and 4/10 classification settings, respectively. At 2K2K, BRC also produces smaller mean prediction sets in 8/10 settings. We acknowledge that gains concentrate at 2K2K and do not claim it is universally optimal. Minimum CAL support and the cell-to-regime-to-global fallback constrain feasible refinement.

Clarified candidate-grid rationale (Sec. 3.1). We examined consecutive levels 1,2,3,4\\{1,2,3,4\\} before adding 66 to investigate larger, potentially nonmonotone refinement responses. The final B=1,2,3,4,6\mathcal B=\\{1,2,3,4,6\\} was fixed independently of TEST and is neither exhaustive nor theoretically optimal.

Upstream geometry, audit fairness, and clarity.

Original Apps. C and H already specified regime construction, uncertainty ordering, common audit groups, and sensitivity to forced KK, non-KMeans constructors, alternative uncertainty coordinates, support, and fallback. Frozen P×U10\mathcal P\times U_{10} audit groups provide a common evaluation target across allocations, not invariance to every possible partition. To address presentation concerns, revised Sec. 2 explicitly defines r:X→1,…,Kr:\mathcal X\to\\{1,\ldots,K\\} and Pk=x:r(x)=kP_k=\\{x:r(x)=k\\} at first use; Sec. 3 distinguishes r(x)r(x), U(x)U(x), and BkB_k, and clarifies OOF and audit terminology. Residual repetition and typographical inconsistencies will be corrected.

Summary.

Original safeguards establish TEST isolation; new theory and REFIT evidence address selection noise and fresh-data transfer.

Reviewer_amFTOpenReview ↗

Follow-up after rebuttal

Thank you for the additional experiments and revisions. The new REFIT results help address my concern about fitting noise in the STRUCTURE split, especially on Adult and Folktables, and Sections 2–3 explain the method more clearly.

My main remaining concern is the practical motivation for fixing the number of calibration cells in advance and how a practitioner should choose the budget and candidate grid. I understand that the method optimizes allocation conditional on these choices, but saying that they were fixed without using TEST data does not explain when those choices are appropriate. This matters because the reported gains are concentrated at the smallest budget and largely disappear at larger budgets.

The revised paper is easier to follow, but some important details still require too much reconstruction from the appendix. For example, the selection bound on p. 6 is stated without explaining there what EE and η\eta represent or that the bound concerns models fitted within folds rather than the final calibrated predictor. The classification regime construction describes its inputs only as predictive-uncertainty information, and the FROZEN experiment does not clearly specify how the STRUCTURE observations are perturbed. These details matter for understanding the guarantee and interpreting the experiments.

The additional experiments and clearer definitions improve the paper, but the exposition and presentation still leave much to be desired, and I remain unconvinced that it establishes a sufficiently compelling practical contribution. I am therefore keeping my weak-reject rating.

AuthorsOpenReview ↗

Practical Justification, Budget Selection, and Presentation Clarity

Response to amFT: Practical Motivation, Budget Choice, and Clarity

We thank the reviewer for recognizing the REFIT results and improved clarity. We agree that avoiding TEST leakage alone does not justify CC or B\mathcal B. We clarify their practical selection and why gains depend on budget.

1. Why does a fixed cell budget matter in practice?

CC need not remain fixed as CAL size changes. Two questions are distinct: How much calibration resolution can be used? and Where is it most useful? BRC addresses the second, given a selected, support-feasible CC. Matching CC between BRC and Uniform isolates allocation from the effect of adding cells. This is a controlled comparison, not a requirement that future deployments use the same number of cells.

This question remains relevant even with ample CAL observations. More data can make a pooled threshold more precise without separating subgroups within its cell: 90% pooled coverage may conceal 80% and 100% coverage in equally prevalent subgroups. Refinement may help, but its benefit differs across regimes and reduces per-cell support.

Knowing that more cells are feasible does not tell us where they are useful. BRC estimates each regime's coverage response to refinement on common frozen OOF audit groups, then allocates cells accordingly.

2. How would a practitioner choose CC and B\mathcal B?

Given planned CAL size NCALN_{\mathrm{CAL}}, estimated regime mass p^k\widehat p_k, and desired direct-cell support mmin⁡m_{\min}, splitting regime kk into bb approximately equal-mass cells gives expected support NCALp^k/bN_{\mathrm{CAL}}\widehat p_k/b. Candidates can be screened using these counts with allowance for sampling variation. If every cell needs direct calibration, Cmmin⁡≤NCALCm_{\min}\le N_{\mathrm{CAL}} is necessary but insufficient.

More CAL can permit larger budgets. Feasible budgets could then be compared on independent validation data using prespecified WPC, set-size, and complexity criteria, before final CAL fitting. This is practical guidance, not an evaluated BRC budget-selection algorithm.

Our grid B=1,2,3,4,6\mathcal B=\\{1,2,3,4,6\\} begins with consecutive refinement levels and adds a larger option to probe responses that need not be monotone. Budgets C∈2K,3K,4KC\in\\{2K,3K,4K\\} examine different degrees of constraint. Neither the grid nor 2K2K is universally optimal.

Learned conformal partitions [1] address where groups are formed; fixed-count binning [2] separates bin-count selection from partition optimization. BRC instead determines how a selected total resolution is distributed across preformed regimes.

3. Why are improvements strongest at 2K2K?

At C=KC=K, each regime receives one cell and there is no allocation choice. 2K2K is our smallest evaluated nontrivial budget. Under tight budgets, assigning another cell to one regime necessarily reduces the resolution available elsewhere.

One CIFAR-100 ViT regime had Ek(1)=0.0247E_k(1)=0.0247, Ek(2)=0.0280E_k(2)=0.0280, Ek(3)=0.0232E_k(3)=0.0232, illustrating nonmonotone estimated refinement benefits.

BRC improved mean TEST WPC over Uniform in 9/10, 7/10, and 4/10 classification comparisons at 2K2K, 3K3K, and 4K4K, with four ties at 4K4K. As Uniform gains cells, the additional benefit of reallocating them may diminish. This does not make 2K2K universally preferable.

The representative 2K2K gains were not driven by fallback. App. H.6 reports zero fallback and substantial CAL support in representative CIFAR-100 and Folktables experiments (smallest reported mean minimum cell count: 645).

At fixed 2K2K, App. H.8 reports positive mean WPC differences with 25%, 50%, and 100% CAL: 0.01240.0124, 0.04430.0443, 0.06060.0606 on CIFAR-100 and 0.05200.0520, 0.09880.0988, 0.12510.1251 on Folktables.

These results show that the allocation advantage persisted with different CAL sizes and in settings with substantial support. They also illustrate why increasing CAL size and deciding how to use the resulting resolution are separate questions.

4. Presentation.

We recognize that these examples reflect broader presentation concerns. Revised Sec. 3 defines the 2η2\eta bound quantities and its fold-fitted OOF scope, distinct from final CAL; App. C.2 specifies backbone and uncertainty inputs; Sec. 4 clarifies that FROZEN subsamples 80% of STRUCTURE without replacement, retaining geometry while recomputing responses.

We will review notation, assumptions, and cross-references throughout, making essential details accessible in the main text.

In summary, CAL size informs feasible resolution, while BRC identifies where that resolution is most useful. We hope these clarifications address the remaining concerns.

References

[1] Kiyani et al., Conformal Prediction with Learned Features, ICML 2024.

[2] Toccaceli, CRPS-Optimal Binning for Univariate Conformal Regression, COPA 2026.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.