- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies the problem of allocating a given number of calibration cells to diverse, already identified predictive regimes. Proposed Budgeted Regime Calibration (BRC) minimizes worst-case empirical deviations from the target coverage level by estimating regime-specific calibration-error responses using OOF evaluation.
Critical strengths
- The framing of the optimization problem for calibration cell budget allocation is straightforward, while the conformal guarantee follows immediately. Hence, it may be practically useful.
Critical weaknesses
-
In my opinion, the technical novelty of the paper is very limited. The strategy proposed for the budget allocation is an empirical optimization problem without any fundamental technical or methodological contribution. The calibration step, once cells are allocated, is the usual conformal guarantee.
-
The proposed BRC is not supported by any consequential theoretical guarantee. The main result (Proposition 1) is a rearrangement statement. These results are about empirical minimization only, not at the population level. Empirical minimization is not supported by its uncertainty analysis.
Questions for authors
-
Could authors identify distinct technical/ methodological novelty in the paper apart from solving the budget allocation problem by known methods?
-
Can you comment on the data requirements utilized in the overall BRC strategy? How practical is it? Do CAL cell requirements meet in general settings? Isn't it a cyclic problem we might face for regimes with low resolution?
3 comments
I thank the authors for the revisions and their response. It has improved the clarity and presentation of the paper. Empirical evidence is useful. However my main concerns remain. Firstly, authors identify their optimization as a distinct problem form the existing works. However, it is not clear why the proposed algorithm demonstrates a substantial new methodological insight. Authors also affirm that neither standard solvers nor conformal quantiles are novel. New theoretical analysis is useful but it neither establishes strict improvement over uniform allocation nor OOF bound implies final calibration. I appreciate empirical studies which show favourable performance in limited budgets. My concern about the CAL cell requirements remain unresolved. I understand that whenever there are insufficient number of examples in CAL within a particular regime, there is a fallback strategy. But, this is exactly the regime where we need method to work. In my opinion, the method is off course correct and might be useful in practice , however the concern about limited technical/methodological novelty for ICLR still remains. Hence, I maintain my initial assessment of the paper.
Clarifying Methodological Novelty, Theoretical Guarantees, and Calibration Feasibility
Response to KtJ7: Methodological Contribution, Theory, and CAL Feasibility
We thank the reviewer. A distinct optimization formulation alone does not establish methodological significance. BRC's contribution is estimating the reliability benefit of additional calibration resolution, rather than inferring it from predictive uncertainty or sample support.
1. Methodological insight. Uncertainty suggests where refinement may occur; support determines whether it is feasible. Neither measures how local coverage error changes when cells are added. Finer partitions may reduce heterogeneity but weaken quantile estimation, producing nonmonotone responses.
BRC estimates this response for each frozen regime and candidate cell count using STRUCTURE-side out-of-fold (OOF) calibration. Crucially, all candidate partitions are evaluated on identical frozen audit groups, so their coverage deviations refer to the same reliability target. BRC then chooses under , lexicographically minimizing maximum local deviation before aggregate deviation. The allocation is frozen before independent CAL fitting. The contribution is comparable response estimation and coupled worst-group allocation, not the solver or quantile.
This differs from learned conformal partitions [1], fixed-count binning [2], and miscoverage-level allocation [3]. BRC allocates cell counts among preformed regimes according to their measured coverage responses, separating refinement placement from refinement utility.
2. New evidence isolating this mechanism (Appendix L.5). We compared BRC with prespecified mean-uncertainty, uncertainty-spread, support, Uniform, and Random allocators across five seeds. Within each seed, predictors, regimes, boundaries, CAL/TEST data, audit groups, and budgets were matched.
At on Folktables TabNet, BRC improved paired TEST worst-group coverage (WPC) over mean-uncertainty allocation by and over Uniform by , positive in all five seeds. Against Uniform, marginal coverage was equal at and mean set size was smaller ( vs. ). On CIFAR-100 ViT, BRC improved WPC over mean-uncertainty allocation by , positive across five seeds. This is a test of the information used for allocation, not simply another benchmark: uncertainty magnitude did not substitute for estimated calibration-refinement responses in these comparisons. Appendix L.5 discloses all budgets and its independent predictor-cache protocol.
Earlier mechanism evidence supports this distinction. A CIFAR-100 regime exhibited , , (App. G), demonstrating a nonmonotone estimated refinement response. On Folktables at , the full lexicographic objective achieved WPC versus for maximum-deviation-only and for weighted-only selection (App. I). Both the response signal and allocation criterion therefore matter.
3. Theoretical contribution. Proposition 1 connects the STRUCTURE-side allocation objective to empirical worst-group shortfall and includes matched Uniform as feasible. The conditional result bounds OOF selection uncertainty under explicit assumptions. Final CAL conformal validity is a separate rank-based property; the OOF bound is not claimed to establish strict final-CAL population superiority. Independent TEST and REFIT analyses evaluate the selected allocations beyond their selection data. The selection analysis and calibration guarantee thus have distinct scopes.
4. CAL support. A constrained total cell count differs from insufficient support in an individual selected cell. The total budget can reflect CAL availability; BRC allocates that budget rather than estimating local thresholds from arbitrarily few observations. The prespecified cell-to-regime-to-global hierarchy is a safeguard, not cyclic reallocation or part of the allocation objective. Appendix H.6 reports zero fallback and substantial local CAL support in representative settings, establishing that their observed gains did not arise from fallback. Appendix H.8 reports positive mean matched WPC differences at , , and CAL availability in both settings.
In summary, BRC estimates the reliability consequence of refinement on a common evaluation target and uses it to allocate a shared resolution resource. Appendix L.5 directly examines the significance of this decision signal relative to uncertainty and support proxies. We ask the reviewer to reconsider its methodological significance.
References: [1] Kiyani et al., Conformal Prediction with Learned Features, ICML 2024. [2] Toccaceli, CRPS-Optimal Binning for Univariate Conformal Regression, COPA 2026. [3] Xu et al., Aggregating Conformal Prediction Sets via Alpha-Allocation, Biometrika 2026.
Clarifying Methodological Novelty, Theoretical Guarantees, and Calibration Feasibility
Response to KtJ7. We distinguish original evidence from rebuttal additions.
Q1. What is novel beyond empirical optimization?
Distinct methodological decision (original Secs. 2–3; App. A.5–A.7). Calibration resolution is a statistical resource: finer cells may improve localization but reduce support. Kiyani et al. [1] study learned partitions; Toccaceli [2] examines bin-count selection; Boström and Johansson [3] discuss limited support in Mondrian categories. Xu et al. [4] allocate confidence levels, while Das et al. [5] allocate training versus calibration data. These differ from allocating a fixed cell budget across preformed regimes, whose support and responses vary. BRC estimates regime-specific responses and jointly allocates cells; neither standard solvers nor conformal quantiles are novel.
Mechanism already presented (original Sec. 3; App. C). Five-fold STRUCTURE OOF estimates resolution-dependent error Ek(b) using identical frozen audit groups across candidates. Uncertainty orders refinement within each regime; the OOF response determines its cell allocation. A joint minimax criterion selects allocation before independent CAL fitting.
Evidence already in the submission (Sec. 4; Apps. F, G, H, I). A CIFAR-100 ViT regime has Ek(1)=0.0247, Ek(2)=0.0280, Ek(3)=0.0232: increasing resolution is not monotonically beneficial. At 2K, BRC yields higher mean TEST WPC in 9/10 matched classification settings and smaller mean sets in 8/10, with all comparison factors except allocation fixed. Appendix F compares Random and GlobalU; App. I ablates the objective (Folktables 2K WPC: full 0.6927 vs. weighted-only 0.5201); App. H.3 tests non-KMeans regimes. These controls and ablations were already reported in the original submission, not added after review. Larger-budget results are mixed.
New uncertainty analysis (revised App. C.7). Proposition 1 concerns empirical optimization; standard conformal validity does not prove superiority. Under frozen label-independent geometry/folds and conditional independence, we now establish, with probability ≥1−δ,
\eta=\max_g\sum_{f:n_{gf}>0}\frac{n_{gf}}{n_g}\sqrt{\frac{\log(2M/\delta)}{2n_{gf}}}.$$ For fitted BRC and Uniform evaluated on independent validation data, we also derive $$\Delta\mathrm{WPC}_{\mathrm{pop}}\ge\widehat\gamma-\widehat\tau-\widehat h-2\epsilon.$$ The first bounds OOF selection uncertainty, not final-CAL WPC; it is conservative and label stratification requires justification. The second is a conditional certificate without demonstrated positivity. **New five-seed FOLD/FROZEN/REFIT tests (App. L.1)** assess transfer. At $2K$, REFIT mean TEST gains were **+0.1592** (Adult), **+0.1290** (Folktables), **+0.0040** (CIFAR-100), and **+0.0160** (ImageNet, separate pipeline); larger-budget effects can be nonpositive. **Q2. What data are required, and does low support cause a cycle?** **Original data requirements (Apps. C, D.7, H.6, H.8, K).** Separate STRUCTURE and CAL cost data. The submitted protocol specified minimum STRUCTURE regime sizes of **100** (classification) and **50** (regression), OOF support **80**, CAL leaf support **30** (tabular), **50** (CIFAR-100), or **20** (ImageNet/regression), and parent support **30**. Unsupported leaves follow a fixed *cell → regime → global* fallback, **not cyclic reallocation**. Fallback reduces localization. Original App. H.6 reported zero fallback in representative CIFAR-100/Folktables sweeps; App. H.8 found positive mean $2K$ WPC differences at **25%, 50%, 100%** CAL fractions; App. K reported small allocation overhead. These tests were already in the submitted paper; feasibility is not universal. **New direct data-cost test (App. L.2).** Uniform-FULL uses **STRUCTURE plus CAL** to fit thresholds; BRC uses STRUCTURE for selection and CAL for calibration. With independently TRAIN-frozen common geometry, BRC had positive five-seed mean $2K$ WPC differences in **all eight** dataset/score comparisons, including CIFAR-100 ViT APS **$+0.0416\pm0.0044$** and Folktables TabNet LAC **$+0.0854\pm0.0080$**. This is evidence under the reference-frozen protocol, not a universal guarantee. **Revised presentation.** Revised Secs. 2–3 clarify notation and audit groups; remaining typos will be corrected. **References.** [1] Kiyani et al., *Conformal Prediction with Learned Features*, ICML 2024. [2] Toccaceli, *CRPS-Optimal Binning for Univariate Conformal Regression*, COPA 2026. [3] Boström and Johansson, *Mondrian Conformal Regressors with Overlapping Categories*, COPA 2026. [4] Xu et al., *Aggregating Conformal Prediction Sets via Alpha-Allocation*, Biometrika 2026. [5] Das et al., *On Optimal Data Splitting for Split Conformal Prediction*, arXiv:2606.31600.