acceptodds
Under review as a conference paper at ICLR 2027

When Errors Cancel: Recovering Hidden Error Structure for Conformal Prediction

Abstract

Locally adaptive conformal prediction normalises the nonconformity score by a learned scale, and essentially every construction learns this scale by regressing it on the observed absolute residual. We show that this default target is corrupted when the residual is a signed aggregate of latent elementary errors that partially cancel. Detection-based counting makes this exact. Over- and under-detections subtract, so the observed count error reports what survived cancellation rather than what the model got wrong. This corrupts the regression target used to fit the scale. The error’s predictable component, its latent magnitude, is masked by a survival factor that the input predicts far less well, and the intervals end up wider than they need to be. We propose cancellation-aware normalisation, which replaces the residual target by an exact factorisation into a latent error magnitude and a survival rate (the fraction of latent error mass that survives cancellation), fitted separately and multiplied. This preserves the coverage guarantee exactly, and shrinks the uncertainty intervals. Our central finding is about how this auxiliary information must enter the scale. Supplying it as an extra input feature, the standard practice in the literature, does not reproduce the gain. Fitting it as a factor of the regression target does. We verify the mechanism directly by intervening on the detector confidence threshold, which controls the survival rate, and evaluate it across three counting benchmarks at very different imaging scales.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
conformal prediction, uncertainty quantification, locally adaptive normalisation, conditional coverage, object counting, object detection, distribution-free inference

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_JwuXOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper addresses the regression task of counting the number of objects in an image and applies a conformal prediction (CP) procedure to provide an instance-based uncertainty interval around that count. A standard way is based of training a network to predict the estimation error and \sigma(x) and then applying CP to the normalized error. The paper writes the error as a product of two factors and trains a separate network for each factor. The paper shows that this way we obtain an improved CP calibration procedure in the sense that the average prediction set is smaller.

Critical strengths

The experiments support a real but moderate improvement over the specific residual-normalization baseline. They do not establish a large or universal advance over existing conformal regression methods.

Critical weaknesses

The conformal construction and coverage guarantee are standard. The new element is the choice of training targets for the normalization scale: predicting total detection errors and the fraction surviving cancellation separately. The factorization is elementary, and the paper does not establish a theoretical improvement in interval width. The paper proposes a sensible modification to the scale estimator used in standard normalized conformal prediction. However, the methodological novelty appears limited, and the broader significance of the proposed factorization remains insufficiently established beyond object counting. Stronger theoretical analysis or validation on another task involving cancellation would strengthen the contribution. I think the paper in its current form is more suitable for a workshop in a computer vision conference than for ICLR

Questions for authors

See weaknesses above.

1 comment
AuthorsOpenReview ↗

Response to Reviewer JwuX

Thank you for the clear summary and for rating soundness 3. Here is why we think it is relevant to this audience, and what we added since your review.

1. A blind spot in a well-studied area. For twenty years, locally adaptive conformal prediction has improved the estimator of the scale (neighbours, ensembles, partitions, kernels, self-supervised features). Almost none of it questions what the scale is fitted to. We show that when errors are signed sums, this target loses much of the predictable signal: cancellation hides 38–66% of the elementary error, and features predict DD far better than ∣E∣|E| (R2R^2 gap +0.33 on GWHD, +0.56 on LIVECell; null on VisDrone).

2. Simple, exact, checkable. The identity ∣E∣=b⋅D|E| = b \cdot D is exact, validity is untouched, the fix is two regressions, and a cheap pre-hoc test indicates where the mechanism is strongest.

3. New theory (no width claim). By the law of total variance,

Var(∣E∣∣x)=E[D2 Var(b∣x,D)∣x]+Var(D E[b∣x,D]∣x).\mathrm{Var}(|E| \mid x) = \mathbb{E}\big[D^2\,\mathrm{Var}(b \mid x, D) \mid x\big] + \mathrm{Var}\big(D\,\mathbb{E}[b \mid x, D] \mid x\big).

Even a predictor that knows DD cannot have squared error below the first term, which grows with D2D^2. Knowing DD raises the out-of-fold R2R^2 of ∣E∣|E| by 0.43 on LIVECell and 0.34 on VisDrone, but only 0.03 on GWHD, the benchmark where our gain is smallest.

4. Another task with cancellation (your request). Hourly electricity load, 313 clients: the total error is a signed sum over clients and 34% of DD cancels. C3 is narrowest in all three settings at coverage 0.899–0.903: +5.9% vs residual normalisation on the full pool (narrower on 30/30 splits, p < 0.001 after correction), +4.6% and +1.5% on 1,600-hour subsets (not significant). A log target gets most of it (+5.1%), so the effect is modest here too.

5. It combines with current methods. Adding the factor estimates narrows a width-optimised normaliser in 10 of 10 cells; using the factorised scale instead of the residual scale narrows Kandinsky CP in 9 of 10 cells and CTI in 7 of 10, and a log-residual scale with GBM in 5 of 5 settings (small differences; details in our reply to Reviewer rT3h). Counting is what practitioners report for cells, crops, traffic and crowds, and other signed aggregates (load forecasting, portfolio profit and loss) plausibly meet the two requirements.

What the evidence supports, plainly. Against residual normalisation, C3 is narrower in 6 of 6 cells (+13.3% and +8.5% LIVECell; +7.9% and +6.4% VisDrone; +1.2% and +0.4% GWHD), winning on 74–89% of splits on LIVECell and VisDrone. After the Nadeau–Bengio correction for overlapping splits these differences are not significant at 0.05, and much of the gain is also obtained by a log-residual target. The correction’s extra term does not shrink with more splits, so its power depends on pool size: on electricity, the same kind of effect gives p = 0.34 on 1,600-hour subsets and p < 0.001 on the full 17,520-hour pool. We state all this in the paper.

Request. You rate soundness 3 and call the modification sensible; your open concerns were novelty, theory and scope. We added a variance identity, a non-vision task, corrected statistics and evidence that the scale combines with current methods. If these move the contribution above the bar for you, we would be grateful if you reconsidered your rating. If not, what single piece of evidence would?

Reviewer_rT3hOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that in normalized split conformal prediction the usual target for learning the scale, the absolute residual, is corrupted when the error is a signed sum of elementary errors that can cancel. In detector-based counting the count error is FP - FN, so ∣E∣|E| equals bb times DD exactly, where DD is FP + FN counts all detection errors and bb is the fraction that survives cancellation. The claim is that DD tracks scene difficulty and is much easier to predict than ∣E∣|E|, while bb is only weakly predictable. So they fit the scale to DD, or to the product of separately fitted bb and DD, which leaves the coverage guarantee intact. On three counting benchmarks the product version gives narrower intervals than the standard residual-based scale. The point the authors push hardest is that feeding the predicted DD to the scale model as an extra input feature, the usual way side information is added, does not reproduce the gain, i.e. such information belongs in the target, not the inputs.

Critical strengths

The work provides a clean, exact reformulation of what a locally adaptive scale should be regressed on, in a construction that pays no coverage price for trying it.

Critical weaknesses
  • The statistical argument could be wrong, and "safeguarded / safe default / never meaningfully worse" is false in general. sec 3.2 claims constant E[b∣x]\mathbb{E}[b \mid x] gives E[∣E∣∣x]∝E[D∣x]\mathbb{E}[|E| \mid x] \propto \mathbb{E}[D \mid x]. With Z=ϕ(X)Z=\phi(X),

\begin{equation*} \mathbb{E}[|E| \mid Z]=\mathbb{E}[b \mid Z] \mathbb{E}[D \mid Z]+\operatorname{Cov}(b, D \mid Z), \end{equation*}

and the covariance has no reason to vanish in a setting that is about cancellation. The scale invariance of section 3.1 only absorbs a global constant, not an input-dependent distortion

  • With ridge, σC3(z)=(a0+\sigma_{\mathrm{C} 3}(z)=\left(a_0+\right. a⊤z)(d0+d⊤z)\left.a^{\top} z\right)\left(d_0+d^{\top} z\right) is quadratic in zz; C2 is c0+c⊤z+λD^(z)c_0+c^{\top} z+\lambda \hat{D}(z), still affine. C2 also trains on a noisy cross-fitted feature while C3 gets direct supervision for both factors. And since D^(z)\hat{D}(z) is a function of zz once the selection fold is frozen, E[∣E∣∣z,D^(z)]=E[∣E∣∣z]\mathbb{E}[|E| \mid z, \hat{D}(z)]=\mathbb{E}[|E| \mid z]: there's no obstruction to the feature path, only that this stacking instance failed. generally, the supportable conclusion is e.g. "this baseline did not recover the gain," not the general principle in the abstract and conclusion
Questions for authors
  • Splits are stratified on deciles of the true nn. What is the exchangeability argument for the eq (4) guarantee under a label-dependent split?

  • Was the feature-family selection (Rresid 2(D)R_{\text {resid }}^2(D), table 3) run on the same pool later split into calibration/test? If so, how does that square with "no calibration or test information reaches any fitted object"?

  • Ramalingam et al. (2024) conformalize detector-derived counts; Bellotti (2020) and Bai et al. (2022) learn normalizers by optimizing efficiency. How do the "no prior work" and "every construction regresses ∣E∣|E|" claims stand against these?

1 comment
AuthorsOpenReview ↗

Response to Reviewer rT3h

Thank you for a precise review. Both weaknesses are correct, so we re-ran everything they touch and rewrote the claims.

W1 — Section 3.2 and “safe”. You are right; the proportionality argument is removed. Within quintiles of predicted count, E[bD]/(E[b] E[D])\mathbb{E}[bD]/(\mathbb{E}[b]\,\mathbb{E}[D]) is 0.84–0.96 (GWHD), 1.08–1.27 (LIVECell), 1.16–1.26 (VisDrone): the covariance is not negligible and its sign differs by dataset. “Never meaningfully worse” is removed. What we now state:

  • In 22 ridge/GBM cells, C3 is never wider than B1 by more than 0.3%.
  • MLP: with an unstandardised target C3 diverges (so does C1); standardising removes it (LIVECell/FPN: C3 207.8, C1 185.8, B1 219.6). Our earlier explanation was wrong.
  • Highest-DD bin: C3 under-covers more than B1 (Table 15). We now write “worse”.
  • VisDrone, ridge: C3 (82.1) is wider than constant width (79.6); B1 (89.2) is wider still.

Corrected tests. All paired tests now use the Nadeau–Bengio correction for overlapping splits:

CellC3 vs B1Splits wonp naive → corrected
GWHD, ridge+1.2%61%1.6e-2 → 0.81
GWHD, GBM+0.4%50%0.49 → 0.95
LIVECell, ridge+13.3%89%1.4e-19 → 0.26
LIVECell, GBM+8.5%89%5.2e-23 → 0.20
VisDrone, ridge+7.9%74%2.0e-10 → 0.48
VisDrone, GBM+6.4%84%4.2e-14 → 0.38

No difference is significant at 0.05 after correction, and B1 vs constant width is not either. The correction adds a term that does not shrink with the number of splits, so power is set by pool size. Electricity shows this: p = 0.34 and 0.67 on 1,600-hour subsets (our vision pool size), p < 0.001 on the full 17,520-hour pool.

W2 — Feature path. We agree and withdraw the principle. We ran a capacity-matched C2 (predicted DD, predicted bb, their product, and predicted DD × 10 principal components). With the raw label ∣E∣|E| it never recovers the gain (LIVECell ridge: −9.8% to −12.8% vs B1). With a log label it recovers most of it or more: +9.2% to +11.6% (LIVECell ridge; C3 +13.3%), +13.0% (VisDrone ridge; C3 +7.9%). The obstacle is the heavy-tailed raw label, not the route by which DD enters.

The resulting ablation: log-B1 fits log⁡(∣E∣+1)\log(|E|+1); log-C3 fits log⁡(D+1)\log(D+1), back-transforms, and multiplies by the same predicted bb as C3. Gain of log-C3 over log-B1:

GWHD/confLIVECell/FPNLIVECell/confVisDrone/FPNVisDrone/conf
Ridge−0.6%+3.4%+0.6%−1.3%−3.4%
GBM+1.0%+5.1%+3.2%+4.2%+2.8%

With GBM the factorised target is narrower in 5 of 5 settings; with ridge it is mixed. None is significant after correction (p ≥ 0.29). A log target alone closes most of the gap on LIVECell (log-B1 185.3 / 175.9 vs raw B1 213.0 / 194.2). The part specific to cancellation is smaller than we first claimed, and we now say so.

Q1 — Stratified splits. We have no exact argument for label-stratified splits; the guarantee is exact for uniform splits. We keep stratification in the main tables and report a uniform-split check (ridge, 100 splits): coverage 0.903–0.904, C3 gains +0.6% / +12.4% / +6.7% (GWHD / LIVECell / VisDrone), against +1.2% / +13.3% / +7.9%.

Q2 — Family selection. Yes, selection used the whole pool; the sentence “no calibration or test information reaches any fitted object” is corrected. Checks: (a) family chosen inside each selection fold: gains +0.3% / +12.1% / +5.0%, but the choice is unstable on VisDrone; (b) with each family fixed (App. C), C3 beats B1 in 8 of 8 LIVECell cells and 5 of 6 VisDrone cells, and by at most +1.3% on GWHD. “The mechanism holds there too” is removed for GWHD.

Q3 — Prior work. We now cite Ramalingam et al. (2024), who propagate conformal sets through programs over detector outputs, and dropped “no prior work”. To our knowledge, the decomposition of the count error has not been used as a target for the scale. Bellotti (2020) and Bai et al. (2022) optimise width instead of regressing ∣E∣|E|, so “every construction” is restricted to regression-based normalisation. A width-optimised normaliser in the spirit of Bellotti is narrower than C3 on 8 of 10 vision cells, so we do not claim to beat direct width optimisation. But adding predicted log⁡D\log D and log⁡b\log b as inputs makes it narrower again in 10 of 10 cells (+0.3% to +3.6%): the information in DD is useful beyond the choice of target.

What we claim after revision. (1) The exact factorisation, valid at no coverage cost. (2) A measurement of what the raw-∣E∣|E| default costs: up to 13% of width, mostly recovered by a log target. (3) A small gain from DD and bb beyond that, not significant after correction. Does this scope resolve your soundness concerns? If one point still blocks a positive assessment, which one?

Reviewer_RhX3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes cancellation-aware normalisation, a method to recover hidden error structure in conformal prediction by factorizing corrupted residuals into a latent error magnitude and a survival rate. This approach preserves exact coverage guarantees while shrinking uncertainty intervals. Experiments on counting benchmarks demonstrate the effectiveness of the proposed method.

Critical strengths
  1. The paper is easy to follow.
  2. The experimental settings are clearly explained.
  3. Extensive experiments are conducted to study the performance of the proposed cancellation-aware normalisation.
Critical weaknesses
  1. The technical contributions are relatively weak and lack theoretical support from the conformal perspective.
  2. The method considers only the detection-based counting task and cannot be directly extended to regression tasks.
  3. More up-to-date baselines should be included.
  4. The writing should be improved, e.g., “Datasets sizes” in line 294 and “At test step” in line 167.
Questions for authors
  1. The introduction had better include more references.
  2. What is the definition of a failure mode? It needs a more detailed description or a formal definition.
  3. Please consider evaluating the method on classification tasks, where TP/TN can also be computed.
  4. Please include more new baselines, such as [1] and [2].

[1] Kandinsky Conformal Prediction: Beyond Class- and Covariate-Conditional Coverage
[2] Conformal thresholded intervals for efficient regression

1 comment
AuthorsOpenReview ↗

Response to Reviewer RhX3

Thank you for the careful reading. We have now run the two baselines you asked for (Q4) and a non-vision task (W2); results are below and in the revised PDF.

W1 — Contribution and theory. The contribution is one precise change: the regression target of the scale. Earlier work improves the estimator of the scale but keeps fitting it to ∣E∣|E|. For signed aggregates, ∣E∣=b⋅D|E| = b \cdot D gives other targets, and validity is kept (the guarantee holds for any scale fitted on a disjoint fold). We add two exact statements, with no claim on interval width:

E[bD∣x]=E[b∣x] E[D∣x]+Cov(b,D∣x),\mathbb{E}[bD \mid x] = \mathbb{E}[b \mid x]\,\mathbb{E}[D \mid x] + \mathrm{Cov}(b, D \mid x),

which replaces the incorrect argument of Sec. 3.2, and

Var(∣E∣∣x)=E[D2 Var(b∣x,D)∣x]+Var(D E[b∣x,D]∣x).\mathrm{Var}(|E| \mid x) = \mathbb{E}\big[D^2\,\mathrm{Var}(b \mid x, D) \mid x\big] + \mathrm{Var}\big(D\,\mathbb{E}[b \mid x, D] \mid x\big).

Even a predictor that knows DD exactly cannot predict ∣E∣|E| with squared error below the first term, which grows with D2D^2. This is why ∣E∣|E| is a poor label. Knowing DD raises the out-of-fold R2R^2 of ∣E∣|E| by 0.43 on LIVECell and 0.34 on VisDrone, but only by 0.03 on GWHD, where our gain is smallest.

W2 — Beyond counting. The method needs (R1) an error that is a signed sum of elementary errors and (R2) elementary errors computable on annotated data. With per-unit errors eke_k logged on the fitting fold: E=∑kekE = \sum_k e_k, D=∑k∣ek∣D = \sum_k |e_k|, b=∣E∣/Db = |E|/D. We tested hourly electricity load (UCI, 313 clients): 34% of DD cancels. C3 is narrowest in all three settings, at coverage 0.899–0.903 (width reduction vs B1, positive = narrower):

SettingConstant widthlog-target B1C3 (ours)
Full pool, 17,520 h, ridge, 30 splits−31.5%+5.1%+5.9% (30/30 splits, p < 0.001)
1,600 h, ridge, 100 splits−34.0%+2.2%+4.6% (85% of splits, p = 0.34)
1,600 h, GBM, 100 splits−38.9%−0.5%+1.5% (66% of splits, p = 0.67)

Adaptation matters here (constant width is 31–39% wider than B1). A log target gets most of the gain, so the effect of the factorisation is modest (+0.9% over log-B1 on the full pool, p = 0.46). On a blocked temporal split, where exchangeability fails, constant width loses coverage (0.82) while C3 stays near target and is 9.3% narrower than B1 with ridge (equal with GBM; single split, no test).

W3 / Q4 — Baselines, now run (our implementations, code in the supplement; 10 vision cells = 5 dataset–feature settings × ridge/GBM).

  • Kandinsky CP [1], 8 overlapping groups. With the C3 scale it is narrower than with the B1 scale in 9 of 10 cells (e.g. VisDrone/FPN 96.0 vs 106.5). Worst-group coverage is 0.85–0.87, against 0.68–0.74 for constant-width split CP. Marginal coverage is 0.91–0.92, so widths are not comparable with plain split CP.
  • CTI [2], a different (asymmetric) interval shape. It is narrower than every symmetric interval on VisDrone (62.0 vs 71.3 for the best one) and GWHD (12.9 vs 13.6), and wider on LIVECell. With the C3 scale plugged in, CTI is narrower than with the B1 scale in 7 of 10 cells. Scale target and interval shape are complementary.

W4, Q1 — Writing and references. Typos fixed (“Dataset sizes”, “At test time”) and full proofreading done. Added to the introduction and related work: Shafer & Vovk (2008), Angelopoulos & Bates (2023), Bellotti (2020), Bai et al. (2022), Andéol et al. (2025), Lempitsky & Zisserman (2010), Ramalingam et al. (2024), and your [1], [2].

Q2 — Failure mode. Two levels. (i) Cancellation: D>∣E∣D > |E| on a non-negligible share of inputs. It is checkable on annotated data and removes 38–66% of DD in our benchmarks, 34% in the electricity task. (ii) Failure of the default target: ∣E∣=b⋅D|E| = b \cdot D carries the factor bb, which features predict poorly (R2R^2 0.06–0.13), adding label noise that a DD-based target does not carry. The pre-hoc diagnostic R2(D)−R2(∣E∣)>0R^2(D) - R^2(|E|) > 0 flags it; it is indicative only (null on VisDrone, where the method still helps). The consequence is never a loss of coverage, only wider intervals.

Q3 — Classification.

  • Single-label classification. One image gives one outcome, so nothing cancels: D=∣E∣D = |E|, b=1b = 1, and we recover the standard method.
  • Multi-class detection and counting. Within scope: (R1) and (R2) hold per class. For each class, with matched detections, E=FP−FNE = FP - FN, D=FP+FND = FP + FN, b=∣E∣/Db = |E|/D. Cancellation can occur within each class and for the total across classes; the same two regressions give one scale per class. We have not run this (our tables use one count per image) and list it as the most direct extension.
  • TP/FP/FN vocabulary. These are the elementary outcomes behind our decomposition; in binary classification the batch-level analogue is quantification (counting positives), i.e. our setting with images instead of boxes.

If you meant another setting, please tell us.

Do these additions address your concerns? If not, which point should we prioritise?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.