- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper addresses the regression task of counting the number of objects in an image and applies a conformal prediction (CP) procedure to provide an instance-based uncertainty interval around that count. A standard way is based of training a network to predict the estimation error and \sigma(x) and then applying CP to the normalized error. The paper writes the error as a product of two factors and trains a separate network for each factor. The paper shows that this way we obtain an improved CP calibration procedure in the sense that the average prediction set is smaller.
Critical strengths
The experiments support a real but moderate improvement over the specific residual-normalization baseline. They do not establish a large or universal advance over existing conformal regression methods.
Critical weaknesses
The conformal construction and coverage guarantee are standard. The new element is the choice of training targets for the normalization scale: predicting total detection errors and the fraction surviving cancellation separately. The factorization is elementary, and the paper does not establish a theoretical improvement in interval width. The paper proposes a sensible modification to the scale estimator used in standard normalized conformal prediction. However, the methodological novelty appears limited, and the broader significance of the proposed factorization remains insufficiently established beyond object counting. Stronger theoretical analysis or validation on another task involving cancellation would strengthen the contribution. I think the paper in its current form is more suitable for a workshop in a computer vision conference than for ICLR
Questions for authors
See weaknesses above.
Response to Reviewer JwuX
Thank you for the clear summary and for rating soundness 3. Here is why we think it is relevant to this audience, and what we added since your review.
1. A blind spot in a well-studied area. For twenty years, locally adaptive conformal prediction has improved the estimator of the scale (neighbours, ensembles, partitions, kernels, self-supervised features). Almost none of it questions what the scale is fitted to. We show that when errors are signed sums, this target loses much of the predictable signal: cancellation hides 38–66% of the elementary error, and features predict D far better than ∣E∣ (R2 gap +0.33 on GWHD, +0.56 on LIVECell; null on VisDrone).
2. Simple, exact, checkable. The identity ∣E∣=b⋅D is exact, validity is untouched, the fix is two regressions, and a cheap pre-hoc test indicates where the mechanism is strongest.
3. New theory (no width claim). By the law of total variance,
Var(∣E∣∣x)=E[D2Var(b∣x,D)∣x]+Var(DE[b∣x,D]∣x).
Even a predictor that knows D cannot have squared error below the first term, which grows with D2. Knowing D raises the out-of-fold R2 of ∣E∣ by 0.43 on LIVECell and 0.34 on VisDrone, but only 0.03 on GWHD, the benchmark where our gain is smallest.
4. Another task with cancellation (your request). Hourly electricity load, 313 clients: the total error is a signed sum over clients and 34% of D cancels. C3 is narrowest in all three settings at coverage 0.899–0.903: +5.9% vs residual normalisation on the full pool (narrower on 30/30 splits, p < 0.001 after correction), +4.6% and +1.5% on 1,600-hour subsets (not significant). A log target gets most of it (+5.1%), so the effect is modest here too.
5. It combines with current methods. Adding the factor estimates narrows a width-optimised normaliser in 10 of 10 cells; using the factorised scale instead of the residual scale narrows Kandinsky CP in 9 of 10 cells and CTI in 7 of 10, and a log-residual scale with GBM in 5 of 5 settings (small differences; details in our reply to Reviewer rT3h). Counting is what practitioners report for cells, crops, traffic and crowds, and other signed aggregates (load forecasting, portfolio profit and loss) plausibly meet the two requirements.
What the evidence supports, plainly. Against residual normalisation, C3 is narrower in 6 of 6 cells (+13.3% and +8.5% LIVECell; +7.9% and +6.4% VisDrone; +1.2% and +0.4% GWHD), winning on 74–89% of splits on LIVECell and VisDrone. After the Nadeau–Bengio correction for overlapping splits these differences are not significant at 0.05, and much of the gain is also obtained by a log-residual target. The correction’s extra term does not shrink with more splits, so its power depends on pool size: on electricity, the same kind of effect gives p = 0.34 on 1,600-hour subsets and p < 0.001 on the full 17,520-hour pool. We state all this in the paper.
Request. You rate soundness 3 and call the modification sensible; your open concerns were novelty, theory and scope. We added a variance identity, a non-vision task, corrected statistics and evidence that the scale combines with current methods. If these move the contribution above the bar for you, we would be grateful if you reconsidered your rating. If not, what single piece of evidence would?