- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper studies how energy weighting affects the size of conformal classification sets after recalibration. Two constructions show efficiency failures while maintaining coverage. Independent Gaussian nuisance features can alter energy weights without changing the label posterior. Almost constant weights can greatly enlarge APS sets when the scoring temperature is sufficiently small.
The proposed method, pooled cost projection (PCP), pools calibration inputs with one query and counts candidate labels under base and target scores. It clips their count difference, uses the base score to break ties, and recalibrates the transformed keys for that query. The transformation uses no labels and treats all pooled inputs symmetrically.
Under conditional exchangeability, with fitting and selection choices fixed using external data, PCP preserves marginal coverage and bounds expected added labels relative to the separately calibrated base. The expectation averages calibration and query randomness. Adaptive linear shrinkage receives the same formal guarantee.
Experiments examine constructed failures, an energy fine-tuned CIFAR-100 model, and transfers between classifiers on ImageNetV2. The authors report smaller mean sets than protected linear controls in the main settings, with additional analyses of budgets, faults, and computation.
Critical strengths
-
Theorem 3 gives a useful finite-sample bound on expected added labels relative to the same separately calibrated base. The proof controls movement of both candidate scores and the recalibrated threshold, and handles ties explicitly. Including recalibration in the certificate is an important part of the result. The label-free, symmetric transformation also gives a clear coverage argument under conditional exchangeability, with fitting and selection completed on external data.
-
The two constructions explain distinct failure mechanisms: nuisance features can dominate density without changing the label posterior, and small weight changes can overwhelm narrow APS score gaps. The appendices make the temperature and calibration restrictions explicit. The high-precision checks show that the second construction persists when very small score gaps are retained. The finite-calibration example gives a concrete application of the addition bound. The linear control with the same formal addition budget, fixed-direction ablations, and separate counts of additions and removals help interpret the empirical gains.
Critical weaknesses
-
The evidence for efficiency gains remains specific to the chosen base and settings. In the exploratory LAC-base experiments in Table 12, PCP does not improve mean set size over the base (11.678 versus 11.485 for WRN, and 1.626 versus 1.596 for JEM). At 99% on ResNet, the comparison with linear shrinkage has mixed signs across the three splits (Appendix J). These observations do not establish a significant deterioration, but they leave limited evidence for gains from a competitive base or across operating points. The repeated splits and additional partitions reuse images and checkpoints, so they mainly measure partition sensitivity rather than independent replications across models or data pools. The paper acknowledges these limits and treats the reported standard deviations descriptively. The remaining concern is the breadth of the empirical support, not the expected-addition guarantee.
-
Practical cost and experimental reproducibility need more evidence. The stated direct implementation costs time and memory per query. With 2,000 calibration inputs and 1,000 classes, the pool contains 2,001,000 candidates for each score. Calibration outputs can be cached, but the supplied timing panel covers 100-class constructed examples and does not establish per-query latency or peak memory for the exact procedure on ImageNetV2. I could not find an implementation or replay archive on the submission page, while the reproducibility statement promises a future release. This leaves the reported score-order checks and mask replays difficult to verify independently. An anonymous implementation with frozen configurations and replay files, together with realistic timing and memory measurements, would make the experimental evidence easier to assess.
Questions for authors
-
Can auxiliary data identify when PCP improves on an already competitive base, given the LAC-base results and mixed 99% ResNet comparisons? Would independent checkpoints or image pools support the observed gains beyond partition sensitivity?
-
Please report exact per-query conformal-update latency and peak memory for ImageNetV2, using cached calibration scores and separating this cost from model inference.
-
Please provide an anonymous archive with the implementation, frozen configurations, score-order checks, and files needed to reproduce the reported mask replays.
Thanks for your answer
Thanks for sharing the ImageNetV2 timing and memory measurements, and for making it easy to reproduce the results. This really helps with my concerns about how it was implemented and whether it can be repeated. The fact that you did multiple training runs on your own backs up the idea of SAPS, and having a fallback option like LAC is really useful. I do not raise my rating though, because how well it works depends on the specific setup, especially when comparing to 99% on ImageNetV2. Can you please add the new results and the link to the revised paper?