- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper suggests two approaches to improve the efficiency of conformal prediction in terms of interval length. While conformal prediction guarantees valid marginal coverage under the exchangeability assumption between calibration and test data, its prediction intervals may be unnecessarily large. The first approach they suggest is Efficient Conformal Prediction with Flexible Quantile Allocation (ECP-FQ). This method chooses how to divide the error budget between the lower and upper residual tails. The second approach they suggest is ECP-FQ+, which uses estimated prediction error magnitude to adjust interval centers and widths across inputs. The parameters that control the interval shape are chosen using held-out data, disjoint from the calibration set used to ensure valid marginal coverage. The authors show empirically and theoretically that interval length reduction is achieved when allowing intermediate adjustments between constant-width intervals and widths proportional to estimated error.
Critical strengths
-
The paper addresses the important problem of improving the efficiency of conformal prediction in terms of interval length. Indeed, standard conformal prediction methods encounter the limitation of unnecessarily large intervals and adapting interval center and width can lead to more informative intervals.
-
The proposed methods are straightforward to understand and implement and do not involve substantial computational complexity. The parameters that control the interval shape have a clear interpretation. In particular, the scale exponent intuitively controls the amount of adaptation to estimated error.
-
The paper provides an extensive and rigorous theoretical analysis of the proposed methods. The paper also provides a comprehensive empirical evaluation, including comparisons with many conformal prediction baselines. The experiments cover both synthetic and real-world datasets, different prediction models, and multiple coverage levels. The authors also provide meaningful ablation studies.
-
The paper is written concisely and is well organized. The motivation, proposed methods, and theoretical and empirical results are well structured.
Critical weaknesses
-
The methodological novelty of the paper is quite limited. The proposed methods build on existing ideas in conformal prediction, such as residual normalization and selecting conformity scores using held-out data. In particular, the main approach introduces a scale exponent into the normalized conformity scores and selects it using held-out data. While the theoretical analysis is very interesting and thorough, the proposed method itself does not seem very novel.
-
The paper includes an extensive evaluation against several conformal prediction baselines. However, comparisons with additional methods specifically designed to optimize prediction interval length, such as CPL and Ad-EffOrt, would further strengthen the empirical evaluation. The authors explicitly note that these methods were not evaluated.
-
Although the paper introduces both ECP-FQ and ECP-FQ++, the empirical results suggest that most of the improvements arise from scale-exponent selection. In particular, ECP-FQ does not demonstrate clear improvements over fixed signed-tail allocations on real datasets, and jointly optimizing interval centers and widths seems to provide little additional benefit over scale-only adaptation. Thus, I question the practical value of the additional components.
Questions for authors
None