- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
The paper studies local model comparison. That is, given a random pair and two fitted models and , it seeks to identify where each model performs better via a comparison score , representing a generalized difference between the models' losses. The sign of indicates the superiority of model or .
The paper addresses two questions. First, the authors propose conformal method that fits generic machine learning algorithm, and constructs an upper prediction bound for future score via classic conformal method. Model is declared the winner only when . Under exchangeability, the authors establish This result provides a high-probability guarantee against misidentifying the better model. Here, the probability taken includes the randomness of the new observation pair . So, it does not certify superiority at a fixed . In view of this constraint, the second contribution is that the authors construct a neighborhood-average estimator , and show it consistently estimates where . Consequently, its sign consistently identifies the local expected winner at a fixed . Then, the authors apply the framework using TabPFN localizer and evaluate it on synthetic and real datasets.
Critical strengths
The biggest strengths of this paper is the proposed conformal method in securing a high probability identification of the better model. This is the pathway to the novelty of this paper. Its overall idea and design are on track. Given the authors could do more along this way, it should substantially improve this paper.
Critical weaknesses
The paper writing is a big concern for me. I will list a few. The paper could be improved significantly if the introduction can be more well written. I will list a few. For example, it is very hard to understand what basic question the paper is trying to address until Section 3 or 4. Moreover, the paper is organized in a way of itemizing all definitions and results, without elaborating their meaning and implications. Moreover, many important components of this paper is delegated to appendix, which damages the flow of this paper. I understand that ICLR has a rigorous page limit, but some high level summary can be done.
Another weakness is the authors claim conformal comparison of models yet the it is only valid taking the probability of new observation . Thus, the more interesting question of local comparison at has no conformal solution. In contrast, a kernel estimation analogy is proposed to establish consistency, which reduces the novelty of this paper.
Questions for authors
Is it possible to obtain a conformal guarantee conditional on , at least for in a subset of the support of ?
In Result 2 of the simulation part, the selection rate and power appear to remain largely unchanged in several scenarios as the nominal level increases. To my understanding, this usually suggests that the constructed high probability bound is overly conservative. Could the authors explain this?
2 comments
Thank you for the detailed response, the revisions to the presentation, and the additional experiments. I appreciate the authors' efforts to address my concerns.
Regarding the guarantees conditional on , I recognize the fundamental challenges of obtaining a local distribution-free coverage and appreciate the clarification authors provided. However, this does not fully address my original concern: the identification of the local expected winner at a fixed is not conformalized and is supported only by asymptotic consistency, rather than a finite-sample guarantee. Given the paper’s emphasis on conformalized local model comparison, this distinction remains important to my assessment of its contribution. While I understand the difficulty of the problem, I still find the novelty somewhat limited.
I also appreciate the clarification regarding the selection and power curves and the additional empirical coverage results. Overall, my main concern about the scope and novelty of the contribu- tion remains. I therefore maintain my original rating.
Response to Reviewer Pens
We thank the reviewer for recognizing our conformal model-selection framework as the core novelty of this work.
W1 (presentation).
Thank you for the constructive suggestions. In the revised PDF, we have reworked the introduction, the presentation of the definitions and results, and the balance between the main text and the appendix, so that the question the paper addresses and the meaning of each result are clear early on.
W2, Q1. Can conformal guarantees be conditional on X?
Our method already adapts to X, but its finite-sample guarantee is marginal. The distinction is important:
At a specific x: With continuous, high-dimensional X, observations rarely share exactly the same x. Therefore, distribution-free conformal calibration cannot generally guarantee nontrivial pointwise conditional coverage (Foygel Barber et al., 2021).
Our approach: We address this challenge by estimating the local mean μ^(x) and scale s^(x) of the loss difference, then applying split conformal calibration to the standardized residuals. The resulting bound adapts to x, while retaining finite-sample marginal validity (Proposition 1).
Conditional on a region: Stronger guarantees are possible for pre-specified subsets of X. By calibrating residuals separately within each region (Mondrian conformal prediction), we obtain finite-sample error control conditional on region membership, though not at every individual x. The trade-off is fewer calibration observations per region.
New experiments (Appendix D.2): We implement this region-conditional extension using two pre-specified regions.
Thus, our framework supports both locally adaptive marginal calibration and, through region-specific calibration, conditional guarantees at the region level.
Q2 (selection and power flat in α).
Our conformal upper bound is tightly calibrated rather than conservative. We computed the empirical coverage P(S≤U^α(X)), the probability that the upper bound covers the score of a test point. A conservative bound would give coverage clearly above 1−α. Instead, our observed coverage closely matches the nominal level (Appendix D.1):
Flat curves occur where detection already reaches its ceiling. In Case 1, A dominates almost everywhere, and the power of Local scaled CP is already 0.98 at α=0.05. In Case 2, A wins in only 4.3% of the covariate space, so accurate error control keeps the selection rate below 0.043 across α.
Where headroom exists, power increases with α:
We hope these new results and clarifications address all of your questions.