acceptodds
Under review as a conference paper at ICLR 2027

Who Wins Where? Conformal Model Comparison for Local Superiority

Abstract

Standard model comparison is global, aggregating losses across the covariate space to declare a single winner. This can obscure heterogeneous performance, where different models are preferable in different regions. We introduce conformalized local model comparison, a split-sample framework for constructing calibrated local winner maps. Under exchangeability, our goal is to provide finite-sample, distribution-free control of the marginal false winner rate while allowing declarations to depend on covariates; the guarantee is marginal because pointwise conditional control is unattainable in this setting. Given a model comparison score, such as the difference between two squared losses, the method estimates local centers and scales from out-of-sample scores and conformally calibrates the remaining uncertainty. At a test point, the procedure declares a local winner only when a one-sided conformal upper bound falls below zero. We instantiate the framework with a pretrained TabPFN localizer whose regression head is fine-tuned to maximize conformal gain. For any choice of localizer, we prove that the error control applies to one-sided false declarations on the realized future comparison score. We further give sufficient conditions for pointwise recovery of the expected winner away from tie boundaries, show that aggregate comparison can sharply contradict the prevalence of local superiority, and derive a squared-loss bias–variance decomposition of local wins. Synthetic and real-world experiments show that the method recovers heterogeneous winner regions and abstains under uncertainty, identifying local gains that global comparison can miss. We provide our code and data in the supplementary material.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Conformal prediction, local model comparison, model selection, uncertainty quantification

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_PensOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

The paper studies local model comparison. That is, given a random pair (X,Y)(X,Y) and two fitted models AA and BB, it seeks to identify where each model performs better via a comparison score S(X,Y)S(X,Y), representing a generalized difference between the models' losses. The sign of S(X,Y)S(X,Y) indicates the superiority of model AA or BB.

The paper addresses two questions. First, the authors propose conformal method that fits generic machine learning algorithm, and constructs an upper prediction bound Uα(x)U_\alpha(x) for future score S(X,Y)S(X,Y) via classic conformal method. Model AA is declared the winner only when Uα(x)<0U_\alpha(x)<0. Under exchangeability, the authors establish P(Uα(X)<0, S(X,Y)≥0)≤α.\mathbb{P}\bigl(U_\alpha(X)<0,\ S(X,Y)\geq 0\bigr)\leq\alpha. This result provides a high-probability guarantee against misidentifying the better model. Here, the probability taken includes the randomness of the new observation pair (X,Y)(X,Y). So, it does not certify superiority at a fixed X=x0X = x_0. In view of this constraint, the second contribution is that the authors construct a neighborhood-average estimator μ^(x0)\hat{\mu}(x_0), and show it consistently estimates μ(x0)\mu(x_0) where μ(x0):=E[S(X,Y)∣X=x0]\mu(x_0) := \mathbb{E}[S(X,Y)\mid X=x_0]. Consequently, its sign consistently identifies the local expected winner at a fixed x0x_0. Then, the authors apply the framework using TabPFN localizer and evaluate it on synthetic and real datasets.

Critical strengths

The biggest strengths of this paper is the proposed conformal method in securing a high probability identification of the better model. This is the pathway to the novelty of this paper. Its overall idea and design are on track. Given the authors could do more along this way, it should substantially improve this paper.

Critical weaknesses

The paper writing is a big concern for me. I will list a few. The paper could be improved significantly if the introduction can be more well written. I will list a few. For example, it is very hard to understand what basic question the paper is trying to address until Section 3 or 4. Moreover, the paper is organized in a way of itemizing all definitions and results, without elaborating their meaning and implications. Moreover, many important components of this paper is delegated to appendix, which damages the flow of this paper. I understand that ICLR has a rigorous page limit, but some high level summary can be done.

Another weakness is the authors claim conformal comparison of models yet the it is only valid taking the probability of new observation (X,Y)(X, Y). Thus, the more interesting question of local comparison at X=x0X = x_0 has no conformal solution. In contrast, a kernel estimation analogy is proposed to establish consistency, which reduces the novelty of this paper.

Questions for authors

Is it possible to obtain a conformal guarantee conditional on X=x0X=x_0, at least for x0x_0 in a subset of the support of XX?

In Result 2 of the simulation part, the selection rate and power appear to remain largely unchanged in several scenarios as the nominal level α\alpha increases. To my understanding, this usually suggests that the constructed high probability bound is overly conservative. Could the authors explain this?

2 comments
AuthorsOpenReview ↗

Response to Reviewer Pens

We thank the reviewer for recognizing our conformal model-selection framework as the core novelty of this work.

W1 (presentation).

Thank you for the constructive suggestions. In the revised PDF, we have reworked the introduction, the presentation of the definitions and results, and the balance between the main text and the appendix, so that the question the paper addresses and the meaning of each result are clear early on.

W2, Q1. Can conformal guarantees be conditional on XX?

Our method already adapts to XX, but its finite-sample guarantee is marginal. The distinction is important:

  • At a specific xx: With continuous, high-dimensional XX, observations rarely share exactly the same xx. Therefore, distribution-free conformal calibration cannot generally guarantee nontrivial pointwise conditional coverage (Foygel Barber et al., 2021).

  • Our approach: We address this challenge by estimating the local mean μ^(x)\hat\mu(x) and scale s^(x)\hat s(x) of the loss difference, then applying split conformal calibration to the standardized residuals. The resulting bound adapts to xx, while retaining finite-sample marginal validity (Proposition 1).

  • Conditional on a region: Stronger guarantees are possible for pre-specified subsets of XX. By calibrating residuals separately within each region (Mondrian conformal prediction), we obtain finite-sample error control conditional on region membership, though not at every individual xx. The trade-off is fewer calibration observations per region.

New experiments (Appendix D.2): We implement this region-conditional extension using two pre-specified regions.

Thus, our framework supports both locally adaptive marginal calibration and, through region-specific calibration, conditional guarantees at the region level.

Q2 (selection and power flat in α\alpha).

Our conformal upper bound is tightly calibrated rather than conservative. We computed the empirical coverage P(S≤U^α(X))\mathbb P(S\le\hat U_\alpha(X)), the probability that the upper bound covers the score of a test point. A conservative bound would give coverage clearly above 1−α1-\alpha. Instead, our observed coverage closely matches the nominal level (Appendix D.1):

Nominal α\alphaTarget 1−α1-\alphaObserved coverage range (5 cases, both local rules)
0.050.950.947–0.953
0.100.900.897–0.906
0.150.850.847–0.861
0.200.800.802–0.812

Flat curves occur where detection already reaches its ceiling. In Case 1, AA dominates almost everywhere, and the power of Local scaled CP is already 0.98 at α=0.05\alpha=0.05. In Case 2, AA wins in only 4.3% of the covariate space, so accurate error control keeps the selection rate below 0.043 across α\alpha.

Where headroom exists, power increases with α\alpha:

CaseRulePower from α=0.05\alpha=0.05 to α=0.20\alpha=0.20
2 (nonlinear tail)Local CP0.19 → 0.64
3 (middle island)Local scaled CP0.30 → 0.83
4 (high noise)Local CP0.11 → 0.49
5 (heteroscedastic)Local scaled CP0.33 → 0.58

We hope these new results and clarifications address all of your questions.

Thank you for the detailed response, the revisions to the presentation, and the additional experiments. I appreciate the authors' efforts to address my concerns.

Regarding the guarantees conditional on x0x_0, I recognize the fundamental challenges of obtaining a local distribution-free coverage and appreciate the clarification authors provided. However, this does not fully address my original concern: the identification of the local expected winner at a fixed x0x_0 is not conformalized and is supported only by asymptotic consistency, rather than a finite-sample guarantee. Given the paper’s emphasis on conformalized local model comparison, this distinction remains important to my assessment of its contribution. While I understand the difficulty of the problem, I still find the novelty somewhat limited.

I also appreciate the clarification regarding the selection and power curves and the additional empirical coverage results. Overall, my main concern about the scope and novelty of the contribu- tion remains. I therefore maintain my original rating.

Reviewer_KzfXOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies local model comparison: deciding which of two fitted predictors, AA or BB, is better near a covariate x0x_0, rather than on average. The target is the sign of the conditional mean comparison score μ(x)=E[S(X,Y)∣X=x]\mu(x) = \mathbb{E}[S(X,Y)\mid X=x] (e.g. S=LA−LBS = L_A - L_B; negative favors AA), with majority and quantile winners as secondary descriptors. The proposed conformalized local model comparison uses a three-way split: IfitI_{\text{fit}} trains A,BA,B; IestI_{\text{est}} fits a localizer returning a center μ^(x)\hat\mu(x) and scale s^(x)\hat s(x) from out-of-sample scores; IcalI_{\text{cal}} calibrates the standardized residuals (Si−μ^(Xi))/s^(Xi)(S_i-\hat\mu(X_i))/\hat s(X_i). At a test point, U^α(x0)=μ^(x0)+s^(x0)q^1−α\hat U_\alpha(x_0) = \hat\mu(x_0) + \hat s(x_0)\hat q_{1-\alpha} and AA is declared the local winner iff U^α(x0)<0\hat U_\alpha(x_0)<0.

Experiments: five 1-D synthetic designs (LightGBM vs. linear regression, n=5000n=5000 split four ways, 5 seeds) show false winner rates below α\alpha, large accuracy gains over the global mean in a global–local mismatch case (0.299 vs. 0.956), and a nuisance-dimension sweep over seven localizers where TabPFN-gain has the best average power at d=10,30,100d=10,30,100; four regression and three classification datasets (MLP vs. linear/logistic regression) show Local CP selecting small regions with the highest conditional gain while Global CP abstains entirely.

Critical strengths

S1. Clear problem formulation with assumption-light guarantee. The expected/majority/quantile winner targets are cleanly separated from the localizer used to estimate them; Proposition 2, though elementary, makes a point practitioners routinely miss (aggregate averaging weighs score magnitude, not prevalence). The declaration rule is a direct application of locally scaled split conformal prediction and the paper is explicit that the guarantee is a prediction bound on the realized score, marginal over XX, not a confidence statement about μ(x0)\mu(x_0). The multi-model extension via a single max-residual quantile (App. A.5) handles multiplicity without Bonferroni and yields interpretable rank intervals.

S2. Thorough and well-organized empirical study. The synthetic suite has an explicit post-fit oracle, separates identification from certification (Fig. 1), stress-tests validity (Case 4) and heteroscedasticity (Case 5), includes a bias–variance diagnostic matching Proposition 3 (Fig. 3), a seven-localizer sweep over nuisance dimensions (Table 6), a score-family ablation (Fig. 8), and both regression and classification real-data tasks. Gain-guided fine-tuning of the TabPFN head is a sensible idea that gives consistent, if modest, power improvements over the pretrained head at every dimension.

Critical weaknesses

The certified object does not match the stated target, and the error metric is close to vacuous when the method is selective. Definition 1 targets the expected winner μ(x0)<0\mu(x_0)<0, but Proposition 1 certifies that the realized score is negative, i.e. effectively a (1−α)(1-\alpha)-quantile winner. With noisy scores these differ sharply: Table 4 shows quantile-winner accuracy of only 0.414 (Case 4) and 0.651 (Case 5), and in Case 2 the quantile winner collapses to the global mean. Moreover the controlled quantity P(declare S≥0)≤α\mathbb{P}(\text{declare}~ S\ge 0)\le\alpha has all test points in the denominator (App. B.2). The error rate among declarations is only bounded by α/selection rate\alpha/\text{selection rate}, which is ≥1\ge 1 whenever selection ≤α\le\alpha. The paper cites Jin & Candès (2023) but does not control an FDR-type error among declared winners, and the "false winner rate" of non-conformal baselines (which target μ\mu, not SS) is measured against the realized score, which makes the comparison in Fig. 2 apples-to-oranges: Local non-CP has 0.993 expected-winner accuracy in Case 4 yet is reported as "over-declaring."

Questions for authors

Q1. Please report the error rate conditional on declaring, P(S≥0∣w^α=A)\mathbb{P}(S\ge0\mid\hat w_\alpha=A) and P(μ(X)>0∣w^α=A)\mathbb{P}(\mu(X)>0\mid\hat w_\alpha=A), per case and dataset. How do you reconcile marginal control with selection rates at or below α\alpha?

Q2. Definition 1 targets μ(x0)<0\mu(x_0)<0 but Proposition 1 certifies the (1−α)(1-\alpha)-quantile of the realized score. Why not calibrate for the expected winner directly (e.g., Giacomini–White, a local CLT on neighborhood-averaged scores, or e-values), or reframe the contribution as quantile-winner certification? Case 4's quantile-winner accuracy of 0.414 suggests the two can be far apart.

Q3. Can you add external baselines — conditional predictive ability tests, a kernel-weighted tt-test on SS, and Jin & Candès-style conformal selection with FDR control — on at least the synthetic cases?

Q4. For gain fine-tuning: ablate the 5% risk caps and TT; report whether fine-tuning changes the conditional error among declarations; give compute cost; and state whether the Table 1 gains over pretrained TabPFN are significant given overlapping standard deviations.

Q5. Real data: what fraction of Local CP declarations have positive realized gain, and how does conditional gain compare to a non-CP rule thresholded to the same selection rate?

Q6. The models A,BA,B are trained on ∣Ifit∣=n/4|I_{\text{fit}}| = n/4. How should a practitioner transfer the winner map to models retrained on all data? Have you tried cross-fitting / jackknife+ to recycle IfitI_{\text{fit}}?

Q7. For M>2M>2 (App. A.5), how fast does the max-residual quantile inflate with M(M−1)M(M-1) pairs? Fig. 7 shows LR/MLP mostly undetermined even for M=3M=3.

Q8. How was TabPFN used as a localizer on BlogFeedback (∣Iest∣≈13|I_{\text{est}}|\approx 13k) — was the in-context set subsampled, and does this affect power?

3 comments
AuthorsOpenReview ↗

Response to Reviewer KzfX (1/3)

We thank the reviewer for a precise summary, and for finding the problem formulation clear and the empirical study thorough.

W(a), Q2 (certified object versus target).

We have clarified the distinct roles of μ(x)\mu(x) and the realized score SS in the revised text. Specifically, μ(x)\mu(x) is the local comparison target estimated by the localizer, while the conformal step certifies the realized score. Thus, the certified object is closer to a quantile winner, or an expected winner with a margin.

This brings two benefits. First, the rule is stricter than the expected-winner condition μ^(x)<0\hat\mu(x)<0, as each declaration requires the estimated advantage to exceed the margin s^(x)q^1−α\hat s(x)\hat q_{1-\alpha}, which is positive in all our experiments. Second, certifying SS directly provides a distribution-free, finite-sample guarantee without relying on asymptotic approximations or negligible-smoothing-bias assumptions.

W(b), W(c), Q1 (error among declarations).

(i) Realized-score error P(S≥0∣declared)\mathbb P(S\ge0\mid\text{declared}) is comparable to or below that of an oracle map. For any covariate-based rule, realized outcomes carry irreducible observation noise. Even an oracle map declaring the exact region {μ(x)<0}\{\mu(x)<0\} incurs this noise. Across five seeds at α=0.10\alpha=0.10, the error of our rules is comparable to or below this reference (real-data results in Appendix D.6 and Q5):

CaseLocal CPLocal scaled CPOracle map
10.0060.0070.005
20.274–0.257
30.0900.0630.129
40.1240.1280.200
50.5710.0030.180

(Note: in Case 5, the unscaled Local CP declares only 14 points over the five seeds. Heteroscedastic noise makes the scaled rule essential, reducing the conditional error to 0.003.)

(ii) Expected-winner error P(μ(X)>0∣declared)\mathbb P(\mu(X)>0\mid\text{declared}) highlights the necessity of conformal calibration. Evaluating all methods directly on the expected-winner target μ(x)\mu(x) shows that the uncalibrated non-conformal rule (μ^(x)<0\hat\mu(x)<0) has a high error when the true winner region is localized (Cases 2 and 3), whereas our conformal rules reduce μ\mu-errors to near zero:

CaseLocal CPLocal scaled CPNon-conformal map
10.0020.0010.004
20.068–0.599
30.0000.0000.290
40.0000.0000.007
50.0000.0020.080
AuthorsOpenReview ↗

Response to Reviewer KzfX (2/3)

Q3 (external baselines and FDR calibration).

We evaluated the suggested baselines alongside our framework at d=1d=1 and d=100d=100, averaged over the five cases at α=0.10\alpha=0.10 (per-case breakdown in Appendix D.4):

  • Kernel tt-test, BH. The Giacomini–White conditional test localized with a kernel and BH-corrected across test points, targeting μ(x)\mu(x).
  • Batch FDR calibration. We apply conformal selection (Jin and Candès, 2023) directly to conformal p-values from our TabPFN localizer, targeting FDR among declarations on SS.
MethodPower, d=1d=1Power, d=100d=100P(μ>0∣decl)\mathbb P(\mu>0\mid\text{decl}), d=1d=1P(μ>0∣decl)\mathbb P(\mu>0\mid\text{decl}), d=100d=100
Kernel tt-test, BH0.820.770.050.21
TabPFN localizer + Batch FDR CP0.550.530.0150.015
TabPFN localizer + Local scaled CP0.450.430.0010.001

The results show the following:

  • High-dimensional robustness versus the kernel tt-test. While the kernel tt-test achieves higher power by targeting the mean asymptotically, its validity relies on a negligible-smoothing-bias condition. With nuisance covariates (d=100d=100), kernel smoothing bias inflates its false expected-winner rate to 21%, whereas our TabPFN-based conformal rule maintains 0.001 error and stable power (0.45 → 0.43).
  • Online point-by-point versus offline batch deployment. Our framework directly supports FDR control via conformal selection (Jin and Candès, 2023), which achieves higher average power by spending the full α=0.10\alpha=0.10 budget among declarations. The two approaches are complementary in deployment: BH-based conformal selection requires an offline batch of test points where each decision depends on the rest of the batch, whereas our rule makes instant, point-by-point online decisions independently for each incoming xx.

Q4 (gain fine-tuning).

(i) Significance. The seeds differ in data and split, so we compare the two heads paired by seed. TabPFN-gain improves the five-case average power of pretrained TabPFN in 20 of 20 seed–dimension pairs (paired tt-test p=0.017p=0.017, 0.030, 0.033 and 0.016 at d=1,10,30,100d=1,10,30,100). Over the 100 combinations of case, seed and dimension, power is higher in 66 and lower in only 1. Power of Local scaled CP, five-case average:

Localizerd=1d=1d=10d=10d=30d=30d=100d=100
TabPFN (pretrained)0.4040.3850.4340.394
TabPFN-gain0.4510.4250.4690.428
Default forest0.4280.4240.4330.415

The relative gain is largest in the high-noise Case 4, where power nearly doubles (per case in Appendix B.5).

(ii) Error among declarations. The error against the expected winner and the realized error are nearly unchanged by fine-tuning (Appendix D.5).

(iii) Compute. Training the head is a one-time cost per dataset, from 11 s to 291 s on one A100 GPU (Appendix D.5).

(iv) Caps and temperature. We reran the gain training with caps of 2% and 10% and with temperatures of 0.05 and 0.2. Power of Local scaled CP, five-case average (Appendix D.5):

Settingd=1d=1d=100d=100
Caps 2%0.4410.415
Caps 5% (paper)0.4510.428
Caps 10%0.4630.433
T=0.05T=0.050.4520.432
T=0.1T=0.1 (paper)0.4510.428
T=0.2T=0.20.4460.424

Every setting is within one standard deviation across seeds of the setting used in the paper (0.038 at d=1d=1, 0.027 at d=100d=100), and every setting stays above the pretrained head (0.404 and 0.394). Thus, gain fine-tuning yields consistent improvements that are not sensitive to these hyperparameters.

AuthorsOpenReview ↗

Response to Reviewer KzfX (3/3)

Q5 (real data).

At the points declared by Local CP, the linear model has the smaller realized loss 74% of the time on average over the seven datasets, against 34% over all test points (Appendix D.6). At the same selection rate, the non-conformal map ranks points similarly well because both rank points by a TabPFN center (Appendix D.6). Our conformal calibration determines this cutoff automatically, without knowing the selection rate in advance.

Q6 (models retrained on all data; cross-fitting).

Because the guarantee applies to any two given fitted models, IfitI_{\mathrm{fit}} is not needed when AA and BB are pretrained. When AA and BB are refitted on more data, empirically transferring the learned declarations still keeps the false winner rate below α\alpha (Appendix D.8). Maintaining a formal guarantee for refitted models requires either a fresh calibration split or cross-fitting/jackknife+ (Barber et al., 2021), which we note as an extension.

Q7 (more than two models).

The joint threshold is a quantile of a maximum over the pairs, so for light-tailed residuals it grows at most on the order of log⁡K\sqrt{\log K} in the number KK of pairs. With M=3M=3 it is 1.61, against 1.07 for a single comparison (Appendix D.8). In Figure 7, LR and MLP are ordered at 80% of the test points; the order changes along xx, so the two models swap ranks 2 and 3.

Q8 (TabPFN on BlogFeedback).

No subsampling at inference. The in-context set uses the full training part of IestI_{\mathrm{est}}, and test queries are processed in batches without affecting power. Subsampling of queries is used only during head fine-tuning; the revision states this.

We hope these new results and clarifications address all of your questions.

Reviewer_aRUXOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes \textit{conformalized local model comparison} to identify covariate regions where one fitted predictor outperforms another. Given a point-wise loss differential S(X,Y)S(X,Y) (e.g., LA(X,Y)−LB(X,Y)L_A(X,Y) - L_B(X,Y)), local superiority is defined by the sign of μ(x)=E[S(X,Y)∣X=x]\mu(x) = \mathbb{E}[S(X,Y)\mid X=x] (plus majority and quantile variants). Using a three-way split (IfitI_{\text{fit}} to train models, IestI_{\text{est}} to fit a local center-scale model (μ^,s^)(\hat{\mu}, \hat{s}), and IcalI_{\text{cal}} to calibrate residuals Ri=(Si−μ^(Xi))/s^(Xi)R_i = (S_i - \hat{\mu}(X_i))/\hat{s}(X_i)), the method declares model AA superior at x0x_0 if the one-sided split-conformal bound U^α(x0)=μ^(x0)+s^(x0)q^1−α<0\hat{U}_\alpha(x_0) = \hat{\mu}(x_0) + \hat{s}(x_0)\hat{q}_{1-\alpha} < 0, ensuring P(U^α(Xn+1)<0,Sn+1≥0)≤α\mathbb{P}(\hat{U}_\alpha(X_{n+1}) < 0, S_{n+1} \ge 0) \le \alpha (Proposition1). The localizer is instantiated with TabPFN-3.5, fine-tuning its regression head via an episodic Lagrangian objective (\texttt{TabPFN-gain}). Section5 provides Nadaraya-Watson sign consistency (Theorem1), a global-vs.-local prevalence counterexample (Proposition2), and an \textit{ex-ante} MSE bias-variance decomposition (Proposition~3), complemented by synthetic and tabular experiments.

Critical strengths

\begin{enumerate}[leftmargin=1.5em, itemsep=0.3em] \item \textbf{[Priority 1] Relevant Problem Formulation:} Moving beyond global average loss to identify local covariate regions where a structured model beats a black-box predictor addresses a genuine practical need. \item \textbf{[Priority 2] Simple, Loss-Agnostic Wrapper:} Applying one-sided split conformal calibration to loss differences S(X,Y)S(X,Y) unifies regression, classification, and partial ordering of M>2M > 2 models (Appendix~A.5). \end{enumerate}

Critical weaknesses

\begin{enumerate}[leftmargin=1.5em, itemsep=0.4em] \item \textbf{[Priority 1] Minimal Methodological & Theoretical Novelty:} \begin{enumerate}[label=1.\arabic*, leftmargin=1.8em, itemsep=0.2em] \item \textit{Trivial Split-Conformal Application & Uninformative Error Denominator:} Proposition1 is a direct application of standard split-conformal quantile coverage to Ri=(Si−μ^(Xi))/s^(Xi)R_i = (S_i - \hat{\mu}(X_i))/\hat{s}(X_i). Crucially, the controlled ``false winner rate'' P(w^α=A,S≥0)≤α\mathbb{P}(\hat{w}_\alpha = A, S \ge 0) \le \alpha includes \textbf{abstentions in the denominator} (Table3) rather than controlling selective error conditional on declaration, P(S≥0∣w^α=A)\mathbb{P}(S \ge 0 \mid \hat{w}_\alpha = A). When selection rates are 2–10%2\text{--}10\% (Figure5), bounding the joint probability at α=0.10\alpha=0.10 leaves the actual false discovery rate among declared regions uncontrolled. \item \textit{Textbook & Disconnected Theory (Section5):} Theorem1 (and its 2-page proof in AppendixA.2) re-proves standard Nadaraya-Watson consistency, which is never used in the main method and fails in high dimensions dd due to OP((nrnd)−1/2)\mathcal{O}_P((n r_n^d)^{-1/2}) rates and lack of adaptivity (k-NN collapses in Table6). Proposition2 is an elementary expectation-vs.-sign example, and Proposition3 is the textbook MSE bias-variance identity proved by reintroducing training randomness (\textit{ex-ante}), contradicting the fixed-predictor (\textit{ex-post}) conditioning of Sections3--4. No theory quantifies how finite-sample estimation errors ∣μ^(x)−μ(x)∣|\hat{\mu}(x) - \mu(x)| and ∣s^(x)−s(x)∣|\hat{s}(x) - s(x)| degrade power or conditional coverage. \end{enumerate}

\item \textbf{[Priority 2] Unjustified \texttt{TabPFN-gain} Overhead, Unassessed Model Switching, and Missing Literature:}
\begin{enumerate}[label=2.\arabic*, leftmargin=1.8em, itemsep=0.2em]
    \item \textit{Opaque \& Empirically Unjustified \texttt{TabPFN-gain} Fine-Tuning:} Section~4.2 leaves the index set $I$ in $\mathcal{R}_{\text{MSE}}, \mathcal{R}_G$ undefined and fails to justify episodic fine-tuning over vanilla in-context learning (ICL) or classical forests. In Tables~1 and~6, a default Random Forest (\texttt{Forest}) matches \texttt{TabPFN-gain} at $d=100$ ($0.415 \pm 0.021$ vs.\ $0.428 \pm 0.027$), beats pretrained \texttt{TabPFN} ($0.394$), outperforms \texttt{TabPFN-gain} in Case~3 at $d=100$ ($0.722$ vs.\ $0.636$), and achieves non-zero power in Case~2 where both \texttt{TabPFN} variants score $0.000$. Adaptive local moment theory from Generalized Random Forests (Athey et al., 2019) is ignored.
    \item \textit{No Evaluation of Model Switching/Aggregation or Abstention:} Despite motivating the paper via model routing, the authors never evaluate a switched predictor combining $\hat{f}_A$ and $\hat{f}_B$ to test whether opportunistic switching improves global test risk or preserves post-selection predictive coverage on $Y$, nor do they compare against Giacomini \& White (2006) decision rules or Localized Conformal Model Selection (\texttt{LCP-MS}, Wang \& Wang, 2026). The 4-way split ($\lfloor n/4 \rfloor$) also starves small datasets (e.g., \textit{Auto MPG}, $n=398$), causing erratic curves (Figure~5) and massive undiscussed abstention (Figure~4).
    \item \textit{Omission of Regional Explainability Literature:} Partitioning the covariate space by local predictive loss differences is directly connected to regional explanations and local/regional variable importance (e.g., \textit{regional explanations: bridging local and global variable importance}), which is neither cited nor compared.
\end{enumerate}

\end{enumerate}

Questions for authors

\begin{enumerate}[leftmargin=1.5em, itemsep=0.25em] \item \textbf{[Priority 1 --- Selective Risk & Abstention]:} Since P(w^α=A,S≥0)≤α\mathbb{P}(\hat{w}_\alpha = A, S \ge 0) \le \alpha includes abstentions in the denominator, what is the empirical selective error rate P(S≥0∣w^α=A)\mathbb{P}(S \ge 0 \mid \hat{w}_\alpha = A) when selection rates are low (<10%<10\% in Figure5), why not control this selective risk directly, and what is the decision rule when both one-sided bounds abstain? \item \textbf{[Priority 2 --- Opportunistic Switching & Aggregation]:} If test points are routed between f^A\hat{f}_A and f^B\hat{f}_B using your conformal declarations, does the resulting hybrid predictor improve overall test loss compared to f^A\hat{f}_A alone, uncalibrated local switching (μ^(x)<0\hat{\mu}(x)<0), Giacomini & White (2006), and Wang & Wang (2026)? \item \textbf{[Priority 3 --- \texttt{TabPFN-gain} Justification]:} What is the index set II in RMSE\mathcal{R}_{\text{MSE}} and RG\mathcal{R}_G (Sec.4.2)? Given that a default Random Forest matches \texttt{TabPFN-gain} at d=100d=100 (Table1) and \texttt{TabPFN-gain} has 0.0000.000 power in Case2 (Table6), what justifies the complexity of episodic Lagrangian fine-tuning? \item \textbf{[Priority 4 --- Estimation Error Impact]:} How do estimation errors ∣μ^(x)−μ(x)∣|\hat{\mu}(x)-\mu(x)| and ∣s^(x)−s(x)∣|\hat{s}(x)-s(x)| impact the variance of q^1−α\hat{q}_{1-\alpha}, conditional coverage, and the high across-seed instability in Figure4? \end{enumerate}

3 comments
AuthorsOpenReview ↗

Response to Reviewer aRUX (1/3)

We thank the reviewer for the detailed comments and for recognizing the practical importance of local model comparison.

W1.1, Q1 (selective risk, abstention, and novelty).

  • Realized selective error is comparable to or below that of an oracle map. To directly evaluate selective risk, we now report P(S≥0∣declared)\mathbb P(S\ge0\mid\text{declared}) across all cases and datasets (Appendix D.3 and D.6). Because covariate-based rules decide before observing YY, realized outcomes carry irreducible noise, captured by an oracle map declaring the exact region {μ(x)<0}\{\mu(x)<0\}. Across five seeds at α=0.10\alpha=0.10, the error of our rules is comparable to or below this reference:
CaseLocal CPLocal scaled CPOracle map
10.0060.0070.005
20.274–0.257
30.0900.0630.129
40.1240.1280.200
50.5710.0030.180

(Note: in Case 5, the unscaled Local CP declares only 14 points over the five seeds. Heteroscedastic noise makes the scaled rule essential, reducing the conditional error to 0.003.)

  • Direct FDR control within our framework. Controlling selective risk among declarations is directly supported on the same localizer: applying BH to conformal p-values (Jin and Candès, 2023) controls FDR at α\alpha in offline batch settings, complementing our instant point-by-point online rule (Appendix D.4).
  • Decision rule under abstention. We declare AA when U^α(x)<0\hat U_\alpha(x)<0 and BB when the symmetric lower bound is positive; when neither holds, the rule abstains. Each direction has its own level-α\alpha guarantee, and calibrating the two directions jointly (Appendix A.5) gives a single α\alpha for both.
  • Methodological novelty. The split-conformal step is deliberately lightweight. It requires only exchangeability, so finite-sample marginal control holds for any comparison score and any localizer fitted on the estimation split. Our core contribution is a framework that brings this guarantee to local model comparison: the comparison score is centered and scaled conditionally on xx, and the localizer is a TabPFN whose regression head is trained specifically for conformal gain.

W1.2, Q4 (theory and estimation error).

We have clarified the distinct roles of μ(x)\mu(x) and the realized score SS in the revised text. Specifically, μ(x)\mu(x) is the local comparison target estimated by the localizer, while the conformal step certifies the realized score. Thus, the certified object is closer to a quantile winner, or an expected winner with a margin.

Section 5 studies the target μ(x)\mu(x). Theorem 1 shows that a neighborhood-average localizer recovers the sign of μ(x)\mu(x), Proposition 2 shows that an aggregate comparison can disagree with the local picture, and Proposition 3 relates the sign of μ(x)\mu(x) to the bias and variance of the two models under squared loss. We have revised Section 5 to state the role of each result and its relation to the guarantee of Proposition 1.

Impact of estimation error (μ^,s^)(\hat\mu,\hat s) and seed variability. Finite-sample validity holds regardless of estimation error in μ^\hat\mu or s^\hat s, as held-out calibration residuals absorb localizer inaccuracy. Estimation error affects power rather than validity: noisier estimates typically inflate q^1−α\hat q_{1-\alpha} and yield fewer declarations, which is what our localizer comparison (Table 6) measures. In Figure 4, across-seed variance stems from two sources: (1) smaller calibration splits on compact datasets (Auto MPG, Concrete), and (2) refitting both the MLP and linear model per seed, which shifts the ground-truth winner map μ(x)\mu(x) itself across seeds.

AuthorsOpenReview ↗

Response to Reviewer aRUX (2/3)

W2.1, Q3 (TabPFN-gain).

(i) Definition of II. The index set II consists of the fold queries from the training part of IestI_{\mathrm{est}}. The revision defines it explicitly.

(ii) Value of fine-tuning. Fine-tuning directly improves upon the pretrained head it modifies. Because random seeds vary in both data draws and splits, we compare the two heads paired by seed. TabPFN-gain improves the five-case average power over pretrained TabPFN in all 20 seed–dimension pairs (paired tt-test p=0.017p=0.017, 0.030, 0.033 and 0.016 at d=1,10,30,100d=1,10,30,100). Over the 100 combinations of case, seed and dimension, power is higher in 66 and lower in only 1. Power of Local scaled CP, five-case average:

Localizerd=1d=1d=10d=10d=30d=30d=100d=100
TabPFN (pretrained)0.4040.3850.4340.394
TabPFN-gain0.4510.4250.4690.428
Default forest0.4280.4240.4330.415

The error among declarations is nearly unchanged by fine-tuning (Appendix D.5).

(iii) Comparison with random forest. Validity does not depend on the localizer, and Table 6 compares localizers under one calibration. TabPFN-gain has the highest five-case average at d=10d=10, 30 and 100. Per case the ranking varies: the forest is ahead in Case 3 at d=100d=100, while in the heteroscedastic Case 5 its power falls with dd and that of TabPFN-gain does not (Appendix B.5). We use TabPFN because it returns a predictive distribution, from which the majority and quantile winner maps are read, and because its head is differentiable and can be trained for the calibrated objective. TabPFN also needs no per-dataset hyperparameter tuning, whereas the power of tree localizers depends on their configuration.

(iv) Zeros in Case 2. The zeros in Case 2 belong to the scaled rule and follow from its target. The scaled rule is close to a quantile winner, and in Case 2 the noise is large relative to the advantage of AA at every xx, so that region is empty at α=0.10\alpha=0.10. With the same TabPFN-gain center, the unscaled Local CP declares in Case 2 at both d=1d=1 and d=100d=100 (Appendix D.5).

(v) Generalized random forests. Generalized random forests can serve as the localizer of our rule. Their pointwise intervals for μ(x)\mu(x) are asymptotic and require the forest's bias to vanish faster than its standard error (Wager and Athey, 2018). The revision states this distinction.

AuthorsOpenReview ↗

Response to Reviewer aRUX (3/3)

W2.2, Q2 (switching, Giacomini–White, LCP-MS, small samples).

(i) Switched predictor. The non-conformal map switches whenever the alternative model has a lower estimated loss, whereas Local CP switches only on the certified subset where false switches are controlled. Using the MLP as the default model and switching to linear regression on declared points (α=0.10\alpha=0.10, five seeds; Appendix D.6), we report the percentage of test points switched and the relative change in the mean test MSE against the MLP alone:

RuleConcreteAuto MPGBlogFeedbackFacebook
Non-conformal map52% / −9.8%58% / −14.9%41% / −8.4%27% / −4.5%
Local CP13% / −7.0%4% / −1.4%8% / −7.5%13% / −4.3%
Share of the non-conformal risk reduction kept71%10%90%95%

Local CP lowers the test MSE on all four datasets with fewer switches. On Concrete, BlogFeedback and Facebook, it keeps 71%–95% of the risk reduction of the non-conformal map while switching one-fifth to one-half as many points. On Auto MPG, with about 100 calibration points, it declares 4% of the points.

(ii) Giacomini and White (2006). We ran its local form, the kernel-weighted test of μ(x)\mu(x) at each test point with BH across test points (Appendix D.4). While targeting the mean yields more declarations, its validity relies on negligible-smoothing-bias assumptions. With 99 nuisance covariates (d=100d=100), 21% of its declarations are wrong, compared with 0.1% for our rule.

(iii) LCP-MS (Wang and Wang, 2026). It selects, at each xx, the model whose localized conformal interval for YY is shortest, and its guarantee is coverage of YY. It does not bound how often the selected model has the larger loss, which is the error our rule controls.

(iv) Small samples. In standard deployment where pretrained models AA and BB are given, IfitI_{\mathrm{fit}} is not required and all data goes to estimation and calibration. Even under the full four-way split at n=400n=400 (about 100 calibration points, as in Auto MPG), our conformal rules keep the false winner rate below α\alpha (Appendix D.8). The revision discusses sample size.

W2.3 (regional explanations).

Regional explainability methods (e.g., Amoukou and Brunel, 2025) partition the feature space into regions of homogeneous variable importance to interpret a single model. In contrast, our framework identifies certified regions of predictive superiority between competing models with finite-sample error control, addressing a distinct decision problem. The revision cites this line of work.

We hope these new results and clarifications address all of your questions.

Reviewer_Fb6FOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper aims to identify regions of the input space for local model comparison where one predictive model outperforms another. The authors propose a conformal framework that estimates the local mean and scale of the loss differences between two models and constructs one-sided conformal prediction bounds to declare local winners. The framework provides finite-sample marginal control of incorrect declarations w.r.p.t. realized loss differences. To improve declaration efficiency, the authors introduce a gain-guided fine-tuning procedure for TabPFN-3.5 that optimizes the realized gain from conformal decisions. The theoretical results establish asymptotic consistency of kernel-based local winner estimation, characterize discrepancies between global and local model superiority, and provide a bias–variance decomposition of local loss differences. Experiments on synthetic and real-world tabular datasets evaluate the proposed method in terms of false winner rates, selection efficiency, and the identification of local winner regions.

Critical strengths

The paper addresses the limitations of global model comparison by identifying regions where different models perform better, which is useful for model selection and routing. Besides, the proposed fine-tuning objective directly optimizes the gain from conformal decisions. Experiments show improvements over pretrained TabPFN across different settings.

Critical weaknesses

Mismatch between the winner definition and theoretical guarantees.

The paper defines a local winner based on the conditional expected loss difference, μ(x)=E[S∣X=x]\mu(x)=\mathbb{E}[S\mid X=x]. However, Proposition 1 only guarantees marginal control of false declarations with respect to the realized loss difference SS, rather than μ(x)\mu(x). Furthermore, Theorem 1 only establishes standard kernel regression consistency and does not address this mismatch for the proposed method. Therefore, the main theoretical guarantee does not directly support the paper's definition of local superiority.

Limited novelty and insufficient comparisons with prior work.

Existing conformal routing methods [1,2] have already studied input-dependent model selection with statistical guarantees, including approaches based on conformal bounds for loss differences. The paper does not sufficiently distinguish its contributions from these methods or compare against them experimentally. Moreover, the improvements from gain-guided fine-tuning over pretrained TabPFN are relatively modest, leaving its practical advantages unclear.

[1] Uddin I, Khider M, Bauer A. Proactive Routing to Interpretable Surrogates with Distribution-Free Safety Guarantees[J]. arXiv preprint arXiv:2603.14623, 2026.

[2] Uddin I, Bauer A. Conformal LLM Routing with Distribution-Free Safety Guarantees[C]//Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (ACL 2026). 2026: 791-799.

Questions for authors
  • Can the authors clarify how Proposition 1 & Theorem 1 support the identification of local expected winners defined in Definition 1?
  • How does the proposed method differ from existing conformal routing approaches [1, 2]?
  • Can the authors compare gain-guided fine-tuning against standard MSE/NLL fine-tuning of the same TabPFN regression head to isolate the contribution of the proposed gain-guided objective?
3 comments
AuthorsOpenReview ↗

Response to Reviewer Fb6F (1/3)

We thank the reviewer for recognizing the value of local model comparison. We appreciate the opportunity to clarify our theoretical guarantees, methodological contributions, and new experimental evidence.

W1, Q1. Relationship between Definition 1, Proposition 1, and Theorem 1

There is no contradiction: the two theoretical results address different questions, and the declaration rule connects them.

  • Definition 1 and Theorem 1: Define the expected local winner using μ(x)=E[S∣X=x]\mu(x)=\mathbb{E}[S\mid X=x]. Theorem 1 proves that a kernel-based estimator consistently identifies the true expected local winner as sample size increases, under the stated assumptions.

  • Proposition 1: Provides finite-sample marginal control of false declarations based on the realized loss difference, rather than the expected loss difference. This guarantee follows the standard split-conformal prediction framework. Our contribution is to adapt conformal calibration to local comparisons of the performance of two competing models, rather than constructing prediction bounds for a single model. As in standard conformal prediction, the guarantee is marginal over new test observations, not conditional on a fixed covariate value.

  • How they connect: The rule is stricter than the expected-winner condition μ^(x)<0\hat\mu(x)<0, as each declaration requires the estimated advantage to exceed the margin s^(x)q^1−α\hat s(x)\hat q_{1-\alpha}, which is positive in all our experiments. Empirically, the error against the expected winner, P(μ(X)>0∣declared)\mathbb P(\mu(X)>0\mid\text{declared}), is at most 0.068 for our rules in every synthetic case, against 0.599 and 0.290 for the non-conformal rule μ^(x)<0\hat\mu(x)<0 in Cases 2 and 3 (Appendix D.3).

This distinction is stated in Section 1 and immediately after Proposition 1 (Section 4.1) of the original paper, and we have made the different roles of the two results clearer in the revision.

AuthorsOpenReview ↗

Response to Reviewer Fb6F (2/3)

W2, Q2. Differences from existing conformal routing methods [1,2]

Thank you for pointing out these relevant studies. While both approaches use conformal methods to compare models, they address different research questions.

  • [1,2]: Safe model substitution. Their objective is to replace an expensive or complex model with a cheaper or simpler model whenever possible, while controlling excessive performance degradation among the substituted predictions. They train a binary safe/unsafe routing classifier and calibrate its threshold. Their conformal analysis is for the surrogate's violation rate on average over the routed set, rather than local winner identification.

  • Our paper: Local model superiority. We compare two competing models throughout the feature space, without assuming either is globally better, cheaper, or simpler. We estimate the continuous conditional loss difference to identify where each model performs better. We then apply locally scaled split conformal prediction to control false declarations on realized loss differences, marginally over test observations.

  • Additional contributions. We introduce gain-guided fine-tuning to improve conformal declaration power and extend the framework to multiple-model comparisons (Appendix A.5). We acknowledge that [1] also considers a regression-conformal baseline.

New experiments directly comparing with [1,2] (Appendix D.7). We implemented their regression-conformal baseline and binary routing method on our synthetic datasets. The table reports the fraction of the oracle expected-winner region recovered at α=δ=0.10\alpha=\delta=0.10.

CaseRidge CP [1]Routing [1,2]Our Local CPOur Scaled CP
1. Homogeneous1.001.000.930.99
2. Nonlinear tail0.000.000.410.00
3. Middle island0.000.000.880.64
4. High noise0.000.000.310.21
5. Heteroscedastic0.000.670.000.41

The key finding is that our approach identifies local winner regions missed by both baselines in Cases 2–4, particularly the nonlinear middle island (88% versus 0%). The methods have different statistical guarantees, so these results demonstrate complementary strengths rather than uniformly better safety.

We have added these related studies, clarified the methodological distinctions, and included the new experiments in the revised paper.

AuthorsOpenReview ↗

Response to Reviewer Fb6F (3/3)

W2, Q3 (gain-guided fine-tuning).

  • Consistent and statistically significant gains over pretrained TabPFN. Fine-tuning directly improves upon the pretrained head it modifies. Because random seeds vary in both data draws and splits, we compare the two heads paired by seed. TabPFN-gain improves the five-case average power over pretrained TabPFN in all 20 seed–dimension pairs (paired tt-test p=0.017p=0.017, 0.030, 0.033 and 0.016 at d=1,10,30,100d=1,10,30,100). Over the 100 combinations of case, seed and dimension, power is higher in 66 and lower in only 1. Power of Local scaled CP, five-case average:
Localizerd=1d=1d=10d=10d=30d=30d=100d=100
TabPFN (pretrained)0.4040.3850.4340.394
TabPFN-gain0.4510.4250.4690.428
Default forest0.4280.4240.4330.415

The relative gain is largest in the high-noise Case 4, where power nearly doubles. The error among declarations is nearly unchanged by fine-tuning (Appendix D.5).

  • Comparison of training objectives. We ran the comparison you asked for. The same TabPFN head is fine-tuned from the same pretrained weights, on the same data, with the same optimizer and learning rate, under three objectives. MSE and Gaussian NLL fine-tuning use standard early stopping on their own held-out loss. Power of Local scaled CP, five-case average (Appendix D.5):
Headd=1d=1d=100d=100
TabPFN (pretrained)0.4040.394
MSE fine-tuning0.4000.402
Gaussian NLL fine-tuning0.3870.372
Gain objective0.4510.428

The gain objective is above both alternatives in all five seeds at both dimensions (paired tt-test p=0.012p=0.012 and 0.031 against MSE, p=0.021p=0.021 and 0.013 against NLL, at d=1d=1 and d=100d=100). MSE and NLL fine-tuning lower their own held-out loss, yet their power stays close to that of the pretrained head, whereas the gain objective raises it by 0.03 to 0.05. The improvement therefore comes from the gain objective, not from fine-tuning itself.

We hope these clarifications and new empirical comparisons thoroughly address your concerns, and we welcome any further feedback.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers

We thank all four reviewers for their constructive comments. We have conducted additional experiments and revised the manuscript to clarify the theoretical contributions, practical implications, and relationship to prior work.

The central challenge: Conformal model comparison in high-dimensional feature spaces.

Our goal is to identify where two models differ in performance across the feature space XX, rather than merely comparing their global average performance. This is challenging because, with continuous high-dimensional XX, almost every observed xx is unique. We cannot directly apply split conformal calibration separately at each xx, and distribution-free pointwise conditional coverage is generally unattainable.

Our approach addresses this challenge by learning the local center and scale of the loss difference, then applying split conformal calibration to obtain finite-sample marginal validity. Gain-guided fine-tuning further improves the power of these local declarations.

Key clarifications and new evidence:

  • Theoretical guarantees (all reviewers): We clarify that μ(x)\mu(x) defines expected local superiority, while Proposition 1 controls false declarations on realized loss differences, marginally over XX.

  • Selective errors (KzfX, aRUX): We now report errors among declarations and errors relative to the oracle expected-winner map (Appendices D.3, D.6).

  • Prior work (Fb6F, KzfX, aRUX): We add comparisons with conformal routing, local mean tests, and conformal selection (D.4, D.7).

  • Gain-guided fine-tuning (Fb6F, KzfX, aRUX): Our method improves over pretrained TabPFN in all 20 paired seed–dimension comparisons and outperforms MSE/NLL fine-tuning in the reported matched experiments (D.5).

  • Practical value and robustness: We add model-switching evaluations, matched-selection comparisons, groupwise calibration, coverage diagnostics, and robustness analyses (D.1–D.2, D.6–D.8).

To address the reviewers' comments, in the revised PDF, Section 1 is rewritten for clarity, some sentence-level clarifications are added in the other sections, and the detailed results are in the new Appendix D.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.