acceptodds
Under review as a conference paper at ICLR 2027

Difficulty-Calibrated Conformal Prediction for Transfer Learning

Abstract

State-of-the-art machine learning models, including foundation models, are predominantly non-Bayesian, and reliable methods for quantifying their predictive uncertainty remain underdeveloped. This challenge becomes particularly important in transfer learning, where models are adapted to downstream tasks with limited data. Conformal prediction (CP) provides model-agnostic uncertainty quantification with finite-sample coverage guarantees, but conventional CP does not exploit uncertainty-related information acquired during pretraining. We introduce Difficulty-Calibrated Conformal Prediction (DC-CP), a framework that transfers source-derived task difficulty to target-task calibration. DC-CP requires neither paired source–target observations nor a shared output space and learns only a one-dimensional link between source difficulty and target prediction error from limited target data. With exact covariate-shift weights, DC-CP combined with weighted CP retains finite-sample marginal coverage under distribution shift while adapting uncertainty estimates to input difficulty. Experiments on two open-domain question-answering benchmarks for large language models and a simulation-to-real polymer science benchmark demonstrate effective difficulty transfer across tasks and domains. Our results establish a model-agnostic framework for transferring not only predictive knowledge but also uncertainty-related information to downstream tasks.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
conformal prediction, transfer learning, uncertainty quantification, covariate shift, difficulty transfer, foundation models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_XL17OpenReview ↗
Rating
3
Confidence
1
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes a difficulty-calibrated conformal prediction method for transfer learning. The proposed method assumes covariate shift between calibration and test data. Unlike existing conformal prediction methods for transfer learning, the proposed method uses the input difficulty score obtained by the source pre-trained model to provide model-agnostic uncertainty quantification. The proposed method retains the finite-sample marginal coverage guarantee of existing methods and, unlike them, can also be expected to reduce the conditional coverage error (CCE), which measures how far coverage within difficulty strata deviates from the nominal level. Experiments using synthetic and real-world datasets (two open-domain question-answering benchmarks and a simulation-to-real benchmark in polymer materials science) show the effectiveness of the proposed method.

Critical strengths
  • First, my expertise is in transfer learning, and I'm not very familiar with conformal prediction. Thus, I'm not confident that I correctly understood the details of the contents in the paper. Nevertheless, I think the idea of using an input-difficulty function obtained from a source pre-trained model to perform target conformal prediction under covariate shift is interesting. The proposed method retains the theoretical guarantee of existing weighted conformal prediction and theoretically shows that it can reduce CCE under the difficulty transfer hypothesis (DTH). The experimental results provide empirical support for this theoretical claim. In addition, the proposed method has many attractive points: It can be applied to arbitrary pretrained and fine-tuned models without modifying their architectures or training objectives, requires neither paired source–target observations nor a shared output space.
Critical weaknesses
  • In the theoretical explanation, the proposed method assumes DTH, that the target loss can be fully explained by a transformation of the source input-difficulty function. This assumption seems somewhat strong to me. Can the authors provide further discussion about this point? Further, the proposed method assumes that exact importance (density-ratio) weights can be available. However, density ratio estimation is generally difficult, especially when the input dimensions are high. Was this effect observed empirically in the experiments? In addition, importance weighting usually assumes that the support of the test distribution is contained within that of the calibration distribution. If this assumption is violated, do the theoretical guarantees and empirical performance of the proposed method still hold?
Questions for authors
  • I would appreciate it if the authors could address the points raised in the Critical Weaknesses section.
5 comments
AuthorsOpenReview ↗

Response to Reviewer XL17 [1/2]

We thank the reviewer for the positive assessment and for raising these important questions about the assumptions and robustness of our method.

Q1: "the proposed method assumes DTH, that the target loss can be fully explained by a transformation of the source input-difficulty function. This assumption seems somewhat strong to me."

DTH is not required for the marginal-coverage guarantee. Since DD, μT\mu_T and hh are estimated independently of the calibration set, DC-CP with weighted CP retains finite-sample marginal coverage for any DD, with exact weights (Section 2.3, Line 167). DTH is used only to explain when conditional coverage improves (Theorem 1), and in practice it only needs to hold approximately.

To show what happens when DTH does not hold, we added a controlled sweep (Figure 9, Appendix D.10, Line 1760). A target-specific part of weight λ\lambda is mixed into the target error scale (λ=0\lambda=0: exact DTH; λ=1\lambda=1: DD is uninformative). CCE under shift:

λ\lambda00.250.50.751
WCP.070.059.054.059.069
LWCP.033.029.027.029.033
DC-CP.029.028.031.050.063

The results indicate:

  • DC-CP achieves the lowest CCE for λ≤0.25\lambda \le 0.25, but LWCP performs better for λ≥0.5\lambda \ge 0.5.
  • DC-CP outperforms WCP across the evaluated range. Even at λ=1\lambda=1, where DD carries no information, DC-CP is no worse than WCP (0.0630.063 vs 0.0690.069).
  • Exact DTH is not assumed or established in the real benchmarks (e.g., LLM answer entropy as DD), and DC-CP still has the lowest CCE there.

Revision: Section 2.3 (Line 203): "In a controlled sweep (Figure 9), DC-CP matches LWCP while the target-specific share of the scale is at most 0.25 and falls behind it from 0.5." See also Section 5, Limitations (Line 482).

AuthorsOpenReview ↗

Response to Reviewer XL17 [2/2]

Q2: "the proposed method assumes that exact importance (density-ratio) weights can be available. However, density ratio estimation is generally difficult, especially when the input dimensions are high. Was this effect observed empirically in the experiments?"

Our real-data studies use estimated weights. In particular, the LLM experiments estimate density ratios using a logistic discriminator trained on 1024-dimensional E5 embeddings. We added two sensitivity studies.

LLM benchmarks (Table 17b), CCE, 30 seeds. CC is the inverse regularisation of the discriminator:

WeightsNQ: WCPLWCPMondrianDC-CPTriviaQA: WCPLWCPMondrianDC-CP
C=0.01C=0.01.157.153.147.057.103.099.097.031
C=1C=1 (paper).145.134.068.059.102.099.091.038
C=10C=10.065.065.090.070.092.088.090.066
C=1C=1, cross-fit.149.138.089.063.104.099.094.037
w≡1w \equiv 1.158.154.147.057.103.099.097.030

Synthetic (Table 16b), coverage on target OOD (nominal 0.9):

WeightsWCP (Ackley)DC-CP (Ackley)WCP (Levy)DC-CP (Levy)
true ww.832.901.801.889
logistic, linear.962.924.995.931
GBM classifier.877.913.888.902
w≡1w \equiv 1.945.922.996.897

The results indicate:

  • DC-CP exhibits smaller variation in target OOD coverage across the tested weighting schemes (approximately 0.890.89–0.930.93) than WCP (approximately 0.800.80–1.001.00).
  • On the LLM benchmarks, DC-CP keeps the lowest CCE except at C=10C=10 on NQ-Open. There the discriminator nearly separates the two samples, and 57% (NQ-Open) and 32% (TriviaQA) of test points get infinite thresholds for every method.

Revision: Appendix D.10 (Lines 1804 and 1825); Section 2.3 (Line 167) now points to these tables.

Q3: "importance weighting usually assumes that the support of the test distribution is contained within that of the calibration distribution. If this assumption is violated, do the theoretical guarantees and empirical performance of the proposed method still hold?"

We agree that the support condition P~X≪PX\tilde P_X \ll P_X is essential for the standard weighted-CP validity guarantee.

  • Theory. If the condition is violated, the guarantee does not hold, for DC-CP as for any weighted CP method. DC-CP changes only the score, so it neither fixes nor worsens a support violation.
  • In practice. Poor overlap often shows up as a few extreme weights. For the affected test points, the weighted quantile can reach the +∞+\infty atom and the method returns an infinite (trivial) interval. This is a conservative behaviour for those points, but it does not restore the guarantee under a support violation. In Table 17(b), a nearly separating discriminator (C=10C=10) produces such infinite thresholds for every method.
  • Two different overlap conditions. Our link-consistency result (Appendix B.3) uses overlap only in the one-dimensional D(X)D(X). This does not replace the input-space condition P~X≪PX\tilde P_X \ll P_X, which is still needed for validity.

Revision: Section 2.1 (Line 91): "This requires P~X≪PX\tilde P_X \ll P_X, so that ww is defined on the test support; the guarantee below also assumes exact weights." Appendix B.3 (Line 908): "This difficulty-space overlap is sufficient for transferring link consistency and concerns only the one-dimensional D(X)D(X); the separate input-space condition P~X≪PX\tilde P_X \ll P_X required by weighted conformal validity remains necessary."

Thank you for the response and the additional results. Most of my concerns have been addressed. But, I have one remaining question. In the LLM benchmark experiments, w=1w=1 showed the lowest CCE in the DC-CP. Does this mean that importance weighting is unnecessary in these experiments?

Reviewer_tRsVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Difficulty-Calibrated Conformal Prediction (DC-CP), a framework that transfers a source-derived difficulty score into target-domain conformal calibration. The method learns a one-dimensional mapping between source difficulty and target prediction error scale and combines it with weighted conformal prediction under covariate shift. The authors provide theoretical analysis under a proposed Difficulty Transfer Hypothesis (DTH) and evaluate the method on experiments with multiple benchmarks.

Critical strengths
  • The paper explores a novel direction for a common challenge in transfer learning: leveraging uncertainty-related information acquired during pretraining to improve uncertainty quantification on downstream tasks.
  • The proposed framework imposes relatively weak assumptions, requiring neither paired source-target observations nor a shared output space.
  • The paper provides promising theoretical justification showing that the proposed approach can improve coverage calibration across different difficulty levels. The empirical evaluation spans multiple domains and offers encouraging evidence that transferred difficulty information can enhance uncertainty estimation under distribution shift.
Critical weaknesses
  1. The implementation and results are based on monotonic link: Although the framework is presented as allowing an arbitrary positive one-dimensional mapping (h), the actual implementation relies on isotonic regression and assumes a monotonic relationship between source difficulty and target loss. The paper does not discuss and experiment situations where this relationship may be non-monotonic.
  2. Dependence on labeled target data: The target tuning set and calibration set both require labeled target-domain observations to fit (h) and perform conformal calibration. This raises questions about the practical setting in which the method is most useful. Transfer learning is often motivated by scarcity of labeled target data, but the proposed approach still depends on separate labeled target subsets for tuning and calibration. The paper would benefit from a discussion of label requirements and sensitivity to target-data availability.
  3. Unclear role of weighted conformal prediction in the main setup: The paper motivates the method through covariate shift and repeatedly combines DC-CP with weighted conformal prediction. However, the construction of the calibration set suggests that calibration samples are already drawn from the target domain. If calibration and deployment data already originate from the same target distribution, it is unclear why weighted conformal prediction is needed. The relationship between the target tuning/calibration datasets and the deployment distribution should be clarified.
  4. The baselines are limited. The proposed method demonstrated reduced conditional coverage error across difficulty strata. However, the baseline methods (WCP, LWCP, and CQR) are primarily designed to achieve marginal coverage rather than balanced coverage across difficulty levels. Consequently, using these as baseline directly may not provide a fair comparison. A stronger baseline would be conformal procedures calibrated within difficulty groups or stratified/group-wise conformal prediction methods. Such comparisons would help isolate whether the gain comes from transferring difficulty information or simply from conditioning calibration on difficulty.
Questions for authors
  • In the paper setting, is there a covariate shift between the calibration set DTcalD_T^{cal} and test point (Xn+1,Yn+1)(X_{n+1}, Y_{n+1})?
5 comments
AuthorsOpenReview ↗

Response to Reviewer tRsV [1/3]

We thank the reviewer for the careful assessment and constructive suggestion. We first clarify the distributional setting, as this also addresses the concern raised in W3.

Q1: "In the paper setting, is there a covariate shift between the calibration set and test point?"

Yes. In our framework, "target" refers to the downstream task, not the test distribution. The tuning and calibration inputs follow PXP_X whereas the test inputs follow a distinct distribution P~X\tilde P_X:

  • LLM study: calibration questions and test questions come from different semantic clusters.
  • Polymer study: the test set consists of a held-out polymer class.
  • Synthetic study: the test inputs are shifted by μshift\mu_{\rm shift}.

This distinction motivates the use of weighted CP to account for covariate shift. Without shift (Tables 8 and 10), DC-CP reduces to split CP with the new score and w≡1w \equiv 1.

Revision: We have added the following sentence to Section 2.2 (Line 130): "Here, 'target' refers to the downstream task; the tuning and calibration inputs follow PXP_X, whereas test inputs follow P~X\tilde P_X." We also state in Section 3 (Line 279) that LWCP and CQR are fitted on PXP_X and applied on P~X\tilde P_X.

W3: "Unclear role of weighted conformal prediction in the main setup ... If calibration and deployment data already originate from the same target distribution, it is unclear why weighted conformal prediction is needed."

As clarified above, the calibration data are generated from the target task but not from the test distribution. Weighted CP corrects for this shift, while DC-CP only replaces the nonconformity score. We have clarified this distinction in Section 2.2 (Line 130) to make the role of importance weighting explicit.

W1: "The implementation and results are based on monotonic link ... The paper does not discuss and experiment situations where this relationship may be non-monotonic."

Monotonicity is not a requirement of the DC-CP framework; it is a modeling restriction of the isotonic link used in our default implementation. To address this, we newly evaluated a scenario where the target error scale σT\sigma_T is U-shaped with respect to the source difficulty, g(d)=1+16(d−0.5)2g(d) = 1 + 16(d-0.5)^2, while the source link remains monotone (Figure 10, Appendix D.10, Line 1782). All links are fitted on the same tuning set. CCE without shift:

ntunen_{\rm tune}501002004001600
WCP.094.093.093.093.094
LWCP.060.056.054.050.042
Mondrian.079.072.061.049.031
DC-CP, isotonic.105.097.089.088.085
DC-CP, spline.057.048.036.032.029
DC-CP, GBM.070.052.039.031.021

The results indicate:

  • Isotonic DC-CP cannot represent the non-monotone relation and therefore performs similarly to WCP.
  • Flexible alternatives such as one-dimensional spline or gradient-boosting machine (GBM) links accommodate the non-monotonic relationship. One of these variants achieves the lowest CCE at every ntunen_{\rm tune}.
  • Under covariate shift, spline and GBM links perform comparably to LWCP and Mondrian (e.g., 0.0430.043, 0.0460.046, 0.0450.045 and 0.0430.043 at ntune=400n_{\rm tune}=400).

Revision: Section 2.2 (Line 139) has been updated: "Flexible one-dimensional links can accommodate non-monotonic relations (Figure 10)."

AuthorsOpenReview ↗

Response to Reviewer tRsV [2/3]

W4: "The baselines are limited ... A stronger baseline would be conformal procedures calibrated within difficulty groups or stratified/group-wise conformal prediction methods. Such comparisons would help isolate whether the gain comes from transferring difficulty information or simply from conditioning calibration on difficulty."

We agree that group-wise calibration provides an important control for distinguishing the benefit of transferred difficulty information from the benefit of conditioning alone. Accordingly, we have added Mondrian conformal regression (Boström et al., 2020) to every comparison. It calibrates separately within KK equal-count bins of σ^\hat\sigma (the same target-side scale model as LWCP), fitted on the same tuning set using identical weights and calibration data (Appendix C.6, Line 1251). Thus, it conditions on difficulty without transferring it.

Language-model benchmarks (Tables 13–15), where CCE is the main metric, 30 seeds:

NQ-Open CCENQ-Open ∞\inftyTriviaQA CCETriviaQA ∞\infty
WCP.1450.1020
LWCP.1340.0990
Mondrian (K=5K=5).06833%.0914%
DC-CP.0590.0380

∞\infty: fraction of test points with an infinite threshold. The paired tt-statistics of DC-CP against Mondrian are −3.0-3.0 (NQ-Open) and −25.2-25.2 (TriviaQA).

Synthetic shift (Table 2), coverage on target ID / target OOD (nominal 0.9):

WCPCQRLWCPMondrianDC-CP
Ackley.972 / .832.826 / .995.947 / .861.963 / .864.908 / .901
Levy1.00 / .801.854 / .964.979 / .823.976 / .845.928 / .889

Polymer (Table 3). Each calibration set holds only about 15–55 points, so even with K=2K=2 bins, 50–86% of Mondrian thresholds are infinite.

The results indicate:

  • Conditioning on a target-side difficulty does not close the gap; Mondrian has higher CCE than DC-CP on both LLM benchmarks.
  • Mondrian improves CCE over WCP and LWCP, but at the cost of infinite thresholds for up to 33% of test points, which DC-CP never returns.
  • On the synthetic shift, Mondrian over-covers the in-distribution half and under-covers the shifted half, like LWCP.

Revision: Mondrian is added to Tables 2, 3, 5–14 and 17 and to Figures 3, 4, 9 and 10; see Section 3 (Line 281), Section 4 (Line 310), Section 4.1 (Lines 330, 342), Section 4.2 (Line 407) and Section 4.3 (Line 456).

AuthorsOpenReview ↗

Response to Reviewer tRsV [3/3]

W4 (continued): Signal or conditioning? To probe this, Table 15 also bins the transferred DD instead of σ^\hat\sigma ("Source-D Mondrian"), and varies the bin count KK (Appendix D.10, Lines 1738–1743).

NQ-Open CCE∞\inftyTriviaQA CCE∞\infty
Mondrian on σ^\hat\sigma, K=2/5/10K=2/5/10.122 / .068 / .0773% / 33% / 69%.099 / .091 / .0840% / 4% / 32%
Source-D Mondrian, K=2/5/10K=2/5/10.124 / .076 / .0681% / 17% / 44%.062 / .045 / .0570% / 1% / 12%
DC-CP.0590%.0380%

The results indicate:

  • With the same K=5K=5, binning the transferred DD instead of σ^\hat\sigma lowers TriviaQA CCE from 0.0910.091 to 0.0450.045. This supports the usefulness of the transferred difficulty signal for group-wise calibration, although the two variants allocate the target samples differently (Source-D Mondrian calibrates on all 865 points; Mondrian on σ^\hat\sigma uses 425 for tuning and 440 for calibration).
  • DC-CP is lower still. One explanation consistent with this result is that DC-CP shares a single threshold through a continuous link, whereas Mondrian splits the calibration points into bins.
  • Larger KK lowers CCE mainly by returning more infinite thresholds.
  • To be transparent: Source-D Mondrian can beat DC-CP when each bin has many effective samples (e.g., 0.0110.011 vs 0.0230.023 in the DTH-exact world; caption of Table 15). We state this in Section 5 (Line 476).

W2: "Dependence on labeled target data ... The paper would benefit from a discussion of label requirements and sensitivity to target-data availability."

Our LLM experiments do not require ground-truth answer labels. Neither tuning nor calibration uses reference answers or correctness labels; the nonconformity scores are computed entirely from the target model's own generated answers and log probabilities. The LLM results therefore show label-free difficulty calibration, not merely a reduction in the number of labelled examples. This is not specific to DC-CP, since the baselines in this study use the same model-generated scores. What is specific to DC-CP is how efficiently it uses the available target samples (see below).

For supervised regression and classification, DC-CP requires labelled target tuning and calibration data, as do the adaptive baselines LWCP, CQR and Mondrian. The difference is what DC-CP learns from the tuning set: a one-dimensional link hh, rather than a scale or quantile model on the full input space.

We further evaluated sensitivity to target-data availability:

  • LLM benchmarks (Table 17a): With total target-sample budgets of only 100, 400 or 865 unlabelled examples, DC-CP has the lowest CCE among all compared methods at every budget on both benchmarks.
  • Synthetic covariate shift (Table 9): DC-CP keeps its advantage with as few as ntune=20n_{\rm tune}=20 labelled target examples.

Thus, DC-CP can be used without ground-truth labels when model-generated scores are available. In the supervised settings studied here, it also retains its advantage with small target tuning sets (as few as ntune=20n_{\rm tune}=20).

Revision: Table 17(a) and its description (Appendix D.10, Line 1825).

Reviewer_zmjVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes an extension of conformal prediction, DC-CP, that transfers source-derived task difficulty to uncertainty quantification of target tasks. The source-side uncertainty information is calculated as a single scalar difficulty score and mapped to target prediction error using a small target tuning set, which is then used to normalize target prediction error and realize difficulty-adaptive conformal uncertainty.

Critical strengths
  1. The proposed DC-CP tries to transfer the uncertainty learned during the pretraining, which is a novel and interesting idea.

  2. The proposed approach is lightweighted and does not require paired source-target observations or a shared output space, which makes it practical to be deployed in resource limited applications.

  3. The experiment result covers a reasonable set of baselines and the result shows performance gain compared to the baselines.

Critical weaknesses
  1. The transfer of the uncertainty relies on the quality of D(x) and the h() mapping, where a held-out source dataset and a target tuning dataset are used. Although the cost is not the main issue here, the transferring performance could be sensitive to the coverage of the two sets. The paper should clarify on this.

  2. The setting of the experimental study is relatively simple. More challenging scenarios, such as those with more complex target-specific difficulties and less shared difficulty with source-derived difficulty.

  3. The proposed DC-CP has several major strong assumptions (e.g., distribution drift) and limitations (not guaranteed difficulty transfer and low-dimensional target link), which are also acknowledged in the paper. This make the scope of the work limited and the technical contribution of the paper incremental.

Questions for authors

Please check the listed weaknesses.

4 comments
AuthorsOpenReview ↗

Response to Reviewer zmjV [1/3]

We thank the reviewer for recognizing the novelty and practical relevance of our work. We have addressed each concern below and describe the corresponding revisions to the manuscript.

W1: "The transfer of the uncertainty relies on the quality of D(x) and the h() mapping, where a held-out source dataset and a target tuning dataset are used. ... the transferring performance could be sensitive to the coverage of the two sets."

We agree that the quality and coverage of both datasets may influence the effectiveness of difficulty transfer. We therefore conducted additional sensitivity analyses for the source held-out set used to construct DD and the target tuning set used to estimate hh.

(a) Source held-out set behind DD. In Table 16(a), we vary the source held-out residuals used to estimate DD, on the Ackley and Levy setting of Table 2, while keeping all other experimental settings fixed. We report coverage for the in-distribution (target ID) and the shifted (target OOD) subsets, with nominal coverage 0.90.9, over 100100 different random seeds.

Residual set behind DDWCP (Ackley) ID / OODDC-CP (Ackley) ID / OODWCP (Levy) ID / OODDC-CP (Levy) ID / OOD
25 points.972 / .832.956 / .8501.00 / .801.982 / .826
100 points.972 / .832.938 / .8611.00 / .801.954 / .851
600 points (paper).972 / .832.907 / .8971.00 / .801.917 / .892
4000 points.972 / .832.893 / .9191.00 / .801.917 / .894
inner 25% of source region.972 / .832.948 / .8511.00 / .801.985 / .820

The results indicate:

  • Reducing the size or diversity of the source residual set weakens the improvement of DC-CP (Ackley OOD decreases from 0.8970.897 to 0.8500.850).
  • Nevertheless, DC-CP remains closer to the nominal coverage level than WCP on the shifted subset.
  • Marginal validity is not affected, because DD is fixed before calibration.

Revision: We have added this to Appendix D.10 (Line 1804) and to the Limitations section (Section 5, Line 487).

AuthorsOpenReview ↗

Response to Reviewer zmjV [2/3]

(b) Target tuning set used to estimate hh. We investigated sensitivity to target-sample availability in two complementary settings.

Tuning-set size under covariate shift (Table 9), coverage on target OOD (nominal 0.9):

ntunen_{\rm tune}WCPLWCPMondrianDC-CP
Ackley 20.839.865.859.896
Ackley 400.841.868.882.897
Levy 20.808.819.826.860
Levy 400.807.832.861.897

Fixed total target-sample budget on the language-model benchmarks (Table 17a), CCE (lower is better), split evenly between tuning and calibration:

BudgetNQ: WCPLWCPMondrianDC-CPTriviaQA: WCPLWCPMondrianDC-CP
100 points.109.100.100.067.093.088.100.066
400 points.140.121.069.060.100.094.086.043
865 points.145.131.067.058.102.098.092.038

From the results, we observe:

  • Under shift, DC-CP retains its advantage even with ntune=20n_{\rm tune}=20.
  • On the LLM benchmarks, DC-CP achieves the lowest CCE at every budget. With a budget of 100 points, nearly all Mondrian thresholds are infinite.
  • One plausible explanation is that DC-CP estimates only a one-dimensional link hh, whereas LWCP and Mondrian rely on a scale model σ^\hat\sigma estimated from high-dimensional inputs (e.g., 1024-dimensional embeddings).
  • Without covariate shift on the synthetic benchmark functions, LWCP achieves lower CCE at every tested tuning-set size (Table 10, ntune=10n_{\rm tune}=10–400400). This finding is discussed in Section 5 (Line 472).

Revision: Table 17 and its description have been added to Appendix D.10 (Lines 1825–1836); Mondrian has been added to Tables 9 and 10.

AuthorsOpenReview ↗

Response to Reviewer zmjV [3/3]

W2: "The setting of the experimental study is relatively simple. More challenging scenarios, such as those with more complex target-specific difficulties and less shared difficulty with source-derived difficulty."

We agree that the relationship between source-derived and target-specific difficulty is an important factor in assessing the scope of the method. Our real-world benchmarks already involve substantial differences between source and target tasks. In particular, the polymer simulation-to-real study transfers difficulty from simulated refractive-index prediction to experimental nematic-phase classification. These tasks differ in both physical properties and data-generating processes, so difficulty transfer is far from guaranteed. The LLM study likewise transfers difficulty from the base model before fine-tuning to a fine-tuned model evaluated on held-out question clusters.

To directly evaluate the reviewer's concern, we added a controlled DTH-violation experiment (Figure 9, Appendix D.10, Line 1760), where

σT(X)=(1−λ) g(D⋆)+λ g(u),ℓT=σT(X)∣Z∣.\sigma_T(X)=(1-\lambda)\,g(D^\star)+\lambda\, g(u),\qquad \ell_T=\sigma_T(X)|Z|.

Here, uu is a target-specific difficulty, and λ\lambda controls how much difficulty is shared (λ=0\lambda=0: exact DTH; λ=1\lambda=1: none). We evaluated CCE against the true target error scale σT\sigma_T, rather than the source difficulty DD. CCE under shift (ntune=100n_{\rm tune}=100, 100 seeds):

λ\lambda00.250.50.751
WCP.070.059.054.059.069
LWCP.033.029.027.029.033
Mondrian.055.048.044.047.054
DC-CP.029.028.031.050.063
oracle σT\sigma_T.015.015.015.015.015

DC-CP achieves the lowest CCE for λ≤0.25\lambda\le 0.25 and outperforms WCP across the evaluated range, though it falls behind LWCP from λ=0.5\lambda=0.5. We also evaluated a non-monotone difficulty relation (Figure 10, Appendix D.10, Line 1782). Together, these experiments show both the practical challenge of transferring difficulty and the limits of its effectiveness.

W3: "The proposed DC-CP has several major strong assumptions (e.g., distribution drift) and limitations ... This make the scope of the work limited and the technical contribution of the paper incremental."

Here, we categorize the assumptions by their specific role.

  • Covariate shift is the standard setting of weighted CP (Tibshirani et al., 2019) and is shared by all baselines. DC-CP imposes no additional assumptions here. Without shift, DC-CP reduces to split CP with a new score and requires no weighting.
  • DTH is not required for validity. Since DD, μT\mu_T and hh are estimated independently of the calibration set, marginal coverage holds with exact weights for any DD and any positive hh (Section 2.3, Line 167). DTH serves strictly to explain when conditional coverage improves (Theorem 1). The newly added Figure 9 demonstrates how the performance gain diminishes when DTH is violated.
  • The monotone link belongs to our isotonic implementation, not to the method (Section 2.2, Line 139; Figure 10).

Conventional transfer-learning approaches primarily transfer predictions or learned representations. Our central contribution is not a new conformal validity theorem, but rather a framework to transfer uncertainty structure across different tasks even with different outputs (e.g., from simulated refractive index to experimental nematic phase), where source residuals cannot be reused as target residuals. DC-CP transfers uncertainty information to target calibration without requiring paired data or a shared output space, while preserving the guarantee of weighted CP. In the newly added experimental result, conditioning on a target-side difficulty (Mondrian) does not reach DC-CP's CCE on the LLM benchmarks (e.g., TriviaQA CCE 0.091 vs 0.038; Tables 13–15), which supports the value of the transferred signal.

Thanks the authors for the response and the additional experiments. My concerns regarding Weaknesses 1&3 are resolved, although I remain unconvinced of the novelty/contribution due to the strong assumptions made. For Weakness 2, the performance gain under more challenging settings seems to be marginal. For these reasons, I will keep my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.