- Rating
- 3
- Confidence
- 1
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes a difficulty-calibrated conformal prediction method for transfer learning. The proposed method assumes covariate shift between calibration and test data. Unlike existing conformal prediction methods for transfer learning, the proposed method uses the input difficulty score obtained by the source pre-trained model to provide model-agnostic uncertainty quantification. The proposed method retains the finite-sample marginal coverage guarantee of existing methods and, unlike them, can also be expected to reduce the conditional coverage error (CCE), which measures how far coverage within difficulty strata deviates from the nominal level. Experiments using synthetic and real-world datasets (two open-domain question-answering benchmarks and a simulation-to-real benchmark in polymer materials science) show the effectiveness of the proposed method.
Critical strengths
- First, my expertise is in transfer learning, and I'm not very familiar with conformal prediction. Thus, I'm not confident that I correctly understood the details of the contents in the paper. Nevertheless, I think the idea of using an input-difficulty function obtained from a source pre-trained model to perform target conformal prediction under covariate shift is interesting. The proposed method retains the theoretical guarantee of existing weighted conformal prediction and theoretically shows that it can reduce CCE under the difficulty transfer hypothesis (DTH). The experimental results provide empirical support for this theoretical claim. In addition, the proposed method has many attractive points: It can be applied to arbitrary pretrained and fine-tuned models without modifying their architectures or training objectives, requires neither paired source–target observations nor a shared output space.
Critical weaknesses
- In the theoretical explanation, the proposed method assumes DTH, that the target loss can be fully explained by a transformation of the source input-difficulty function. This assumption seems somewhat strong to me. Can the authors provide further discussion about this point? Further, the proposed method assumes that exact importance (density-ratio) weights can be available. However, density ratio estimation is generally difficult, especially when the input dimensions are high. Was this effect observed empirically in the experiments? In addition, importance weighting usually assumes that the support of the test distribution is contained within that of the calibration distribution. If this assumption is violated, do the theoretical guarantees and empirical performance of the proposed method still hold?
Questions for authors
- I would appreciate it if the authors could address the points raised in the Critical Weaknesses section.
5 comments
Response to Reviewer XL17 [2/2]
Q2: "the proposed method assumes that exact importance (density-ratio) weights can be available. However, density ratio estimation is generally difficult, especially when the input dimensions are high. Was this effect observed empirically in the experiments?"
Our real-data studies use estimated weights. In particular, the LLM experiments estimate density ratios using a logistic discriminator trained on 1024-dimensional E5 embeddings. We added two sensitivity studies.
LLM benchmarks (Table 17b), CCE, 30 seeds. is the inverse regularisation of the discriminator:
| Weights | NQ: WCP | LWCP | Mondrian | DC-CP | TriviaQA: WCP | LWCP | Mondrian | DC-CP |
|---|---|---|---|---|---|---|---|---|
| .157 | .153 | .147 | .057 | .103 | .099 | .097 | .031 | |
| (paper) | .145 | .134 | .068 | .059 | .102 | .099 | .091 | .038 |
| .065 | .065 | .090 | .070 | .092 | .088 | .090 | .066 | |
| , cross-fit | .149 | .138 | .089 | .063 | .104 | .099 | .094 | .037 |
| .158 | .154 | .147 | .057 | .103 | .099 | .097 | .030 |
Synthetic (Table 16b), coverage on target OOD (nominal 0.9):
| Weights | WCP (Ackley) | DC-CP (Ackley) | WCP (Levy) | DC-CP (Levy) |
|---|---|---|---|---|
| true | .832 | .901 | .801 | .889 |
| logistic, linear | .962 | .924 | .995 | .931 |
| GBM classifier | .877 | .913 | .888 | .902 |
| .945 | .922 | .996 | .897 |
The results indicate:
- DC-CP exhibits smaller variation in target OOD coverage across the tested weighting schemes (approximately –) than WCP (approximately –).
- On the LLM benchmarks, DC-CP keeps the lowest CCE except at on NQ-Open. There the discriminator nearly separates the two samples, and 57% (NQ-Open) and 32% (TriviaQA) of test points get infinite thresholds for every method.
Revision: Appendix D.10 (Lines 1804 and 1825); Section 2.3 (Line 167) now points to these tables.
Q3: "importance weighting usually assumes that the support of the test distribution is contained within that of the calibration distribution. If this assumption is violated, do the theoretical guarantees and empirical performance of the proposed method still hold?"
We agree that the support condition is essential for the standard weighted-CP validity guarantee.
- Theory. If the condition is violated, the guarantee does not hold, for DC-CP as for any weighted CP method. DC-CP changes only the score, so it neither fixes nor worsens a support violation.
- In practice. Poor overlap often shows up as a few extreme weights. For the affected test points, the weighted quantile can reach the atom and the method returns an infinite (trivial) interval. This is a conservative behaviour for those points, but it does not restore the guarantee under a support violation. In Table 17(b), a nearly separating discriminator () produces such infinite thresholds for every method.
- Two different overlap conditions. Our link-consistency result (Appendix B.3) uses overlap only in the one-dimensional . This does not replace the input-space condition , which is still needed for validity.
Revision: Section 2.1 (Line 91): "This requires , so that is defined on the test support; the guarantee below also assumes exact weights." Appendix B.3 (Line 908): "This difficulty-space overlap is sufficient for transferring link consistency and concerns only the one-dimensional ; the separate input-space condition required by weighted conformal validity remains necessary."
Thank you for the response and the additional results. Most of my concerns have been addressed. But, I have one remaining question. In the LLM benchmark experiments, showed the lowest CCE in the DC-CP. Does this mean that importance weighting is unnecessary in these experiments?
Response to Reviewer XL17 [1/2]
We thank the reviewer for the positive assessment and for raising these important questions about the assumptions and robustness of our method.
Q1: "the proposed method assumes DTH, that the target loss can be fully explained by a transformation of the source input-difficulty function. This assumption seems somewhat strong to me."
DTH is not required for the marginal-coverage guarantee. Since D, μT and h are estimated independently of the calibration set, DC-CP with weighted CP retains finite-sample marginal coverage for any D, with exact weights (Section 2.3, Line 167). DTH is used only to explain when conditional coverage improves (Theorem 1), and in practice it only needs to hold approximately.
To show what happens when DTH does not hold, we added a controlled sweep (Figure 9, Appendix D.10, Line 1760). A target-specific part of weight λ is mixed into the target error scale (λ=0: exact DTH; λ=1: D is uninformative). CCE under shift:
The results indicate:
Revision: Section 2.3 (Line 203): "In a controlled sweep (Figure 9), DC-CP matches LWCP while the target-specific share of the scale is at most 0.25 and falls behind it from 0.5." See also Section 5, Limitations (Line 482).