- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper evaluates conformal prediction methods under distribution shift between source and target data. It proposes a width-matched evaluation protocol that compares methods using both source and target calibration data against a target-only conformal baseline at the same average prediction-set size. Experiments suggest that most gains in nominal coverage are attributable to wider prediction sets: gated source-reuse methods retain small benefits only when very few target labels are available, whereas pooling and label-free reweighting generally fail to improve over target-only calibration.
Critical strengths
- Careful empirical study. The experiments evaluates multiple TFMs, classical models, score functions, and real-shift benchmarks using paired comparisons. Its negative evidence—that target-only calibration is difficult to beat—is practically useful.
- Useful bias–variance diagnosis. The analysis shows that source reuse can reduce calibration-draw variance at small label budgets but becomes harmful when source bias dominates.
Critical weaknesses
-
Limited conceptual novelty. Comparing methods at equal prediction-set size is a natural extension of the classical coverage–efficiency tradeoff, so the proposed metric is not particularly surprising as a standalone contribution. Moreover, wider sets are not inherently undesirable, since practitioners may accept them for better coverage. Width-matched evaluation should therefore complement, rather than replace, nominal-level coverage, weakening the paper’s main novelty claim.
-
Insufficient justification for conformalizing TFMs and a missing native-uncertainty baseline. TFMs already provide posterior predictive distributions (PPDs), which prior work has found to be reasonably well calibrated marginally. However, the paper does not compare native TFM predictive sets or intervals with their conformalized counterparts, either in distribution or under shift. Consequently, it does not establish that conformal prediction is necessary or beneficial for uncertainty quantification when TFMs already provide native probabilistic uncertainty.
Questions for authors
- I would to see comparison with TFM PPD both in distribution and under shift setting
- Is prediction-set size matched separately for every setting and calibration draw, or only after averaging across settings and draws? If matching is aggregated, could reflect redistribution of width across heterogeneous settings rather than information gained from the source?
- Because SASC selects its mixing weight using the target calibration labels, its point version does not retain the usual finite-sample conformal guarantee. Does the reported advantage persist if weight selection and final quantile estimation use disjoint target samples?
- The “oracle temper” selects among estimated weights using target test labels but is not based on the true density ratio. How do the authors distinguish failure of covariate-shift correction from failure to estimate the density ratio accurately?
2 comments
Thank you for your detailed rebuttal and the substantial revisions. The native TFM baseline, width-matching procedure, and controlled experiments with known proportions address several of the concerns I raised in my first review.
I now view the paper primarily as an empirical study of calibration under tabular distribution shift. The technical novelty beyond standard coverage-efficiency comparisons remains limited, and the main findings are largely consistent with what one would expect from small calibration samples and poor overlap. Two changes would make the paper stronger.
-
If this is meant to be an empirical evaluation paper, the authors need to justify why the proposed evaluation protocol is a reasonable choice. I don’t think the paper does this yet.
-
Right now, the paper only reports how existing methods perform. It would benefit from more concrete practical guidance: when should practitioners reuse source data, when should they turn to methods that use unlabeled data, and when is it worth collecting target labels?
Based on the new evidence, I have raised my score from 1 to 2. In my view, addressing the remaining concerns would require reframing the study around a clear empirical contribution, justifying the evaluation protocol, and establishing what decisions the results can inform in practice. I see this as comparable in scope to developing a new paper, and I think a fresh submission would be more appropriate. My recommendation on the current submission therefore remains below the acceptance threshold.
Response to Reviewer iwUF
Response to Reviewer iwUF
Thank you for the careful review. We ran every analysis you asked for, on the submission's cached predictions. Changes are in blue; new analyses are in Appendix Z.
W1. Novelty; wider sets are not bad; the control should complement nominal coverage. We agree and now say so in Sec. 1. Equal-size comparison is standard practice; our claim is narrower. Source-reuse rules are justified by nominal-level gains, and our comparator asks whether target labels alone, at a higher level, buy the same width. Table 3 reports both readings. The control is now two-sided (see Q2), so it also credits rules narrower than naive.
W2 / Q1. Native TFM uncertainty (new Table 2, App. Z.1).
So in distribution conformal adds mainly a guarantee; under shift target labels fix coverage. At matched width the native set acts like the source-reusing rules: +0.004 to +0.011 per draw at k=50, but 0.033 to 0.040 below the control per setting.
Q2. Matching unit. The submitted match was pooled: one comparator level for all settings. We now also match per setting and per setting and seed (Table 3, last column; App. Z.3). The comparator randomizes between adjacent ranks, so its mean size equals the rule's. Per-draw matching is not defined: on one draw the set size fixes the threshold.
Q3. Disjoint samples for selection and estimation (App. Z.2). With w chosen on half the labels and the threshold built from the other half, SASC's gain is gone from k=30 (-0.007 to -0.002 at k=50). NexCP-gated keeps point estimates of +0.003 to +0.013, significant under both schemes only for GAMformer. The gain does not require dropping the guarantee. The floored rule max(SASC, naive) never falls below a valid split-conformal threshold, so it has finite-sample marginal coverage however w is chosen. It keeps the gain: +0.029 to +0.034 at k=20 and +0.008 to +0.016 at k=50, with intervals above zero under both block structures except TabICLv2 at k=50 under task blocks ([-0.001, +0.018]). max(NexCP-gated, naive) gives +0.021 to +0.026 at k=50.
Q4. Ratio-estimation error versus failure of covariate-shift correction. We built a twin of each real shift with a known ratio (new Table 4, App. Z.4): target rows resampled with probability proportional to 1/w(x), so P(X) moves as estimated and P(Y|X) stays the target's. On the twin only overlap and estimation can fail.
We welcome your view of the revised claims.