acceptodds
Under review as a conference paper at ICLR 2027

What Source Calibration Buys Under Real Tabular Shift

Abstract

Real tabular shift breaks the conformal coverage guarantee of tabular foundation models (TFMs). Once a rule reuses the source calibration set, its tail coverage at the same nominal level need not reflect better calibration, since a higher threshold buys coverage with width. We study what the source calibration set is worth for several TFMs on real shifts from US Census, wine and UCI tables. We propose a width-matched control, a reporting protocol that calibrates on the target labels alone at an inflated quantile rank, matches each rule on its mean set size, and summarizes coverage per calibration draw with paired cluster bootstraps over settings and tasks. For source-anchored rules, a bias-variance comparison gives a crossover budget below which the source quantile has lower risk, and it orders the observed crossover across settings, though retrospectively. This supports equal-width comparisons of pooling, prior and gated rules. Experiments show that nominal-level reporting is optimistic for the gated rules, whose gains are mostly width: against the control, a per-draw gain survives at small label budgets and fades as labels grow, pooling rules fall below the control, and the closest prior method sits close to it. Label-free reweighting does not recalibrate real shift under stronger density-ratio estimators or label-shift weights, and the same holds for gradient boosting and logistic regression. Code is available at https://anonymous.4open.science/r/TFM-1-ICLR2027-576E/.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Conformal prediction, Distribution shift, Tabular foundation models, Covariate shift reweighting, Calibration evaluation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_iwUFOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper evaluates conformal prediction methods under distribution shift between source and target data. It proposes a width-matched evaluation protocol that compares methods using both source and target calibration data against a target-only conformal baseline at the same average prediction-set size. Experiments suggest that most gains in nominal coverage are attributable to wider prediction sets: gated source-reuse methods retain small benefits only when very few target labels are available, whereas pooling and label-free reweighting generally fail to improve over target-only calibration.

Critical strengths
  1. Careful empirical study. The experiments evaluates multiple TFMs, classical models, score functions, and real-shift benchmarks using paired comparisons. Its negative evidence—that target-only calibration is difficult to beat—is practically useful.
  2. Useful bias–variance diagnosis. The analysis shows that source reuse can reduce calibration-draw variance at small label budgets but becomes harmful when source bias dominates.
Critical weaknesses
  • Limited conceptual novelty. Comparing methods at equal prediction-set size is a natural extension of the classical coverage–efficiency tradeoff, so the proposed metric is not particularly surprising as a standalone contribution. Moreover, wider sets are not inherently undesirable, since practitioners may accept them for better coverage. Width-matched evaluation should therefore complement, rather than replace, nominal-level coverage, weakening the paper’s main novelty claim.

  • Insufficient justification for conformalizing TFMs and a missing native-uncertainty baseline. TFMs already provide posterior predictive distributions (PPDs), which prior work has found to be reasonably well calibrated marginally. However, the paper does not compare native TFM predictive sets or intervals with their conformalized counterparts, either in distribution or under shift. Consequently, it does not establish that conformal prediction is necessary or beneficial for uncertainty quantification when TFMs already provide native probabilistic uncertainty.

Questions for authors
  1. I would to see comparison with TFM PPD both in distribution and under shift setting
  2. Is prediction-set size matched separately for every setting and calibration draw, or only after averaging across settings and draws? If matching is aggregated, could Δk\Delta_k reflect redistribution of width across heterogeneous settings rather than information gained from the source?
  3. Because SASC selects its mixing weight using the target calibration labels, its point version does not retain the usual finite-sample conformal guarantee. Does the reported advantage persist if weight selection and final quantile estimation use disjoint target samples?
  4. The “oracle temper” selects among estimated weights using target test labels but is not based on the true density ratio. How do the authors distinguish failure of covariate-shift correction from failure to estimate the density ratio accurately?
2 comments
AuthorsOpenReview ↗

Response to Reviewer iwUF

Response to Reviewer iwUF

Thank you for the careful review. We ran every analysis you asked for, on the submission's cached predictions. Changes are in blue; new analyses are in Appendix Z.

W1. Novelty; wider sets are not bad; the control should complement nominal coverage. We agree and now say so in Sec. 1. Equal-size comparison is standard practice; our claim is narrower. Source-reuse rules are justified by nominal-level gains, and our comparator asks whether target labels alone, at a higher level, buy the same width. Table 3 reports both readings. The control is now two-sided (see Q2), so it also credits rules narrower than naive.

W2 / Q1. Native TFM uncertainty (new Table 2, App. Z.1).

  • In distribution the PPDs are close to calibrated. A native set with predictive mass exactly 0.9 (randomized at the boundary) covers 0.893 to 0.895 in every family. The plain native set (smallest with mass at least 0.9) covers 0.98, with sets 20 to 29% larger than split conformal.
  • Under real shift the randomized native set covers 0.885 (census and wine) and 0.821 (UCI), tails 0.863 and 0.724. Source-calibrated conformal has a worse tail (0.817 and 0.001). The plain native set keeps tails of 0.970 and 0.833 only with far wider sets. Fifty target labels restore a tail of 0.89 in both.
  • Native regression intervals [q.05, q.95] cover 0.88 to 0.90 in distribution and 0.87 to 0.89 under shift; target CQR (50 labels) covers 0.90.

So in distribution conformal adds mainly a guarantee; under shift target labels fix coverage. At matched width the native set acts like the source-reusing rules: +0.004 to +0.011 per draw at k=50, but 0.033 to 0.040 below the control per setting.

Q2. Matching unit. The submitted match was pooled: one comparator level for all settings. We now also match per setting and per setting and seed (Table 3, last column; App. Z.3). The comparator randomizes between adjacent ranks, so its mean size equals the rule's. Per-draw matching is not defined: on one draw the set size fixes the threshold.

  • The gated rules' gains are unchanged or slightly larger under the finer units. SASC at k=50: pooled +0.011 to +0.016, per setting and seed +0.015 to +0.018. NexCP-gated: +0.023 to +0.024 and +0.021 to +0.024. Under both setting and task blocks the intervals exclude zero for all five TFMs for NexCP-gated, and for four of five for SASC at k=50 (five of five matched per setting and seed). The gain is not width moved between settings.
  • Percentiled over settings, no rule beats the control by more than 0.01 under any unit: the gain is draw-to-draw stability.
  • We also found that the submitted grid held only inflated ranks, so a rule narrower than naive was compared with naive itself. The two-sided grid fixes this: gated-rule numbers move by at most 0.007, and Pooled-DR and WCC-CSPD now beat the control at k=20.

Q3. Disjoint samples for selection and estimation (App. Z.2). With w chosen on half the labels and the threshold built from the other half, SASC's gain is gone from k=30 (-0.007 to -0.002 at k=50). NexCP-gated keeps point estimates of +0.003 to +0.013, significant under both schemes only for GAMformer. The gain does not require dropping the guarantee. The floored rule max(SASC, naive) never falls below a valid split-conformal threshold, so it has finite-sample marginal coverage however w is chosen. It keeps the gain: +0.029 to +0.034 at k=20 and +0.008 to +0.016 at k=50, with intervals above zero under both block structures except TabICLv2 at k=50 under task blocks ([-0.001, +0.018]). max(NexCP-gated, naive) gives +0.021 to +0.026 at k=50.

Q4. Ratio-estimation error versus failure of covariate-shift correction. We built a twin of each real shift with a known ratio (new Table 4, App. Z.4): target rows resampled with probability proportional to 1/w(x), so P(X) moves as estimated and P(Y|X) stays the target's. On the twin only overlap and estimation can fail.

  • With the true ratio, weighted conformal reaches 0.904 to 0.918 on the census twins. On UCI it gives 0.925 to 0.964, with full sets for 30 to 61% of test points: at that overlap even a known ratio fails.
  • Re-estimating a sharp ratio from unlabeled rows costs 0.02 to 0.03 of coverage on census and 0.06 to 0.20 on UCI.
  • Folktables lists the state code as a predictor in PublicCoverage and TravelTime, so 16 spatial settings have disjoint support and no ratio exists. Wine is nearly disjoint, so its ratio is extreme. We now report these settings separately.
  • On the 29 shared-support settings the real shift costs 0.010 of coverage, no more than its twin. So within these controls, poor overlap and ratio-estimation error can reproduce much of the loss. The twins do not give the true ratio of the original shifts and cannot identify a conditional component; per setting, twin and real coverage agree only weakly (rank correlation 0.28 on census, 0.39 on UCI). Sec. 4.2 states this limit.

We welcome your view of the revised claims.

Thank you for your detailed rebuttal and the substantial revisions. The native TFM baseline, width-matching procedure, and controlled experiments with known proportions address several of the concerns I raised in my first review.

I now view the paper primarily as an empirical study of calibration under tabular distribution shift. The technical novelty beyond standard coverage-efficiency comparisons remains limited, and the main findings are largely consistent with what one would expect from small calibration samples and poor overlap. Two changes would make the paper stronger.

  • If this is meant to be an empirical evaluation paper, the authors need to justify why the proposed evaluation protocol is a reasonable choice. I don’t think the paper does this yet.

  • Right now, the paper only reports how existing methods perform. It would benefit from more concrete practical guidance: when should practitioners reuse source data, when should they turn to methods that use unlabeled data, and when is it worth collecting target labels?

Based on the new evidence, I have raised my score from 1 to 2. In my view, addressing the remaining concerns would require reframing the study around a clear empirical contribution, justifying the evaluation protocol, and establishing what decisions the results can inform in practice. I see this as comparable in scope to developing a new paper, and I think a fresh submission would be more appropriate. My recommendation on the current submission therefore remains below the acceptance threshold.

Reviewer_NfPPOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the value of reusing source calibration data for conformal prediction with tabular foundation models (TFMs) under real distribution shift. The authors propose a width-matched control protocol that separates genuine calibration improvement from mere increase in prediction set width, a confound they identify in prior evaluation practices. They also introduce SASC (Source-Anchored Shrinkage Conformal) as a best-case probe, derive a crossover budget k⋆=κ/b2k^\star = \kappa / b^2 characterizing when source anchoring helps, and validate extensively across 5 TFMs, 2 classical models, and 57 real shift settings.

Critical strengths

S1. Width-matched control addresses a genuine evaluation confound

The central methodological insight—that a rule returning wider sets gains tail coverage without better calibration, and that prior work conflated these two—is both correct and consequential. The proposed protocol (inflated-rank target-only comparator, per-draw tail statistics, paired cluster bootstraps) is well-designed and should improve how the community evaluates conformal methods under shift.

S2. Extensive, well-controlled empirical validation

The experimental design is thorough: 5 frozen TFMs from 3 pretraining lineages (TabICLv2, TabPFNv2/v3, TabDPT, GAMformer), 46 Census+Wine shifts, 11 UCI shifts, 2 classical baselines (XGBoost, logistic regression), multiple score functions, and controlled covariate-shift sanity checks.

S3. Crossover budget provides actionable theoretical insight

Proposition 1 (k⋆=κ/b2k^\star = \kappa/b^2) gives a clean, interpretable characterization of when source anchoring has lower MSE than the naive quantile. The conditions (κ>0\kappa > 0 requires fT(mT)/fT(qT)>1.67f_T(m_T)/f_T(q_T) > 1.67 at α=0.1\alpha=0.1, satisfied by LAC but not APS/RAPS) explain empirical observations and correctly predict the failure of anchoring on cumulative-mass scores. The Spearman ρ=+0.73\rho = +0.73 between predicted and observed crossovers is encouraging.

Critical weaknesses

W1. SASC serves as both probe and method, creating circularity risk

SASC is described as "the best case for the source that we could construct, not a proposal" (p.3), yet it is evaluated alongside proposed methods and its gains are reported as evidence of what the source can buy. Because SASC is handcrafted to maximize source benefit (location-corrected anchor, coverage-gated weight selection on target labels), comparing other methods against it sets a bar that is specific to SASC's design choices rather than a general upper bound on source value. A method that uses the source in a fundamentally different way (e.g., through the score function itself, or through adaptive support-set construction) could outperform SASC without invalidating the paper's conclusions—but the paper's framing makes this difficult to discern.

W2. Scope of "real shift" is narrower than claimed; residual shift uncharacterized

The 46 Census+Wine settings are predominantly geographic/temporal shifts within the same country and domain. The 11 UCI extensions include only 2 sensor-drift settings. The paper explicitly states "we do not claim that the residual measured below is concept shift" (p.6) and acknowledges that three attempts to identify the shift type "failed their own control" (Appendix D.5). This means the central negative result—label-free reweighting does not recalibrate real shift—lacks a mechanistic explanation. Is the residual due to concept shift, overlap failure, or ratio estimation error?

W3. Presentation is excessively dense; key arguments are fragmented

The paper is very difficult to read. Notation is introduced rapidly (q^src\hat{q}_{\text{src}}, q^naive\hat{q}_{\text{naive}}, q^SASC(w)\hat{q}_{\text{SASC}}(w), Δk\Delta_k, κ\kappa, bb, vnv_n, vav_a, cc, neffn_{\text{eff}}, nfraceffn_{\text{frac}}^{\text{eff}}, etc.) often without sufficient motivation before use. Key arguments are split between the main text and appendices in ways that force the reader to jump back and forth (e.g., the crossover budget is introduced informally in §3.3, formalized in Appendix D.3, and tested in §4.3). The three takeaways are strong, but they are buried in dense technical prose.

Questions for authors

See weaknesses

1 comment
AuthorsOpenReview ↗

Response to Reviewer NfPP

Response to Reviewer NfPP

Thank you for the clear summary and the constructive points. Changes are in blue in the revised PDF. New analyses are in Appendix Z, recomputed from the same cached predictions and calibration draws as the submission.

W1. SASC as both probe and method. We agree the framing was unclear and have changed it (Sec. 1 and 3.2).

  • SASC is not the bar. Every rule, SASC included, is scored against the same target-only control (target labels at an inflated or deflated rank, matched on set size). SASC is never the reference.
  • SASC is not an upper bound, and the text no longer calls it "the best case". NexCP-gated transfers the source differently (one flat source weight) and keeps a larger gain at every budget: +0.045 to +0.053 at k=20 and +0.023 to +0.024 at k=50, against +0.024 to +0.032 and +0.011 to +0.016 for SASC.
  • Other uses of the source are reported against the same control. These include pooling, density-ratio pooling, WCC-CSPD, conditional anchoring per confidence bin, resampling the in-context support toward the target and embedding-space ratios (App. Y.6). New: the TFM's own predictive distribution as a native set (Table 2). None changes the picture. Gains at matched width are draw-to-draw stability at small k, and percentiled over settings no rule beats the control by more than 0.01.
  • Our conclusions are stated about the control, not about SASC. A rule that uses the source through the score function could beat SASC without changing them, as you note; it would be read against the same control.

W2. Scope of real shift, and what the residual is. This was the most useful point. A new analysis answers part of it (Sec. 4.2, Table 4, App. Z.4). We built a twin of each real shift with a known ratio. Target rows are resampled with probability proportional to 1/w(x), so P(X) moves as estimated and P(Y|X) stays the target's. On the twin only overlap and ratio estimation can fail.

  • Ratio estimation error. With the true ratio, weighted conformal reaches 0.904 to 0.918 on the census twins. Re-estimating a sharp ratio costs 0.02 to 0.03 (census) and 0.06 to 0.20 (UCI).
  • Overlap failure. On UCI even the true ratio gives full sets to 30 to 61% of test points. In census, folktables lists the state code as a predictor in PublicCoverage and TravelTime, so 16 spatial settings have disjoint support and no ratio exists. Wine is nearly disjoint, so its ratio is extreme.
  • Conditional shift. On the 29 shared-support settings the real shift loses 0.010 of coverage, no more than its twin, and label-free rules tie 50 target labels at matched width. The twins show that overlap and estimation error can reproduce much of the loss. They do not identify the contribution of conditional shift in the original data: per setting, twin and real coverage agree only weakly (rank correlation 0.28 and 0.39). We say this in Sec. 4.2 and App. Z.4.

On scope: our shifts are geographic and temporal shifts in census, wine and six UCI tables, two of them sensor drift. The Conclusion now says so, and we make no claim about other shift types.

W3. Presentation.

  • A notation table (Table 1, Sec. 2) lists every symbol in order of first use, with a pointer to where each is defined.
  • A reading guide closes the introduction. The crossover budget now has one home in Sec. 3.3, with the formal statement in App. D.3 and the test in Sec. 4.3, and the guide says so.
  • Two figures (tail coverage against budget, and the two labeling strategies) moved to App. Z.6 to free space for the new results. Their text is unchanged.
  • The protocol paragraph now states the matching unit, the comparator and why per-draw matching is not defined, in plain terms.

Two corrections we made while revising, so you can check them:

  1. The submitted control held only inflated ranks, so a rule narrower than naive was compared with naive itself. The control is now two-sided. Gated-rule numbers move by at most 0.007, and Pooled-DR and WCC-CSPD now beat the control at k=20.
  2. ACS PublicCoverage and TravelTime include the state code (and PUMA) as features, so 16 of the 46 settings have disjoint covariate support. We report them, with the nearly disjoint Wine, separately.

Thank you again. We would welcome any further suggestions on readability.

Reviewer_yJ26OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the calibration of tabular models under real-world distribution shifts. The authors argue that the increased coverage obtained by reusing source-domain data may simply result from wider prediction sets rather than genuinely better calibration. To address this issue, the paper introduces a width-matched evaluation protocol and investigates when source-domain calibration data can provide real benefits.

Critical strengths
  1. The paper identifies an easily overlooked issue: higher coverage may simply come from wider prediction sets. The proposed width-matched comparison is reasonable and provides a fairer evaluation.
  2. The evaluation considers not only the average prediction-set size but also poor outcomes from individual calibration draws. The use of cluster bootstrap confidence intervals helps prevent average results from hiding the instability of few-shot calibration.
Critical weaknesses
  1. SASC selects its mixing weight using the same target-domain labels, so it no longer retains a strict conformal coverage guarantee. Once a rigorous guarantee is added, it provides almost no advantage over target-only calibration.
  2. The paper claims that label-free reweighting methods cannot recalibrate real-world distribution shifts. However, only a limited range of methods is evaluated, and most experiments are concentrated on a small number of ACS tasks. Therefore, this conclusion appears overly broad.
Questions for authors

See Weaknesses.

1 comment
AuthorsOpenReview ↗

*Response to Reviewer yJ26

Response to Reviewer yJ26

Thank you for the review. Both points led to new analyses, which changed the paper. Changes are in blue; new analyses are in Appendix Z, recomputed from the same cached predictions and calibration draws as the submission.

W1. SASC selects w on the same target labels, so it has no finite-sample guarantee; with a rigorous guarantee it gives almost no advantage. We agree that the point rule has no finite-sample guarantee; the paper states this, and SASC is a probe, not a proposal. Whether a guarantee removes the advantage depends on which guarantee. We now test three (App. Z.2), all against the width-matched control with 95% cluster-bootstrap intervals:

  1. Finite-sample marginal guarantee (new). The floored rule max(SASC, naive) never falls below the split-conformal quantile of the k target labels. So its coverage is at least 1 - alpha in finite samples, however w is chosen. It keeps the gain: +0.029 to +0.034 at k=20 and +0.008 to +0.016 at k=50. The intervals exclude zero under both setting and task blocks for all five TFMs, except TabICLv2 at k=50 under task blocks ([-0.001, +0.018]). max(NexCP-gated, naive) gives +0.021 to +0.026 at k=50. Sets are 3 to 6% larger than naive, and the control is matched at that width. The anchor raises the threshold on draws whose target quantile came out low, which is where width helps most.
  2. Disjoint selection and estimation (new). Choosing w on one half of the labels and building the threshold from the other half removes SASC's gain from k=30 (-0.007 to -0.002 at k=50). The estimate then rests on half the labels. NexCP-gated keeps point estimates of +0.003 to +0.013, significant under both bootstrap schemes only for GAMformer.
  3. Training-conditional (PAC) guarantee (in the submission). The certified form adds nothing over certified naive at matched width, as you say.

The split forms are ablations of label reuse, not valid rules, because their blend still contains a biased anchor. Sec. 3.2 now says which guarantee each form has.

So with a marginal guarantee a small per-draw gain remains (one to five points of tail coverage at k <= 50). With a PAC guarantee it does not. We agree the advantage is small, and that is the paper's message: most of what source reuse appears to buy at the nominal level is width. Sec. 3.2, Table 3 and Takeaway 1 now state this.

W2. The claim that label-free reweighting cannot recalibrate real shift is too broad. We agree it was too broad. We tested it more widely and narrowed it (Sec. 4.2, Table 4, App. Z.4 and Z.5).

  • More methods. We added fifteen label-free rules on all 57 settings: uLSIF and RuLSIF direct ratio estimation, weight clipping at the 95th and 99th percentiles, ratios without the state code, a threshold implied by the TFM's own predictive distribution (a soft pseudo-calibration), and KL-robust conformal. We also added native TFM sets (Table 2) and ran the four ratio estimators of the submission on the UCI family.
  • Beyond ACS. On the 11 UCI shifts every label-free rule loses to 50 target labels by 0.08 to 0.65 of tail coverage. The one exception is KL-robust conformal, which ties only by returning nearly the full label set.
  • Where the claim does not hold, and why. Folktables lists the state code as a predictor in two ACS tasks. So in 16 of 46 settings the supports are disjoint and no density ratio exists; Wine is nearly disjoint, so its ratio is extreme. On the other 29 the shift is mild: source conformal covers 0.890. There every reweighting rule, plain source conformal included, ties 50 target labels at matched width.
  • What the twins show. In a twin of each shift with a known ratio, the true ratio reaches 0.90 on census but not at UCI overlap, and re-estimating a sharp ratio costs 0.02 to 0.20. So poor overlap and ratio-estimation error can reproduce much of the loss. The twins do not identify the contribution of conditional shift in the original data.

The claim now reads: across the evaluated benchmarks, none of the tested label-free rules consistently improves tail coverage over target-only calibration at matched width. Some reach nominal mean coverage (uLSIF 0.904) but lose in the tail at equal width. Where supports overlap, label-free rules tie fifty labels; where they do not, the ratio is missing or extreme. The Conclusion states the scope: geographic and temporal shifts of census, wine and six UCI tables.

On the PC chairs' note: the paper already carries a "Use of large language models" statement (p. 11), which we keep.

Thank you again. We hope the narrower claim and the new evidence address your concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.