acceptodds
Under review as a conference paper at ICLR 2027

Amortizing Label Shift for In-Context Tabular Prediction

Abstract

Prior Fitted Networks have emerged as a disruptive paradigm enabling the pre-training of Tabular Foundation Models (TFMs) entirely on synthetic data. While these models have proven successful across a broad variety of real-world predictive tasks, they still make strong assumptions about the train and test distributions being fixed. This assumption is often unrealistic, as many applications exhibit substantial distribution shifts. In particular, a drawback of current TFMs for real-world deployment is their vulnerability to label shift, where class proportions change between train and test time. In this work, we introduce ShiftICL, the first Tabular Foundation Model that explicitly amortizes label-shift adaptation during pre-training. ShiftICL is trained on synthetic tasks spanning a broad range of label distributions and shift severities, enabling it to adapt its predictive distribution to previously unseen label shifts without retraining or access to test labels. Across a large collection of 205 OpenML tabular datasets under controlled label shift, ShiftICL substantially improves predictive accuracy and calibration over existing state-of-the-art methods tailored for distribution drifts, with gains becoming increasingly pronounced as the shift grows.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
tabular foundation model, label shift, prior fitted networks

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Bv3FOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes ShiftICL, a prior-fitted tabular model that handles label shift between the labeled context and the unlabeled query set. During pre-training, the query class prior is obtained by perturbing the context prior in log-odds space, and candidate shifts are selected with a QDA-based curriculum. Architecturally, the row-mixing blocks of a small TabPFNv3-style backbone get two learned inducing tokens that summarize the query set, so each prediction can depend on the whole unlabeled batch. The authors also restate the identifiability result of Garg et al. (2020) and give a consistency result for the transductive PPD. On 209 OpenML datasets with synthetic label shift, ShiftICL degrades much less than a Base model, DistPFN variants and Drift-Resilient TabPFN in accuracy, NLL and ECE as shift grows. A small ablation on Gaussian mixtures is used to attribute the gains to query-set conditioning.

Critical strengths

[1] The idea is simple and well motivated.

Under label shift, the unlabeled query marginal carries information about the new class prior, and a PFN is a natural way to amortize both the prior estimation and the posterior correction. The inducing-token summary is also a cheap way to let the model see the query population without making the context representations depend on the query, and App. A.3 explains this clearly. I think this is a reasonable direction for tabular foundation models under shift.

[2] The gains in the main experiments are large and consistent.

Across all four shift families, ShiftICL degrades much less than the other methods as shift grows. For example, at the largest shift its normalized NLL stays around 0.67 while Base goes to about 1.38 (Fig. 2), and the calibration results in Fig. 3 show the same pattern. If these gains hold against stronger baselines, this would be a useful result.

Critical weaknesses

[1] The baselines are weak, and the source of the gains is not clearly identified.

Almost every non-oracle competitor is built on the authors' own Base, a small 8M parameter model trained for a short time. The classical label-shift estimators cited in Sec. 2.1.1, such as EM/MLLS with calibration and BBSE, are never evaluated, even though they can be applied to Base directly and are usually stronger than simple reweighting. There is also no stronger public TFM such as TabPFNv2 or TabICLv2, with or without correction. Therefore, I think describing the comparison as against "state-of-the-art methods tailored for distribution drifts" is an overstatement.

Also, Sec. 4.1 admits that Base and ShiftICL differ in both the training prior and the architecture. The only experiment that separates the two is App. C.2, which uses a different and much smaller model on Gaussian mixtures with a single training seed. The results there are mixed (for example, ShiftICL is below both Base variants at K = 2 for small shifts), and the Table 3 configuration caps the number of classes at 5 while Fig. 4 and 5 evaluate K = 7 and 10. Thus, the claim in Sec. 5 that the gains come from query-set conditioning rather than the shifted prior is not established at the scale of the main results. A shift-trained Base on the 209 datasets is the missing experiment.

[2] The transductive behaviour and the robustness of the results are not studied.

Predictions depend on the size and content of the query set, and the theory is about the number of query rows going to infinity, yet all evaluations use a fixed batch of 512 rows. There is no experiment with small batches, single queries, or query batches drawn from mixed priors. I would also like to see what happens when the label-shift assumption is violated, since the model is now trained to interpret any change in the query marginal as a prior change.

The statistical reporting is also thin. Error bars come from five evaluation seeds after averaging over datasets, with a single pre-training seed and no per-dataset comparison. At zero or weak shift ShiftICL is slightly worse than Drift-Resilient TabPFN and has higher ECE than most methods, so the claim that it keeps strong performance without shift needs a per-dataset breakdown.

[Smaller issues]

The "Oracle" is Base reweighted by the true query proportions, which is not Bayes-optimal unless Base is well calibrated, so I would not call it the Bayes optimal PPD. The theory is light. Cor. 3.1 restates Garg et al., and Prop. 3.3 assumes standard posterior-consistency conditions, while the step to arbitrary n and m in App. B needs more care. There are also many typos (for example "ShifICL", "syntetic", "checkoint").

Questions for authors

Q1) Could you add EM/MLLS with calibration and BBSE on top of Base, and ideally a corrected stronger public TFM such as TabPFNv2 or TabICLv2, under the same 209-dataset protocol? This would show whether the gain comes from the method or from a weak reference model.

Q2) Could you train a shift-trained Base at the 8M scale on the same shifted prior and add it to Fig. 2 and 3? Without this, it is not possible to attribute the gains to query-set conditioning rather than to the training prior.

Q3) How does ShiftICL behave as the query set size changes (for example from 1 to 512 rows), and when the query batch comes from a mixture of priors? If it does not fall back to Base performance for small batches, this would limit its practical use.

Q4) The tiny model in Table 3 allows at most 5 classes. How were the K = 7 and K = 10 results in Fig. 4 and 5 produced?

2 comments
AuthorsOpenReview ↗

Added the missing baselines and ablation studies in the revised version.

Thank you for your thorough feedback and comments. We added these baselines and ablations in the revised version, we'll however briefly address each point below:

  1. We added EM (Saerens et al., 2002) and BBSE (lipton et al., 2018) with the same experimental protocol as in Lee 2026 (Section 4, Figure2 in the updated main body). The picture slightly changes: EM is a competitive baseline, mostly matching ShiftICL in terms of accuracy. However, ShiftICL achieves the best performance in terms of normalized NLL and ECE in all families. Furthermore, ShiftICL requires no heldout split or prior estimation steps. As for the stronger base model, TabPFNv2 and TabICLv2 are much larger than ShiftICL, which has only 8M parameters. We pre-trained a base model with the same number of parameters for comparisons to be fair. We do openly list the small scale of our models as a limitation of this work. Training a TabICLv2 backbone on shifted episodes is left as future work.

  2. Yes, we scaled up the ablations from appendix C to 8M parameters, ran them on the OpenML benchmark, and moved them into the main body as a separate “ablation studies” section (section 5). Results are summarized in Figure 6. We also updated the results on the synthetic tasks, which we left them in the appendix D.1. It separates the contribution of the transductive formulation from that of the shifted prior. The prior alone does contribute: it accounts for roughly two thirds of ShiftICL’s accuracy gain over Base in the Inverse-frequency family. But it does not explain the whole gain. ShiftICL outperforms Base + shifted episodes in the Inverse-frequency, Dirichlet and Spike families. In the Swap family the two are on par in accuracy and ECE, and ShiftICL is better in NLL. In addition, Base + shifted episodes loses about three accuracy points when there is no shift, whereas ShiftICL stays close to Base.

  3. We added an ablation in Appendix D.2 that answers precisely this question. We ablated both the query set size and also a variant where we fix the query size at 256 rows and mix the query prior with the context prior, with different proportions. We found that:

    (i) When the shift is strong, scores improve as the query set grows. When there’s no shift, Base tends to perform better when the query set is smaller: ShiftICL was trained on query sets that were at least 103 rows, so very small query sets are out of distribution.

    (ii) When the query set contains a mix of priors, ShiftICL outperforms the baselines when there’s moderate and severe label shift, and stays close to Base when there’s no shift.

    This way we empirically verify that the improvement comes from the information the query carries and not only how many rows it contains.

  4. Since we scaled up the ablations this Table is no longer in the paper.

Thank you for the detailed response and additional experiments.

My main concerns have been adequately addressed, and I will increase my score accordingly.

(Additional detailed assessment is provided below.)

Reviewer_g8WoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ShiftICL, a tabular foundation model designed to adapt to label shift, where class proportions change between training and test distributions while class-conditional feature distributions remain unchanged. The method incorporates label shifts into synthetic pretraining episodes and introduces an inducing-token mechanism that aggregates information from the unlabeled query set. The authors formulate prediction under label shift as Bayesian inference, discuss conditions for predictive identifiability, and establish consistency of the ideal Bayesian posterior predictive under stated assumptions. Experiments on 209 OpenML datasets evaluate inverse-frequency, Dirichlet, spike, and swap shifts. Additional synthetic ablations investigate the contribution of query-set conditioning.

Critical strengths
  1. The label shifts problem is well motivated. The paper explains why labeled context data alone cannot generally determine the target predictive distribution and how unlabeled query examples provide additional evidence about target class proportions.
  2. The query-set mechanism is matched to the inference problem. ShiftICL summarizes context-conditioned query representations using two inducing tokens per row-interaction block. This provides a concrete mechanism for extracting target-population information while adding only (O(mr)) attention interactions per block.
Critical weaknesses
  1. Base is pretrained without label shift, whereas ShiftICL introduces both shifted training episodes and query-set conditioning. The authors explicitly acknowledge that their comparison changes both factors (p. 8, lines 378–380). Appendix C.2 provides relevant ablations, but these use approximately one-million-parameter models and only 20 synthetic Gaussian-mixture tasks per setting. The main results use approximately eight-million-parameter models on real tabular datasets. Consequently, the ablations do not adequately explain the gains in the main benchmark. A complete comparison crossing shifted/unshifted pretraining with enabled/disabled query-set conditioning is needed at the main experimental scale.
  2. Corollary 3.1 is explicitly a direct consequence of existing identifiability results. Proposition 3.3 establishes consistency for an ideal Bayesian posterior predictive under well-specification and the stated consistency conditions. For the actual network, however, the amortization error in Equation (4) remains uncontrolled. Its convergence to zero is assumed rather than established (p. 6, lines 278–279). Appendix B appropriately acknowledges that the result alone does not establish consistency of the trained network (p. 14, lines 723–727). The paper also does not verify the required conditions for its implemented synthetic prior or provide finite-sample results. The theory therefore motivates the formulation but offers limited support for the reliability of the implemented method.
  3. The central mechanism depends on aggregating evidence from the query population, yet the main experiments largely fix the query size at 512, and the synthetic ablations fix it at 256. The paper does not establish how performance changes with query-set size or how sensitive predictions are to dividing observations from the same target population into separate batches. There is also an unresolved configuration discrepancy: Table 3 specifies a maximum of five classes, while Figures 4 and 5 include seven- and ten-class tasks. This could reflect a table error or an undocumented extrapolation setting. The distinction matters for interpreting the degradation at larger class counts and should be clarified.
  4. The reported uncertainty reflects five evaluation sampling seeds, while pretraining uses a single reported seed. The authors clearly disclose this limitation. Nevertheless, these error bars do not quantify training variability or establish how consistently improvements occur across datasets. Reporting per-dataset paired comparisons, win rates, and results from repeated training runs in the appendix or code repository would strengthen the empirical evidence.
Questions for authors
  1. How does ShiftICL compare with calibrated EM/MLLS and BBSE applied to the same Base model? Do the improvements persist when the competing methods explicitly estimate target class proportions?
  2. Can the authors provide a complete ablation crossing shifted/unshifted pretraining with enabled/disabled query-set conditioning, using the main model configuration on the OpenML benchmark? Do the conclusions from the smaller synthetic experiments carry over? The shift prior is constructed by perturbing log-odds relative to a reference class. Is that reference randomized during pretraining? For multiclass tasks, is the induced shift distribution invariant to class relabeling, and has prediction sensitivity to label permutations been evaluated?
3 comments
AuthorsOpenReview ↗

Added missing experiments and baselines, and fixed the argument in Section 3.

Thank you for your insightful feedback. We updated the paper with the requested baselines and ablations, and also fixed our argument about the amortization error in Section 3. All details are in the revision.

  1. We added EM (Saerens et al., 2002) and BBSE (Lipton et al., 2018) on the same Base model, following Lee et al. (2026) (see Section 4, Figure 2 of the revised version). Both explicitly estimate the target class proportions. Against EM, the accuracy improvement does not persist: EM matches ShiftICL at every shift level. The improvement in normalized NLL does persist in all four families, and so does the advantage at zero shift, where EM and BBSE fall below Base. BBSE is weaker than ShiftICL on all three metrics. Regarding calibration, both baselines are uncalibrated, as in Lee et al. 2026: calibration requires held-out labeled rows, and the context has at most 512. The fact that our model supports small datasets is listed as a limitation in Section 6.

  2. We extended the main body with this ablation (results are summarized in Figure 6). We pre-trained a Base model with about 8M parameters on shifted episodes (Base+shifted episodes) and collected results on the OpenML benchmark suite. We found that, even though the shifted synthetic prior contributes to the improvement in performance on shifted datasets, ShiftICL outperforms Base + shifted episodes in the Inverse-frequency, Dirichlet and Spike families. In the Swap family ShiftICL is better in Normalized NLL, and on-a-par on Accuracy and ECE. Moreover, in weak shifts Base+shifted episodes scores worse than ShiftICL, which falls back to Base. Regarding the reference class, it is fixed to the last class and not randomized. It only fixes the parameterization: the perturbation has full support, so every target prior is reachable whichever class is the reference, and the set of shifts seen in pre-training is invariant to relabeling. Our four evaluation shift families select the shifted classes independently of their index, on real datasets with arbitrary class order, and ShiftICL’s gains hold there. We didn’t measure sensitivity to label permutations. Randomizing the reference class is a natural fix for the short term.

On the weakness cited in point 2, that’s a good catch. After delving into this point we realized that what we assumed about the amortization error converging to 0 as n,mn,m tend to inf⁡\inf is not entirely correct. That said, this amortization error is not really uncontrolled: in their seminal work, Muller et al. 2022, Corollary 3.1, show that the prior-data loss equals the KL divergence between the network and the Bayesian PPD, which is the loss that the pre-training procedure minimizes. Now, as you correctly pointed out, this doesn’t mean the error will go to zero (this is an argument Nagler formally presents in his PMLR paper from 2023). Rather than being uncontrolled, the amortization error is then governed by the pre-training procedure (and not by n,mn,m). We fixed this argument in the revised version of the paper, lines 274-282. What we aim at addressing in Section 3 is the Bayesian estimation error, not the full error. As for Corollary 3.1, we acknowledge in the paper that it’s a direct consequence of Garg et al. 2020 (line 149). We stated it as such for the purpose of rigorously justifying the transductive formulation later on.

Area_Chair_PHP2OpenReview ↗

Following Up

Hi Reviewer g8Wo,

Has the authors' response addressed your concerns?

Regards, AC

AuthorsOpenReview ↗

Added calibrated EM and BBSE; strong results for shifticl

We added calibrated EM and calibrated BBSE (BCTS with 5-fold cross-fitting on the context, so the query predictions still use the full context) and compared all methods across context sizes. We fixed the query size to m=512m=512, swept the context size over n∈{128,256,512}n\in\{128, 256, 512\}, and report accuracy, normalized NLL and ECE. All methods receive the same episodes, and both models are the paper checkpoints (note that 128 rows lies within the pre-training range). Full results are in Tables 3–5 (Appendix E of the revised PDF, the PDF is otherwise unchanged).

Two things stand out. First, the calibrated baselines are strong: with the full context of 512 rows, calibrated EM matches ShiftICL in accuracy and is comparable or better in NLL under Dirichlet, spike and swap shifts, while ShiftICL remains clearly better under inverse-frequency shift. Second, the gap in favor of ShiftICL grows as the context shrinks. The table counts, out of 21 shift settings (family and severity), those where ShiftICL is significantly better than the second best method (paired t-test across datasets, not counting privileged known-prior baselines). In parentheses: settings where ShiftICL has the best mean.

Context rowsAccuracyNormalized NLLECE
12810 (13)12 (15)12 (14)
2565 (12)4 (13)3 (13)
5123 (10)4 (9)3 (7)

Conversely, no post-hoc correction is significantly better than ShiftICL in accuracy at any context size, and in at most two NLL or ECE settings per context size, all under spike shift. In short, ShiftICL matches calibrated post-hoc corrections when labeled context is plentiful and outperforms them when it is scarce, in a single forward pass.

Reviewer_Z4TLOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes ShiftICL, a tabular foundation model pretrained to adapt to label shift, where class proportions change between context and query sets while class-conditional feature distributions remain unchanged. ShiftICL combines synthetic pretraining episodes with varying class proportions and a transductive architecture that summarizes unlabeled query rows through learned inducing tokens. This enables query-set-dependent predictions in a single forward pass without test labels or parameter updates.

The paper formulates a Bayesian posterior predictive target, discusses identifiability, and establishes consistency of the ideal Bayesian predictor under stated assumptions. Experiments cover 209 OpenML datasets under four controlled shift families. ShiftICL improves accuracy, normalized negative log-likelihood, and calibration over the evaluated baselines, particularly under stronger shifts.

Critical strengths
  • The method provides a coherent integration of label-shift adaptation into tabular in-context learning. The synthetic training distribution and transductive architecture address complementary requirements: learning how class proportions may change and obtaining evidence about those changes from unlabeled query features. The inducing-token mechanism also avoids quadratic query-to-query attention.

  • The evaluation is broad within the controlled label-shift setting. Experiments span 209 datasets and multiple shift constructions, rather than relying on a single resampling scheme. The paper evaluates probabilistic quality alongside accuracy, uses shared evaluation episodes across methods, and clearly states that its error bars measure evaluation-sampling variability. The improvements under severe shifts are substantial and consistent across several metrics.

Critical weaknesses
  • The baseline coverage is insufficient to establish superiority over established label-shift adaptation methods. Although the paper discusses classical approaches, the experiments primarily compare against DistPFN, its temperature-scaled variant, and Drift-Resilient TabPFN. They do not include established approaches such as EM-based prior estimation or confusion-matrix-based label-shift estimation applied to the same Base predictions.
  • The main comparison confounds architecture and pretraining. ShiftICL changes both the training distribution and query-set conditioning. Controls separating these factors are limited to smaller models on synthetic tasks.
Questions for authors
  • How does ShiftICL compare with EM-based and confusion-matrix-based label-shift correction using the same Base model?

  • Can the architecture-versus-pretraining ablation be repeated on the main OpenML benchmark?

  • How sensitive is performance to query-batch size and modest violations of the label-shift assumption?

1 comment
AuthorsOpenReview ↗

Added the required benchmarks and ablations in the revised version.

Thank you for your comments and feedback. We've updated the paper with the required experiments and ablations, so we'll reply each question with pointers to the relevant sections.

  1. We added both such benchmarks in Section 4. The picture slightly changes for us: EM (Saerens et al., 2002) is a strong benchmark, and it catches up with ShiftICL in terms of accuracy. However, it lags behind ShiftICL in terms of Normalized NLL and ECE (Figure 2 in the revised paper). The confusion-matrix-based method, BBSE (Lipton et al. 2018) lags behind ShiftICL in general.

  2. We added a new section in the revised version (Section 5) where we carry out ablations that isolate the contribution from the transductive formulation from that of the synthetic shifted prior. Figure 6 summarizes the results on the OpenML benchmark: the prior alone does contribute but it doesn’t explain all the gain. ShiftICL outperforms Base + shifted episodes in the Inverse-frequency, Dirichlet and Spike families. In the Swap family the two are on par in terms of accuracy and ECE, but ShiftICL is better in terms of Normalized NLL. Also, ShiftICL has gains of about three accuracy points over Base+shifted episodes in scenarios where there’s no shift, staying close to Base.

  3. We also added an ablation to study the effect of the query set in the predictive performance of ShiftICL, both in terms of size but also composition (mixing the query and context priors with different proportions), in Appendix D.2. The results (Figures 7 - 9) show results for the size ablation. The stronger the shift, the more consistently ShiftICL outperforms other baselines for queries of smaller sizes. When there’s no shift, Base tends to perform better. This is due to the fact that ShiftICL was trained with query sets no smaller than 103 rows. As a result, query sets of small size are out of its training distribution. Figures 10-12 show results for the prior mixing ablation: ShiftICL outperforms other baselines when there’s moderate or strong shift, and remains fairly close to Base when there’s not.

Regarding modest violations of label shift, this work hinges on the assumption that the class conditional laws of query and context are the same. Therefore, other kinds of distribution shift are out of the scope of this work. We do however openly state this as a limitation in the conclusion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.