- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes ShiftICL, a prior-fitted tabular model that handles label shift between the labeled context and the unlabeled query set. During pre-training, the query class prior is obtained by perturbing the context prior in log-odds space, and candidate shifts are selected with a QDA-based curriculum. Architecturally, the row-mixing blocks of a small TabPFNv3-style backbone get two learned inducing tokens that summarize the query set, so each prediction can depend on the whole unlabeled batch. The authors also restate the identifiability result of Garg et al. (2020) and give a consistency result for the transductive PPD. On 209 OpenML datasets with synthetic label shift, ShiftICL degrades much less than a Base model, DistPFN variants and Drift-Resilient TabPFN in accuracy, NLL and ECE as shift grows. A small ablation on Gaussian mixtures is used to attribute the gains to query-set conditioning.
Critical strengths
[1] The idea is simple and well motivated.
Under label shift, the unlabeled query marginal carries information about the new class prior, and a PFN is a natural way to amortize both the prior estimation and the posterior correction. The inducing-token summary is also a cheap way to let the model see the query population without making the context representations depend on the query, and App. A.3 explains this clearly. I think this is a reasonable direction for tabular foundation models under shift.
[2] The gains in the main experiments are large and consistent.
Across all four shift families, ShiftICL degrades much less than the other methods as shift grows. For example, at the largest shift its normalized NLL stays around 0.67 while Base goes to about 1.38 (Fig. 2), and the calibration results in Fig. 3 show the same pattern. If these gains hold against stronger baselines, this would be a useful result.
Critical weaknesses
[1] The baselines are weak, and the source of the gains is not clearly identified.
Almost every non-oracle competitor is built on the authors' own Base, a small 8M parameter model trained for a short time. The classical label-shift estimators cited in Sec. 2.1.1, such as EM/MLLS with calibration and BBSE, are never evaluated, even though they can be applied to Base directly and are usually stronger than simple reweighting. There is also no stronger public TFM such as TabPFNv2 or TabICLv2, with or without correction. Therefore, I think describing the comparison as against "state-of-the-art methods tailored for distribution drifts" is an overstatement.
Also, Sec. 4.1 admits that Base and ShiftICL differ in both the training prior and the architecture. The only experiment that separates the two is App. C.2, which uses a different and much smaller model on Gaussian mixtures with a single training seed. The results there are mixed (for example, ShiftICL is below both Base variants at K = 2 for small shifts), and the Table 3 configuration caps the number of classes at 5 while Fig. 4 and 5 evaluate K = 7 and 10. Thus, the claim in Sec. 5 that the gains come from query-set conditioning rather than the shifted prior is not established at the scale of the main results. A shift-trained Base on the 209 datasets is the missing experiment.
[2] The transductive behaviour and the robustness of the results are not studied.
Predictions depend on the size and content of the query set, and the theory is about the number of query rows going to infinity, yet all evaluations use a fixed batch of 512 rows. There is no experiment with small batches, single queries, or query batches drawn from mixed priors. I would also like to see what happens when the label-shift assumption is violated, since the model is now trained to interpret any change in the query marginal as a prior change.
The statistical reporting is also thin. Error bars come from five evaluation seeds after averaging over datasets, with a single pre-training seed and no per-dataset comparison. At zero or weak shift ShiftICL is slightly worse than Drift-Resilient TabPFN and has higher ECE than most methods, so the claim that it keeps strong performance without shift needs a per-dataset breakdown.
[Smaller issues]
The "Oracle" is Base reweighted by the true query proportions, which is not Bayes-optimal unless Base is well calibrated, so I would not call it the Bayes optimal PPD. The theory is light. Cor. 3.1 restates Garg et al., and Prop. 3.3 assumes standard posterior-consistency conditions, while the step to arbitrary n and m in App. B needs more care. There are also many typos (for example "ShifICL", "syntetic", "checkoint").
Questions for authors
Q1) Could you add EM/MLLS with calibration and BBSE on top of Base, and ideally a corrected stronger public TFM such as TabPFNv2 or TabICLv2, under the same 209-dataset protocol? This would show whether the gain comes from the method or from a weak reference model.
Q2) Could you train a shift-trained Base at the 8M scale on the same shifted prior and add it to Fig. 2 and 3? Without this, it is not possible to attribute the gains to query-set conditioning rather than to the training prior.
Q3) How does ShiftICL behave as the query set size changes (for example from 1 to 512 rows), and when the query batch comes from a mixture of priors? If it does not fall back to Base performance for small batches, this would limit its practical use.
Q4) The tiny model in Table 3 allows at most 5 classes. How were the K = 7 and K = 10 results in Fig. 4 and 5 produced?
2 comments
Thank you for the detailed response and additional experiments.
My main concerns have been adequately addressed, and I will increase my score accordingly.
(Additional detailed assessment is provided below.)
Added the missing baselines and ablation studies in the revised version.
Thank you for your thorough feedback and comments. We added these baselines and ablations in the revised version, we'll however briefly address each point below:
We added EM (Saerens et al., 2002) and BBSE (lipton et al., 2018) with the same experimental protocol as in Lee 2026 (Section 4, Figure2 in the updated main body). The picture slightly changes: EM is a competitive baseline, mostly matching ShiftICL in terms of accuracy. However, ShiftICL achieves the best performance in terms of normalized NLL and ECE in all families. Furthermore, ShiftICL requires no heldout split or prior estimation steps. As for the stronger base model, TabPFNv2 and TabICLv2 are much larger than ShiftICL, which has only 8M parameters. We pre-trained a base model with the same number of parameters for comparisons to be fair. We do openly list the small scale of our models as a limitation of this work. Training a TabICLv2 backbone on shifted episodes is left as future work.
Yes, we scaled up the ablations from appendix C to 8M parameters, ran them on the OpenML benchmark, and moved them into the main body as a separate “ablation studies” section (section 5). Results are summarized in Figure 6. We also updated the results on the synthetic tasks, which we left them in the appendix D.1. It separates the contribution of the transductive formulation from that of the shifted prior. The prior alone does contribute: it accounts for roughly two thirds of ShiftICL’s accuracy gain over Base in the Inverse-frequency family. But it does not explain the whole gain. ShiftICL outperforms Base + shifted episodes in the Inverse-frequency, Dirichlet and Spike families. In the Swap family the two are on par in accuracy and ECE, and ShiftICL is better in NLL. In addition, Base + shifted episodes loses about three accuracy points when there is no shift, whereas ShiftICL stays close to Base.
We added an ablation in Appendix D.2 that answers precisely this question. We ablated both the query set size and also a variant where we fix the query size at 256 rows and mix the query prior with the context prior, with different proportions. We found that:
(i) When the shift is strong, scores improve as the query set grows. When there’s no shift, Base tends to perform better when the query set is smaller: ShiftICL was trained on query sets that were at least 103 rows, so very small query sets are out of distribution.
(ii) When the query set contains a mix of priors, ShiftICL outperforms the baselines when there’s moderate and severe label shift, and stays close to Base when there’s no shift.
This way we empirically verify that the improvement comes from the information the query carries and not only how many rows it contains.
Since we scaled up the ablations this Table is no longer in the paper.