- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The authors propose a new method for compressing the context of tabular PFN models for classification tasks. They test their method on a subset of TALENT and on MacOData benchmarks with TabICLv2 and TabPFNv1 models, where their algorithm outperforms naive and optimized baselines, including inference on the full-context. Broadly, their method works by estimating the importance of each sample and feature via iterative bagging and scoring on the validation set.
Critical strengths
- Clearly written and easy to follow, addresses an important problem. Scaling tabular PFNs to larger contexts is an important problem. The paper exposition is clearly written and easy to follow.
- Proposed algorithm is intuitive and appears sound. It is an iterative procedure that tries to estimate the importance of each sample by including and then removing it from the training set and scoring on validation.
- Empirical performance on the reported set of benchmarks appears to be strong. On the classification, the authors report 1.54 to 14.68% improvement with even more impressive gains on the data noise setup (>100% improvements).
Critical weaknesses
- Limited to classification. While this is an important subset of tasks, it would be great if the method could be extended to regression. In particular, it seems if we swap auc-roc with on line 10 in Algorithm 1, the method should natively support regression too. Could the authors explain in more details what's the reason for excluding regression?
- Outdated backbones. The authors test with TabPFNv1, which is an outdated tabular PFN. Can the authors also include more recent models such as TabPFNv2 or even v3.5? I wonder if the results would still hold there. In particular, for the latest tabular PFNs which have scaled to the data up to 1M rows, the claim that "larger contexts degrade performance" is no longer true.
- Benchmark. This is not critical, but I do find the authors' choice of datasets a bit arbitrary. TALENT benchmark has many more classification datasets (120 binary classification, 80 multi-class), out of which the authors choose to select 38. The inclusion of OddBench is helpful.
- Reliance on the validation data. I view this as the major limitation of the current work. Tabular PFN models are attractive, because they don't need validation data to fit. Even if given validation data, they usually just concatenate it directly to the context (as done in TabArena and BeyondArena benchmarks). In particular, the H3 baseline is not done correctly, the TFM should be fitted on both train and validation: VIP-COP in some sense instills information from the validation into the context, so it makes sense to give the baseline methods access to it as well. Besides that, I think VIP-COP can overfit validation when run for many iterations. Does it make sense possibly to use it with cross-validation?
Questions for authors
See above. To re-iterate, I would be interested to understand:
- Why omit regression? It seems they have all the right tools to tackle it as well.
- What made the authors choose these specific datasets / benchmarks? Can they extend the results?
- The choice of TFM models seems arbitrary. Can the authors include the latest TFM models such as TabPFNv3.5?
- It seems their estimate for H3 baseline (full-context) is unnecessarily pessimistic. Can you re-run H3 by also including the validation in the context?
- Related to the above, it seems the method is sensitive to the quality of the validation data. What if we have temporal or grouped splits (e.g. TabReD or BeyondArena benchmarks), where the validation data does not supply enough good signal about the test-time performance. How would the proposed method fare on such datasets? In general, can the method benefit from cross-validation?
7 comments
Continued
Weakness 3 / Question 2: - “What made the authors choose these specific datasets / benchmarks? Can they extend the results?”
The selection follows a fixed protocol. We take all publicly available TALENT classification datasets with n > 1000 & d < 100 for small-d, and n > 1,000 & d > 100 for Large-d, then further group small-d into small-d (n<50K) and small-d (n>50K) to cover different backbone context sizes (Section 4.1, Appendix A.1). We excluded the TALENT-EXTENSION many-class datasets because their sources were not publicly accessible. For outlier detection, we use all 240 OddBench datasets with more than 6,000 samples.
Paper Revision: The evaluation is extended in three directions: 1) 22 regression datasets above, 2) new TabPFN-v3.5-fast backbone, and 3) a “small-data” regime in which all samples and features fit within the context.
Weakness 4 / Question 4 - "It seems their estimate for H3 baseline (full-context) is unnecessarily pessimistic. Can you re-run H3 by also including the validation in the context?"
We thank the reviewer for this point and fully agree that the full-context baseline should get access to the same data as VIP-COP. However, feeding train + val both would only serve as data augmentation, and not explicit optimization. Nevertheless, we added a new baseline, H3_full (shown in blue in all results Appendix B, Table 6-19), which places the full training set plus the validation set in the context, giving it the same data budget as VIP-COP. Note that H3 is default to H3_full in all TabPFNv3.5 results.
| Backbone | Avg. % improv. over H3 (full) | p-value |
|---|---|---|
| TabPFNv1 | +13.82% | <1e-6 |
| TabICLv2 | +5.20% | 9e-5 |
| TabPFN-v3.5-fast | +5.64% | 3e-5 |
The above results show that VIP-COP still outperforms H3_full overall (p-value <= 3e-5) under all backbone settings, demonstrating the effectiveness of VIP-COP under matched data access.
Paper Revision: We added H3_full comparison that feeds both train and val data as context, please refer to Tables 6-19 in Appendix B for full results. Also we’ve updated aggregated results in Figure 2, Table1 and Figure 4 now also involving H3_full.
Weakness 4 / Question 5: - "it seems the method is sensitive to the quality of the validation data. Can the method benefit from cross-validation?"
Regarding Sensitivity to validation quality. All results are reported on a held-out test set never seen during selection, so overfitting to validation data would surface as lower test performance; instead, test performance improves monotonically over rounds (Figs. 5 and 7).
Context selection is a form of model selection, and reserving labeled validation data for model selection is standard ML practice. When validation poorly predicts test performance (e.g., temporal or grouped shifts as in TabReD), VIP-COP may indeed favor contexts whose gains do not transfer, but validation–test alignment is a generalization concern shared by all methods that tune or select on held-out data, not one specific to VIP-COP.
Regarding cross-validation. We remark that CV does not directly apply to our context optimization setting: Standard CV scores one shared (hyperparameter) configuration across folds and then refits, whereas VIP-COP selects specific rows and columns. Since the eligible rows would differ per fold under CV, each fold would likely yield a different context (consisting of different rows and columns), then, averaging fold scores does not determine which single context to deploy. A CV extension would therefore need a principled way to merge fold-specific contexts. Further, it would have to improve generalization enough to justify the added optimization cost over multiple folds.
We hope our responses clearly address the reviewer’s questions. If the main concerns have been resolved, we would greatly appreciate your reconsideration of the overall assessment. We would also welcome any other questions or suggestions that could strengthen the paper further.
Response
Thank you for your extensive answers! The new experiments have helped clarify some of the doubts I had during the review.
A few things are still holding me back from reconsidering my score
-
The novelty claim. Since as you have mentioned, the work of [1] is very close to yours, I see VIP-COP's main additions to be adaptive sampling using the fitted scores and extension to columns (feature selection). There are also works such as [2], [3], can you discuss those? Related to the adaptive sampling, do you have an ablation showing the benefits of it? For example, I think the cleanest way to test this would be to run the same algorithm twice. In one run, sample subsets uniformly. In the other, let the learned scores guide sampling.
-
Noise ablation is weak. The synthetic noise ablations may make the selection problem easier than it is in practice. For example, replacing targets with independent draws breaks the relationship between features and labels, whereas real datasets often contain features that are redundant, weakly predictive, or useful only in certain periods. If possible, I would like to see an evaluation on Sberbank Housing from TabReD, where a small subset of features appears to carry much of the predictive signal. I am curious if VIP-COP can select a compact context and feature subset that improves test score?
[1] Rundel et al. Rundel, D., Kobialka, J., von Crailsheim, C., Feurer, M., Nagler, T., & Rügamer, D. (2024). Interpretable Machine Learning for TabPFN. World Conference on Explainable Artificial Intelligence (xAI), 465–476.
[2] Yoon, J., Arik, S., & Pfister, T. (2020). Data Valuation using Reinforcement Learning. International Conference on Machine Learning (ICML), PMLR 119, 10842–10851. Proceedings. proceedings.mlr.press
[3] Killamsetty, K., Sivasubramanian, D., Ramakrishnan, G., & Iyer, R. (2021). GLISTER: Generalization based Data Subset Selection for Efficient and Robust Learning. Proceedings of the AAAI Conference on Artificial Intelligence, 35(9), 8110–8118. Proceedings.
We thank the reviewer for the thorough comments and suggestions. We address each Weakness and Question below. Furthermore, a revised manuscript, with changes highlighted in blue, has been uploaded to OpenReview.
Weakness 1 / Question 1: "the method could be extended to regression."
We agree that VIP-COP can be used towards regression natively by replacing AUROC with R2 on Line 10 of Algorithm 1. We evaluate the TabICLv2 regressor on 22 large regression datasets: 4 from TALENT and 18 from OpenML, ranging from 51K to 5.5M samples. On 19 of them, the training set exceeds TabICLv2’s 48K context limit. We consider our original setting and a noisy setting in which 30% of the training targets are corrupted (see Settings in Section 4.5 in blue in the revision).
The results are shown as follows:
Original: VIP-COP improves only marginally over random selection (it outperforms H1 on 14 of 22 datasets, p=0.006) and trails H2 and H3_full. We think the reason is that clean data contain very few harmful samples, so there is little for selection to remove.
Noisy: Here, the context selection matters, as VIP-COP outperforms H1 on 21 of 22 datasets and H2 on 20 of 22 significantly (both p<1e-6), recovering 59% of the R2 lost to noise on average. It is on par with H3_full (p=0.63). Per-dataset results are in Appendix B, Tables 20 and 21 of the revised paper.
Paper Revision: We added Section 4.5 in the revised paper; and included regression results on 22 datasets with both original and noisy settings.
Weakness 2 / Question 3: "The choice of TFM models seems arbitrary. Can the authors include the latest TFM models such as TabPFNv3.5?"
We notice this as a common question among two reviewers and have revised our manuscript to incorporate TabPFN-v3.5-fast, with context of 1M samples and 6,000 features (see blue text in 4.1 Experiment Setup). This backbone is a strong test of the reviewer’s point, because every one of our datasets fits in its context.
For such models, the motivation of avoiding oversized contexts no longer applies. Section 2.2 lists a second purpose of context optimization, removing irrelevant or harmful samples even when the data readily fit context limits.
Under the original and augmentation setting, VIP-COP improves over H1–O2 by approximately 2.0% and 1.52%, respectively. Under sample noise, the improvement rises to 13.40%, and VIP-COP reduces the performance degradation by at least 8 percentage points relative to the baselines. Thus, context selection remains valuable even when capacity is sufficient, particularly when the data contain noise.
Paper Revision: In the main paper, we revise all experiments to now involve TabPFN v3.5-fast results. Individual results are given in Table 8,12.17. Aggregated stats are in Figure 6 and Table 18.
Continued in next comment