acceptodds
Under review as a conference paper at ICLR 2027

VIP-COP: Black-box Context Optimization for Tabular Foundation Models

Abstract

Tabular foundation models (TFMs) have emerged as a powerful paradigm for in-context learning on structured data, enabling direct prediction on new tabular tasks without task-specific training. However, their effectiveness is constrained by context length limits, restricting application to medium-scale data and degrading performance when inference-time data exceed pretraining (size) distributions. Our work introduces VIP-COP, estimating the Value of Importance for Prediction of training examples and features for hard Context OPtimization for TFMs. Its explicit selection mechanism suppresses noise and isolates influential data, enabling the model to also benefit from data augmentation by prioritizing high-value augmented samples and features. VIP-COP is (i) fast, boosting performance often within minutes of optimization, based on an online KernelSHAP based regression with iterative refinement, value-guided context sampling, and multi-fidelity pruning; (ii) budget-aware and any-time, improving with additional test-time compute unlike heuristics that produce fixed context; and (iii) model-aware yet fully black-box, requiring no access to model internals, making it compatible with both proprietary and open-source TFMs; (iv) interpretable, identifying discrete “Very Important Predictors” (samples and features) that maximize signal-to-noise; which makes it (v) robust, isolating high-value data from noise. In contrast,soft-prompt optimization requires model gradients, produce abstract latent tokens,and lack explicit signal discrimination. Extensive experiments show that VIP-COP consistently outperforms heuristic and optimized baselines across large-scale high-dimensional testbeds, including data augmentation and data noise settings, establishing a new state of the art in test-time context refinement for TFMs.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Tabular Foundation Models, Data Augmentation, Context Optimization

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_4Em7OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors propose a new method for compressing the context of tabular PFN models for classification tasks. They test their method on a subset of TALENT and on MacOData benchmarks with TabICLv2 and TabPFNv1 models, where their algorithm outperforms naive and optimized baselines, including inference on the full-context. Broadly, their method works by estimating the importance of each sample and feature via iterative bagging and scoring on the validation set.

Critical strengths
  1. Clearly written and easy to follow, addresses an important problem. Scaling tabular PFNs to larger contexts is an important problem. The paper exposition is clearly written and easy to follow.
  2. Proposed algorithm is intuitive and appears sound. It is an iterative procedure that tries to estimate the importance of each sample by including and then removing it from the training set and scoring on validation.
  3. Empirical performance on the reported set of benchmarks appears to be strong. On the classification, the authors report 1.54 to 14.68% improvement with even more impressive gains on the data noise setup (>100% improvements).
Critical weaknesses
  1. Limited to classification. While this is an important subset of tasks, it would be great if the method could be extended to regression. In particular, it seems if we swap auc-roc with R2R^2 on line 10 in Algorithm 1, the method should natively support regression too. Could the authors explain in more details what's the reason for excluding regression?
  2. Outdated backbones. The authors test with TabPFNv1, which is an outdated tabular PFN. Can the authors also include more recent models such as TabPFNv2 or even v3.5? I wonder if the results would still hold there. In particular, for the latest tabular PFNs which have scaled to the data up to 1M rows, the claim that "larger contexts degrade performance" is no longer true.
  3. Benchmark. This is not critical, but I do find the authors' choice of datasets a bit arbitrary. TALENT benchmark has many more classification datasets (120 binary classification, 80 multi-class), out of which the authors choose to select 38. The inclusion of OddBench is helpful.
  4. Reliance on the validation data. I view this as the major limitation of the current work. Tabular PFN models are attractive, because they don't need validation data to fit. Even if given validation data, they usually just concatenate it directly to the context (as done in TabArena and BeyondArena benchmarks). In particular, the H3 baseline is not done correctly, the TFM should be fitted on both train and validation: VIP-COP in some sense instills information from the validation into the context, so it makes sense to give the baseline methods access to it as well. Besides that, I think VIP-COP can overfit validation when run for many iterations. Does it make sense possibly to use it with cross-validation?
Questions for authors

See above. To re-iterate, I would be interested to understand:

  1. Why omit regression? It seems they have all the right tools to tackle it as well.
  2. What made the authors choose these specific datasets / benchmarks? Can they extend the results?
  3. The choice of TFM models seems arbitrary. Can the authors include the latest TFM models such as TabPFNv3.5?
  4. It seems their estimate for H3 baseline (full-context) is unnecessarily pessimistic. Can you re-run H3 by also including the validation in the context?
  5. Related to the above, it seems the method is sensitive to the quality of the validation data. What if we have temporal or grouped splits (e.g. TabReD or BeyondArena benchmarks), where the validation data does not supply enough good signal about the test-time performance. How would the proposed method fare on such datasets? In general, can the method benefit from cross-validation?
7 comments

We thank the reviewer for the thorough comments and suggestions. We address each Weakness and Question below. Furthermore, a revised manuscript, with changes highlighted in blue, has been uploaded to OpenReview.


Weakness 1 / Question 1: "the method could be extended to regression."

We agree that VIP-COP can be used towards regression natively by replacing AUROC with R2R^2 on Line 10 of Algorithm 1. We evaluate the TabICLv2 regressor on 22 large regression datasets: 4 from TALENT and 18 from OpenML, ranging from 51K to 5.5M samples. On 19 of them, the training set exceeds TabICLv2’s 48K context limit. We consider our original setting and a noisy setting in which 30% of the training targets are corrupted (see Settings in Section 4.5 in blue in the revision).

The results are shown as follows:

SettingBaselineAvg. δR2\delta R^2 (VIP-COP − baseline)Win / Tie / Losep-value
OriginalH1 (Rand)+0.002014 / 2 / 60.006
OriginalH2 (Ens)−0.00362 / 2 / 180.997
OriginalH3 full−0.00462 / 1 / 190.998
NoisyH1 (Rand)+0.044721 / 0 / 1<1e-6
NoisyH2 (Ens)+0.041520 / 0 / 2<1e-6
NoisyH3 full−0.00199 / 1 / 120.625

Original: VIP-COP improves only marginally over random selection (it outperforms H1 on 14 of 22 datasets, p=0.006) and trails H2 and H3_full. We think the reason is that clean data contain very few harmful samples, so there is little for selection to remove.

Noisy: Here, the context selection matters, as VIP-COP outperforms H1 on 21 of 22 datasets and H2 on 20 of 22 significantly (both p<1e-6), recovering 59% of the R2R^2 lost to noise on average. It is on par with H3_full (p=0.63). Per-dataset results are in Appendix B, Tables 20 and 21 of the revised paper.

Paper Revision: We added Section 4.5 in the revised paper; and included regression results on 22 datasets with both original and noisy settings.


Weakness 2 / Question 3: "The choice of TFM models seems arbitrary. Can the authors include the latest TFM models such as TabPFNv3.5?"

We notice this as a common question among two reviewers and have revised our manuscript to incorporate TabPFN-v3.5-fast, with context of 1M samples and 6,000 features (see blue text in 4.1 Experiment Setup). This backbone is a strong test of the reviewer’s point, because every one of our datasets fits in its context.

For such models, the motivation of avoiding oversized contexts no longer applies. Section 2.2 lists a second purpose of context optimization, removing irrelevant or harmful samples even when the data readily fit context limits.

Setting (TabPFN-v3.5-fast)VIP-COp vs. H1–O2
Original (Table 8)+2.00%
Sample augmentation (Table 12)+1.52%
Sample noise (Table 17)+13.40%

Under the original and augmentation setting, VIP-COP improves over H1–O2 by approximately 2.0% and 1.52%, respectively. Under sample noise, the improvement rises to 13.40%, and VIP-COP reduces the performance degradation by at least 8 percentage points relative to the baselines. Thus, context selection remains valuable even when capacity is sufficient, particularly when the data contain noise.

Paper Revision: In the main paper, we revise all experiments to now involve TabPFN v3.5-fast results. Individual results are given in Table 8,12.17. Aggregated stats are in Figure 6 and Table 18.

Continued in next comment

Continued


Weakness 3 / Question 2: - “What made the authors choose these specific datasets / benchmarks? Can they extend the results?”

The selection follows a fixed protocol. We take all publicly available TALENT classification datasets with n > 1000 & d < 100 for small-d, and n > 1,000 & d > 100 for Large-d, then further group small-d into small-d (n<50K) and small-d (n>50K) to cover different backbone context sizes (Section 4.1, Appendix A.1). We excluded the TALENT-EXTENSION many-class datasets because their sources were not publicly accessible. For outlier detection, we use all 240 OddBench datasets with more than 6,000 samples.

Paper Revision: The evaluation is extended in three directions: 1) 22 regression datasets above, 2) new TabPFN-v3.5-fast backbone, and 3) a “small-data” regime in which all samples and features fit within the context.


Weakness 4 / Question 4 - "It seems their estimate for H3 baseline (full-context) is unnecessarily pessimistic. Can you re-run H3 by also including the validation in the context?"

We thank the reviewer for this point and fully agree that the full-context baseline should get access to the same data as VIP-COP. However, feeding train + val both would only serve as data augmentation, and not explicit optimization. Nevertheless, we added a new baseline, H3_full (shown in blue in all results Appendix B, Table 6-19), which places the full training set plus the validation set in the context, giving it the same data budget as VIP-COP. Note that H3 is default to H3_full in all TabPFNv3.5 results.

BackboneAvg. % improv. over H3 (full)p-value
TabPFNv1+13.82%<1e-6
TabICLv2+5.20%9e-5
TabPFN-v3.5-fast+5.64%3e-5

The above results show that VIP-COP still outperforms H3_full overall (p-value <= 3e-5) under all backbone settings, demonstrating the effectiveness of VIP-COP under matched data access.

Paper Revision: We added H3_full comparison that feeds both train and val data as context, please refer to Tables 6-19 in Appendix B for full results. Also we’ve updated aggregated results in Figure 2, Table1 and Figure 4 now also involving H3_full.


Weakness 4 / Question 5: - "it seems the method is sensitive to the quality of the validation data. Can the method benefit from cross-validation?"

Regarding Sensitivity to validation quality. All results are reported on a held-out test set never seen during selection, so overfitting to validation data would surface as lower test performance; instead, test performance improves monotonically over rounds (Figs. 5 and 7).

Context selection is a form of model selection, and reserving labeled validation data for model selection is standard ML practice. When validation poorly predicts test performance (e.g., temporal or grouped shifts as in TabReD), VIP-COP may indeed favor contexts whose gains do not transfer, but validation–test alignment is a generalization concern shared by all methods that tune or select on held-out data, not one specific to VIP-COP.

Regarding cross-validation. We remark that CV does not directly apply to our context optimization setting: Standard CV scores one shared (hyperparameter) configuration across folds and then refits, whereas VIP-COP selects specific rows and columns. Since the eligible rows would differ per fold under CV, each fold would likely yield a different context (consisting of different rows and columns), then, averaging fold scores does not determine which single context to deploy. A CV extension would therefore need a principled way to merge fold-specific contexts. Further, it would have to improve generalization enough to justify the added optimization cost over multiple folds.


We hope our responses clearly address the reviewer’s questions. If the main concerns have been resolved, we would greatly appreciate your reconsideration of the overall assessment. We would also welcome any other questions or suggestions that could strengthen the paper further.

Reviewer_4Em7OpenReview ↗

Response

Thank you for your extensive answers! The new experiments have helped clarify some of the doubts I had during the review.

A few things are still holding me back from reconsidering my score

  1. The novelty claim. Since as you have mentioned, the work of [1] is very close to yours, I see VIP-COP's main additions to be adaptive sampling using the fitted scores and extension to columns (feature selection). There are also works such as [2], [3], can you discuss those? Related to the adaptive sampling, do you have an ablation showing the benefits of it? For example, I think the cleanest way to test this would be to run the same algorithm twice. In one run, sample subsets uniformly. In the other, let the learned scores guide sampling.

  2. Noise ablation is weak. The synthetic noise ablations may make the selection problem easier than it is in practice. For example, replacing targets with independent draws breaks the relationship between features and labels, whereas real datasets often contain features that are redundant, weakly predictive, or useful only in certain periods. If possible, I would like to see an evaluation on Sberbank Housing from TabReD, where a small subset of features appears to carry much of the predictive signal. I am curious if VIP-COP can select a compact context and feature subset that improves test score?

[1] Rundel et al. Rundel, D., Kobialka, J., von Crailsheim, C., Feurer, M., Nagler, T., & Rügamer, D. (2024). Interpretable Machine Learning for TabPFN. World Conference on Explainable Artificial Intelligence (xAI), 465–476.

[2] Yoon, J., Arik, S., & Pfister, T. (2020). Data Valuation using Reinforcement Learning. International Conference on Machine Learning (ICML), PMLR 119, 10842–10851. Proceedings. proceedings.mlr.press

[3] Killamsetty, K., Sivasubramanian, D., Ramakrishnan, G., & Iyer, R. (2021). GLISTER: Generalization based Data Subset Selection for Efficient and Robust Learning. Proceedings of the AAAI Conference on Artificial Intelligence, 35(9), 8110–8118. Proceedings.

Reviewer_KoKJOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

VIP-COP selects training examples and features as context for a frozen tabular foundation model. It scores candidate contexts on validation data and uses the results to guide selection. The paper reports gains over its baselines on original classification data, augmented and noisy data, and outlier detection.

Critical strengths
  1. I like the idea of selecting useful samples and features without needing access to the TFM's internals.
  2. The introduction and Section 2 explain the idea well. I also appreciate having the algorithm written out and broken down in Section 3.
  3. The reported results look strong. I particularly liked the contamination analysis and how it checks which outliers the method removes.
Critical weaknesses
  1. The main comparison needs more attention. I would like a clearer comparison with established methods for selecting samples and features. The analysis should focus more on original classification and show where VIP-COP helps compared with the strongest alternatives [1-4]. The analysis should focus more on classification using the original, unmodified datasets and show where VIP-COP helps compared with strong alternatives under comparable computational budgets.

  2. Parts of the paper are hard to follow. I found Section 3 and the experimental sections difficult to read in places. A simpler explanation of the method and Figure 1 would help. I would keep only the key numbers in the text and move the rest to tables or the appendix.

  3. More backbones and smaller datasets would help. Testing on more relevant TFM backbones would make the claim that the method is backbone agnostic more convincing. I would also like results on smaller datasets, which are an important part of tabular learning.

[1] Feuer, B., Hegde, C., and Cohen, N. “Scaling TabPFN: Sketching and Feature Selection for Tabular Prior-Data Fitted Networks.” NeurIPS Table Representation Learning Workshop, 2023.

[2] Feuer, B., et al. “TuneTables: Context Optimization for Scalable Prior-Data Fitted Networks.” NeurIPS, 2024.

[3] Rundel, D., et al. “Interpretable Machine Learning for TabPFN.” World Conference on Explainable Artificial Intelligence (xAI), 2024.

[4] Thomas, V., et al. “Retrieval & Fine-Tuning for In-Context Tabular Models.” NeurIPS, 2024.

Questions for authors

How does VIP-COP compare with other strong sample- and feature-selection methods in the main classification setting? (see weaknesses)

Could you give a clearer walkthrough of the method, including the loss and how contexts are sampled?

11 comments

We sincerely thank the reviewer for the constructive comments and valuable suggestions on comparisons with other established baselines and evaluation on the latest backbones. We address each point below. Please find the revised manuscript, with changes highlighted in blue, now uploaded to OpenReview.


Regarding Weakness 1 / Question 1 - Additional Baselines:

The reviewer hopes to see “how VIP-COP helps compared with the alternatives [1-4]”: We want to first clarify how VIP-COP differs from the four proposed alternatives along two key axes: (1) model awareness and (2) blackbox nature.

  • Model awareness: The sketching and feature-selection methods in [1] (k-means, coreset, PCA, mutual information) are model-agnostic, as they select context heuristically, based on data properties alone. We have readily included a similar baseline, which is O1 (KMeansReps).

  • Blackbox nature: TuneTables [2] and retrieval with fine-tuning [4] are model-aware but require gradient access or weight updates. Therefore, they are not compatible with the black-box setting that we address.

Therefore, Rundel et al. [3] (DataShapley) is the only directly comparable alternative to VIP-COP. Unlike VIP-COP, however, DataShapley selects samples only, in a non-adaptive single pass. It can therefore be viewed as a restricted alternative to VIP-COP. We added DataShapley [3] as an additional baseline, denoted O3 (Appendix A.4), and evaluated it on classification tasks across all three settings and across backbones under comparable computational budgets (Appendix Tables 6–19, blue O3 columns).

Results show that VIP-COP outperforms O3 significantly across settings (Fig. 2; Appendix Figs. 6, 7; Tables 1, 18, 19).

The summarized pairwise permutation tests of VIP-COP versus O3 are listed below:

ComparisonTabICLv2TabPFN-v1TabPFN-v3.5-fast
p-value<1e-6<1e-68e-05

These results demonstrate that VIP-COP significantly outperforms O3, consistent across all three backbones.

Paper Revisions: Thanks to the reviewer’s suggestions, we have now revised Section 2.3 (Related Work) to include discussions of [1-4], and we have included O3 comparison results for all-settings and all-backbones in the experiments. Please see Appendix Tables 6–17, updated blue texts for individual dataset results.


Regarding Weakness 2 / Question 2 - Readability:

The reviewer commented that “Section 3 and the experimental sections are too hard to read and a simpler explanation of the method and Figure 1”: To improve clarity, we added a step-by-step walkthrough of VIP-COP alongside Figure 1 and added two key operations to aid understanding:

Loss functions (Alg. 1, Line 13; Fig. 1, Step 4): Each sampled context CbC_b has validation AUROC PbP_b and a binary inclusion vector cb\mathbf{c}_b . VIP-COP learns item (feature and sample) importance values by minimizing the squared error between actual performance and the sum of selected items' VIPs:

ℓb=(pb−cb⊤ϕ)2\ell_b = (p_b - \mathbf{c}_b^\top \boldsymbol{\phi})^2

Context sampling (Alg. 1, Lines 6–8; Fig. 1, Step 2): VIP-COP samples nCn_C examples and dCd_C features with probabilities softmax⁡(ϕ/τ)\operatorname{softmax}(\boldsymbol{\phi}/\tau). Sampling starts uniformly and progressively favors high-VIP items over iterations, as their importance estimates are updated.

Paper Revisions: We have moved redundant experimental details into the appendix and updated a simple overview of VIP-COP alongside Figure 1 (blue texts on pg. 4).

Continued in Next Comment

Continued


Weakness 3 - Backbones & Small Datasets:

Regarding the comment that “more backbones and smaller datasets would help”, we want to first clarify what “smaller datasets” mean for TFMs and context selection settings. Whether a dataset is "small" depends on the TFM backbone's context limit. We consider a dataset small when all its samples and features fit within that backbone's context. In this case, our submission already contains results with TabICLv2.

Specifically, our TabICLv2 results show that VIP-COP's selection helps even when no compression is needed. All 16 small-d datasets are smaller than TabICLv2's context (see Table 3, Appx. A), for which VIP-COP still improves over all baselines by a significant margin (Table 7, 11, and 16, Appx. B), signaling the effectiveness of VIP-COP to small datasets with large backbones like TabICLv2:

SettingVIP-COP vs. H1–O2VIP-COP vs. H3_fullVIP-COP vs. O3
Original+3.95%+3.65%+1.10%
Augmented+4.17%+4.41%+2.40%
Noise+7.82%+3.96%+9.42%

Experiments with latest backbones: As requested, we have also provided additional experiments with a larger backbone, the latest TabPFN-v3.5-fast. With 1M samples and 6,000 features, it can accommodate every dataset in this evaluation without compression. We summarize the results for TabPFN-v3.5-fast in the results below:

Setting (TabPFN-v3.5-fast)VIP-COP vs. H1- O2VIP-COP vs. O3
Original (Table 8)+2.00%+0.31% (+1.02% on larger datasets)
Sample augmentation (Table 12)+1.52%+0.49%
Sample noise (Table 17)+13.40%+7.95%

Note that TabPFN-v3.5’s context limit is larger than all of these training sets. When the whole training pool fits, every one of H1-O2 baselines ends up passing all the data and making the same prediction. Our results show that VIP-COP using TabPFN-v3.5-fast outperforms other baselines in ALL settings. Especially under sample noise setting (Table 17, Appx.), the benefit of selection is substantially more pronounced. VIP-COP outperforms H1–O2 by 13.40% and O3 (DataShapley baseline) by 7.95%, and it helps recover at least 8% of the performance decline due to noisy data.

Together, these results demonstrate VIP-COP’s generalizability and effectiveness on large-context backbones such as TabPFN-v3.5-fast, even when context compression appear unnecessary.

Paper Revisions: We have revised our manuscript with the additional comparison results using TabPFN-v3.5-fast. Please refer to Table 18 and Figure 6 for details.


We hope this rebuttal clearly addresses your questions: especially regarding 1) the addition of the O3 baseline and 2) replicating all experiments using TabPFN-v3.5-fast. All experiments in the manuscript are now updated with these new additions. If the main concerns have been resolved, we would greatly appreciate your reconsideration of the overall assessment. We would also welcome any other questions or suggestions that could strengthen the paper further.

Thank you for the rebuttal and additional experiments.

The DataShapley comparison is useful, but I remain unconvinced that it is the main relevant alternative. Being model-aware does not itself establish an advantage over other selection methods. Other method might work better without being model-aware. Therefore, other black-box sample- and feature-selection methods remain relevant, and k-means alone does not adequately cover these alternatives.

Also, the practical importance of the black-box restriction also needs clearer justification.

Thank you for the added the clearer explanation of how the importance scores are learned and used to select contexts.

Adding TabPFN-v3.5-fast is helpful. However, my question about smaller datasets concerned cases with fewer training examples. Showing that the data fit within a larger model’s context does not fully address this.

Moreover, on the new backbone, the improvement over DataShapley on the original data is small and not statistically significant. The larger gains under added noise are interesting, but leave my concern about the main classification comparison mostly unaddressed.

Reviewer_DwwaOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

In this paper, the authors propose a context optimizer for Tabular Prior-Fitted Networks. Specifically, given a black-box TabPFN-like model, and a context dataset containing many samples and features, VIP-COP selects which objects and features should be deleted from the context. This procedure can not only increase the speed of inference, but also improve the performance of the downstream in-context transformer.

To build VIP-COP, authors assign values phi to rows and columns, then sample subsets with probabilities obtained as softmax of values proportional to phi_i. After that, the authors evaluate the model using this sample, and update values phi based on the predictions on the validation subset. At any time, a user could stop this procedure and get a reduced context by choosing top-k features. The paper is evaluated on the TALENT benchmark, where it demonstrates improved performance over the evaluated baselines. The authors also demonstrate interpretability of their method on the experiments with noisy and augmented data.

Critical strengths
  1. The most important quality of the proposed technique is its ability to work with any tabular in-context predictors, including proprietary ones. Additionally, robust performance across different backbones makes VIP-COP a universal component for tabular prediction on large data.
  2. Native support for controlling both the duration of context optimization and the degree to which the context is compressed. This makes it useful in applications where a specific context size is required by computational resources.
Critical weaknesses
  1. The main weakness I see in this algorithm is its use of the predictive performance of a black-box model on the validation subset as an objective. This could lead to unfair comparisons to other compared baselines, which did not use validation subset in any form except for the hyperparameter tuning. Additionally, this over-adaptation to the validation set could reduce the usefulness of values phi as signals of how informative a particular feature is.
Questions for authors
  1. How transferable are the feature-importance values phi across different architectures, including non-foundational models?
  2. Could the results of the study be replicated when using held-out part of the train set instead of the validation set?
2 comments

We thank the reviewer for the thorough comments and suggestions. We address each Weakness and Question below. Furthermore, a revised manuscript, with changes highlighted in blue, has been uploaded to OpenReview.


Weakness 1 - “main issue with validation subset and unfair comparison”

We want to point out that with the backbone model frozen, context optimization is effectively model selection, so choosing a context on held-out validation data is standard practice; it is also cheap in our setting, where input data far exceeds the context capacity, part of which is simply used for validation.

Furthermore, not all baselines can exploit validation data, since they perform no optimization: H1 uses a random context (averaged over samples), H2 ensembles random contexts, and H3 ingests all training data regardless of context limits.

Where applicable, baselines do use validation data: k-means selects the best of 5 centroid initializations by validation balanced accuracy (Appendix A.4), and DT+TFM tunes its DT component on validation data, which keeps the comparisons fair and practically appropriate.

For a fairer comparison we nonetheless add H3_full, which feeds both train + validation data as context; note that here, validation data can only serve as augmented context, and unlike VIP-COP, not for model-aware optimization. In short, VIP-COP still significantly outperforms this H3_full variant, as shown in the results below:

BackboneAvg. % improv. over H3_fullp-value
TabPFNv1+13.82%<1e-6
TabICLv2+5.20%9e-5
TabPFN-v3.5-fast+5.64%3e-5

Paper Revision: We added H3_full comparison which feeds both train and val data as contexts, please refer to blue text in Tables 7-20 in Appendix for full results. We also updated aggregated results in Figure 2, Table1 and Figure 4 with H3_full.


Question 1 - "how transferable are the feature-importance values phi across different architectures, including non-foundational models?"

We thank the reviewer for this inquisitive question, which motivated us to investigate how “universal” or domain-specific selected features are vs. how backbone-specific they are. To this end, we investigate the transferability of VIP_COP selected features to different models by 2 controlled experiments using high-dimensional datasets (d > 100).

Experiment 1: Cross-TFM transfer (TabICLv2 → TabPFN-v1). We run VIP-COP with only feature selection on TabICLv2 . We then transfer its top 100 features to test on TabPFN-v1, and compare performance with transferred features against 100 randomly selected features, averaged over five draws:

DatasetTabPFNv1 (Transferred 100)TabPFNv1 (Random 100)% improvements
gina_agnostic0.9310.843+10.44%
RELATHE_10.8120.631+28.68%
madeline0.7440.613+21.37%
Average+20.2%

Experiment 2: Non-foundational model transfer (TabICLv2 → XGBoost). Similarly, transfer TabICLv2's top 100 features to test on XGBoost. We compare XGBoost performance with transferred features against 100 random features, averaged over five draws:

DatasetXGBoost (Transferred 100)XGBoost (Random 100)% improvements
gina_agnostic0.9280.888+4.5%
RELATHE_10.7950.607+31.1%
madeline0.8420.765+10.2%
Average+15.3%

Transferred features outperform random selection in both TFM and non-foundational models, suggesting that feature subsets selected by VIP-COP on TabICLv2 remain useful across different model architectures. This is an interesting finding that we will include in our revised manuscript.

Continued in next comment

Continued


Question 2 - "Could the results of the study be replicated when using held-out part of the train set instead of the validation set?"

Validation and train are part of the same labeled input data. In other words, validation data comes from the same pool which train data comes from. We use the official train/val/test splits as common best-practice in machine learning.

“Using the held-out part of the train set instead of the validation set” is essentially just another way of splitting the dataset, so we don't expect the results to change in any meaningful way. However, we may not have fully understood the reviewer’s concern and would greatly appreciate it if the reviewer could clarify further.


We hope this rebuttal clearly addresses your questions. If the main concerns have been resolved, we would greatly appreciate your reconsideration of the overall assessment. We would also welcome any other questions or suggestions that could strengthen the paper further.

Comments on the paper

2 comments

CONTINUED

ReviewerRequestResultDetails at
KoKJModel-agnostic selection baselines from Feuer et al. [1]We add CoreSet (greedy k-center), mutual-information (MI) feature selection, and CoreSet+MI, each choosing its own budget on validation data, on 24 datasets in the original, augmented, and noisy settings with TabICLv2. VIP-COP significantly outperforms all three in all three settings (p ≤ 3.3e-3 for all nine comparisons).Response to Reviewer KoKJ
KoKJSmall datasets with few training examplesWe take all classification datasets with n<1000, d<100, and at most 10 classes (the TabICLv2 classifier limit), 25 datasets in all, to create data-starvation conditions. VIP-COP achieves the best mean balanced accuracy in the original, augmented, and noisy settings, improving on the strongest baseline in each by 2.1%, 4.6%, and 4.3% respectively. It improves performance even when the full dataset fits in context, particularly under data noise.Response to Reviewer KoKJ, Table A
4Em7RegressionNew Section 4.5 reports regression on 22 datasets in the original and noisy settings, with R2 replacing AUROC as the objective. VIP-COP improves over baselines, particularly under data noise (both p < 1e-6).Revised Appendix B, Tables 20–21
KoKJ, 4Em7More recent backbones beyond TabPFN-v1 and TabICLv2We extend our evaluation to TabPFN-v3.5-fast, with a context capacity of 1M samples and 6,000 features. VIP-COP continues to outperform baselines by 0.49%–13.40%, demonstrating its effectiveness across backbone architectures and context sizes.Revised Appendix B, Tables 8, 12, 17, 18; Figure 6
4Em7Real, temporally shifted data with weakly predictive features, on Sberbank Housing from TabReDWith the TabReD default temporal split (392 features) and the TabICLv2 regressor, VIP-COP co-optimizes samples and features while using only ~28% of the labeled rows. It lowers RMSE by 25.7% and raises R2 by 165%Response to Reviewer 4Em7
4Em7An ablation on score-guided samplingWe ran VIP-COP twice on 8 large datasets (n>50k, TabICLv2, same 10k-row super-round and query budget), changing only uniform vs. score-guided importance sampling (IS). IS improves the context steadily across rounds, while uniform sampling never improves significantly and falls below its starting context in two-thirds of rounds. The effect carries over to test data . We will add this ablation to the revised manuscript.Response to Reviewer 4Em7

Why VIP-COP

  • Generality. It works across backbones (4), across settings (many), across tasks (3). Backbones include Tabpfn-v1, TabICL-v2, Outformer, Tabpfn-v3.5; settings include original, noise, augmentation, small/large n and d datasets, etc; and tasks including classification, regression, outlier detection.
  • It beats the strongest alternatives, not only simple heuristics. VIP-COP significantly outperforms data valuation (DataShapley), model-agnostic selection (CoreSet, MI), and a full context that includes the validation data.
  • It needs only black-box access. Many foundation models are proprietary, license-restricted, served through an API, or too costly to fine-tune. VIP-COP needs only model queries, so it applies where gradient-based methods such as TuneTables cannot.
  • It still matters when everything fits in context. On TabPFN-v3.5-fast and on small datasets, VIP-COP helps most when the data contain harmful samples, and it does not hurt on clean data. A practitioner who cannot tell in advance whether their data are clean can apply it safely.
  • It transfers beyond classification. Swapping the objective to R2 carries VIP-COP to regression, including a real temporally shifted task where it uses only ~28% of the labeled rows.
  • Its adaptive search distinguishes it. Score-guided sampling keeps improving the context throughout the any-time process, while uniform sampling does not.

We are grateful to the reviewers, whose requests made the paper clearer and better tested. The new results are in Section 4.5 and Appendix B of the uploaded revision, with changes highlighted in blue.

We thank the reviewers for their constructive feedback and questions, and we have attempted to address every concern raised. Reviewer 4Em7 found the additional Sberbank Housing results impressive, extensive experiments helpful, and will reconsider their score. Reviewer KoKJ noted that additional CoreSet and mutual-information comparisons and the new backbone address most of their concerns.


TL;DR.

Tabular foundation models (TabPFN, TabICL, etc.) predict from a context of labeled rows. Real tables are often too large for that context, or contain noisy and weakly predictive samples and features that hurt predictions even when everything fits. VIP-COP optimizes the context itself. It co-selects samples and features with only black-box queries to any frozen backbones, and as stated by Reviewer Dwwa, provides "native support for controlling both the duration of context optimization and the degree to which the context is compressed."

In the revision, VIP-COP significantly outperforms additional baselines: data valuation (DataShapley), model-agnostic selection (CoreSet, mutual information), and a full-context baseline given the validation data. It continues to perform well on the newest backbone, TabPFN-v3.5-fast, on both large and now, also, small datasets, as well as on regression, including a real temporally-shifted task. VIP_COp selected features are also shown to be beneficial when transferred to other models.


List of requests that reviewers have asked and our new results:

ReviewerRequestResultDetails at
Dwwa, 4Em7A full-context baseline that sees both train and validation dataWe add H3_full, which places the full training set plus the validation set in context. VIP-COP outperforms it by 13.82% on TabPFN-v1, 5.20% on TabICLv2, and 5.64% on TabPFN-v3.5-fast (p ≤ 9e-5).Revised Appendix B, Tables 6–19
DwwaTransfer VIP_COp selected features to other models (including Non-TFMs)Features selected on TabICLv2 outperform random features by 20.2% on TabPFN-v1 and 15.3% on XGBoost.Response to Reviewer Dwwa: Q1
KoKJA clearer walkthrough of the methodSection 3 now gives a step-by-step overview alongside Figure 1 and clarifies the loss function, the VIP updates, and importance-guided context sampling.Revised Algorithm 1; Respone to Reviewer KoKJ: W2/Q2
KoKJComparison with the strong alternatives, Feuer et al. [1], TuneTables [2], Rundel et al. [3], Thomas et al. [4]We clarify how VIP-COP differs from the four suggested alternatives and add Rundel et al. [3] (DataShapley) as baseline O3. VIP-COP significantly outperforms O3 by 12.98% on TabPFN-v1 (Appendix Tables 6, 9–10, 13–15), 6.74% on TabICLv2 (Appendix Tables 7, 11, 16), and 2.92% on TabPFN-v3.5-fast (Appendix Tables 8, 12, 17).TabPFN-v1 (Revised Appendix Tables 6, 9–10, 13–15); TabICLv2 (Revised Appendix Tables 7, 11, 16); TabPFN-v3.5-fast (Revised Appendix Tables 8, 12, 17)

CONTINUED IN NEXT POST

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.