acceptodds
Under review as a conference paper at ICLR 2027

PACE: Plug-and-Play Contextual Embedding for Feature Screening with Pretrained Tabular Foundation Models

Abstract

In high-dimensional tabular learning, feature screening provides a lightweight, model-agnostic way to remove irrelevant features before model fitting. However, scoring raw values directly can miss nonlinear or distributional structure. We introduce PACE (Plug-and-Play Contextual Embedding), which inserts a frozen tabular foundation model (TFM) column encoder before an existing feature-scoring rule, expanding each feature into a higher-dimensional contextual representation. Across controlled studies, PACE improves raw-space screening of complex nonlinear dependence with only modest additional encoding cost. These gains translate to downstream prediction on TALENT datasets: PACE-DC improves binary AUC by 0.077 and multiclass macro-AUC by 0.064, with a median normalized RMSE improvement of 0.063 across ten learners. Matched random-weight and random-feature controls show that PACE gains from pretrained structure beyond generic dimensional expansion. PACE further achieves favorable performance–time trade-offs against task-fitted selectors and attribution-based methods, positioning pretrained column geometry as a reusable upstream primitive for tabular learning.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Feature Screening, Tabular Foundation Models, Pretrained Representations, High-Dimensional Data

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_6KkjOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors study feature screening with the help of tabular foundation models (TFMs). Specifically, they propose using TFMs (TabICLv1 in particular) to extract high-dimensional features and then applying standard feature screening methods to select the most relevant features for downstream tasks. Experiments on both synthetic and nuisance-augmented real-world datasets demonstrate the effectiveness of the proposed method.

Critical strengths
  1. The proposed method is reasonable and sound.
  2. The paper is generally well-written and easy to follow.
Critical weaknesses
  1. The idea of using pre-trained models to extract high-dimensional features and then applying feature screening methods does not seem novel to me. Why do the authors need to use TFMs to implement this idea? In addition, as the authors state in Section 3.3, modern TFMs incorporate both feature-wise and sample-wise attention, while the proposed method seems to be built only on frameworks that precede feature-wise attention. Hence, this approach is not fully aligned with the ideas behind modern TFMs. Moreover, TabICLv1 is a relatively old model, and I do not see how the proposed framework could integrate modern TFMs.

  2. The experiments on nuisance-augmented real-world datasets are not convincing. First, in real-world scenarios, most features usually contain some information about the target variable, making this experimental setting unrealistic. Second, compared with TabPFN-Wide and GOTabPFN, the proposed method somehow utilizes the additional information of how many features are useful. I would expect the authors to conduct experiments on real-world datasets without nuisance augmentation and report the performance of the proposed method for different values of kk.

  3. Several parts of the paper are unclear. X\mathcal{X} is used without being defined in Equation (2). The FF-norm and the corresponding inner product are not defined in Line 152. I also do not understand what "not new selection candidates" means in Line 135.

Questions for authors

See the weakness part.

5 comments

Thank you for the questions about the contribution, modern TFM compatibility, and realistic screening budgets. We clarify the role of the pretrained column encoder and add natural-data evidence across multiple feature budgets. Revisions in the manuscript are highlighted in blue for ease of reference.

Weakness 1 — Why use pretrained tabular representations?

TFM column encoders combine tabular-data compatibility, transferable nonlinear representations, and low deployment cost. Their native preprocessing accommodates numerical and categorical variables and missing entries, while pretraining supplies a representation learned across diverse tabular tasks. This makes them well suited to heterogeneous screening problems, where a single prespecified transformation may poorly represent some feature types.

PACE uses the frozen column encoder without task-specific training, so the added computation fits the role of screening as an inexpensive preprocessing step. The resulting ranking retains original-feature identities and can be reused across downstream predictors. This combination of mixed-data support, pretrained geometry, and reusable low-cost screening motivates our choice of TFMs.

Weakness 1 — Why TabICL v1, and how are newer TFMs supported?

We focus mainly on TabICL v1 because its native column encoder matches the feature-wise structure of marginal screening (also known as sure independence screening): each feature is represented using its own observations and then scored against the response. This keeps importance attributable to the original variable and preserves the interpretation of screening as ranking individual features before discarding low-scoring ones.

The ablations in Section 4.5 also cover TabPFN v2.6, TabICL v2, and TabPFN v3. For TabICL v2 and TabPFN v3, we extract response-blind feature representations before cross-feature mixing, using copies of a single column when grouped inputs are required. We expanded Appendix G to explain each extraction procedure, including the group-level representations used in the TabPFN v2.6 layer ablation.

Weakness 2 — Natural data and feature budgets

Thank you for suggesting this evaluation. The TALENT budget uses the original table width to set a common retained dimensionality. To evaluate screening directly on natural tables across different budgets, we added the SelectArena study below.

The study covers 71 natural SelectArena datasets without injected features, including 49 binary, 12 multiclass, and 10 regression datasets. We compare raw/PACE SIS, DC, and HSIC against all 15 published SelectArena selectors. Every method uses the same five dataset-specific budgets, three matched outer folds, and four predictors: TabICL v2, random forest, LightGBM, and a linear model.

For original width pp and j=1,…,5j=1,\ldots,5, the SelectArena budgets are

\lceil pj/6\rceil,&p\leq100,\\ \lceil p^{j/6}\rceil,&p>100. \end{cases}$$ These budgets depend on the observed width, without requiring knowledge of the number of relevant features. On the 61 classification datasets, error reduction relative to random selection is: | Method | Budget 1 | Budget 2 | Budget 3 | Budget 4 | Budget 5 | Five-budget mean | | ------------- | --------: | --------: | --------: | --------: | --------: | ---------------: | | Raw-SIS | 31.12 | 28.68 | 20.37 | 14.90 | 10.82 | 21.18 | | PACE-SIS | 28.14 | 29.68 | 22.44 | 16.72 | 11.60 | 21.72 | | Raw-DC | 31.29 | 27.77 | 21.37 | 14.89 | 9.71 | 21.00 | | PACE-DC | 33.36 | 32.36 | 25.18 | **17.77** | **12.05** | 24.15 | | Raw-HSIC | 33.61 | 31.25 | 23.36 | 15.56 | 10.57 | 22.87 | | **PACE-HSIC** | **35.40** | **33.36** | **25.35** | 17.22 | 11.79 | **24.62** | Entries are percentage error reduction relative to random selection. PACE-DC and PACE-HSIC show clear improvements over their raw counterparts at all five budgets. **SIS gains are concentrated in classification**, where PACE-SIS improves over raw SIS at four of the five budgets. The paired mean PACE-minus-Raw effects below average matched folds, budgets, and predictors within each dataset. Brackets give pointwise 95% intervals from 20,000 paired dataset-bootstrap resamples; effects are percentage points of error reduction relative to random selection. | Evaluation | SIS | DC | HSIC | | -------------------------- | ------------------: | ---------------------: | ---------------------: | | All 71 datasets | −0.08 [−2.23, 1.61] | **+2.94 [1.55, 4.50]** | **+1.71 [0.52, 3.15]** | | 61 classification datasets | +0.54 [−0.94, 2.01] | **+3.14 [1.65, 4.77]** | **+1.75 [0.58, 3.11]** | All three utilities show positive mean gains on classification tasks, and DC and HSIC show clear improvements on both evaluations. These results demonstrate the value of PACE on natural features across a range of budgets determined from the observed table width. We added a dedicated discussion in **Section 4.4**, with the protocol and paired statistical analysis in **Appendix H**. As a complementary robustness check, we also evaluated row-permuted copies of original features as nuisance columns, preserving their empirical marginal distributions. We have added this experiment and its retention and predictive results to **Appendix E.3, “Row-permuted nuisance features” (Table 11)**. ### Weakness 3 — Notation and original selection candidates Thank you for the careful reading and for identifying these ambiguities. We have revised **Section 3.2** to define the feature input space and Frobenius inner product and norm, and to clarify that screening selects original features using their embedding-based scores.

Dear Reviewer 6Kkj,

We appreciate the time and care you have devoted to reviewing our work. Following our responses and supplementary experiments, we wanted to ask whether you have any remaining questions or concerns. We would be glad to discuss them and provide any additional clarification that would be helpful.

Thank you again for the constructive discussion.

Reviewer_fCytOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes PACE, a plug-and-play feature screening framework that leverages frozen Tabular Foundation Model (TFM) column representations to improve feature ranking in high-dimensional tabular data. Instead of directly applying conventional screening utilities to raw features, PACE transforms individual columns into pretrained contextual embeddings while preserving their original identities. This is a good paper from a representation learning perspective with some decent theory. The paper proposes a feature screening or ranking mechanism for high-dimensional tabular data in an interesting way. The authors provide a theoretical information-preservation perspective and extensive evaluations across controlled simulations, TALENT benchmark datasets, and natural high-dimensional benchmarks. The results demonstrate improved feature screening, downstream predictive performance, and computational efficiency across multiple screening utilities and predictive models.

Critical strengths
  • The paper introduces an interesting approach to leveraging pretrained TFM representations for feature screening without modifying downstream predictors or requiring task-specific encoder training.
  • Proposition 1 provides a mathematically sound information boundary characterization, explaining how response blind transformations can preserve marginal information while modifying feature geometry.
  • The experiments cover 25 controlled settings, 300 nuisance-augmented TALENT benchmark datasets, 10 natural high-dimensional datasets, and multiple downstream models, including recent TFMs and their variants.
  • Comparisons against random-weight encoders, random Fourier features, and alternative representation extraction strategies provide evidence that the improvements arise from pre-trained representations rather than dimensional expansion alone.
  • PACE demonstrates favorable performance-time trade-offs, including feature dimensionality scaling experiments involving up to 100K features in controlled simulations.
Critical weaknesses
  • Clarification regarding TabDPT. In Related Work, L105-L106, the paper mentions that TabDPT also supports a high feature count through continued pre-training. But actually, TabDPT uses PCA as an internal layer to reduce the dimensionality. TabDPT uses PCA-based dimensionality reduction when the number of features exceeds its supported limit, rather than handling all high-dimensional features directly through continued pre-training.

  • Incomplete explanation of nuisance augmentation. In Section 4.3, L284-L295, the paper mentions some feature augmentation techniques, and this part deserves some more description and clarity. This introduces Gaussian nuisance features to study screening robustness. While appropriate for the controlled experiment, the construction and its limitations deserve further discussion. Specifically, there are already some established feature augmentation methods for representation learning. Maybe an ablation against recent feature augmentation methods (e.g., AugTab [1], although its objectives differ from nuisance injection) would benefit the paper and strengthen its contribution.

[1]. Habib et al. "AugTab: Learnable Feature Augmentation for Low-Dimensional Tabular Data." ECML PKDD 2026.

  • Unclear hyperparameter protocol for GOTabPFN. Figure 3 and Appendix Table 12 compare PACE-based pipelines with GOTabPFN. However, the manuscript does not clearly specify whether the GOTabPFN baseline follows its recommended dataset-specific hyperparameter optimization protocol, including Optuna-based [2] tuning. Clarification is important for reproducibility and interpreting the comparison. However, other TFMs such as TabPFN, TabICL, or TabPFN-Wide do not require dataset-specific hyperparameter tuning as per their recommended settings. But GOTabPFN mentions in their paper that their model excels when dataset-wise Optuna-based [2] hyperparameter optimization or tuning is adopted. Can the authors confirm that they followed this recommended protocol or that a fixed configuration was used for all datasets? If dataset-specific tuning has been done, then I would recommend that the authors add all dataset-wise tuned best hyperparameters in the appendix. Otherwise, we will probably not see the best result of this model.

[2]. Akiba et al. "Optuna: A Next-Generation Hyperparameter Optimization Framework." KDD 2019.

  • Additional high-dimensional baselines. Although the paper compares several feature screening and selection methods, the evaluation could be strengthened by including additional high-dimensional feature selection approaches. The paper should compare against some high-dimensional, specific tabular models also, not just TFMs. However, the paper mentions STG [3] in the related work, but I believe that, along with STG [3], some other high-dimensional oriented models such as LLSPIN [4] and ProtoGate [5] should also be added as comparative baselines. Their task-specific and instance-wise selection mechanisms should be distinguished from PACE's reusable global feature screening.

[3]. Yamada et al. "Feature Selection Using Stochastic Gates." ICML 2020.

[4]. Yang et al. "Locally Sparse Neural Networks for Tabular Biomedical Data." ICML 2022.

[5]. Jiang et al. "Protogate: Prototype-based Neural Networks with Global-to-Local Feature Selection for Tabular Biomedical Data." ICML 2024.

  • Scope of theoretical guarantees. The theoretical proposition establishes an information boundary for deterministic response blind transformations but does not guarantee active feature recovery or sure screening consistency. Although the authors acknowledge this limitation, a stronger connection between the theoretical analysis and empirical screening improvements would further strengthen the contribution.

  • Minor presentation and bibliographic issues. In Section 4.1, L246-L249 in Evaluation metrics, I believe it’s better to not use OqO_q here, as it visually conflicts with the Big OO notation of computational cost. The notation OqO_q for original feature indices in Section 4.1 could be revised to avoid visual ambiguity with computational Big OO notation. Additionally, some references do not contain their original publication venues. For example, GOTabPFN (L545-L547) should be ICML 2026; BeyondArena (L621-624) should be NeurIPS 2026 ED Track; TabICL (L629-L630) should be ICML 2025; TabICLv2 (L631-L632) should be ICML 2026.

  • Missing architectural illustration: The paper does not provide an architectural diagram or workflow illustration of the proposed PACE framework. Although the methodology is mathematically described, a schematic showing the frozen TFM column encoder, contextual feature representations, screening utilities, and downstream prediction pipeline would substantially improve the clarity and accessibility of the proposed approach. The authors could consider using a tool such as draw.io (https://app.diagrams.net/ ) to create a high quality, publication ready architectural figure.

Questions for authors

Please see the points raised in Critical Weaknesses along with the following questions:

  1. Can the authors provide more details on the Gaussian nuisance augmentation protocol and discuss its relationship to other feature augmentation approaches?

  2. Was GOTabPFN evaluated using dataset-specific Optuna hyperparameter optimization or fixed configurations? If tuned, could the authors provide the best configurations for each dataset?

  3. Could the authors consider additional high-dimensional feature selection baselines, including STG, LLSPIN and ProtoGate, while accounting for their different selection mechanisms?

  4. Can the theoretical analysis be extended to establish sufficient conditions for reliable feature screening or support recovery?

  5. Can the authors clarify whether PACE's screening improvements remain consistent under more challenging feature dependencies and distribution shifts beyond independent Gaussian nuisance contamination?

  6. Can the authors clarify the role of PCA in TabDPT's handling of high-dimensional features and fix their writing?

  7. Could the authors include a high quality architectural diagram to better illustrate the overall PACE framework and its workflow?

8 comments

Thank you for your positive assessment of our work and for the constructive suggestions on the evaluation protocol, additional selectors, and presentation. We address each point below. Revisions in the manuscript are highlighted in blue for ease of reference.

Weakness 2 / Questions 1 and 5 — Nuisance construction and distributional robustness

Thank you for flagging our wording. Our use of augmentation was misleading because it suggested that we were constructing predictive features. We append independent standard Gaussian noise features to test how well screening retains original features. We now call this Gaussian nuisance contamination in Section 4.3 and Appendix E.1. AugTab's learnable predictive-feature augmentation serves a different purpose.

To address your questions about feature dependence and distribution shifts, we also evaluated row-permuted copies of real features as nuisance columns, preserving their marginal distributions while randomizing response alignment. Across 98 binary and 57 multiclass datasets, PACE improved original-feature retention for SIS, DC, and HSIC in both tasks; binary AUC gains were +1.11, +0.41, and +0.68 percentage points, respectively. We have added this experiment, including original-feature retention and predictive gains, to Appendix E.3, “Row-permuted nuisance features” (Table 11). To complement these controls, we added 71 natural SelectArena datasets without artificial nuisance, evaluated at five budgets in Section 4.4, with protocol and statistical details in Appendix H.

Weakness 3 / Question 2 — GOTabPFN tuning

Our original GOTabPFN baseline used a fixed configuration to match the tuning-free evaluation of the other frozen TFMs. We acknowledge that this choice could understate GOTabPFN's performance with dataset-specific tuning. We therefore additionally evaluated Optuna tuning on nine datasets from the natural high-dimensional comparison, with five splits each.

We used TPE over the authors' 14-parameter binary search space for both classification tasks, allowing up to 50 trials and 360 seconds per split. Hyperparameters were selected by validation accuracy, followed by refitting on training plus validation data and evaluation on the held-out test split. All original features were available to GOTabPFN. This uses our existing validation partitions rather than reproducing the authors' repeated-cross-validation example.

PipelineMean accuracyPaired mean advantage over tuned GOTabPFN [95% CI]
GOTabPFN, fixed86.94%—
GOTabPFN, Optuna-tuned88.97%—
PACE-SIS + TabPFN v391.44%+2.48 [0.14, 5.32]
PACE-SIS + TabICL v291.39%+2.43 [0.32, 5.20]

All rows use the same nine datasets; PACE uses the manuscript's 30% feature budget. We average five splits within each dataset and use paired dataset-bootstrap intervals. Tuning increases GOTabPFN's mean accuracy by 2.03 percentage points [−0.43, 6.39]. PACE-SIS retains a positive paired advantage with both predictors under this tuning budget. We have added the tuning protocol and results at the end of Appendix E.6, “Natural high-dimensional classification.”

Weakness 4 / Question 3 — Additional high-dimensional selectors

Thank you for suggesting these comparisons. We evaluated STG and ProtoGate using their authors' implementations on 103 binary TALENT datasets with fewer than 20,000 training observations, with three seeds under the 2,000-feature nuisance protocol.

STG learns global gates jointly with its predictor. ProtoGate couples instance-wise gates to prototype-based prediction. We evaluated their native prediction procedures and also constructed global rankings for matched downstream comparisons: STG uses its global gate parameters, and ProtoGate uses gate values averaged over training observations.

With their native predictors, STG obtains mean AUC 0.7420 and accuracy 78.82%, retaining 648 active global features on average. ProtoGate obtains mean AUC 0.6443 and accuracy 77.10%, with 482 active features per observation on average. These counts describe different selection mechanisms.

For the reusable-ranking comparison, all methods retain k=min⁡(3porig,2000)k=\min(3p_{\mathrm{orig}},2000) features—90 on average—and use the same downstream predictor:

PredictorSTG AUCProtoGate AUCPACE-SIS AUCMedian PACE-SIS gain over STG [95% CI]Median gain over ProtoGate [95% CI]
TabPFN v30.80350.74480.8425+0.76 [0.35, 1.36]+4.87 [2.37, 9.27]
TabICL v20.80570.74880.8419+0.45 [0.24, 0.95]+3.19 [1.40, 7.79]

The table reports mean AUC and median paired gains in AUC percentage points. Comparisons match datasets, seeds, feature budgets, and predictors, with dataset-bootstrap confidence intervals. PACE-SIS improves over both added selectors when their rankings are evaluated at the same budget with the same downstream predictor.

Weakness 5 / Question 4 — Scope of the theory

Thank you for this important question. We explored sufficient conditions for screening consistency, but the main obstacle is that the column encoder processes observations jointly, inducing dependence among their representations. Its data-dependent attention and pretrained structure also complicate uniform control of the screening scores. We do not yet have a clear route to resolving these difficulties and regard this as an interesting direction for future theoretical work.

Weakness 1 / Question 6 — TabDPT and PCA

Thank you for raising this point. We would like to clarify that our manuscript does not describe TabDPT as using continued pretraining; that discussion concerns TabPFN-Wide. TabDPT uses PCA to reduce tables exceeding its feature limit to a compatible input dimension. It is pretrained on real tabular datasets, including datasets with more than 10,000 features, such as la1s.wc (13,195 features) and OVA_Lung (10,935 features), as documented in their paper(Sections 3.4 and 4.1 and Table B.1). When nuisance features account for substantial variance, PCA can discard lower-variance directions carrying predictive signal. This provides a possible explanation for why screening can help: it removes irrelevant features before TabDPT computes the principal components.

Weaknesses 6 and 7 / Question 7 — Notation, references, and workflow

Thank you for your helpful suggestions on notation and references. We have checked both and made corresponding revisions. We also added a workflow diagram in Appendix B to illustrate representation extraction, feature scoring, and downstream prediction.

I thank the authors for their detailed responses and additional experiments. I appreciate their efforts to address my concerns. However, I have two remaining questions:

I understand the motivation for a tuning-free comparison, but evaluating a model without its officially recommended dataset-specific optimization may underestimate its performance. Both tuned and untuned comparisons are valuable. Could the authors clarify whether the 360 second budget was sufficient to complete all 50 Optuna trials and whether the optimization showed convergence?

I appreciate the authors' acknowledgment that the original augmentation terminology was misleading. While the additional row-permuted experiments strengthen the evaluation, they do not fully address realistic feature dependencies. Could the authors consider an ablation with varying correlations between informative and nuisance features to examine whether PACE's advantages remain consistent?

Overall, the rebuttal strengthens my positive impression, but these methodological concerns remain important for interpreting the results.

Reviewer_3k6GOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PACE is a plug-and-play adapter for screening marginal features. Each feature column goes through a frozen column encoder from a pretrained tabular foundation model (TabICL v1, layer 2, d=128) without seeing the labels. An existing screening utility is then applied to the embedding instead of the raw values. Proposition 1 shows that a fixed transform that never sees Y cannot make an independent feature dependent. The method is evaluated on 25 controlled simulations (minimum model size against a known active set), on 300 TALENT datasets padded with Gaussian noise columns (10 downstream learners), on 10 natural high-dimensional datasets, and in ablations against random-weight and random-Fourier-feature controls.

Critical strengths

In simulations where the true active features are known, PACE recovers them much earlier in hard cases (e.g., raw SIS MMS 2179 → 4 in nonlinear classification, 86,096 → 15 at p=100K). Win/tie/loss counts and paired tests are reported per utility (Table 4), so the gains are not hidden in averages.

Matched random-weight encoders and random Fourier features with the same output dimension both do clearly worse than the pretrained encoder (e.g., a 4.91x MMS reduction over random weights for TabICL v1-L2, Fig. 12). So the gain comes from pretraining, not just from adding dimensions- the ablations test the obvious alternative explanation.

Critical weaknesses
  • W1: The +0.077 AUC is for DC only. On the same benchmark, PACE-SIS gains only +0.010 and PACE-HSIC +0.008 (Tables 6–7). Raw DC is unusually poor here: in binary tasks it keeps only 35% of the original features and its recovery is −0.111, meaning screening with raw DC does worse than not screening at all (Table 10). This points to an issue with how raw DC handles real features, not a general advantage of PACE. The noise columns are also i.i.d. Gaussian, so they look very different from real features. Since rankings use uncalibrated dependence scores, the encoder could separate noise from real columns by distribution shape alone. On the natural high-dimensional datasets, which are the realistic test, the gains nearly disappear: PACE-DC 92.5 vs. raw DC 92.4, and PACE-HSIC 92.1 vs. raw HSIC 92.3 (Table 12).

  • W2: Every simulated design uses additive marginal effects (linear, quadratic, exp, log, class shifts). None has a pure interaction signal such as Y = X1·X2, which no marginal screener can detect but SHAP, LOCO, or tree importance can. The cross-paradigm comparisons (Figs. 5–9) are therefore run only on cases built for marginal screening, so they overstate how PACE compares with model-based selectors.

Questions for authors
  1. Why does raw DC perform worse than no screening on TALENT? How are raw features scaled and how are categorical features encoded before DC? If raw DC is fixed (e.g., with standardization or a rank transform), how much of the +0.077 remains? This directly affects the main claim.

  2. Can you repeat the TALENT study with permuted copies of real columns as noise features instead of Gaussian noise? Permuted columns keep each feature's distribution but remove any link to Y. This would show whether PACE is detecting dependence on Y or just the difference between real and Gaussian columns.

3 comments

Thank you for identifying the anomalous DC baseline and the lack of interaction experiments. We acknowledge the DC error and provide corrected results, a distribution-matched nuisance control, new interaction experiments, and a larger natural-data evaluation. Revisions in the manuscript are highlighted in blue for ease of reference.

Weakness 1 / Question 1 — Correction of the DC results

You were right to question the raw-DC results. We apologize for the error and have corrected all affected raw-DC and PACE-DC results, figures, tables, and conclusions. The corrected PACE-DC gains are +0.0032 binary AUC and +0.0012 multiclass macro-AUC, with both 95% confidence intervals above zero. PACE therefore provides a further predictive improvement over the competitive, corrected raw-DC baseline. These corrections are reflected in Section 4.3 and Appendix E.

The gains should also be interpreted under the matched evaluation: the number of retained features, downstream predictor, and data splits are held fixed. In this setting, even a 0.004 AUC gain can be practically valuable because it comes from improving the screening representation without increasing the selected feature count or changing the predictor. The roughly 0.008 or 0.01 gains highlighted in your review thus provide useful evidence for the adapter, particularly given its modest computational overhead. The SelectArena results below further assess this benefit against competitive selectors on natural tables.

For preprocessing, we use feature-wise mean imputation for missing numerical entries and one-hot encoding for categorical features. Continuous features retain their original scales before utility calculation. DC uses normalized distance correlation with Euclidean distances and a shared response representation; classification responses are one-hot encoded. Standardizing the raw continuous features gives identical results, consistent with the scale invariance of normalized distance correlation for a scalar feature. In the additional rank-transform check, downstream performance differs by at most 0.0003, with no statistically significant difference. We clarified the preprocessing in Appendix G to make this comparison explicit.

Weakness 1 / Question 2 — Row-permuted nuisance and original-feature retention

Following your suggestion, we repeated the classification experiment with independently row-permuted copies of original features as nuisance columns. Each copy preserves its source column's empirical distribution, categorical levels, and missingness while randomizing its alignment with the response. Permutations are performed separately within each data split.

The experiment includes 98 binary and 57 multiclass datasets with fewer than 20,000 observations and at most 10 classes. Raw and PACE methods use identical feature budgets. Predictive gains are averaged over three seeds and four learners—LightGBM, MLP, RealMLP, and TabDPT—within each dataset, then equally across datasets. The table combines original-feature retention, shown as Raw → PACE, with PACE-minus-Raw AUC gains in percentage points. Multiclass tasks use macro-AUC; brackets give 95% intervals from 10,000 dataset-bootstrap resamples.

TaskUtilityGaussian retention: Raw → PACERow-permuted retention: Raw → PACEGaussian AUC gain [95% CI]Row-permuted AUC gain [95% CI]
BinarySIS61.78% → 71.81%61.27% → 69.81%+0.94 [0.54, 1.47]+1.11 [0.53, 1.82]
BinaryDC68.94% → 70.50%68.34% → 70.94%+0.87 [0.57, 1.20]+0.41 [0.09, 0.80]
BinaryHSIC65.30% → 66.99%66.07% → 67.41%+1.25 [0.76, 1.76]+0.68 [0.31, 1.07]
MulticlassSIS79.13% → 84.06%78.60% → 82.90%+0.37 [0.15, 0.62]+0.33 [0.00, 0.66]
MulticlassDC81.05% → 81.92%81.08% → 81.89%+0.17 [0.05, 0.31]+0.20 [−0.04, 0.48]
MulticlassHSIC77.96% → 80.19%79.34% → 79.70%+0.61 [0.39, 0.86]+0.42 [0.17, 0.67]

PACE improves retention for every utility in both tasks under row-permuted nuisance. SIS has the largest retention gains: 8.54 percentage points for binary and 4.30 for multiclass classification. Predictive gains also persist, with the largest binary improvements for SIS and HSIC. All three binary AUC intervals and the multiclass HSIC interval are above zero. These results support the value of pretrained screening geometry even when original and nuisance columns share the same marginal distributions. We have added the experimental setup and the retention and predictive results to Appendix E.3, “Row-permuted nuisance features” (Table 11).

We agree that original-feature retention is a proxy for rejecting injected nuisance: original columns can themselves be irrelevant, so retaining every original feature need not yield the best prediction. We therefore assess retention alongside downstream performance. The joint improvements above provide evidence that PACE retains more original features while improving predictive usefulness.

Weakness 2 — Interaction signals and marginal screening

We conducted three additional regression experiments with product interactions and report them below. Each uses 512 observations and 80 paired repetitions. The first two have 100 independent standard normal features and responses Y=X1X2+ϵY=X_1X_2+\epsilon and Y=1.5X1X2X3+ϵY=1.5X_1X_2X_3+\epsilon. The third has 500 features with independent standard normal logarithms and response Y=1.5(log⁡X1)(log⁡X2)(log⁡X3)+ϵY=1.5(\log X_1)(\log X_2)(\log X_3)+\epsilon. In each case, ϵ∼N(0,1)\epsilon\sim\mathcal N(0,1) independently.

The table counts repetitions in which all active features appear among the top 10; higher is better.

MethodTwo-way Gaussian productThree-way Gaussian productThree-way log-product
Raw-SIS8/803/800/80
PACE-SIS9/806/800/80
Raw-DC69/8067/8039/80
PACE-DC80/8079/8072/80
Raw-HSIC80/8079/8059/80
PACE-HSIC80/8079/8077/80
LightGBM-Gain79/8069/8027/80
XGBoost-Gain4/801/800/80

These examples distinguish zero marginal correlation from marginal independence. Here, E[Y∣Xj]=0\mathbb E[Y\mid X_j]=0 for every active feature, so SIS has zero population correlation, including after a response-blind transformation of that feature. However, the conditional response variance changes with the active feature, allowing DC and HSIC to detect dependence. PACE improves DC recovery in all three settings and HSIC recovery in the log-product setting. The benefit of the PACE adapter therefore depends on the screening utility: DC and HSIC can exploit the distributional dependence in these interactions, whereas the zero conditional mean limits correlation-based SIS. Matching the utility to the form of dependence allows the pretrained representation to improve active-feature recovery, including beyond additive signals. We have added these designs and their recovery results to Appendix C.3, “Recovery under product interactions” (Table 6).

Weakness 1 — Evidence on natural datasets

We added SelectArena experiments on 71 natural datasets without injected nuisance, covering five feature budgets, three matched folds, four downstream predictors, and 21 selectors. PACE-HSIC and PACE-DC rank second and third overall by Elo and lie on the observed performance–time frontier.

The table reports paired mean PACE-minus-Raw gains in percentage points of error reduction relative to random selection. We average matched predictors, folds, and budgets within each dataset and obtain pointwise 95% intervals from 20,000 paired dataset-bootstrap resamples.

EvaluationSISDCHSIC
All 71 datasets−0.08 [−2.23, 1.61]+2.94 [1.55, 4.50]+1.71 [0.52, 3.15]
61 classification datasets+0.54 [−0.94, 2.01]+3.14 [1.65, 4.77]+1.75 [0.58, 3.11]

PACE yields positive mean gains over raw screening for all three utilities on classification tasks. SIS gains are concentrated in classification, where PACE-SIS improves over raw SIS at four of the five feature budgets, with a mean paired gain of +0.54 points. PACE-DC and PACE-HSIC show clear improvements over their raw counterparts on both the classification subset and the full benchmark. Together, these results demonstrate the predictive value of the representation adapter across different screening utilities.

To address this concern, we added a dedicated SelectArena evaluation in Section 4.4, with the experimental protocol and paired statistical analysis in Appendix H.

Dear Reviewer 3k6G,

Thank you again for your thoughtful feedback. We have provided detailed responses and additional experimental results addressing your comments. We would appreciate hearing whether these resolve your concerns or whether there are any points you would like us to clarify further.

Thank you for your time and consideration.

Reviewer_4kmwOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes PACE, which inserts a frozen tabular foundation model (TFM) column encoder before an existing feature-screening score. Each feature column is mapped to a higher-dimensional representation without using the response, and existing screening methods such as sure independence screening (SIS), distance correlation (DC), and HSIC are applied to this representation instead of the raw values. The authors evaluate PACE on 25 simulated settings with known active features, TALENT datasets augmented with Gaussian noise columns to reach 2,000 features, and ten natural high-dimensional classification datasets. In the simulations, PACE often reduces the number of top-ranked features needed to recover all active features. On TALENT, PACE-DC improves binary AUC by 0.077, multiclass macro-AUC by 0.064, and median normalized RMSE by 0.063 across ten downstream learners.

Comparisons with random-weight encoders and random Fourier features of matched dimension suggest that the gains come from pretraining rather than dimensional expansion alone. The paper also compares PACE with feature-importance methods, attribution-based methods, and learned selectors, reporting favorable performance–time trade-offs.

Critical strengths
  1. Controls show that pretraining, not just added dimensions, contributes to the gains.

    The paper compares PACE with the same TabICL v1 encoder using random weights. This preserves the architecture, representation dimension, and extraction procedure while removing the effect of pretraining. Across settings, PACE needs a median of 6.38 times fewer top-ranked features than raw input to recover the active set, and 4.91 times fewer than the random-weight encoder. It also outperforms random Fourier features of matched dimension in all six task–utility summaries. The same pattern holds for TabICL v2 and TabPFN v3. These controls address two important alternative explanations: that the gains come from the encoder architecture alone or from generic nonlinear expansion. They support the paper's central claim that pretrained column representations provide useful structure for feature screening.

  2. Broad coverage and low computational cost.

    PACE works with 13 existing screening scores without changing their definitions. It is evaluated on 25 simulated settings with up to 100,000 features and compared with tree-based feature importance, SHAP, SAGE, LOCO, and five learned selectors. The method is also fast. At 2,000 features, PACE takes about 2 seconds, compared with 2,497 seconds for TabICL-SHAP and 17,769 seconds for SAGE-CatBoost. Its runtime scales roughly linearly with the number of features, although DC and HSIC have roughly quadratic costs in the sample size. On the nuisance-augmented TALENT benchmark, all three PACE variants lie on the time–performance frontier, and PACE-HSIC achieves the best average rank. For tables too wide for direct TFM processing, PACE could therefore serve as an inexpensive first screening step without requiring a task-specific model.

Critical weaknesses
  1. The TALENT benchmark may reward distinguishing original columns from noise rather than finding relevant features.

The TALENT study pads 300 datasets with independent N(0,1)N(0,1) noise columns to reach 2,000 features. Real features are often discrete, skewed, or heavy-tailed, so a pretrained encoder could rank original columns above the noise based on their distributions, even when they carry no useful signal. The main metric, original-feature retention, counts every original column as correct, regardless of whether it is informative. There are signs that this may affect the results. Raw DC has negative recovery on binary tasks (-0.111) and regression (-0.020), meaning that screening with raw DC produces worse predictions than keeping all 2,000 columns. It also has the worst ranks and lowest retention among the methods in Table 11.

The headline gains (+0.077 binary AUC, +0.064 multiclass AUC, and +0.063 normalized RMSE) are all reported for PACE-DC against this baseline. PACE-SIS and PACE-HSIC improve AUC by only about 0.01 (Tables 6 & 7). A better control can use row-permuted copies of the original features as nuisance columns. This would preserve each feature's marginal distribution while removing its association with the response. If the gains are consistent under this construction, the evidence that PACE identifies relevant features would be much stronger.

  1. The gains are smaller or mixed for nonlinear screening scores, and the evidence on natural data is limited.

In the simulations, the largest gains occur for SIS, MV, and CAVS, which measure shifts in location and cannot detect quadratic or bimodal signals. Against methods designed to detect nonlinear dependence, the results are mixed: HSIC wins 3, ties 11, and loses 2 of 16 classification settings. In regression, both HSIC and PC lose 3 of 9 settings. On TALENT, PACE-SIS ranks worse than raw SIS in regression (9.43 vs. 8.86), while PACE-HSIC performs similarly to raw HSIC (6.82 vs. 7.38).

The natural high-dimensional study is the only real-data evaluation without added noise, but it covers just ten datasets, some with very few samples. The differences between PACE and raw screening are about one percentage point or less and have mixed signs:

  • TabPFN v3: Raw DC achieves 91.8 vs. 91.7 for PACE-DC; raw HSIC achieves 91.3 vs. 91.2 for PACE-HSIC.
  • TabICL v2: Raw HSIC achieves 92.3 vs. 92.1 for PACE-HSIC; raw DC achieves 92.4 vs. 92.5 for PACE-DC.

No confidence intervals are reported, so it is unclear to me whether these differences are meaningful. Overall, the paper shows that PACE can improve screening in controlled settings, but does not establish that it consistently improves feature screening on natural datasets.

Questions for authors
  1. Does PACE help when nuisance columns resemble real features? Could the authors rerun the TALENT study using row-permuted copies of the original features as nuisance columns? This would preserve each feature's marginal distribution while breaking its association with the response. Please report feature retention and downstream predictive performance for raw and PACE versions of SIS, DC, and HSIC. Also, why does raw DC perform worse than retaining all features on the current benchmark, as indicated by its negative recovery values in Table 10?

  2. How is the response excluded from each backbone's representation? The response-blind property is clear for TabICL v1's column-wise encoder. However, for the TabICL v2 and TabPFN v2.6/v3 representations used in the backbone ablations, how exactly is the response withheld during feature embedding extraction? Is each feature still processed independently at these extraction points?

  3. Are the improvements on natural data statistically meaningful? In Table 12, the differences between PACE and raw screening are approximately one percentage point or less, with improvements in both directions. Could the authors report uncertainty across datasets and data splits, and ideally evaluate PACE on a larger collection of natural high-dimensional datasets without artificially added noise?

4 comments

Thank you for the careful evaluation and the helpful suggestions on the DC baseline, nuisance construction, and natural-data evidence. We address the corresponding weaknesses and questions below.

Weakness 1 / Question 1 — Correction of the TALENT DC results

You were right to question the originally reported raw-DC results. We acknowledge the error and apologize for the inaccurate numbers. We have rechecked the results throughout the paper and corrected the affected raw-DC and PACE-DC results, together with the associated figures, tables, rankings, and conclusions. The corrected matched comparisons are:

TaskOriginal-feature retention, Raw-DC → PACE-DCPACE-minus-Raw downstream effect [95% CI]
Binary classification0.649 → 0.682AUC: +0.00315 [0.00135, 0.00485]
Multiclass classification0.795 → 0.806Macro-AUC: +0.00119 [0.00030, 0.00207]
Regression0.762 → 0.767Normalized RMSE improvement: +0.00228 [−0.00056, 0.00521]

Classification effects are mean paired gains across datasets; the regression effect is the median clean-table-normalized RMSE improvement. The corrected raw-DC baseline is already competitive, and PACE provides a further positive gain on both classification tasks, with confidence intervals above zero. The regression gain is relatively smaller. We have revised the TALENT discussion in Section 4.3 and the corresponding results in Appendix E to reflect these corrected comparisons. Revisions in the manuscript are highlighted in blue for ease of reference.

Weakness 1 / Question 1 — Distribution-matched nuisance and original-feature retention

Following your suggestion, we repeated the classification experiment with independently row-permuted copies of original features as nuisance columns. Each copy preserves its source column's empirical distribution, categorical levels, and missingness while randomizing its alignment with the response. Permutations are performed separately within each data split.

The experiment includes 98 binary and 57 multiclass datasets with fewer than 20,000 observations and at most 10 classes. Raw and PACE methods use identical feature budgets. Predictive gains are averaged over three seeds and four learners—LightGBM, MLP, RealMLP, and TabDPT—within each dataset, then equally across datasets. The table combines original-feature retention, shown as Raw → PACE, with PACE-minus-Raw AUC gains in percentage points. Multiclass tasks use macro-AUC; brackets give 95% intervals from 10,000 dataset-bootstrap resamples.

TaskUtilityGaussian retention: Raw → PACERow-permuted retention: Raw → PACEGaussian AUC gain [95% CI]Row-permuted AUC gain [95% CI]
BinarySIS61.78% → 71.81%61.27% → 69.81%+0.94 [0.54, 1.47]+1.11 [0.53, 1.82]
BinaryDC68.94% → 70.50%68.34% → 70.94%+0.87 [0.57, 1.20]+0.41 [0.09, 0.80]
BinaryHSIC65.30% → 66.99%66.07% → 67.41%+1.25 [0.76, 1.76]+0.68 [0.31, 1.07]
MulticlassSIS79.13% → 84.06%78.60% → 82.90%+0.37 [0.15, 0.62]+0.33 [0.00, 0.66]
MulticlassDC81.05% → 81.92%81.08% → 81.89%+0.17 [0.05, 0.31]+0.20 [−0.04, 0.48]
MulticlassHSIC77.96% → 80.19%79.34% → 79.70%+0.61 [0.39, 0.86]+0.42 [0.17, 0.67]

PACE improves retention for every utility in both tasks under row-permuted nuisance. SIS has the largest retention gains: 8.54 percentage points for binary and 4.30 for multiclass classification. Predictive gains also persist, with the largest binary improvements for SIS and HSIC. All three binary AUC intervals and the multiclass HSIC interval are above zero. These results support the value of pretrained screening geometry even when original and nuisance columns share the same marginal distributions. We have added the experimental setup and the retention and predictive results to Appendix E.3, “Row-permuted nuisance features” (Table 11).

We agree that original-feature retention is a proxy for rejecting injected nuisance: original columns can themselves be irrelevant, so retaining every original feature need not yield the best prediction. We therefore assess retention alongside downstream performance. The joint improvements above provide evidence that PACE retains more original features while improving predictive usefulness.

Weakness 2 — Mixed results for nonlinear screening utilities

PACE involves a practical “no free lunch” trade-off: no single representation improves every utility in every setting, and we acknowledge this limitation. The ties nevertheless have a useful interpretation: all 11 HSIC classification ties have median MMS equal to the active-set size, the best possible value of this metric. In these settings, the representation preserves the optimal median support-recovery performance already achieved by raw HSIC. There is no room to reduce median MMS further.

The broader utility comparison shows substantial gains where raw screening has more difficulty. SIS improves in 14 of 16 classification settings and all nine regression settings; DC improves in eight classification settings with eight ties, and in eight of nine regression settings. The losses for HSIC and PC show that the representation remains utility-dependent. Appendix C.1 reports the complete comparisons, and Section 5 identifies the dependence on utility and representation as a limitation. The natural-data results below further test whether PACE improves strong nonlinear utilities in practice.

Weakness 2 / Question 3 — Natural-data improvements and uncertainty

We added SelectArena experiments on 71 natural datasets: 49 binary, 12 multiclass, and 10 regression datasets, without artificial nuisance. We compare raw/PACE SIS, DC, and HSIC against all 15 published SelectArena selectors. Every method uses five dataset-specific feature budgets, three matched outer folds, and four predictors: TabICL v2, random forest, LightGBM, and a linear model.

The table reports paired mean PACE-minus-Raw gains in percentage points of error reduction relative to random selection. We average matched predictors, folds, and budgets within each dataset and use 20,000 paired dataset-bootstrap resamples for pointwise 95% intervals.

EvaluationSISDCHSIC
All 71 datasets−0.08 [−2.23, 1.61]+2.94 [1.55, 4.50]+1.71 [0.52, 3.15]
61 classification datasets+0.54 [−0.94, 2.01]+3.14 [1.65, 4.77]+1.75 [0.58, 3.11]

Errors are measured by 1−AUC1-\mathrm{AUC} for binary classification, log-loss for multiclass classification, and RMSE for regression. PACE yields positive mean gains over raw screening for all three utilities on classification tasks. SIS gains are concentrated in classification: PACE-SIS improves over raw SIS at four of the five budgets, with a mean paired gain of +0.54 points across classification datasets. DC and HSIC show clear improvements over their raw counterparts on both the classification subset and the full benchmark. We added a dedicated discussion of these utility-specific gains in Section 4.4, with the protocol and paired statistical analysis in Appendix H.

Question 2 — Response exclusion and column locality

Thank you for asking us to make the extraction procedure explicit. The encoder receives feature values only; training responses enter when the screening utility scores the resulting representations. We have expanded Appendix G, “Preprocessing and information access,” to explain how this is implemented for each backbone.

For TabICL v1, extraction occurs before cross-feature interaction. For TabICL v2 and TabPFN v3, we bypass the addition of training-label embeddings and pass the feature projection directly to the column Transformer or feature-distribution embedding blocks. When a block expects a feature group, its slots contain copies of one original feature. The resulting representation therefore depends on that feature across observations.

For TabPFN v2.6, the projection-only ablation also processes each feature separately. The layer ablation supplies zeros to the target encoder, skips cross-group attention, and reads only feature tokens. Features within a native group share a representation, so this ablation is response-blind but has group-level rather than strict per-column locality. We appreciate the opportunity to clarify this distinction.

Thank you for correcting the DC results and for the new experiments. The row-permuted control and the SelectArena results mostly address my questions, and the DC and HSIC gains on natural data are promising. However, after the correction, the TALENT gains are small (PACE-DC: +0.003 binary AUC, +0.001 multiclass). I will hold my rating.

Thank you for your careful review and thoughtful follow-up. We appreciate your time and constructive feedback.

We would like to add that a 0.003 AUC gain (0.3 percentage points) is a clear and practically meaningful improvement over an already strong raw-DC baseline, achieved with the same feature budget and downstream predictor. Its value is also reflected in the normalized nuisance-recovery results. The row-permuted control and SelectArena experiments provide further evidence that PACE’s benefits persist across different evaluation settings.

We are currently extending the SelectArena experiments to assess screening stability and noise discrimination, and will share the additional results when available. We hope you will consider revisiting your rating in light of this additional evidence.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Manuscript Revisions

We thank each reviewer for their valuable feedback. Their suggestions led us to clarify the method and broaden the empirical evaluation.

  • Clarified the method, its scope, and terminology. We explain how the frozen encoder represents each feature without access to the response, how PACE ranks the original features, and when marginal screening can detect interaction signals. We have refined the relevant descriptions and revised some imprecise wording (Section 3 and Appendices E and G).
  • Corrected the distance-correlation implementation and updated all affected TALENT results, figures, and conclusions. PACE-DC continues to improve predictive performance over corrected raw DC at matched feature budgets and with the same downstream predictors (Section 4.3 and Appendix E).
  • Broadened the evaluation on real data to cover 71 SelectArena datasets and five feature budgets. Relative to their raw counterparts, PACE-DC and PACE-HSIC improved error reduction over random selection by 2.94 percentage points (95% CI [1.55, 4.50]) and 1.71 points (95% CI [0.52, 3.15]), respectively. PACE-SIS improved over raw SIS at four of five budgets on classification tasks (Section 4.4 and Appendix H).
  • Tested sensitivity to nuisance-feature construction. With row-permuted nuisance features that preserve the distributions of the original features, PACE improved binary AUC for all three utilities, with all three paired 95% confidence intervals above zero (Appendix E.3). Additional simulations showed improved signal recovery under strong feature correlation across three nonlinear designs.
  • Extended the high-dimensional comparison with additional selectors (Appendix E.6) and 50-trial GOTabPFN searches without a time limit. PACE-SIS retained 30% of the features and achieved 91.39–91.44% mean accuracy across the two predictors, compared with 91.05% for tuned GOTabPFN. Across the same splits, PACE screening accounted for only about 0.7% of the total time spent on GOTabPFN parameter search.

Manuscript revisions are highlighted in blue.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.