- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
In this paper, the authors study feature screening with the help of tabular foundation models (TFMs). Specifically, they propose using TFMs (TabICLv1 in particular) to extract high-dimensional features and then applying standard feature screening methods to select the most relevant features for downstream tasks. Experiments on both synthetic and nuisance-augmented real-world datasets demonstrate the effectiveness of the proposed method.
Critical strengths
- The proposed method is reasonable and sound.
- The paper is generally well-written and easy to follow.
Critical weaknesses
-
The idea of using pre-trained models to extract high-dimensional features and then applying feature screening methods does not seem novel to me. Why do the authors need to use TFMs to implement this idea? In addition, as the authors state in Section 3.3, modern TFMs incorporate both feature-wise and sample-wise attention, while the proposed method seems to be built only on frameworks that precede feature-wise attention. Hence, this approach is not fully aligned with the ideas behind modern TFMs. Moreover, TabICLv1 is a relatively old model, and I do not see how the proposed framework could integrate modern TFMs.
-
The experiments on nuisance-augmented real-world datasets are not convincing. First, in real-world scenarios, most features usually contain some information about the target variable, making this experimental setting unrealistic. Second, compared with TabPFN-Wide and GOTabPFN, the proposed method somehow utilizes the additional information of how many features are useful. I would expect the authors to conduct experiments on real-world datasets without nuisance augmentation and report the performance of the proposed method for different values of .
-
Several parts of the paper are unclear. is used without being defined in Equation (2). The -norm and the corresponding inner product are not defined in Line 152. I also do not understand what "not new selection candidates" means in Line 135.
Questions for authors
See the weakness part.
5 comments
Weakness 2 — Natural data and feature budgets
Thank you for suggesting this evaluation. The TALENT budget uses the original table width to set a common retained dimensionality. To evaluate screening directly on natural tables across different budgets, we added the SelectArena study below.
The study covers 71 natural SelectArena datasets without injected features, including 49 binary, 12 multiclass, and 10 regression datasets. We compare raw/PACE SIS, DC, and HSIC against all 15 published SelectArena selectors. Every method uses the same five dataset-specific budgets, three matched outer folds, and four predictors: TabICL v2, random forest, LightGBM, and a linear model.
For original width and , the SelectArena budgets are
\lceil pj/6\rceil,&p\leq100,\\ \lceil p^{j/6}\rceil,&p>100. \end{cases}$$ These budgets depend on the observed width, without requiring knowledge of the number of relevant features. On the 61 classification datasets, error reduction relative to random selection is: | Method | Budget 1 | Budget 2 | Budget 3 | Budget 4 | Budget 5 | Five-budget mean | | ------------- | --------: | --------: | --------: | --------: | --------: | ---------------: | | Raw-SIS | 31.12 | 28.68 | 20.37 | 14.90 | 10.82 | 21.18 | | PACE-SIS | 28.14 | 29.68 | 22.44 | 16.72 | 11.60 | 21.72 | | Raw-DC | 31.29 | 27.77 | 21.37 | 14.89 | 9.71 | 21.00 | | PACE-DC | 33.36 | 32.36 | 25.18 | **17.77** | **12.05** | 24.15 | | Raw-HSIC | 33.61 | 31.25 | 23.36 | 15.56 | 10.57 | 22.87 | | **PACE-HSIC** | **35.40** | **33.36** | **25.35** | 17.22 | 11.79 | **24.62** | Entries are percentage error reduction relative to random selection. PACE-DC and PACE-HSIC show clear improvements over their raw counterparts at all five budgets. **SIS gains are concentrated in classification**, where PACE-SIS improves over raw SIS at four of the five budgets. The paired mean PACE-minus-Raw effects below average matched folds, budgets, and predictors within each dataset. Brackets give pointwise 95% intervals from 20,000 paired dataset-bootstrap resamples; effects are percentage points of error reduction relative to random selection. | Evaluation | SIS | DC | HSIC | | -------------------------- | ------------------: | ---------------------: | ---------------------: | | All 71 datasets | −0.08 [−2.23, 1.61] | **+2.94 [1.55, 4.50]** | **+1.71 [0.52, 3.15]** | | 61 classification datasets | +0.54 [−0.94, 2.01] | **+3.14 [1.65, 4.77]** | **+1.75 [0.58, 3.11]** | All three utilities show positive mean gains on classification tasks, and DC and HSIC show clear improvements on both evaluations. These results demonstrate the value of PACE on natural features across a range of budgets determined from the observed table width. We added a dedicated discussion in **Section 4.4**, with the protocol and paired statistical analysis in **Appendix H**. As a complementary robustness check, we also evaluated row-permuted copies of original features as nuisance columns, preserving their empirical marginal distributions. We have added this experiment and its retention and predictive results to **Appendix E.3, “Row-permuted nuisance features” (Table 11)**. ### Weakness 3 — Notation and original selection candidates Thank you for the careful reading and for identifying these ambiguities. We have revised **Section 3.2** to define the feature input space and Frobenius inner product and norm, and to clarify that screening selects original features using their embedding-based scores.Dear Reviewer 6Kkj,
We appreciate the time and care you have devoted to reviewing our work. Following our responses and supplementary experiments, we wanted to ask whether you have any remaining questions or concerns. We would be glad to discuss them and provide any additional clarification that would be helpful.
Thank you again for the constructive discussion.
Thank you for the questions about the contribution, modern TFM compatibility, and realistic screening budgets. We clarify the role of the pretrained column encoder and add natural-data evidence across multiple feature budgets. Revisions in the manuscript are highlighted in blue for ease of reference.
Weakness 1 — Why use pretrained tabular representations?
TFM column encoders combine tabular-data compatibility, transferable nonlinear representations, and low deployment cost. Their native preprocessing accommodates numerical and categorical variables and missing entries, while pretraining supplies a representation learned across diverse tabular tasks. This makes them well suited to heterogeneous screening problems, where a single prespecified transformation may poorly represent some feature types.
PACE uses the frozen column encoder without task-specific training, so the added computation fits the role of screening as an inexpensive preprocessing step. The resulting ranking retains original-feature identities and can be reused across downstream predictors. This combination of mixed-data support, pretrained geometry, and reusable low-cost screening motivates our choice of TFMs.
Weakness 1 — Why TabICL v1, and how are newer TFMs supported?
We focus mainly on TabICL v1 because its native column encoder matches the feature-wise structure of marginal screening (also known as sure independence screening): each feature is represented using its own observations and then scored against the response. This keeps importance attributable to the original variable and preserves the interpretation of screening as ranking individual features before discarding low-scoring ones.
The ablations in Section 4.5 also cover TabPFN v2.6, TabICL v2, and TabPFN v3. For TabICL v2 and TabPFN v3, we extract response-blind feature representations before cross-feature mixing, using copies of a single column when grouped inputs are required. We expanded Appendix G to explain each extraction procedure, including the group-level representations used in the TabPFN v2.6 layer ablation.