acceptodds
Under review as a conference paper at ICLR 2027

SelectArena: A Modern Benchmark for Feature Selection on Real-world Tabular Data

Abstract

Selecting a subset of features (columns) from a tabular dataset is a ubiquitous challenge in real-world applications. Industry experts seek to minimize model training costs, climate scientists require an interpretable subset of factors to understand extreme weather events, and clinicians need to choose among diagnostic tests due to financial and time constraints. When looking at the scientific literature to determine which feature selection method to use, practitioners across domains face an unrepresentative landscape of benchmarking. Despite extensive research, existing feature selection benchmarks provide limited guidance because they rely on outdated models, ad hoc evaluation protocols, or artificially generated data that do not reflect real-world problems. To close the gap between scientific benchmarking and real-world applications, we introduce SelectArena, a modern benchmark for feature selection. We compare 15 widely used feature selection methods on 71 manually curated real-world tabular datasets, covering classification and regression problems, using standardized evaluation protocols and state-of-the-art tabular (foundation) models. Our results reveal clear trade-offs across predictive performance, runtime, stability, and validity. Tree-based methods, CART and RFImportance, along with F-Test are Pareto-optimal when taking into account predictive performance and runtime. F-Test, information-theory-, and distance-based methods lead on stability and validity, with F-Test emerging as a balanced choice across our metrics. Search-based methods are slow, unstable and achieve low performance. SelectArena turns these findings into practical guidance and, as an extensible platform, supports future feature selection research.

Primary area:
datasets and benchmarks
Keywords:
feature selection, benchmark, tabular data, real-world datasets, stability, validity

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_h7fcOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces a modern benchmark for feature selection on tabular data. It contains 15 feature-selection methods on 71 manually curated tabular datasets.The methods are evaluated under four dimensions: predictive performance, stability, validity, and runtime. The paper ends with recommending that F-Test Is strong on all four axes.

Critical strengths
  • There are lots of efforts putting into creating a benchmark, ranging from dataset selection, methodology filtering, etc. The proposed benchmark is considerable larger and more comprehensive than most prior work.
  • The evaluation is comprehensive, covering four axes with 5 downstream models.
  • The paper provides statistical meaningful results by using repetitions and bootstrap intervals.
Critical weaknesses
Questions for authors

See weaknesses.

1 comment
AuthorsOpenReview ↗

Encoding and inductive bias

Dear Reviewer,

Thank you very much for your effort in reviewing and for acknowledging the scale and usefulness of our empirical comparison of feature selection methods. Both weaknesses raised concern aspects we considered carefully when designing SelectArena. Below, we reflect on W1 and W2. Citations appear in a separate comment.

W1: Encoding impact

The ordinal encoding we use is a part of AutoGluon [14] preprocessing pipeline and not a subject to our own design choices. We agree it might be a suboptimal choice for methods that interpret the encoded values as magnitudes (e.g., F-test, Lasso, LaplacianScore or univariate tests and distance-based), since it imposes an arbitrary order on categories that have none. Some of our implementations are invariant to the particular assignment of numbers to categories by default, e.g. MI, where categorical columns are passed to a discrete estimator. For future work we consider ablating frequency and target encoding.

W2: Bias in the tree-based methods

The feature selection methods and models we use in our benchmark reflect different inductive biases and tree-based methods are a good example of this. The papers mentioned by the reviewer show that tree ensembles overestimate the importance of noisy features, leading to systematic bias in feature selection. Our implementation uses the MDI importance of scikit-learn with fully grown trees, which is the setting in which this bias is the strongest. SelectArena confirms these findings, showing relatively low validity for CART (0.59) and RFImportance (0.67) in comparison to the highest validity of F-Test (0.85) when 50% of the provided features are Gaussian noise (Figure D.10). We extended the framework of SelectArena to a living benchmark with a new public leaderboard added on demand of another reviewer with new methods and more analysis. The per model analysis in the performance tab shows that the aforementioned bias does not translate into poor predictive performance. Among the 15 original methods, RFImportance and CART achieve a high improvement over Random for every downstream model, including non-tree models such as LinearModel and TabICLv2. Including unbiased tree-models will be a next step for the living benchmark.

Please reach out if you have any further questions. If you are satisfied with our response to your feedback, we would appreciate it if you could consider raising your rating.

Thank you again for your review.

Reviewer_737LOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper introduces SelectArena, a large-scale, modern benchmark for feature selection on real-world tabular data. It evaluates 15 widely used methods across 71 carefully curated real-world datasets using standardized protocols and five downstream models, including a tabular foundation model, and assesses predictive performance, runtime, stability, and validity. The study finds clear trade-offs, with tree-based selectors (RFImportance, CART) and F-Test on the Pareto front for performance/runtime, and F-Test additionally leading on stability and validity, providing actionable guidance for practitioners.

Critical strengths
  1. This paper conducts a comprehensive evaluation of 15 feature selection methods on 71 manually annotated real-world tabular datasets, covering both classification and regression tasks. The proposed SelectArena benchmark employs a systematic process for organizing datasets and methods, as well as a standardized evaluation pipeline, and makes the code and datasets publicly available, thereby making a significant contribution to the research community. It lays a solid foundation for future research in feature selection.
  2. Unlike many existing benchmarks, which focus primarily on predictive performance, SelectArena comprehensively evaluates predictive performance, runtime, stability, and validity using five different downstream models, including the tabular foundation model TabICLv2. This broader evaluation reveals significant trade-offs among feature selection methods. The final results and findings provide practical insights for feature selection under different priorities.
Critical weaknesses
  1. Although SelectArena outlines multiple criteria for its method of screening representative feature selection approaches, the methods evaluated primarily represent classical approaches, while some newer feature selection paradigms were not included. Therefore, the conclusions in the report may reflect the selected traditional methods more closely than the broader landscape of the feature selection field as it currently stands. Recent research on feature selection has also included methods based on model interpretation, conditional feature importance, stability-based selection, learning-based selection, and feature selection methods tailored for deep tabular models, among others. Although the paper discusses some of these related techniques in the appendix, it does not fully verify whether excluding these methods affects the final conclusions. This seriously undermines the generalizability of the paper’s findings.
  2. The proposed validity metric is poorly designed. It primarily measures the rejection of additional injected Gaussian noise rather than the identification of features that actually carry information. This means that the metric measures the ability to filter out specific types of artificial noise rather than the ability to truly identify useful features. Since the original features themselves may be irrelevant or redundant, high selection accuracy does not necessarily mean that the selected features are actually valid. In short, this may undermine the validity and interpretability of the conclusions.
Questions for authors
  1. Since the one-hour time limit and random fallback affect nearly half of the SBE and SFS runs, and since removing the fallback (Appendix D.2) leads to cardinality violations that prevent a controlled comparison, could the authors clarify whether the poor performance of search-based methods stems from their intrinsic limitations or from the evaluation protocol itself? This design may unfairly penalize computationally expensive methods. If computationally expensive search algorithms still fail to achieve competitive performance even at a higher computational cost such as 10 hours, the comparison results would be even more interesting.
  2. Since model-coupled feature selection methods are uniformly evaluated with LGBM as the proxy model, even when their selected features are later used by substantially different downstream predictors, could this design obscure the benefits of model-specific feature selection and thus fail to support the conclusion that aligning selector and predictor inductive biases provides little advantage?
  3. The inclusion criterion (at least three citations across a fixed set of 33 mostly older references) may systematically favor classical methods, and established recent methods such as Stochastic Gates (Yamada et al., ICML 2020), LassoNet (Lemhadri et al., JMLR 2021), and Concrete Autoencoder (Balın et al., ICML 2019) were excluded. Could the authors clarify whether the 15 selected methods sufficiently represent the current state of feature selection research, and whether the resulting rankings and recommendations can be generalized beyond this method pool?
1 comment
AuthorsOpenReview ↗

New public Leaderboard with new Methods

Dear Reviewer,

Thank you for your review. Your comments motivated a substantial extension: SelectArena now includes three new feature selection methods and is a living benchmark with an public, interactive leaderboard (see W1/Q3) to shape future research. Below, we address weaknesses W1 and W2 as well as your questions Q1-Q3. Citations and * can be found in a separate comment.

W1/Q3: New Methods

We agree that our method selection criteria are biased towards more classical methods due to the citation requirement. Therefore, following TabArena, MulTaBench, and RamanBench [7, 8, 9], as an additional contribution to the paper, we decided to transform SelectArena into a living benchmark by creating a public platform with a leaderboard, which allows users to add and evaluate any feature selection method and provides more detailed results analysis. This way we allow the benchmark to grow organically based on users interest. As a first step and a proof of concept, we added three more recent approaches based on the recommendations: SHAP [11] (model interpretation), CPI [12] (conditional feature importance), and LassoNet [13]. We integrate the results for a single seed in the performance plots (Elo and performance tab) on the anonymized version of our leaderboard. The remaining two seeds of the performance runs, as well as the stability and validity experiments, are still running; we will add them to the leaderboard as soon as they are complete.

W2: Validity & Gaussian Noise

We acknowledge that our validity metric is a proxy and does not perfectly measure the validity of the selected features. However, for our collection of real-world datasets no ground truth for useful features exists, which leaves us with two options only (as stated in Appendix D.4, referenced in Section 4): 1) Apply removal-based methods like ROAR, which relate informativeness to change in performance if selected features are removed. This fails for redundant features common in real-world datasets [5] and there is evidence that 39 of our datasets contain redundant features*; or 2). Assess whether a method can discard features that are uninformative by construction. In our setting, it is more reasonable to assume that features in real-world datasets carry predictive information about the target than to assume we have no redundant features, which is why we implemented the metric as it is. This approach is used across different benchmarking papers in the field [1-4].

Q1: Performance of Search-based Methods

The budgeted, time-limited evaluation protocol is designed this way to reflect practitioner's reality in which preprocessing steps like feature selection are unlikely to be allowed more time than fitting the downstream model. We follow the practices of previous tabular benchmarks like TabArena[7] and set the time limit for both stages to 1h. We acknowledge that this setting does not prioritize predictive performance on the downstream task alone and therefore might not reveal the best performing methods overall but rather under our (realistic) constraints. However, we agree that it might be interesting to investigate the performance of search-based methods beyond the time constraint. First, we analyze the performance of all methods on a subset of 32 datasets on which no method has timed out. The resulting Elo ranking can be found in the performance tab of the new leaderboard. s. We observe that SBE ranks first, revealing its strong predictive abilities, while SFS and LOCO rank the same as on the full experimental grid. However, the scope of this analysis is limited to medium sized and low-dimensional datasets. To complement it, we conduct additional experiments with five high-dimensional datasets, LGBM as the downstream model, one seed and all budgets for all search-based feature selection methods with time limit lifted to 22 hours. We will report results as soon as possible.

Q2: Effects of Proxy Model

We agree that a different proxy model could benefit other downstream models and our conclusions are limited to the LGBM proxy we evaluate. However, in this scope, we find no evidence that alignment helps, both for search- and non-search-based methods (Figure 4). , Search-based methods that use LGBM as as proxy model all achieve their largest improvement over Random with TabICLv2 or RealmMLP (SFS: 18.1% with LGBM vs. 19.3% with TabICLv2; LOCO: 13% vs. 14.8% with RealMLP, SBE: 12.3% vs. 13.2% with TabICLv2). The same holds for non-search-based methods. RFImportance and CART achieve best improvement over Random with TabICLv2 (29% and 25.9%, respectively) rather than with RF or LGBM. Nevertheless, evaluating search methods with other proxy models constitutes a part of our future work.

Please reach out if you have any further questions. If you are satisfied with our response to your feedback, we would appreciate it if you would consider raising your rating.

The authors

Reviewer_tYFeOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces SelectArena, a benchmark for evaluating feature selection methods on real-world tabular data. It compares 15 feature selection methods on 71 curated datasets using five downstream models, including modern foundation models for tabular data. The benchmark evaluates predictive performance, runtime, stability, and feature-set validity under standardized experimental settings.

Critical strengths

The paper evaluates 15 feature selection methods on 71 real-world datasets spanning both classification and regression tasks, and considers multiple evaluation criteria including predictive performance, runtime, stability, and feature-set validity. This large-scale evaluation provides a useful empirical comparison of feature selection methods.

Critical weaknesses
  1. A key concern is that the evaluation protocol may not provide a fair comparison for all feature selection methods. Although the authors investigate the random fallback strategy in Appendix D.2, the ablation does not enforce the same feature budget across methods. In addition, the performance of search-based methods is strongly affected by the one-hour time limit and fallback mechanism. As a result, the reported rankings may partly reflect artifacts of the evaluation protocol rather than the intrinsic feature selection capabilities of the methods.
  2. The paper's empirical findings provide limited new insight beyond confirming well-known properties of existing feature selection methods. The analysis mainly emphasizes aggregate rankings, with limited discussion of why certain methods perform better across different dataset characteristics or downstream models. This weakens the contribution of the benchmark, especially given that its primary value lies in generating actionable insights rather than merely reporting large-scale comparisons.
Questions for authors
  1. How do the one-hour time limit and the random fallback strategy affect the rankings of search-based methods such as SBE, SFS, and LOCO? Are the rankings reliable under these constraints?
  2. How does the use of independent Gaussian noise affect the reliability of the validity evaluation, particularly when irrelevant features are correlated with informative ones?
  3. How does imposing a fixed feature budget affect methods such as Lasso and ElasticNet, and does this constraint materially affect their performance rankings?
  4. The paper investigates hyperparameter tuning for LGBM in Appendix D.2, but it is unclear whether a comparable analysis was conducted for the feature selection methods themselves. Could their hyperparameter settings affect the reported rankings?
  5. The authors conclude that the F-test offers the best overall trade-off among predictive performance, runtime, stability, and validity. Could the authors clarify how these four criteria are aggregated or weighted to support this conclusion, particularly since no explicit weighting scheme is provided?
  6. Beyond confirming known properties of existing feature selection methods, what new insights does the benchmark provide, and is there any theoretical or mechanistic analysis explaining why certain methods perform better under specific dataset characteristics or downstream models?
2 comments
AuthorsOpenReview ↗

Scope reflection and new characteristic-dependent analysis on public leaderboard (1)

Dear Reviewer,

Thank you for your reviewing effort. Your comments helped us clarify the scope of our benchmark and motivated a substantial extension of SelectArena: a publicly available, interactive leaderboard that turns SelectArena into a living benchmark. It offers new insights beyond the paper (see W2/Q6), and it will keep growing as new methods are added. We will include all additional plots in the appendix of our paper. At the same time, we respectfully disagree with some concerns, particularly regarding the validity metric and the evaluation protocol. In this comment, we address W2/Q6 and Q3, in the following comment, we provide our reply to the remaining weaknesses and questions. Citations are provided in a separate comment.

W2/Q6 Limited Insights

We believe our paper offers several new insights, and we strengthened them with additional analyses in response to the reviewer’s comment. In Table 1, we show that the benchmark framework is more comprehensive: the number and characteristics (real-world, i.i.d., predictive) of the datasets, the curated set of feature selection methods, and the multi-axes view do not appear together in any prior work. Therefore, the insights that we share, e.g., in the Recommendations paragraph of Section 5, are unique and more aligned with any real-world scenario a practitioner might encounter than any previous benchmark could provide. Additionally, our results are not just confirming known properties of the feature selection methods but sometimes reveal that the findings of earlier benchmarks don’t generalize on our bigger, real-world dataset collection: Hopf & Reifenrath [18] show in Table 8 that RF methods perform well overall but have higher runtime, and ANOVA (the F-test in our paper) underperforms across performance and stability metrics. Neither is aligned with our findings: F-test is the most balanced choice across all four axes, and RFImportance is Pareto-optimal in runtime and performance (Figure 4). Bakhmach et al. [19] find that information-theoretic methods (JMI, DISR, CMIM) are stable but predictively weak. In contrast, MI in our benchmark is among the only methods that combine high predictive performance with high stability and validity (Figures 5, 6). Regarding dataset characteristics and downstream models, we show, for example, that stability degrades in low-data regimes while validity is less sensitive (Figures 5, 6), and that aligning the inductive biases of the selector and predictor does not help (Figure 4). While we offer explanations where our data supports them (e.g., MarkovBlanket’s preference for removing redundant features), a full mechanistic account of the rankings is beyond the scope of a benchmark paper, whose primary contribution is a standardized, realistic evaluation protocol. To extend our framework, we added a leaderboard, creating a living benchmark. This leaderboard now also includes additional breakdown performance results per dataset size, downstream model, task type, and budget. These largely support our aggregated view, e.g., the ranking by improvement over random is highly consistent across downstream models and budgets (Spearman ρ ≥ 0.83 with the overall ranking), with RFImportance first and Random and LaplacianScore last in all of them. We will add those plots to the appendix of our paper.

Q3 Budget Effect on Regularization-based Methods

To answer the reviewer's question, we conducted a small-scale experiment using native feature subsets (non-zero coefficients) from Lasso and ElasticNet to evaluate the downstream performance of LGBM and a linear model. We use a new metric to compare the new results with five per-budget ones: R(m,d)=Err(all,d)Err(m,d)×100R(m, d) = \frac{\mathrm{Err}(\text{all}, d)}{\mathrm{Err}(m, d)} \times 100 where Err(all,d)\mathrm{Err}(\text{all}, d) is the error of the downstream model on all features of dataset dd and Err(m,d)\mathrm{Err}(m, d) its error on the features selected by method mm. Without a budget, Lasso keeps about as many features as our largest budget, ElasticNet chooses slightly more. Both methods perform similarly to the largest budget with roughly the same number of imposed/selected features. The budget therefore truncates their solution at smaller budgets, as it does for every other scoring method. The corresponding figure can be found in the “without feature budget” section of the performance tab of our leaderboard.

AuthorsOpenReview ↗

Scope reflection and new characteristic-dependent analysis on public leaderboard (2)

W1/Q1 Evaluation Protocol

Although we understand the reviewer’s concern about the evaluation protocol, we disagree that it produces unfair rankings. The evaluation protocol reflects practitioner reality, and we set the time limit and feature budget to ensure a fair comparison. Therefore, the rankings fairly measure how well a method performs under our practical conditions. The time-performance trade-off matters, as shown by TALENT [10], TabArena [7], and BeyondArena [15], which report or impose model fitting time. Practitioners are unlikely to spend more time on a preprocessing step like feature selection than on model fitting; hence we impose the same time limit on both. The fact that search-based methods rank lower under these realistic constraints is a finding, not an artifact. Alternative conditions could also produce interesting insights, but they don't invalidate our benchmark. Regarding the critique of the fallback ablation study: we include the fallback precisely to handle budget violations in timed-out experiments, so the feature budgets differ by default. We designed the ablation study to study how much the timed-out method’s performance depends on the random fallback and which direction this dependency takes, and we present the detailed results in Table D.3 and Figure D.3.

Complementing the budget-axis ablation, we add a time-axis ablation: We analyze the subset of 32 datasets on which no method has timed out and present the results in the ablations section of the “Performance” tab of the leaderboard. SBE ranking first suggests that the time constraint hinders its predictive abilities, and practitioners optimizing only for performance might favor it above RFImportance (rank 2). To see whether this generalizes to bigger datasets, we are conducting a small ablation study on five high-dimensional datasets, using LGBM as the downstream model, with all budgets, a single seed, and for all search-based methods with the time limit lifted to 22 hours. Available within the next few days, the results of these ablations can be interpreted as guidelines for a single-axis (predictive performance) benchmark under alternative conditions.

Q2 Gaussian Noise in Validity

We deliberately use independent Gaussian noise to ensure that the injected features are unambiguously irrelevant, which is a common practice in the literature (as discussed in Section 3.3)[1-4]. Introducing correlations with informative features could cause the injected noise to carry predictive information about the target, making it difficult to establish a clear ground truth for evaluating noise rejection. [16, 17].

Q4 HP Tuning for FS methods

We agree that tuning feature selection methods is an interesting and largely underexplored topic. To our knowledge, prior work mostly focuses on tuning feature selection jointly with the downstream model and not isolating the effect of the method’s own hyperparameters [1, 20]. We use default hyperparameters except for Lasso and ElasticNet, which tune their regularization by internal cross-validation. Tuning against downstream performance would require a validation loop with the downstream model for every configuration, which is too costly within our one-hour limit: [1] report between 1 and 72 hours per tuning experiment. We consider it realistic that practitioners are unlikely to spend more time on tuning for feature selection than on fitting the model itself.

However, we suspect that hyperparameter tuning would affect only some methods. F-Test has no hyperparameters and random forest is known to be robust to its defaults [6]. Search-based methods, already computationally costly, would become even more expensive.

Q5 Weighting scheme between metrics

The weighing scheme we implicitly used is equal weighting: scorem=14(I~m+S~m+V~m+T~m).\mathrm{score}_m = \tfrac{1}{4}\left(\tilde{I}_m + \tilde{S}_m + \tilde{V}_m + \tilde{T}_m\right). F-Test is balanced when comparing the min-max normalized performance (improvement over random II) against the runtime TT of feature selection methods, as well as it proves to have the best stability SS and validity score VV (Appendix Table D.4). Yet, our conclusion holds with different weighting schemes: With equal weights F-Test obtains the highest score (0.98), followed by GainRatio and MI (0.94 each); RF Importance ranks 5th (0.87). Using more performance-focused weights (½ for performance, ⅙ for stability, validity and time), we see F-Test (0.96) at the front in contrast to e.g. RF Importance (0.91). Min-max normalization: x~=x−min⁡(x)max⁡(x)−min⁡(x)\tilde{x} = \frac{x - \min(x)}{\max(x) - \min(x)} (inverse for time)

Please reach out if you have any further questions. If you are satisfied with our reply to your feedback, we would like to kindly ask you to consider raising your rating.

The authors

Comments on the paper

3 comments
AuthorsOpenReview ↗

Central citation comment

In this comment, we provide all papers and explications (*) referenced in our replies:

[1] Valeriia Cherepanova, Roman Levin, Gowthami Somepalli, Jonas Geiping, C Bayan Bruss, Andrew G Wilson, Tom Goldstein, and Micah Goldblum. A performance-driven benchmark for feature selection in tabular deep learning. Advances in Neural Information Processing Systems, 36:41956–41979, 2023.

[2] Isabelle Guyon et al. Design of experiments of the nips 2003 variable selection benchmark. In NIPS2003 workshop on feature extraction and feature selection, volume 253, pp. 40, 2003.

[3] Isabelle Guyon, Steve Gunn, Asa Ben-Hur, and Gideon Dror. Result analysis of the nips 2003 feature selection challenge. Advances in neural information processing systems, 17, 2004.

[4] Isabelle Guyon, Jiwen Li, Theodor Mader, Patrick A Pletscher, Georg Schneider, and Markus Uhr.Competitive baseline methods set new standards for the nips 2003 feature selection benchmark.Pattern recognition letters, 28(12):1438–1444, 2007a.

[5] Sara Hooker, Dumitru Erhan, Pieter-Jan Kindermans, and Been Kim. A benchmark for interpretability methods in deep neural networks. Advances in neural information processing systems, 32, 2019.

[6] Probst, Philipp, Anne-Laure Boulesteix, and Bernd Bischl. "Tunability: Importance of hyperparameters of machine learning algorithms." Journal of Machine Learning Research 20.53 (2019): 1-32.

[7] Nick Erickson, Lennart Purucker, Andrej Tschalzev, David Holzmüller, Prateek Mutalik Desai, David Salinas, and Frank Hutter. Tabarena: A living benchmark for machine learning on tabular data. In The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2025.

[8] Arazi, Alan, et al. "MulTaBench: Benchmarking multimodal tabular learning with text and image." arXiv preprint arXiv:2605.10616 (2026).

[9] Koddenbrock, Mario, et al. "RamanBench: A Large-Scale Benchmark for Machine Learning on Raman Spectroscopy." arXiv preprint arXiv:2605.02003 (2026).

[10] Ye, Han-Jia, et al. "A closer look at deep learning methods on tabular datasets." arXiv preprint arXiv:2407.00956 (2024).

[11] Muschalik, Maximilian, et al. "shapiq: Shapley interactions for machine learning." Advances in Neural Information Processing Systems 37 (2024): 130324-130357.

[12] Chamma, Ahmad, Denis A. Engemann, and Bertrand Thirion. "Statistically valid variable importance assessment through conditional permutations." Advances in neural information processing systems 36 (2023): 67662-67685.

[13] Lemhadri, Ismael, et al. "Lassonet: A neural network with feature sparsity." Journal of Machine Learning Research 22.127 (2021): 1-29.

[14] Erickson, Nick, et al. "Autogluon-tabular: Robust and accurate automl for structured data." arXiv preprint arXiv:2003.06505(2020).

[15] Purucker, Lennart, et al. "Beyond IID: How general are tabular foundation models, really?." arXiv preprint arXiv:2606.30410(2026).

[16] Ron Kohavi and George H John. Wrappers for feature subset selection. Artificial intelligence, 97(1-2):273–324, 1997.

[17] Isabelle Guyon and André Elisseeff. An introduction to variable and feature selection. Journal of machine learning research, 3(Mar):1157–1182, 2003.

[18] Konstantin Hopf and Sascha Reifenrath. Filter methods for feature selection in supervised machine learning applications–review and benchmark. arXiv preprint arXiv:2111.12140, 2021.

[19] Anastasiia Bakhmach, Paul Dufossé, Andrea Vaglio, Florence Monville, Laurent Greillier, Fabrice Barlési, and Sébastien Benzekry. Roofs: Robust biomarker feature selection. arXiv preprint arXiv:2601.05151, 2026.

[20] Binder, Martin, et al. "Multi-objective hyperparameter tuning and feature selection using filter ensembles." Proceedings of the 2020 genetic and evolutionary computation conference. 2020.

*We call a feature redundant if another feature in the same dataset almost determines it: |Spearman ρ| ≥ 0.95 for numeric pairs, bias-corrected Cramér’s V ≥ 0.95 for categorical pairs, and an adjusted correlation ratio ≥ 0.95 between categorical and numeric features, or if it is an exact copy of another feature. Computed on up to 10,000 sampled rows per dataset, 39 of the 71 datasets contain at least one redundant feature. Examples: call minutes and the resulting charges (churn), the debt ratio and net worth to assets (ρ = −1, taiwanese_bankruptcy), exact and average molecular weight (drug_induced_autoimmunity), and a city and its development index (hr_analytics).

AuthorsOpenReview ↗

Additional results from the 22h experiment

Dear Reviewers tYFE and 737L,

in a follow-up to questions W1/Q1 from tYFE and Q1 from 737L, we share results of the search-based methods with a 22-hour time limit, on the five highest-dimensional datasets (12k–22k features), with LGBM as the downstream model, all budgets and a single seed.

LOCO no longer triggered the random fallback, but it scored all features on only one of the five datasets and on the remaining ones roughly 20–25%. Nevertheless, its rank rose from 15 to 11 (out of 18), although it remained well behind the best methods. SFS runs were completed only on the smallest budget for all datasets and on the second smallest on three of five with its average rank still improving from 11.3 to 10.0 across all budgets. SBE did not finish elimination for any dataset or budget, so its output remained entirely random.

This confirms that search-based methods are prohibitively expensive in high-dimensional regimes and their low rank is not merely an artifact of our one-hour limit, since even a 22-fold longer limit does not allow them to complete feature selection on all datasets. More results can be found in the time-constraint ablation section in the performance tab of our leaderboard. We hope that this study helped to clarify your concerns regarding the time limit. All additional analysis and ablations will be added to the appendix.

Please reach out if you have any further questions. The authors

Dear reviewers, dear AC,

Thank you again for your reviews. As the discussion period ends tomorrow, we would like to briefly summarize the main changes to our submission. All new results are now included on our anonymized public leaderboard which we refer to below.

  • New methods: We added SHAP, LassoNet and CPI to broaden the coverage of selection methods and include more recent work [Leaderboard → Elo leaderboard] (737L).
  • Time limit and search-based methods: We showed that even with a 22-hour limit, search-based methods do not complete selection on high-dimensional datasets and do not substantially change the Elo ranking compared to the one-hour limit [Performance → Time-constraint ablation]. We also clarified why time and feature budgets constitute a realistic evaluation protocol (tYFE, 737L).
  • Insights: We highlighted new findings from our results, some of which contradict prior work on smaller or synthetic setups. We also provided a breakdown by dataset characteristics, model, task type and budget [Performance] (tYFE).
  • Validity metric: We explained why using independent Gaussian noise for measuring validity is the most suitable choice for our setup (tYFE, 737L).

The remaining questions were answered in our individual responses to the reviewers.

We would greatly appreciate it if you could let us know whether our replies address your concerns, and if so, update your assessment accordingly.

Sincerely,

The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.