- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper introduces a modern benchmark for feature selection on tabular data. It contains 15 feature-selection methods on 71 manually curated tabular datasets.The methods are evaluated under four dimensions: predictive performance, stability, validity, and runtime. The paper ends with recommending that F-Test Is strong on all four axes.
Critical strengths
- There are lots of efforts putting into creating a benchmark, ranging from dataset selection, methodology filtering, etc. The proposed benchmark is considerable larger and more comprehensive than most prior work.
- The evaluation is comprehensive, covering four axes with 5 downstream models.
- The paper provides statistical meaningful results by using repetitions and bootstrap intervals.
Critical weaknesses
- Encoding may confound comparisons. Categorical features are ordinally encoded and some methods then treat the resulting codes as numerical values. This also has further impact on F-Test and other feature selection methods.
- For tree based methods, it has been studies that feature selection biases are prominent, such as https://proceedings.neurips.cc/paper_files/paper/2019/file/702cafa3bb4c9c86e4a3b6834b45aedd-Paper.pdf and https://par.nsf.gov/servlets/purl/10298437
Questions for authors
See weaknesses.
Encoding and inductive bias
Dear Reviewer,
Thank you very much for your effort in reviewing and for acknowledging the scale and usefulness of our empirical comparison of feature selection methods. Both weaknesses raised concern aspects we considered carefully when designing SelectArena. Below, we reflect on W1 and W2. Citations appear in a separate comment.
W1: Encoding impact
The ordinal encoding we use is a part of AutoGluon [14] preprocessing pipeline and not a subject to our own design choices. We agree it might be a suboptimal choice for methods that interpret the encoded values as magnitudes (e.g., F-test, Lasso, LaplacianScore or univariate tests and distance-based), since it imposes an arbitrary order on categories that have none. Some of our implementations are invariant to the particular assignment of numbers to categories by default, e.g. MI, where categorical columns are passed to a discrete estimator. For future work we consider ablating frequency and target encoding.
W2: Bias in the tree-based methods
The feature selection methods and models we use in our benchmark reflect different inductive biases and tree-based methods are a good example of this. The papers mentioned by the reviewer show that tree ensembles overestimate the importance of noisy features, leading to systematic bias in feature selection. Our implementation uses the MDI importance of scikit-learn with fully grown trees, which is the setting in which this bias is the strongest. SelectArena confirms these findings, showing relatively low validity for CART (0.59) and RFImportance (0.67) in comparison to the highest validity of F-Test (0.85) when 50% of the provided features are Gaussian noise (Figure D.10). We extended the framework of SelectArena to a living benchmark with a new public leaderboard added on demand of another reviewer with new methods and more analysis. The per model analysis in the performance tab shows that the aforementioned bias does not translate into poor predictive performance. Among the 15 original methods, RFImportance and CART achieve a high improvement over Random for every downstream model, including non-tree models such as LinearModel and TabICLv2. Including unbiased tree-models will be a next step for the living benchmark.
Please reach out if you have any further questions. If you are satisfied with our response to your feedback, we would appreciate it if you could consider raising your rating.
Thank you again for your review.