acceptodds
Under review as a conference paper at ICLR 2027

FetNAS: Joint Optimization of Feature Selection and Neural Architecture Search

Abstract

Deep learning is increasingly deployed in resource-constrained settings such as IoT and industrial sensing, where memory and compute are limited. In these settings, input features often correspond to physical sensors that are costly to install and read. Because feature relevance is rarely known a priori and datasets often contain irrelevant or redundant variables, an effective model must select as few input features as necessary while also fitting a constrained memory footprint. Existing methods address only one side of this problem: Neural Architecture Search (NAS) treats the input features as fixed, whereas embedded feature selection assumes a fixed architecture, leaving the joint search space largely unexplored. We address this gap with FetNAS, a weight-sharing NAS method that treats the number of input features as an elastic dimension alongside network depth and width. FetNAS trains a single over-parameterized supernet using progressive shrinking, where each subnetwork uses the top-ranked features according to a gradient-based importance ranking rather than a random ordering. Once the supernet is trained, subnetworks are evaluated without retraining and a multi-objective evolutionary search identifies the Pareto front over predictive performance, parameter count and input dimensionality. Across the TabArena benchmark, FetNAS achieves a better performance-complexity trade-off than the baselines for classification. Up to the third quartile of model size, it typically uses - fewer parameters and - fewer input features. For regression, gains are more modest and budget-dependent. FetNAS improves performance at moderate feature budgets but retains a larger share of the features than for classification. Beyond reducing model complexity, the explicit feature subset lowers data-acquisition cost and eases edge deployment.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Neural Architecture Search, Feature Selection, Tabular Deep Learning, End-to-end Machine Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_77BxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes FetNAS, a weight-sharing NAS method that jointly searches over the number of input features, network depth, and layer width. It adds an elastic-features stage to progressive shrinking and uses a gradient-based feature ranking to construct nested feature subsets. After supernet training, multi-objective evolutionary search finds subnetworks that balance predictive performance, parameter count, and feature count using inherited weights without retraining. Experiments on TabArena show improved performance–complexity trade-offs for classification, while gains for regression are smaller and depend on the feature budget.

Critical strengths

The paper studies a relevant problem by considering feature selection and neural architecture search together for resource-constrained applications. The evaluation covers 51 TabArena datasets and shows some benefits for classification under limited feature and parameter budgets.

Critical weaknesses
  1. FetNAS adds an elastic-features stage to progressive shrinking and combines feature importance ranking with multi-objective evolutionary search. However, the novelty beyond this combination seems somewhat limited.
  2. FetNAS uses the same fixed feature ranking for all subnetworks and searches only over the number of features. This may limit its ability to select suitable feature subsets for different architectures.
  3. The work does not report comparisons of actual training time, search time, or computational resource usage, making it unclear how much computational cost FetNAS saves in practice.
  4. The baselines mainly combine UF or mRMR with Huang’s rule of thumb or evolutionary NAS. Comparisons with embedded feature selection methods and existing methods for joint feature selection and architecture search are missing.
Questions for authors

My questions are reflected in the concerns raised in the Weaknesses part.

3 comments

We thank the reviewer for the valuable comments. Please find our response to your questions below.

77Bx-Q1

Please see "Contribution and novelty" in the general response.

77Bx-Q2

Please see "Fixed ranking vs. arbitrary feature subsets" in the general response. We are including a non-ranking variant which samples over arbitrary feature subset per subnetwork. Random subsets are competitive, though ranking methods still yield better results and serve as a good heuristic (difference of +0.011+0.011 [+0.004+0.004, +0.017+0.017] F1/R2R^2 in favour of the ranking at matched budgets).

77Bx-Q3

Please see "Computational cost" in the general response: FetNAS covers the five feature budgets of the paper with 28% of the GPU time of the sequential pipelines.

77Bx-Q4

For embedded feature selection we added LassoNet, see "Embedded feature-selection baseline" in the general response.

For the joint methods, we agree that prior work exists, e.g. [1], which co-evolves neural architectures and input features. However, we want to point out that this approach trains every candidate model from scratch during the search. In FetNAS, the supernet is trained only once and every candidate inherits its weights, so it is evaluated without retraining. A direct comparison under our protocol is therefore not possible.

[1] Hafiz, Faizal, Jan Broekaert, Davide La Torre, and Akshya Swain. “Co-Evolution of Neural Architectures and Features for Stock Market Forecasting: A Multi-Objective Decision Perspective.” Decision Support Systems 174 (November 2023): 114015.

Thank you for the detailed response. I still have concerns about the methodological novelty. Therefore, I keep my original rating.

We thank the reviewer for taking their time to go to our response.

Thank you for your response. However, we still believe that our proposed approach to jointly optimize feature selection and neural architecture should be considered a contribution to the literature. To the best of our knowledge, no existing NAS method treats feature selection as a searchable dimension in a weight-sharing supernet within the NAS loop. Consequently, FetNAS can sample architectures for arbitrary feature and parameter budgets from a single supernet without requiring retraining or a separate supernet for each feature budget.

In addition, we propose an elastic-features stage that enables subnetworks with different input feature sets to share weights. Unlike depth or width, removing a feature removes information, so sharing weights across subnetworks with different inputs is not a straightforward extension. We also define and study several feature-ranking methods and compare them with each other and with random feature sampling. Finally, through our evaluation on the TabArena benchmark, we show that our method is valuable and competitive with sequential feature-selection-and-NAS pipelines, embedded feature-selection methods and handcrafted architectures.

We will make sure that this is clarified in the revised version of the paper.

Reviewer_d5nAOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

FetNAS extends weight-sharing neural architecture search to input feature count. It trains an MLP supernet through progressive shrinking, then searches over depth, width, and prefixes of a fixed gradient-based feature ranking. Experiments on 51 TabArena datasets show favorable classification trade-offs, while regression gains are more limited. The approach is practical, but the evaluation does not sufficiently isolate the contribution of feature elasticity or establish the reliability of the comparisons.

Critical strengths

Jointly controlling input requirements and model size is relevant to resource-constrained learning. Supporting multiple budgets with one supernet is a useful practical contribution. The evaluation spans 38 classification and 13 regression datasets, with five cross-validation splits and detailed per-dataset results. The main comparison appropriately selects models on validation data before test evaluation. The paper reports promising compact-model results for classification and acknowledges the fixed-ranking restriction, weaker regression gains, and unquantified retraining gap.

Critical weaknesses
  1. The fixed ranking limits the scope of joint optimization.

Equation (1) considers arbitrary feature subsets, whereas FetNAS restricts all architectures to prefixes of one ranking. It jointly optimizes architecture and feature count, but cannot adapt feature identities to architecture. The acknowledged restriction warrants quantification across model capacities and feature interactions. The novelty claim should remain specific to elastic inputs in a weight-sharing supernet, rather than joint feature-architecture optimization generally.

A sequential baseline using the same gradient ranking would help isolate elastic-feature training. Section 4.5 already compares alternative rankings within the elastic pipeline; this alone does not separate ranking quality from the benefit of elasticity. The random ablation also needs clarification: Equation (3) suggests random ranking, while the discussion describes random subset sampling. Fixed random prefixes and independently resampled subsets test different hypotheses. Please specify resampling frequency and input-column mapping, compare these controls, or weaken the claim that importance-ranked nesting is necessary.

  1. The comparisons do not isolate the source of the gains.

Appendix C gives FetNAS 510 training epochs per supernet versus 390 for Evo; RoT also uses a different optimizer and batch size. An equal-compute Evo control and clarification of the RoT training differences are needed to distinguish elasticity from optimization effects. The fixed UF threshold and four-point mRMR sweep motivate a tuned sparse MLP or embedded feature-selection baseline.

Please report total computation across multiple budgets, including Evo-mRMR's four supernets, actual candidate counts, BN recalibration, caching, search time, hardware, and peak memory.

Inherited-weight results warrant validation. On anneal, Appendix D reports RoT-All at F1-macro 0.91 with 6.3K parameters, versus 0.70 with 53.4K for FetNAS at the largest Evo-All budget point. These are fold averages and use different feature subsets/training schemes, so the gap does not isolate its cause. Retrain or fine-tune representative small, medium, and large candidates and report performance gaps and ranking agreement. On amazon employee access, scores near 0.49 also warrant a dummy baseline, per-class recall, and training diagnostics.

  1. The evaluation and uncertainty reporting need clarification.

Section 4.4 selects models on validation data, but Section 4.5 explicitly constructs fronts using test performance to compare ranking strategies. This can describe the candidate pool; it does not directly evaluate a validation-selected deployment policy. Clarify the pool and additionally report test results for a frontier or budget policy frozen on validation data. Specify hypervolume reference points, objective transformations, clipping, treatment of negative R2R^2, and reference-point sensitivity.

Table 2 describes standard deviations across datasets, yet reports RoT-All parameter count as 46.1K ±\pm 0 despite different model sizes in Appendix D. Specify the dataset/fold aggregation formula and distinguish between-dataset variation from cross-validation variability. Report dataset-level paired differences with uncertainty intervals for the main baselines; the current CD diagram only compares ranking variants. Folds and budget points should not be treated as independent datasets.

Questions for authors
  1. Feature count and parameter count are proxies for acquisition and deployment costs. Please qualify claims of actual cost savings unless supported by grouped-sensor acquisition costs or measured latency and memory usage. Likewise, Appendix F’s hypervolume-weighted feature-reduction score does not directly measure acquisition savings; reporting retained feature counts at a fixed predictive-performance tolerance would make the practical trade-off clearer.

  2. Specify scaling, categorical encoding, imputation, and which preprocessing steps are fitted exclusively within each training fold. Gradient rankings depend on feature units; clarify whether feature counts refer to raw variables or encoded columns. In Section~3.2, nested slicing preserves parameter-index alignment but does not by itself guarantee the absence of gradient interference. Please qualify ``preventing interference'' or provide supporting evidence.

  3. Complete the training specification, particularly the task and distillation losses, loss weights, and stage-specific minimum subnetworks, including whether all input features remain active before the elastic-features stage. Equation~(2) measures sensitivity of the task loss rather than of the prediction itself. Appendix~C explicitly includes BN recalibration in ``zero-shot'' evaluation; please make its computational cost explicit.

  4. AppendixC already explains that multiclass mRMR combines class-wise feature sets, while the main-text footnote mentions only rounding. Please state both reasons for exceeding the requested retention fraction in the main text and report the integer feature count at each budget point, clarifying rounded ratios such as k/d=0.00k/d=0.00 and the handling of infeasible budgets. Table3 already notes that UF regression results cover only nine datasets; cross-group comparisons should therefore use a common dataset subset. Please also explain Table4's binary-task ``#Classes'' entries of1 and rename the column if it denotes output units.

6 comments
AuthorsOpenReview ↗

Official Comment by Authors (Part 1)

We thank the reviewer for the valuable comments. Please find our response to your questions below.

d5nA-W1a: Fixed ranking vs. Equation (1)

Please see "Fixed ranking vs. arbitrary feature subsets" in the general response. We agree that the novelty claim should be specific to elastic inputs in a weight-sharing supernet and the zero-shot architecture sampling and will make this clear in the revised version of the paper.

d5nA-W1b: Missing sequential baseline with the same gradient ranking

We are currently running this ablation for each budget a dedicated supernet is trained on the first 20/40/60/80% of the features of the same gradient ranking that FetNAS uses. We will post the results once available.

d5nA-W1c: Random ablation is ambiguous

Please see "Fixed ranking vs. arbitrary feature subsets" in the general response.

d5nA-W2a: Unequal training budgets and optimizers

The difference in epochs is structural rather than arbitrary. FetNAS trains for 510 epochs per supernet: 150 epochs of full training plus one 120-epoch elastic stage each for layers, units and features. Evo trains for 390 epochs: the same full training and only the two elastic stages it supports (layers and units). The gap of 120 epochs is therefore exactly the elastic-features stage.

To check whether the additional epochs explain the difference in performance, we add results for Evo with the same training budget of 510 epochs. Preliminary result on a subset of datasets (the larger ones are still running): the extra epochs help Evo-All slightly and FetNAS remains ahead in classification (+0.02+0.02 F1-macro [+0.004+0.004, +0.042+0.042], better on 15 and worse on 4 datasets), while results for regression remain more or less same.

RoT is a single model trained from scratch and not a supernet, so it uses a different training setup, namely Adam with a batch size of 16 for 150 epochs. To check whether this setup affects the comparison, we are currently running experiments on the NAS variants with the same training settings (optimizer and batch size) as RoT. We will post the results once available.

d5nA-W2b: Weak feature-selection baselines

Please see "Embedded feature-selection baseline" in the general response.

d5nA-W2c: Total computation not reported

Please see "Computational cost" in the general response.

d5nA-W3a: Test-based fronts in Section 4.5

We agree that, under standard practice, model selection should be performed using the validation set. Our intention, however, was not to use the test set for model selection, but rather to provide an unbiased comparison and ranking of the FetNAS variants based on their final test performance. To address this concern and assess the consistency of our conclusions, we will include the same analysis using the validation set.

Fronts are selected on (validation score, parameters, features) and the hypervolume is computed on the test scores of those models. The reference point is the worst value of each objective over all compared variants and splits of a dataset. No additional transformations are made to the objectives.

d5nA-W3b: Aggregation and statistical reporting

Regarding Table 2, the standard deviation is computed over the five seeds and not over the dataset. The metric is first averaged over all datasets and the std is then calculated over the seeds. This is also why RoT-All shows 46.146.1K ±0\pm 0, since its architecture is identical for every seed. The variation between datasets is shown in Appendix D (Tables 5 and 6), which reports the results of FetNAS and all baselines for each dataset separately, so the gain per dataset can be read directly from these tables.

AuthorsOpenReview ↗

Official Comment by Authors (Part 2)

d5nA-Q1: Cost claims

Feature count and parameter count are indeed proxies for, rather than direct measurements of, acquisition and deployment cost and we acknowledge that we do not have measured acquisition-cost for a specific case. We therefore motivate the cost-reduction argument through an application where the proxy is close to the real quantity: in in-mold sensor optimization [1], different features map to a specific physical sensor that must be installed, so feature reduction directly reduces acquisition cost. As mentioned under "Additional experiments" in the general response to all reviewers, we further add mRMR as a redundancy-aware ranking, reducing redundant features and thus the sensors to be acquired. Note that the parameter count is only one possible complexity objective: it can easily be swapped for FLOPs or other runtime metrics such as measured latency or memory usage, since the objective is only used during the evolutionary search and does not change the supernet training. Per the reviewer's suggestion, we will also add the number of features retained at a fixed predictive-performance tolerance for FetNAS compared to baseline methods in the appendix. To summarize: on classification FetNAS reaches the same test score as the sequential pipelines with 38% instead of 57% of the features, on regression with 71% instead of 76%.

[1] Seo, Jaho, Amir Khajepour, and Jan P. Huissoon. “Optimal Sensor Location to Estimate Temperature Distribution in an Injection Mould.” September 14, 2011, 253–60.

d5nA-Q2: Preprocessing and “preventing interference”

We use the official TabArena train/test splits and hold out 20% of the training rows for validation. All preprocessing is fitted on the remaining training rows only. Constant columns are dropped, missing numeric values are imputed with the training median, numeric features and regression targets are standardised with training statistics. Categorical features are coded with the categories seen in training (one extra code for missing or unseen values) and encoded inside the network (one-hot for up to nine codes, a learned 8-dimensional embedding otherwise).

Feature counts refer to raw variables: a categorical feature counts once, is kept or removed as a whole and its ranking score is the mean over its columns. Numeric inputs are standardised, so gradient scores are comparable across features.

Please qualify 'preventing interference' Nested slicing guarantees that a smaller subnetwork uses the same weights and inputs as part of every larger one, which removes one source of conflicting updates but cannot guarantee the absence of gradient interference. We will rephrase to “reduces”.

d5nA-Q3: Complete training specification

Losses. The task loss is cross-entropy (classification) or mean squared error (regression). In the elastic stages we use sandwich sampling with in-place distillation [2]. Per batch the largest subnetwork is trained on the task loss and the smallest and two random subnetworks on the task loss plus a distillation loss (weight 1) towards the detached outputs of the largest. The distillation loss is the KL-divergence at temperature 2 for multiclass tasks and the mean squared error on the logits otherwise. The gradients of the four subnetworks are averaged into one optimizer step.

Minimum subnetworks. A dimension stays at its maximum until its stage starts, so one hidden layer at full width in the elastic-layers stage, one hidden layer of 8 units from the elastic-units stage on and additionally a single feature in the elastic-features stage.

Features before the elastic-features stage. Yes, all features are active in all subnetworks until that stage. The ranking is computed once at its start.

Equation (2). Correct. The score is the gradient of the task loss with respect to the input. We will state this explicitly.

BN recalibration. “Zero-shot” means that no weight is updated by gradient descent after supernet training. The BatchNorm statistics of every candidate are re-estimated with forward passes over 64 fixed training batches, this cost is included in the reported search times and quantified in "Computational cost" under our general response.

[2] Yu, Jiahui, and Thomas S. Huang. “Universally Slimmable Networks and Improved Training Techniques.” 2019, 1803–11.

AuthorsOpenReview ↗

Official Comment by Authors (Part 3)

d5nA-Q4: mRMR budgets, table consistency

mRMR retention. k/dk/d can exceed the requested fraction because f⋅df \cdot d is rounded up and because for multi-class tasks mRMR is run separately for each class (one-vs-rest) and the top-kk features of all classes are combined.

Feature counts. We add the feature count kk next to k/dk/d for every budget point in Appendix D. A ratio of k/d=0.00k/d = 0.00 is a rounding effect on high-dimensional datasets, where a few features out of several hundred round to zero.

UF on nine datasets. For regression, UF keeps features with ∣ρ∣≥0.4|\rho| \geq 0.4 (correlation ratio for categorical features), a common threshold for moderate to very strong correlations [3]. Datasets where UF selects no feature in at least one split are excluded from the UF rows. Within the UF group, FetNAS, Evo and RoT are all compared on the same nine datasets, so this comparison is not affected. Comparisons between feature-budget groups do use different datasets and we therefore add a version of Table 3 in which all groups are averaged over the common nine datasets.

Table 4. The column denotes the number of output units, which is 1 for binary classification and regression. We renamed it from #Classes to #Outputs.

[3] Schober, Patrick, Christa Boer, and Lothar A. Schwarte. “Correlation Coefficients: Appropriate Use and Interpretation.” Anesthesia and Analgesia 126, no. 5 (2018): 1763–68.

Reviewer_y8xEOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This work introduces FetNAS, a joint optimization of feature selection and architecture search. An MLP supernet is trained using all features. Then a subset of networks is optimized. Subsequently, a subset of weights is optimized. Finally, the supernet is trained with subnetworks that use top-k features. Experiments on TabArena demonstrate the effectiveness of FetNAS on both classification and regression tasks.

Critical strengths
  • Optimizing both feature selection and architecture is reasonable for achieving higher performance.
  • The paper is easy to read and understand.
Critical weaknesses
  • Novelty is limited. This paper does not introduce a new search method or architecture. Optimizing subnetwork or subweight has been proposed in [1][2]. Multi-objective evolutionary search has been done before. There is no new technique in the method. Moreover, the authors stated in the introduction that "we extend progressive shrinking with an elastic-features stage, ..", which makes the paper incremental.
  • Lack of ablation study. It is unclear how each step in training the supernet (Section 3.3 and Section 3.5) helps in improving the performance.
  • The motivation for feature sampling is weak, lacking sufficient references, empirical evidence, and theoretical analysis to support the claim.

Reference

[1] Once-for-All: Train One Network and Specialize it for Efficient Deployment, Han Cai et.al.

[2] ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware, Han Cai et.al.

Questions for authors
  1. Can the author provide an ablation study on how each component in Section 3.3 contributes to the performance?
  2. What are the lower-ranked features (lines 242- 243)?
  3. "we run a multi-objective evolutionary search over all candidate architectures using three objectives... This produces a Pareto front from which a model can be selected for any desired complexity budget ". What is the objective for selecting a parent for mutation/crossover?
5 comments

We thank the reviewer for the valuable comments. Please find our response to your questions below.

y8xE-W1: Limited novelty

Please see "Contribution and novelty" in the general response.

y8xE-W2: Lack of ablation study

We are currently running a leave-one-stage-out ablation of the supernet training (each stage is removed in turn). We will post the results in this thread as soon as the runs have finished and add them to the appendix.

y8xE-W3: Weak motivation for feature sampling

We agree that the motivation was stated too briefly. We now cite the established use of gradient [1,2] and weight-based [3,4] sensitivity for ranking inputs. We have also add empirical evidence (see "Fixed ranking vs. arbitrary feature subsets"). An importance-based order is clearly better than a random order and the same architectures score higher in a supernet trained on nested subsets than in one trained on randomly resampled subsets.

[1] Baehrens, David, Timon Schroeter, Stefan Harmeling, Motoaki Kawanabe, Katja Hansen, and Klaus-Robert Müller. “How to Explain Individual Classification Decisions.” The Journal of Machine Learning Research 11 (August 2010): 1803–31.

[2] Simonyan, Karen, Andrea Vedaldi, and Andrew Zisserman. Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps. December 23, 2013.

[3] Olden, Julian D., and Donald A. Jackson. “Illuminating the ‘Black Box’: A Randomization Approach for Understanding Variable Contributions in Artificial Neural Networks.” Ecological Modelling 154, no. 1 (2002): 135–50.

[4] Lemhadri, Ismael, Feng Ruan, Louis Abraham, and Robert Tibshirani. “LassoNet: A Neural Network with Feature Sparsity.” Journal of Machine Learning Research 22, no. 127 (2021): 1–29.

y8xE-Q1: Ablation of Section 3.3 components

Please see our answer to y8xE-W2.

y8xE-Q2: Lower-ranked features

Lower-ranked features are those at the bottom of the ordering produced by the chosen ranking variant. The ranking variants order features from most to least (presumed) informative, so lower-ranked features are treated as contributing little predictive information and may be removed with negligible loss of performance for the supernet.

y8xE-Q3: Parent selection in the evolutionary search

Parents for mutation and crossover are sampled uniformly at random from the non-dominated set of solutions, so every Pareto-optimal candidate is equally likely to reproduce and no objective is favoured. We kept the evolutionary search algorithm simple, as it is not the main contribution, but note that one could use more advanced algorithms such as NSGA-II.

AuthorsOpenReview ↗

y8xE-W2: Lack of ablation study (cont.)

Please find the ablation on the different stages of supernet training (Section 3.3) as asked by the reviewer (W2 and Q1). In each variant one stage is left out, while the other stages keep their epochs (for elastic layers and elastic units, depth or width then stays at its maximum). Every variant uses the same search space and is evaluated zero-shot at the matched budgets of Tables 2 and 3. The tables give the difference to FetNAS with all stages (so negative values mean that the stage helps).

Results show that the elastic layers stage has the largest impact. Without it, performance drops at every budget, most at small parameter budgets (up to −0.13 F1 and −0.39 R2R^2). Without this stage the supernet is only trained at full depth, so shallower subnetworks cannot be used zero-shot. Elastic units mainly matters at small parameter budgets for classification (up to −0.04 F1) and at all budgets for regression (up to −0.37 R2R^2). Elastic features gives a consistent gain when features are removed. For regression the gain grows as the feature budget shrinks (about −0.01 with all features, −0.09 to −0.21 at mRMR f = 20 and UF). For classification the F1 difference is small (about −0.01). Full training has no consistent effect and could technically be left out.

Zero-shot classification results (test F1)

Feature budgetStage left outminQ1Q2Q3max
All− full training+0.001−0.006−0.004−0.005−0.006
− elastic layers−0.126−0.123−0.097−0.067−0.045
− elastic units−0.026−0.0120.000+0.001+0.004
− elastic features−0.002−0.011−0.009−0.013−0.008
mRMR f = 20− full training+0.0080.000−0.004−0.0040.000
− elastic layers−0.124−0.118−0.104−0.073−0.046
− elastic units−0.041−0.020−0.013−0.0020.000
− elastic features0.000−0.009−0.009−0.011−0.009
UF− full training+0.020+0.009+0.003+0.003+0.008
− elastic layers−0.101−0.121−0.114−0.081−0.044
− elastic units−0.034−0.016−0.009+0.001+0.002
− elastic features+0.008−0.004−0.004−0.003−0.002

Zero-shot regression results (test R2R^2)

Feature budgetStage left outminQ1Q2Q3max
All− full training+0.010+0.005−0.001−0.004−0.004
− elastic layers−0.375−0.260−0.204−0.092−0.028
− elastic units−0.369−0.098−0.043−0.037−0.026
− elastic features+0.015−0.007−0.009−0.018−0.014
mRMR f = 20− full training+0.017+0.015+0.009+0.016+0.014
− elastic layers−0.303−0.202−0.163−0.097−0.047
− elastic units−0.305−0.150−0.100−0.084−0.073
− elastic features−0.207−0.114−0.109−0.096−0.086
UF− full training+0.003−0.011−0.026−0.024−0.025
− elastic layers−0.394−0.265−0.201−0.109−0.060
− elastic units−0.369−0.144−0.103−0.088−0.079
− elastic features−0.150−0.112−0.105−0.103−0.104

Because F1 only depends on the predicted class, it is fairly robust to leaving out a stage. Therefore, we also report AUROC for the same subnetworks. The effects of leaving out a stage are more prominent. Without elastic layers AUROC drops by 0.08–0.23, without elastic units by up to 0.08 and without elastic features by 0.02–0.03 in every cell.

Zero-shot classification results (test AUROC)

Feature budgetStage left outminQ1Q2Q3max
All− full training−0.006−0.007−0.004−0.008−0.009
− elastic layers−0.224−0.208−0.159−0.116−0.095
− elastic units−0.069−0.034−0.017−0.017−0.015
− elastic features−0.026−0.026−0.024−0.025−0.020
mRMR f = 20− full training+0.005−0.0050.000−0.0030.000
− elastic layers−0.223−0.204−0.178−0.135−0.089
− elastic units−0.080−0.057−0.033−0.027−0.024
− elastic features−0.022−0.034−0.033−0.028−0.027
UF− full training+0.006−0.003−0.005−0.003−0.001
− elastic layers−0.217−0.214−0.186−0.141−0.092
− elastic units−0.076−0.051−0.038−0.028−0.025
− elastic features−0.017−0.031−0.025−0.025−0.027

We will include these results in the appendix of the revised paper.

Reviewer's weakness 2 also mentions Section 3.5 (which is not present in the original submission). We assume this refers to the feature ranking discussed in Section 3.4, which is already covered under section 4.5 and discussed under "Fixed ranking vs. arbitrary feature subsets" in the general response. We hope this addresses the reviewer's concern.

I thank the authors for providing the response.

After carefully reviewing the response, I believe this work needs revision before publication.

First, the novelty is limited. Even though the authors now state, "our contribution is on the input side", it does not automatically address this concern. Because in the methods section, supernet training has several stages, such as full/elastic layer/etc., these are well known in NAS. As it stands, the reader gains limited insight from this section.

Second, feature sampling does not show a clear advantage over random (fixed vs random), e.g., +0.011 [+0.004, +0.017]. It is hard to tell how meaningful these numbers are. If many components in the paper come from previous works (in a direct or indirect manner), it is hard to find the novelty of this work.

Third, it seems strange that the authors use random sampling to select parents for mutation/crossver. We usually pick the best candidate in a pool. I am not sure we call this evolutionary search.

For these reasons, I keep my rating.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General response to all reviewers

We thank all reviewers for their valuable feedback and comments. Below, we address the main concerns raised across the reviews.

Contribution and novelty

We agree that progressive shrinking and multi-objective evolutionary search are established components and we do not claim them as contributions. Our contribution is on the input side. NAS techniques such as Once-for-All and ProxylessNAS give every subnetwork the same full input, so each feature budget requires a separate feature-selection step and a separate supernet.

FetNAS instead (1) makes the number of input features a searchable dimension of the supernet and (2) allows us to sample architectures zero-shot for any feature and parameter budget. This is not trivial. Once a feature is removed, its information is gone, so the weight sharing depends on which features the smaller subnetworks keep. To make this work, we add an elastic-features stage and investigate different ways to rank the features (Section 4.5). We validate the approach on all 51 datasets of the TabArena benchmark, where it achieves competitive results against the sequential pipelines. To the best of our knowledge, no existing weight-sharing NAS method combines these three: feature selection inside a zero-shot supernet, a study of how to incorporate feature selection in the search procedure, including the feature ranking and competitive results on a large tabular benchmark.

This makes the method directly applicable for practitioners. Each model comes with an explicit feature subset, so they can choose one that fits both their computational budget and the features they can afford to acquire without any retraining. We will rephrase the abstract, introduction and contributions such that the novelty claim is more clear.

This tackles the weakness mentioned by all reviewers.

Fixed ranking vs. arbitrary feature subsets

We want to point out that the broader FetNAS framework is not limited to a fixed ranking. Therefore, we are also introducing a random variant, which does not use a ranking and where the feature subset is randomly sampled per subnetwork, so different architectures can use different feature subsets. Concretely, a new subset of k features is drawn uniformly for every sampled subnetwork in every training batch and for every candidate in the search. Unselected inputs are masked, so each feature always enters through its own input column. The results are as follows:

  • Random subsets are competitive with ranking, but ranking still yields better results: +0.011+0.011 [+0.004+0.004, +0.017+0.017] for all matched budgets, better on 38 of 51 datasets. We will add these findings to the paper, as well.

This tackles the weaknesses mentioned by reviewer d5nA and 77Bx.

Computational cost

Wall-clock GPU time summed over all 255 runs (51 datasets ×\times 5 splits; one Tesla P100 and 9 CPU cores per run): covering the five feature budgets of the paper takes ~190 GPU-hours with the sequential Evo pipelines and ~50 GPU-hours with FetNAS (one supernet), in other words 28% of the cost. Every search evaluates 5,632 candidates (512 initial, 20 generations of 256), without caching. BatchNorm recalibration is part of every candidate evaluation and of the reported search time. It takes 30–44% of the search time on datasets below 1,000 training rows and 68–92% above 2,000.

This tackles the weaknesses mentioned by reviewer d5nA and 77Bx.

Embedded feature-selection baseline

We added LassoNet [1] as an embedded feature-selection baseline, evaluated under the same protocol as UF and mRMR. Preliminary headline result, as the difference FetNAS minus LassoNet per dataset. With all features available LassoNet is ahead (−0.02-0.02 F1-macro, −0.05-0.05 R2R^2). As soon as the feature budget is restricted FetNAS is ahead: +0.02+0.02 / +0.03+0.03 / +0.05+0.05 / +0.07+0.07 F1-macro and +0.03+0.03 / +0.06+0.06 / +0.08+0.08 / +0.07+0.07 R2R^2 at 80% / 60% / 40% / 20% of the features, with fewer parameters. The full table will be posted with the final results.

[1] Lemhadri, Ismael, Feng Ruan, Louis Abraham, and Robert Tibshirani. “LassoNet: A Neural Network with Feature Sparsity.” Journal of Machine Learning Research 22, no. 127 (2021): 1–29.

This tackles the weaknesses mentioned by reviewer d5nA and 77Bx.

Additional experiments

We are currently running additional experiments with an alternative ranking method (minimum Redundancy – Maximum Relevance) [2] and with a smaller batch size for the supernets. The latter follows up on reviewer d5nA's remark that RoT and the supernets are trained with different optimizers and batch sizes. We will post these results once they are available and add them to the revised submission.

[2] Peng, Hanchuan, Fuhui Long, and C. Ding. “Feature Selection Based on Mutual Information Criteria of Max-Dependency, Max-Relevance, and Min-Redundancy.” IEEE Transactions on Pattern Analysis and Machine Intelligence 27, no. 8 (2005): 1226–38.

AuthorsOpenReview ↗

Additional experiment: FetNAS with mRMR as ranking

We additionally evaluated mRMR as the feature ranking of FetNAS. Here, features are ordered by mRMR computed on the training set. Everything else is left unchanged. The motivation is that mRMR penalises redundant features which helps pruning features that contain the same information. On classification, both rankings perform the same (mean difference −0.001 F1). mRMR is slightly better at the smallest feature budgets, whereas the gradient ranking at the larger ones. On regression mRMR is clearly better at small feature budgets (+0.02 to +0.11 R2R^2 at mRMR 20% and UF) and slightly worse at medium budgets with larger subnetworks (up to −0.02).

The gradient ranking nevertheless remains a good heuristic, as it scales better with the number of features dd. It requires a single forward and backward pass of the largest subnetwork over the nn training samples, which is O(n d)\mathcal{O}(n\,d). mRMR scales quadratically with the number of features and linearly with the number of samples. Since FetNAS requires the full ranking rather than the top kk only, the greedy selection runs for dd steps, each relating the newly selected feature to all remaining ones which results in O(n d2)\mathcal{O}(n\,d^2). For most datasets both costs are negligible compared to supernet training, but on high-dimensional data mRMR dominates (in our implementation about 3.4 additional CPU hours per split on hiva_agnostic with 1,617 features).

Zero-shot difference FetNAS-mRMR − FetNAS-G on matched feature budgets for classification (test F1).

Feature budgetminQ1Q2Q3max
All features−0.002−0.007−0.008−0.006−0.004
mRMR 80%−0.001−0.008−0.007−0.006−0.004
mRMR 60%−0.003−0.002−0.006−0.007−0.005
mRMR 40%+0.001+0.001−0.000−0.004+0.000
mRMR 20%+0.009+0.001+0.005+0.002+0.004
UF+0.002+0.004+0.005+0.002+0.001

Zero-shot difference FetNAS-mRMR − FetNAS-G on matched feature budgets for regression (test R2R^2).

Feature budgetminQ1Q2Q3max
All features−0.012−0.004−0.005−0.003−0.001
mRMR 80%−0.011−0.008−0.007−0.007−0.009
mRMR 60%+0.009−0.009−0.015−0.019−0.019
mRMR 40%+0.029−0.005−0.014−0.016−0.016
mRMR 20%+0.061+0.035+0.030+0.023+0.022
UF+0.076+0.110+0.103+0.090+0.081
AuthorsOpenReview ↗

Revised submission

We would like to thank all reviewers for their engagement during the rebuttal. We have revised the manuscript and highlighted all additions and modifications in blue to address the concerns raised during the discussion. The main changes are summarized below:

  • Clarified narrative and contributions. We revised the manuscript to present the problem setting, motivation and core contributions more clearly.

  • Unranked feature sampling (FetNAS (unranked)). We added a variant that retains the joint search over input dimensionality and architecture while replacing the feature ranking with randomly sampled feature subsets.

  • Embedded feature-selection baseline (RoT-LassoNet). We added comparisons against LassoNet as an embedded feature-selection method.

  • Matched-performance evaluation and compute details. We added experiments comparing models at matched predictive performance (Appendix E). We also report wall-clock GPU search and training costs (Appendix C).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.