Steadying Neural Feature Selection through Coupled Residual Learning

Feature selection aims to retain the informative features in a dataset, reducing dimensionality and computational costs while improving model interpretability and performance. Recently, neural-network-based methods have emerged as promising approaches to feature selection. However, the non-identifiability of neural networks can make the selected features unstable, causing difficulties in interpretability. To address this issue, we propose a coupled residual connection, a simple variant of a residual connection that integrates a linear residual component and a nonlinear neural network through a shared learnable weight. This coupling strategy ensures the identifiability of feature-selection-related parameters while remaining compatible with existing neural network architectures, such as multilayer perceptrons and mixtures of experts. Based on this proposal, we introduce a straightforward training-then-trimming algorithm. Experiments on synthetic and real-world datasets demonstrate that the proposed algorithm typically yields more stable results and improves prediction performance by 9%–50%. We further observe that incorporating our approach into an existing method improves its stability and predictive performance.