acceptodds
Under review as a conference paper at ICLR 2027

Steadying Neural Feature Selection through Coupled Residual Learning

Abstract

Feature selection aims to retain the informative features in a dataset, reducing dimensionality and computational costs while improving model interpretability and performance. Recently, neural-network-based methods have emerged as promising approaches to feature selection. However, the non-identifiability of neural networks can make the selected features unstable, causing difficulties in interpretability. To address this issue, we propose a coupled residual connection, a simple variant of a residual connection that integrates a linear residual component and a nonlinear neural network through a shared learnable weight. This coupling strategy ensures the identifiability of feature-selection-related parameters while remaining compatible with existing neural network architectures, such as multilayer perceptrons and mixtures of experts. Based on this proposal, we introduce a straightforward training-then-trimming algorithm. Experiments on synthetic and real-world datasets demonstrate that the proposed algorithm typically yields more stable results and improves prediction performance by 9%–50%. We further observe that incorporating our approach into an existing method improves its stability and predictive performance.

Primary area:
interpretability and explainable AI
Keywords:
Interpretability, Feature Selection, Neural Network, Residual Learning

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_xDfyOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

A coupled residual connection, in which a single shared β\beta gates both the linear residual and the nonlinear backbone The empirical contribution is a body of experiments on synthetic and real-world datasets against Lasso/LassoNet/STG/DFS, plus a demonstration that the connection can be bolted onto an existing selector.

Critical strengths

Clearly written. The architecture is stated precisely, the §4.4 three-way distinction between input-gating / partial-linear / single-index models is genuinely clarifying, and both proofs are short and correct under their stated assumptions.

The idea seems working. Results partially support the claims

Critical weaknesses

For the experimental parts, the evaluation cannot measure the quality the paper claims to improve, and omits the baseline that can beat their method without sparsity included. The real data experiments never inspect the selected features, justified by "no ground truth".

There is no full-feature baseline. In my own run, using all features gives a better MSE for unregularized CORAL with selected features, so selection does not improve accuracy here. Therefore, the method's value rests on interpretability, but the paper offers no interpretability analysis and cites the lack of ground truth.

My main concern is the theoretical component. Propositions 1 and 2 are presented as theoretical contributions, but their proofs are elementary, neither uses the coupling that is the paper’s central design choice, and neither explains the empirical results they are linked to. I elaborate below.

My main concern is that the two propositions add little substance to the paper. Their proofs are elementary, they do not depend on the proposed coupling, and they do not support the claims they are cited for. As detailed below, they read as decoration rather than as theoretical support for the method.

  1. The proofs are elementary and do not reflect the architecture. Proposition 1 relies only on the fact that x⊤(β1−β2)x^\top(\beta_1-\beta_2) is unbounded along a coordinate axis whenever β1≠β2\beta_1 \neq \beta_2, while hΘh_\Theta is bounded. Proposition 2 follows by setting β⋆=1p\beta^\star = \mathbf{1}_p, so that the Hadamard product reduces to the identity, and applying universal approximation to g⋆−x⊤1pg^\star - x^\top\mathbf{1}_p. Neither argument uses depth, architecture, or the Hadamard coupling. In addition, the experiments use ReLU, which falls outside the bounded-activation assumption of Proposition 1, so the stated theory does not cover the reported results. These results are better presented as remarks than as theoretical contributions.

  2. Neither proposition supports the coupling. Replacing hΘ(x∘β)h_\Theta(x\circ\beta) by hΘ(x)h_\Theta(x), both proofs go through verbatim. The results therefore hold for any "linear plus bounded nonlinearity" model and provide no justification for x∘βx\circ\beta, which is the paper's actual design choice. In particular, the claim that "the coupling strategy introduces identifiability" is not supported: the identifiability in Proposition 1 comes entirely from the bounded network being unable to absorb an unbounded linear term, and is equally available to the uncoupled model x⊤β+hΘ(x)x^\top\beta + h_\Theta(x). The coupling currently has no theoretical support in the manuscript.

  3. Proposition 1 cannot explain Fig. 2. Proposition 1 is a statement about exact equality over Rp\mathbb{R}^p, and its content lies entirely in the behavior as ∥x∥→∞\|x\|\to\infty. It provides no stability on bounded domains. On a compact KK, under the hypothesis of Proposition 2, for every ε>0\varepsilon > 0, every β1≠β2\beta_1 \neq \beta_2 with β2\beta_2 having nonzero entries, and every Θ1\Theta_1, there exists Θ2\Theta_2 such that sup⁡K∣f(⋅;β1,Θ1)−f(⋅;β2,Θ2)∣≤ε\sup_K |f(\cdot;\beta_1,\Theta_1) - f(\cdot;\beta_2,\Theta_2)| \le \varepsilon. To see this, apply universal approximation to g(z)=(z⊘β2)⊤(β1−β2)+hΘ1((z⊘β2)∘β1)g(z) = (z\oslash\beta_2)^\top(\beta_1-\beta_2) + h_{\Theta_1}((z\oslash\beta_2)\circ\beta_1) on β2∘K\beta_2\circ K; this is the proof of B.2 with β⋆\beta^\star left free, and it holds for sigmoid as well. (For analytic activations, exact equality on a set with nonempty interior does extend to Rp\mathbb{R}^p, but this offers no robustness to approximation error, which is what matters for estimation.) The only quantitative bounded-domain statement available is the following: if KK contains ±Mej\pm M e_j for all jj and sup⁡K∣hΘr∣≤Br\sup_K |h_{\Theta_r}| \le B_r, then ∥β1−β2∥∞≤(ε+B1+B2)/M\|\beta_1-\beta_2\|_\infty \le (\varepsilon + B_1 + B_2)/M. This bound is vacuous unless BrB_r is controlled, e.g. by regularization. It does not require bounded activations; boundedness only makes BrB_r depend on the output layer alone. The concentration of β^\hat\beta in Fig. 2 therefore cannot be attributed to Proposition 1. The most plausible explanation is norm control through weight decay or the implicit bias of the optimizer, although other factors such as initialization or early stopping cannot be ruled out without ablation.

Questions for authors

What, specifically, does the Hadamard coupling contribute? As argued above, Propositions 1 and 2 do not use the coupling, and the "with vs. without coupling" experiments only compare against h(x∘β)h(x\circ\beta), i.e., they test the presence of the residual term rather than the coupling. The coupling's evident role is to gate features so that βj=0\beta_j = 0 removes feature jj from the whole model, but LassoNet's hierarchy constraint achieves the same gating. Could the authors either provide a theoretical result whose proof genuinely depends on the coupling x∘βx\circ\beta, for example a stability or estimation guarantee that fails for the uncoupled or LassoNet-style model?

Reviewer_gEL7OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper studies variable selection in neural networks by proposing a coupled residual connection, which uses a shared learnable weight to scale the input for both the linear component and the neural network component. The authors show that this coupling structure ensures identifiability of the shared learnable weight, and they propose a two-step feature selection approach that selects the features with the top weight magnitudes after training. Experiments on synthetic and real-world datasets are provided to show improved prediction performance against several baseline methods.

Critical strengths

The coupled residual connection, which shares the learnable weight between the linear component and the neural network input, is a simple and new idea. The partial identifiability result in Proposition 1 is correct under its stated assumptions.

Critical weaknesses
  1. Lack of theoretical guarantees. There is no theoretical guarantee for this method. The paper does not establish feature selection consistency, and if the model is fitted with the selected features, there is no statistical guarantee for the final fitted model.

  2. Insufficient experimental details and evaluation. The paper does not report the network settings or the hyperparameters of the optimizer, and it does not mention how the tuning parameters are selected. The fixed-budget method is only for the feature-selection tuning parameter; it does not work for other tuning parameters, such as the learning rate, weight decay and network structure. Without these details, the experimental results cannot be verified or reproduced, and it is unclear whether the comparison with the baselines is fair. In addition, the experiments do not have a measure for steady selection. Recall and precision may reflect steady selection when both are high, as in Table 1, but in general they are not metrics for steady selection. In the other experiments, only the accuracy or RMSE is reported, without mentioning steady selection.

Questions for authors
  1. What are the network settings and the hyperparameters of the optimizer (e.g., learning rate, weight decay, network structure), and how were these tuning parameters selected? The fixed-budget method only covers the feature-selection tuning parameter.
  2. Can the authors establish feature selection consistency? If the model is fitted with the selected features, what is the statistical guarantee for the final fitted model?
  3. Can the authors provide a measure for steady selection? Recall and precision reflect steady selection only when both are high, as in Table 1, and the other experiments report only accuracy or RMSE.
  4. Which methods does the Introduction sentence "One important class of methods introduces an additional learnable vector β to scale the input vector x before feeding it into the neural network" refer to? Please add citations.
  5. Please cite LassoNet for the sentence "This design separates linear and nonlinear effects, a strategy with a long history in econometrics, statistics, and machine learning."
  6. Can the authors test the performance of the extension in Section 4.3?
  7. Section 5.1 mentions conducting 100 independent experiments, but the coefficient distribution in Figure 2 is over 50 replications. Can the authors explain this inconsistency?
Reviewer_N2drOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper addresses the instability in neural feature selection caused by parameter non-identifiability in input-gated networks hΘ(x∘β)h_\Theta(\mathbf{x} \circ \boldsymbol{\beta}). The authors propose CORAL-Net (Coupled Residual Learning), which couples a linear residual connection with the nonlinear network through a shared feature weight β∈Rp\boldsymbol{\beta} \in \mathbb{R}^p:

$ f(\mathbf{x}; \boldsymbol{\beta}, \Theta) = \mathbf{x}^\top \boldsymbol{\beta} + h_\Theta(\mathbf{x} \circ \boldsymbol{\beta}) $

(extended to multi-output tasks via row norms of B∈Rp×m\mathbf{B} \in \mathbb{R}^{p \times m}). Features are selected via a two-stage training-then-trimming procedure: (β,Θ)(\boldsymbol{\beta}, \Theta) are jointly trained with a standard optimizer, and features with ∣β^j∣>δ|\widehat{\beta}_j| > \delta (or top-ss by ∣β^j∣|\widehat{\beta}_j|) are retained.

  • Contributions: (1) The CORAL architecture and two-stage training-then-trimming algorithm compatible with standard backbones (MLP, MoE) and existing selectors (DFS); and (2) theoretical guarantees showing partial identifiability of β\boldsymbol{\beta} under bounded last-layer activations (Proposition 1) while preserving universal approximation on compact domains (Proposition 2).
  • Validation: Evaluated on 2 synthetic datasets (Friedman, Artificial) and ~20 real-world classification and regression benchmarks, demonstrating lower run-to-run variance and improved downstream predictive performance compared to uncoupled networks and baselines (Lasso, LassoNet, STG, DFS).
Critical strengths
  1. Lightweight, Drop-In Architectural Design with Zero Optimization Overhead: Coupling the linear residual x⊤β\mathbf{x}^\top \boldsymbol{\beta} and input gate x∘β\mathbf{x} \circ \boldsymbol{\beta} adds only pp parameters (or p×mp \times m for mm outputs), requires no custom proximal operators, hierarchy constraints, or temperature annealing schedules (unlike LassoNet or Gumbel/Concrete gating), and can be wrapped around arbitrary differentiable backbones (MLP, MoE) or existing selectors (CORAL-DFS). This is consequential because practical adoption of neural feature selectors is often bottlenecked by brittle hyperparameter tuning and specialized optimization loops; a drop-in module trainable with standard SGD/Adam lowers the barrier to use.
  2. Consistent Empirical Reduction in Run-to-Run Variance and Strong Downstream Performance: Across the evaluated classification and regression benchmarks, adding the coupled residual connection consistently shrinks the standard deviation of downstream performance across runs while improving mean accuracy/RMSE relative to uncoupled counterparts (hΘ(x∘β)h_\Theta(\mathbf{x} \circ \boldsymbol{\beta})) and baselines (Lasso, LassoNet, STG, DFS). This is consequential because it provides direct empirical evidence that anchoring the input gate to a linear residual connection acts as an effective inductive bias for stabilizing feature selection on standard tabular datasets.
Critical weaknesses

Fundamental Gap in the Identifiability Theory (Proposition 1 vs. Proposition 2, ReLU Networks, and Softmax Classification):

  • Unidentifiability on compact data domains: In CORAL-Net, the output is the sum of a linear part x⊤β\mathbf{x}^\top \boldsymbol{\beta} and a neural network part hΘ(x∘β)h_\Theta(\mathbf{x} \circ \boldsymbol{\beta}). Because a neural network is a universal approximator (Proposition 2), on any bounded range of data (e.g., xj∈[−1,1]x_j \in [-1, 1]), hΘh_\Theta can easily learn the negative of the linear part (−x⊤β-\mathbf{x}^\top \boldsymbol{\beta}) and cancel it out. For example, suppose the true function is y=5x1y = 5x_1 (so x1x_1 is useful and x2x_2 is pure noise):

    • Model 1 sets β1=(5,0)⊤\beta_1 = (5, 0)^\top and hΘ1=0h_{\Theta_1} = 0, giving f(x)=5x1f(\mathbf{x}) = 5x_1 and selecting x1x_1.
    • Model 2 sets β2=(1,100)⊤\beta_2 = (1, 100)^\top (giving a huge weight to the noise feature x2x_2) and has the neural network learn hΘ2(z1,z2)≈4z1−z2=4x1−100x2h_{\Theta_2}(z_1, z_2) \approx 4z_1 - z_2 = 4x_1 - 100x_2. Adding the linear part (x1+100x2)(x_1 + 100x_2) and the neural network part (4x1−100x2)(4x_1 - 100x_2) still gives f(x)≈5x1f(\mathbf{x}) \approx 5x_1 across the entire dataset, but now selects the noise feature x2x_2.

    Proposition 1 only rules out Model 2 by assuming hΘh_\Theta has bounded activations (like Sigmoid) and plugging in x2→∞x_2 \to \infty, where the bounded network saturates and can no longer cancel 100x2100x_2. However, training data are always finite and bounded—the loss function never evaluates x2=∞x_2 = \infty, so on the actual training data both models achieve the same loss while selecting completely different features.

  • Failure for ReLU networks and multi-class softmax: All main experiments use unbounded ReLU networks, where hΘ(z)h_\Theta(\mathbf{z}) can represent 4z1−z24z_1 - z_2 via ReLU⁡(z)−ReLU⁡(−z)=z\operatorname{ReLU}(z) - \operatorname{ReLU}(-z) = z globally on all of Rp\mathbb{R}^p, directly violating Proposition 1 even at infinity. Furthermore, in the multi-output extension (Section 4.3), shifting row Bj↦Bj+c1m\mathbf{B}_j \mapsto \mathbf{B}_j + c\mathbf{1}_m under softmax cross-entropy loss leaves class probabilities unchanged while altering the selection score ∥Bj∥2\|\mathbf{B}_j\|_2 arbitrarily.

  • Why consequential: Identifiability of β\boldsymbol{\beta} is the paper's central theoretical claim and primary justification for CORAL; the fact that it fails on compact domains, for ReLU networks, and under multi-class softmax loss undermines the theoretical foundation of the work.

Questions for authors
  1. Mechanism preventing cancellation on bounded data domains and with ReLU: As shown in the proof of Proposition 2, on any bounded dataset domain K⊂RpK \subset \mathbb{R}^p (and globally on Rp\mathbb{R}^p for ReLU networks), hΘ(x∘β)h_\Theta(\mathbf{x} \circ \boldsymbol{\beta}) can approximate g(x)−x⊤βg(\mathbf{x}) - \mathbf{x}^\top \boldsymbol{\beta} for any non-zero β\boldsymbol{\beta}, canceling the linear residual branch. In your experiments, what actually prevents hΘh_\Theta from canceling x⊤β\mathbf{x}^\top \boldsymbol{\beta} during training—is ℓ2\ell_2 weight decay applied to Θ\Theta, is β\boldsymbol{\beta} initialized near zero/one while Θ\Theta is small, or does early stopping / learning-rate dynamics drive the linear branch to fit first?

  2. Shift non-identifiability in multi-class classification (Section 4.3): Under softmax cross-entropy loss, shifting row Bj↦Bj+c1m⊤\mathbf{B}_j \mapsto \mathbf{B}_j + c\mathbf{1}_m^\top and inversely scaling the jj-th first-layer weights of hΘh_\Theta leaves all class probabilities unchanged while changing ∥Bj∥2\|\mathbf{B}_j\|_2 arbitrarily. Is a zero-sum constraint (Bj1m=0\mathbf{B}_j \mathbf{1}_m = 0) or weight decay applied in the classification experiments to make ∥Bj∥2\|\mathbf{B}_j\|_2 identifiable?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.