Self-Regularized Gumbel Sigmoid for Cost-Aware Subset Selection

Subset selection is a cornerstone of machine learning, but selecting useful features under heterogenous candidate cost constraints remains challenging. Differentiable relaxations make it tractable, but many existing methods require tuning an explicit sparsity penalty and treat candidates as equally costly, limiting their applicability when resource costs vary. We introduce the Self-regularized Gumbel Sigmoid (SrGS), a simple framework that replaces penalty-based sparsification with a three-stage structural mechanism: a competitive normalizer over candidates; explicit cost encoding, which rescales the attention weight into retention probabilities of candidates that meet a budget expressed in physical resources such as memory or latency; and stochastic gating. SrGS adds only one learnable scalar per candidate and masks inputs without modifying the backbone network. We show that the default Softmax normalizer exponentially suppresses the gradients of losing candidates, replace it with scale-invariant Absolute Linear Normalization (ALN) and introduce a Utility-Aware Subgradient (UAS) to preserve optimization signals at its singularity, together with clipping that maintains gradient flow for suppressed candidates. For linear models with squared loss, we characterize the induced implicit regularizer and derive the budget allocation that minimizes it for fixed effective coefficients. The minimized regularizer vanishes exactly when the weighted feature support fits within the budget. Across five feature-selection benchmarks spanning click-through-rate prediction (CTR), and several high-dimensional speech, text, image datasets, we evaluate feature selection and cost-aware embedding dimension optimization under a common select-retrain protocol. Across these settings, SrGS is competitive with or outperforms state-of-the-art greedy, pruning, and differentiable baselines, demonstrating that a low-overhead selection mechanism can accommodate diverse domains and resource constraints without specialized backbone architectures.