acceptodds
Under review as a conference paper at ICLR 2027

Self-Regularized Gumbel Sigmoid for Cost-Aware Subset Selection

Abstract

Subset selection is a cornerstone of machine learning, but selecting useful features under heterogenous candidate cost constraints remains challenging. Differentiable relaxations make it tractable, but many existing methods require tuning an explicit sparsity penalty and treat candidates as equally costly, limiting their applicability when resource costs vary. We introduce the Self-regularized Gumbel Sigmoid (SrGS), a simple framework that replaces penalty-based sparsification with a three-stage structural mechanism: a competitive normalizer over candidates; explicit cost encoding, which rescales the attention weight into retention probabilities of candidates that meet a budget expressed in physical resources such as memory or latency; and stochastic gating. SrGS adds only one learnable scalar per candidate and masks inputs without modifying the backbone network. We show that the default Softmax normalizer exponentially suppresses the gradients of losing candidates, replace it with scale-invariant Absolute Linear Normalization (ALN) and introduce a Utility-Aware Subgradient (UAS) to preserve optimization signals at its singularity, together with clipping that maintains gradient flow for suppressed candidates. For linear models with squared loss, we characterize the induced implicit regularizer and derive the budget allocation that minimizes it for fixed effective coefficients. The minimized regularizer vanishes exactly when the weighted feature support fits within the budget. Across five feature-selection benchmarks spanning click-through-rate prediction (CTR), and several high-dimensional speech, text, image datasets, we evaluate feature selection and cost-aware embedding dimension optimization under a common select-retrain protocol. Across these settings, SrGS is competitive with or outperforms state-of-the-art greedy, pruning, and differentiable baselines, demonstrating that a low-overhead selection mechanism can accommodate diverse domains and resource constraints without specialized backbone architectures.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Subset Selection, Feature Selection, Cost-Aware Selection, Gumbel-Sigmoid Relaxation, Implicit Regularization, Straight-Through Estimator

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_VAECOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Self-regularized Gumbel Sigmoid (SrGS) for feature selection under a resource budget. It assigns one learnable score to each candidate, normalizes the scores into a competitive distribution, converts that distribution into retention probabilities using candidate costs, and trains with stochastic Gumbel-Sigmoid gates. At deployment, it constructs a subset whose total cost does not exceed the budget.

Critical strengths

Practical problem formulation. The paper treats feature costs as heterogeneous, which is more appropriate than a fixed feature count when embedding tables have different vocabulary sizes.

Simple integration. SrGS adds one score per candidate and masks inputs without changing the downstream backbone. The select–retrain evaluation helps assess the selected subset rather than the trained gates alone.

Critical weaknesses

1. Limited evidence for the benefit of ALN. The paper motivates ALN as a remedy for Softmax gradient suppression, but its empirical gain is small and inconsistent. In Table 1, ALN improves accuracy by only about 0.10 percentage points on average across the paired settings, and it performs worse than Softmax under one-stage selection with standard annealing (0.9535 versus 0.9561). The paper does not provide gradient or feature-recovery measurements showing that the proposed optimization problem occurs in practice or that ALN resolves it.

2. The UAS rule is insufficiently justified. UAS chooses a particular subgradient when a score is exactly zero, but this choice is not uniquely implied by the objective. When a feature has positive utility disadvantage, zero is locally preferable, yet the rule still moves its score away from zero and can increase the loss on that step. Moreover, UAS acts only when a score is exactly zero, and the paper provides no isolated ablation establishing how often this happens or whether UAS improves the final result.

3. Reproducibility is limited by the absence of code. The manuscript provides pseudocode and experimental settings but no implementation or repository link. Details such as the custom UAS gradient, asymmetric clipping, budget annealing, preprocessing, and select–retrain procedure are implementation-sensitive. Without code, it is difficult to independently verify the reported comparisons.

4.The ablation results are modest and narrow. Table 1 evaluates only MNIST at K=50K=50. The reported average gains from two-stage selection, slow annealing, and ALN are approximately 0.19, 0.37, and 0.10 percentage points, respectively—small relative to some of the reported variation across seeds. Ablations across additional datasets and budgets would provide stronger evidence that each component contributes reliably.

5. The final greedy selection is not justified by the training objective. At inference, the method ranks candidates by retention probability and accepts each one if it fits the remaining budget. This guarantees feasibility but provides no guarantee of selecting the best subset under heterogeneous costs or interacting feature utilities. The deterministic subset can also differ substantially from the stochastic masks used during training. The paper should quantify this training-to-selection gap and compare the greedy rule with alternative budgeted selection procedures.

Questions for authors

Q1: Line 464 cites “REINFORCE (?)”, and the placeholder also appears in Appendix G. What is the intended reference? Please correct these citations.

Q2: How were the marginal gains of 0.19 and 0.37 percentage points reported for Table 1 calculated? Averaging the displayed factorial contrasts appears to give approximately 0.34 points for two-stage selection and 0.52 points for slow annealing. Please clarify or correct these figures. Could you also provide separate ablations of UAS and the asymmetric clipping rule across multiple datasets and budgets?

Q3: During the actual soft-gate training, what is the per-batch distribution of the gate-weighted cost ∑iciwi\sum_i c_iw_i? How much performance is lost when converting the learned gates into a hard-budget subset, both before and after retraining? These measurements would help connect the implementation to the idealized hard-gate, linear-model analysis.

Q4: For the strongest baselines, what hyperparameter ranges were searched at each budget, which settings were selected, and how many trials or how much compute was used? This information is needed to assess the fairness of the comparisons.

2 comments

Thank you for your review and helpful comments. We address your concerns and questions below, together with additional experimental results.

W1/W4/Q2. Could broader ablations show when ALN and the clipping rules improve predictive performance?

The broader normalization and clipping ablation is reported in General response, G1.

W1. Could you provide feature-recovery measurements showing that useful suppressed candidates recover with ALN?

The requested feature-recovery measurements are reported in General response, G2.

We also measured score gradients directly in the MNIST ablation. At K=20K=20, about 250 of the 784 Softmax scores end below the lower clipping bound, and their median score-gradient magnitude is (3.7–4.5)×10−7(3.7\text{--}4.5)\times10^{-7} of that of the retained features. Under ALN, the corresponding gradients remain of order one relative to the retained features. As General response, G1 shows, this difference in gradients does not translate into a consistent accuracy gain.

W4/Q2. Could the component ablations cover additional datasets and budgets?

The ablation is extended to Avazu embedding allocation at three budgets in General response, G3.

W2/Q2. How is the UAS update at zero justified, how often is it activated, and does it improve performance?

We address this point in the General response above.

W3/Q4. Could you provide the implementation and reproducibility details, including baseline hyperparameter ranges, selected settings, trial counts, and compute at each budget?

We plan to release the implementation and configuration files for all reported runs. Completing the release process will take additional time beyond the rebuttal period. We also expect to gain access to the original development materials, which would allow us to check for additional records of hyperparameter selection. We will report any further details that we can verify.

Our recollection is that we generally started from the hyperparameter settings in the original papers. When adjustment was needed, we tried values on a coarse multiplicative scale, with roughly threefold spacing. Complete records of this selection process are unavailable, so we cannot reliably reconstruct the exact search ranges, trial counts, or tuning compute for each method and budget. The current manuscript's Reproducibility and Hyperparameter Details appendix lists the final method-specific configurations. The final evaluation runs used one fixed configuration per method, dataset, and budget across five seeds, with settings largely shared across datasets and budgets. The recorded compute for the published runs is about 5,500 job-hours, including queueing time and both selection and retraining. This figure covers the final evaluation jobs.

W5/Q3. How much performance is lost when learned gates are converted to the final hard-budget subset, before and after retraining, and how is the greedy rule justified?

The comparison of stochastic gate evaluation, final feature selection, and fresh retraining is reported in General response, G4.

The final greedy scan enforces budget feasibility. The evaluation in General response, G4 measures its behavior on MNIST and Avazu, including a case (25% of the reference embedding budget) where the greedy subset departs from the training masks and retraining is required. Optimality for interacting neural utilities requires further analysis. We will state these limits explicitly. Comparisons with alternative final-selection procedures remain future work.

Q3. What is the per-batch distribution of the actual gate-weighted cost during soft gate training?

The per-example and per-batch cost distributions, measured after budget annealing, are reported in General response, G6.

Q1. What is the intended reference for the unresolved REINFORCE citations in the main text and appendix?

The intended REINFORCE reference is Williams (1992), Simple statistical gradient-following algorithms for connectionist reinforcement learning [3]. Sorry for the typo. The current manuscript includes this citation in the main text and in the appendix's Connection to REINFORCE section.

Q2. Why do the marginal gains stated for Table 1 differ from the contrasts calculated from its displayed values?

Your recalculation is correct. Sorry for the typo.

References

[3] Ronald J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 8:229–256, 1992. https://doi.org/10.1007/BF00992696

This version has indeed improved readability and informativeness. However, I did not see any superiority over other baselines, typo exists (see line 464) and the code is not released, so I will raise score to weak rejection.

Reviewer_6796OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes SrGS, a differentiable feature-selection method based on Gumbel-Sigmoid sampling, with the main idea being cost-aware subset selection under a shared resource budget. The budget formulation generalizes K-hot Gumbel-Sigmoid selection to cost-aware Gumbel-Sigmoid selection, replacing the expected number of selected features with an expected total cost constraint. The method normalizes feature scores and rescales the resulting retention probabilities according to their individual costs and the available budget.

The authors also propose Absolute Linear Normalization (ALN) and a Utility-Aware Subgradient (UAS) to address vanishing-gradient issues associated with competitive normalization. They further provide a theoretical characterization of the implicit regularization induced by stochastic masking under a cost constraint.

The method is evaluated on five standard feature-selection benchmarks, as well as heterogeneous-cost embedding dimension optimization on Avazu and Criteo. The results are generally competitive with the baselines.

Critical strengths
  1. Cost-aware feature selection through a shared budget.

The main contribution, in my view, is the shared budget formulation, which generalizes expected K-feature selection to heterogeneous feature costs. I find this a useful idea, particularly in settings where features have very different resource requirements. In the heterogeneous-cost embedding allocation experiments, SrGS performs competitively with the specialized baselines, demonstrating that the proposed budget formulation works in practice.

  1. Competitive performance in the uniform-cost setting.

When heterogeneous feature costs are disabled, the method performs competitively with established feature-selection baselines. This serves as a useful sanity check that the proposed framework does not introduce apparent learning or optimization problems in the uniform-cost special case.

Critical weaknesses
  1. The experimental evaluation does not focus enough on the central cost-aware contribution. The paper is primarily motivated and framed as a cost-aware feature-selection method. However, most of the feature-selection experiments use homogeneous costs, where the distinctive cost-modeling capability is effectively disabled. These experiments establish that SrGS is competitive as a general-purpose feature-selection method, but I find that somewhat less interesting given the paper's motivation. Gumbel-Sigmoid sampling and K-subset selection are already well established, and the proposed method seems fairly elaborate for achieving mostly comparable performance in these ordinary feature-selection settings. The heterogeneous-cost experiments are limited to embedding dimension optimization on two CTR datasets. While these results are encouraging and demonstrate competitive performance, I would have liked to see at least one conventional feature-selection experiment with heterogeneous feature costs. Given that cost-aware feature selection is the main motivation of the paper, I think a broader evaluation of this specific capability would strengthen the contribution. I also find the related work somewhat limited. A quick literature search reveals previous work explicitly addressing cost-aware feature selection, for example Bolón-Canedo et al. (2014) and Jagdhuber et al. (2020). The paper acknowledges classical knapsack optimization, but does not sufficiently discuss the existing cost-aware feature-selection literature. I am not questioning the novelty of the specific proposed formulation, but I think this literature should be discussed to better establish what is new.
  2. The benefit of ALN/UAS is not convincingly demonstrated. I understand ALN and UAS primarily as mechanisms for addressing an optimization difficulty introduced by the competitive normalization used for the shared budget. This is a legitimate contribution, but I think it would be more natural to frame it as a supporting part of the proposed cost-aware selection framework rather than as a largely separate advance in feature-selection optimization. The empirical benefit also seems quite small. The MNIST ablation reports an average improvement of approximately 0.10 percentage points from ALN. If ALN is intended as a broader improvement for ordinary feature selection, I don't find this particularly convincing. More importantly, the ablation is performed only on MNIST with homogeneous costs. There is no corresponding ablation in the heterogeneous-cost setting, which is where I would expect the proposed normalization to be most relevant. The individual effects of UAS and the asymmetric clipping mechanism are also not clearly isolated.
Questions for authors
  1. Heterogeneous-cost evaluation: Could the authors provide an experiment on a standard tabular dataset of the kind commonly used in the feature-selection literature, with an associated cost for each original input feature (either provided or appropriately specified), rather than evaluating heterogeneous costs only through embedding dimension optimization? I think this would strengthen the validation of the paper's central contribution.

  2. ALN/UAS ablation: Could the authors provide an ablation of ALN, UAS, and the clipping mechanism in a heterogeneous-cost setting? The existing MNIST ablation shows only a very small average improvement from ALN, making it difficult to assess the practical significance of these components.

  3. Cost-aware feature-selection literature: Could the authors discuss how SrGS relates to existing methods for feature selection with heterogeneous costs? There appears to be relevant prior work that is not covered in the related work section.

  4. Select-retrain protocol: I would be interested in seeing both the performance of the jointly trained model without retraining and the reported select-retrain results. While the authors argue that retraining avoids selection artifacts, the opposite argument could also be made: retaining the jointly trained model may be preferable, since it has already learned how to utilize the selected features. There is a trade-off between these approaches, and reporting both would be informative.

  5. Theoretical relevance: The implicit-regularization analysis in Section 5 is exact only for linear models with squared loss and hard Bernoulli gates, whereas the actual SrGS experiments use nonlinear classifiers with classification losses and soft gates. Could the authors clarify how these theoretical results apply to the actual SrGS method, and how they specifically support its proposed optimization procedure?

My current assessment is borderline, leaning toward weak rejection, primarily due to the limited experimental validation of heterogeneous-cost selection, which appears to be a central contribution of the paper. However, I find the proposed approach promising and would be open to revising my recommendation toward acceptance if the authors can convincingly address the concerns raised above.

2 comments

Thank you for your review and helpful comments. We address your concerns and questions below, together with additional experimental results.

W1/Q3. How does SrGS relate to existing feature-selection methods with heterogeneous costs?

Thank you for pointing out this gap in the related work. Bolón-Canedo et al. [1] incorporate acquisition costs into filter criteria such as CFS and mRMR, balancing feature relevance and cost. Jagdhuber et al. [2] study feature selection under a cost budget using forward selection and genetic algorithms. These works establish the cost-aware selection setting and provide important points of comparison. SrGS contributes a differentiable shared-budget parameterization that can be trained jointly with a neural backbone, together with an analysis of the objective induced by stochastic masking. We will add this discussion and the corresponding references to the revised paper.

W1/Q1. Could you evaluate heterogeneous costs on a standard tabular dataset with a cost for each original input feature?

Our heterogeneous-cost evaluation currently focuses on embedding allocation. We plan to evaluate additional tabular datasets with original-feature acquisition costs and established cost-aware methods as future work.

W2. ALN and UAS should be framed as supporting components, and their practical benefits need clearer evidence.

The main contribution is a differentiable selection framework with a shared cost budget. It learns one score per candidate, translates the scores into retention probabilities using the budget, and masks inputs while keeping the backbone architecture unchanged. ALN is an optimization choice that supports recovery of suppressed candidates, and Softmax can also work well depending on the task.

The normalization and clipping ablation across MNIST budgets is reported in General response, G1.

W2. The practical benefit of ALN is unclear given the small accuracy gain in the MNIST ablation.

For additional evidence on ALN's practical benefit, please see the controlled feature-recovery experiment in General response, G2.

W2/Q2. Could you provide a normalization and clipping ablation in a heterogeneous-cost setting?

The heterogeneous-cost component ablation requested here is reported in General response, G3.

Q4. Could you compare the jointly trained model on the selected features with the model retrained on the same feature set?

The comparison of the jointly trained predictor on the selected features with a freshly retrained predictor, alongside stochastic gate evaluation, is reported in General response, G4.

Q5. How does the hard gate linear analysis relate to nonlinear soft gate training and support the proposed optimization procedure?

The scope of the hard gate result, its local connection to nonlinear soft gate training, and the allocation experiment are discussed in General response, G5.

References

[1] Verónica Bolón-Canedo, Iago Porto-Díaz, Noelia Sánchez-Maroño, and Amparo Alonso-Betanzos. A framework for cost-based feature selection. Pattern Recognition, 47(7):2481–2489, 2014. https://doi.org/10.1016/j.patcog.2014.01.008

[2] Rudolf Jagdhuber, Michel Lang, Arnulf Stenzl, Jochen Neuhaus, and Jörg Rahnenführer. Cost-Constrained feature selection in binary classification: adaptations for greedy forward selection and genetic algorithms. BMC Bioinformatics, 21:26, 2020. https://doi.org/10.1186/s12859-020-3361-9

Thank you for the detailed response and the substantial effort put into addressing the reviewers' concerns. I appreciate that you conducted several of the requested experiments, particularly the heterogeneous-cost component ablation and the select-retrain comparison. However, given that my main concern remains unresolved, I will maintain my current score of weak rejection.

I still find the core idea of cost-aware Gumbel-Sigmoid selection interesting and potentially useful. However, I am not yet convinced that the method has been sufficiently validated for its intended general-purpose application. My main concern remains the lack of experiments on conventional tabular feature-selection datasets with heterogeneous feature costs. Since cost-aware subset selection is the central motivation of the paper, deferring this evaluation to future work leaves me questioning the general applicability of the proposed framework beyond embedding dimension optimization.

The additional ALN experiments provide useful evidence for improved feature recovery, although the gains in predictive performance remain inconsistent. I also appreciate the clarification of the theoretical analysis, but the finding that the learned allocations remain substantially different from the theoretical optimum, even under the simplified assumptions of the theorem, limits how strongly the theory supports the actual optimization procedure.

Overall, I think this is a promising research direction, and the additional experiments and clarifications have meaningfully strengthened the paper. I also appreciate the willingness to acknowledge limitations and revise the proposed contributions. However, I feel the framework still needs broader validation of its central cost-aware feature-selection capability before I can recommend acceptance.

Reviewer_AmJfOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SrGS, a method for picking a subset of candidates (input features, or embedding blocks in CTR models) under a cost budget given in real units like memory or latency.

In earlier methods there was a sparsity penalty with a λ to tune, each candidate getting a score. The scores are normalized, rescaled by the average cost so the expected cost of the kept set matches the budget, and turned into binary masks with a Gumbel-Sigmoid and straight-through gradients.

The authors argue that Softmax makes the gradients of losing candidates vanish, so they swap in an absolute-value normalizer (ALN). To handle the kink at zero, they add UAS and an asymmetric clip. They also analyze the regularizer induced by the random masks. For linear models with squared loss, it vanishes exactly when the cost of the active features fits the budget, and they give a closed-form optimal allocation. Experiments cover feature selection on Avazu, Criteo, ISOLET, DEXTER and MNIST, embedding-dimension search on Avazu and Criteo, and a small ablation on MNIST.

Critical strengths
  1. Formulation is precise, like one scalar per candidate overhead, the backbone is untouched, lambda tuning is removed, and the final greedy scan takes care of the budget

  2. The same method handles uniform-cost feature selection and heterogeneous-cost embedding-dimension search (c = 4V_j). Most baselines address only one of the two.

Critical weaknesses
  1. UAS ablation is missing, though it is listed as headline contribution, also ALN+UAS vs softmax has very thin support from the results as some have very less gap and some say the opposite

  2. The regularizer analysis is exact only for linear models, squared loss and hard gates, and the optimal allocation it derives (probability ∝ |β|/√c) is a benchmark. SrGS’s own allocation comes from Eq. 3, and I couldn’t find anything showing the trained method approaches that benchmark or that the regularizer matters in the nonlinear experiments

Questions for authors
  1. Which of the paper’s contributions produces actual gains? Can you isolate each of the components like ALN-UAS, regularizer

  2. Clarify more on how UAS is useful like how often does scores hit zero during training

  3. In simple linear setting with known coefficients and heterogeneous cost does SrGS’s learned allocation approach the one in Thm 2

2 comments

Thank you for your review and helpful comments. We address your concerns and questions below, together with additional experimental results.

Q1. Which contributions produce actual gains, and can the effects of ALN, UAS, and the regularizer be isolated?

The main contribution is a differentiable selection framework with a shared cost budget. It learns one score per candidate, translates the scores into retention probabilities using the budget, and masks inputs while keeping the backbone architecture unchanged. ALN is an optimization choice that supports recovery of suppressed candidates, and Softmax can also work well depending on the task.

For a retention probability zz and a small stability constant ϵ>0\epsilon>0, clipping keeps the probability inside [ϵ,1−ϵ][\epsilon,1-\epsilon], where logit⁡(z)=log⁡z−log⁡(1−z)\operatorname{logit}(z)=\log z-\log(1-z) is finite. The logit diverges at the endpoints, and budget-scaled values can exceed one. This explains the numerical role of forward clipping. We discuss the regularizer and its allocation benchmark in the response to W2/Q1/Q3 below.

W1/Q1. Could you provide stronger evidence for the practical gains of ALN over Softmax?

The normalization and clipping ablation is reported in General response, G1.

W1. The empirical support for ALN is limited because its gains over Softmax are small and inconsistent.

The additional test of ALN's recovery behavior under deliberately suppressed initialization is reported in General response, G2.

W1/Q1. Could you clarify the practical gains from ALN across different data settings and budgets?

The ALN versus Softmax comparison in heterogeneous-cost embedding allocation is reported in General response, G3.

W1/Q2. How useful is UAS in practice, and how often do scores become exactly zero during training?

We address this point in the General response above.

W2/Q1/Q3. Does the learned allocation approach Theorem 2 in its linear hard gate setting with heterogeneous costs, and how does this analysis relate to the nonlinear experiments?

The comparison of learned retention probabilities with the allocation predicted by Theorem 2 for the current effective coefficients, including the effect of adding the variance gradient, is reported in General response, G5.

Thank you for bringing clarity to my concerns with extensive experiments.

However some of the contributions like UAS are removed after you perform the ablation. The softmax problem is justified with experiments properly but the results with ALN as a replacement are still mixed. Also as z does not approach q*, so the framing that regularizer is minimized is still ambiguous.

Overall honest findings so will keep my rating to weak accept

Comments on the paper

4 comments

General response (1/4)

We thank all reviewers for their careful reviews, detailed comments, and questions, and apologize for our delayed responses. We have decided to remove UAS from the paper. UAS is invoked when a learned score is exactly zero, where the default absolute-value subgradient is zero. This condition is rare in our experiments, and tests without UAS show negligible performance differences. We have removed UAS from the method and will upload a revised paper shortly.

We are posting the requested additional experiments as OpenReview comments without further delay. These results and explanations will be included in the appendix and discussion of the revised paper.

G1. Normalization and clipping

Reviewers 6796 (W2), VAEC (W1/W4/Q2), and AmJf (W1/Q1) asked for clearer evidence of the practical effects of normalization and clipping across budgets.

We test how normalization and clipping affect subsets selected from MNIST images and labels, using retrained test accuracy and five-seed mean differences in percentage points across budgets.

Methods. We cross Softmax/ALN with the three clipping rules below on MNIST at K=20,50,100K=20,50,100, using five seeds and 50,000 steps each for selection and retraining. Other SrGS settings follow the paper's select–retrain protocol.

All rules use zi=clip⁡(z~i,ϵ,1−ϵ)z_i=\operatorname{clip}(\tilde z_i,\epsilon,1-\epsilon) in the forward pass, with ϵ=10−6\epsilon=10^{-6}. Their backward derivatives di=∂zi/∂z~id_i=\partial z_i/\partial\tilde z_i are:

  • C0C_0 (paper, asymmetric): di=1z~i≤1−ϵd_i=\mathbf 1\\{\tilde z_i\le1-\epsilon\\}.
  • C1C_1 (standard clipping): di=1ϵ≤z~i≤1−ϵd_i=\mathbf 1\\{\epsilon\le\tilde z_i\le1-\epsilon\\}.
  • C2C_2 (gradients pass at both bounds): di=1d_i=1.

Results. Mean accuracy differences are shown in percentage points over five seeds. The ALN −- Softmax rows average the three clipping rules. The clipping rows use ALN and compare each alternative with the paper's C0C_0 rule.

\hline \text{Comparison} & K=20 & K=50 & K=100 \\\\ \hline \text{ALN }-\text{ Softmax} & -0.45 & +0.40 & +0.10 \\\\[3pt] C_1-C_0\text{ under ALN} & +0.14 & +0.05 & -0.11 \\\\[3pt] C_2-C_0\text{ under ALN} & -0.92 & -0.31 & -0.09 \\\\ \hline \end{array}$$ A positive value favors the first method in the comparison. Averaged over budgets and clipping rules, the ALN $-$ Softmax difference is $+0.018$ percentage points. ALN's mean effect depends on budget. The $C_1-C_0$ differences are small and change sign, while $C_0$ consistently outperforms $C_2$. This suggests that limiting further allocation to saturated candidates helps selection even when the normalizer has little average effect. ALN's gains at $K=50$ and 100 follow the direction of the paper's stronger SrGS results against other baselines at moderate and high budgets. ### G2. Feature recovery Reviewer VAEC (W1) requested feature-recovery measurements, and reviewers 6796 (W2) and AmJf (W1) questioned ALN's practical benefit. We test recovery of three initially suppressed relevant features using ALN and Softmax on synthetic Gaussian inputs with known linear targets, measuring true-feature recall, recovery step, and test MSE after least-squares refitting. **Methods.** We use the paper's SrGS settings with 100 independent Gaussian features, 10 relevant features, budget 10, and 4,096 training and test examples each. Three relevant features start at probability $10^{-4}$ or $10^{-6}$ at the target budget, with the remaining allocation redistributed. Both methods share initial probabilities, model weights, data, minibatches, and gate-noise streams. We run 10 seeds for 50,000 steps and refit the final subset by least squares. Recovery is the first of five consecutive 250-step checkpoints selecting all three suppressed features. **Results.** Recall is the mean percentage of the 10 relevant features selected. Test MSE is mean $\pm$ sample standard deviation, and recovery step is the median across seeds. Each row contains 10 runs. $$\begin{array}{ccccc} \hline \text{Initial }q & \text{Method} & \text{Recall (\\%)} & \text{Test MSE} & \text{Recovery step} \\\\ \hline 10^{-4} & \text{ALN} & 100 & 0.0100\pm0.0002 & 6{,}875 \\\\ 10^{-4} & \text{Softmax} & 100 & 0.0100\pm0.0002 & 36{,}500 \\\\[3pt] 10^{-6} & \text{ALN} & 100 & 0.0100\pm0.0002 & 7{,}875 \\\\ 10^{-6} & \text{Softmax} & 70 & 0.3138\pm0.0077 & \text{unrecovered} \\\\ \hline \end{array}$$ At $10^{-6}$, ALN recovers all 10 relevant features in every seed, while Softmax selects seven in every seed. ALN recovers the initially suppressed features earlier at $10^{-4}$ and gives better final recovery and prediction error at $10^{-6}$. These results describe recovery under deliberately suppressed initialization in this controlled task.

General response (2/4)

G2. Feature recovery

This suggests that ALN can provide an optimization benefit when useful features start with very little allocation, a regime that small average accuracy differences can obscure.

G3. Avazu normalization

Reviewer 6796 (W2/Q2) requested heterogeneous-cost component ablations, and reviewers VAEC (W4/Q2) and AmJf (W1/Q1) asked for broader evidence of component gains.

We learn budget-feasible embedding dimensions from Avazu records and vocabulary-based costs under each normalizer/clipping rule, measuring five-seed mean full-test AUC after retraining.

Methods. Each candidate is a four-dimensional embedding block with cost 4Vj4V_j, where VjV_j is its field's vocabulary size. The first block of each field is always retained and charged to the budget, and ρ\rho scales the reference allocation of 20 dimensions per field. We cross Softmax/ALN with C0C_0 (paper asymmetric clipping), C1C_1 (standard clipping), and C2C_2 (gradients pass at both bounds), using the definitions in General response, G1. Other SrGS settings follow the paper. Each configuration has five seeds, for 90 runs in total.

Results. Full-test AUC after retraining is shown as mean ±\pm sample standard deviation over five seeds. The last column gives the mean ALN −- Softmax difference in units of 10−410^{-4} AUC.

\hline \rho & \text{Clip} & \text{ALN AUC} & \text{Softmax AUC} & \text{ALN }-\text{ Softmax} \\\\ \hline 0.25 & C_0 & 0.77200\pm0.00015 & 0.77210\pm0.00011 & -1.0 \\\\ 0.25 & C_1 & 0.77186\pm0.00006 & 0.77229\pm0.00015 & -4.3 \\\\ 0.25 & C_2 & 0.77239\pm0.00023 & 0.77230\pm0.00012 & +0.9 \\\\[3pt] 0.50 & C_0 & 0.77239\pm0.00029 & 0.77238\pm0.00031 & +0.0 \\\\ 0.50 & C_1 & 0.77235\pm0.00018 & 0.77245\pm0.00029 & -0.9 \\\\ 0.50 & C_2 & 0.77157\pm0.00124 & 0.77105\pm0.00146 & +5.2 \\\\[3pt] 1.00 & C_0 & 0.77074\pm0.00199 & 0.76849\pm0.00148 & +22.6 \\\\ 1.00 & C_1 & 0.77148\pm0.00070 & 0.76955\pm0.00068 & +19.4 \\\\ 1.00 & C_2 & 0.77030\pm0.00110 & 0.76964\pm0.00109 & +6.6 \\\\ \hline \end{array}$$ At $\rho=0.25$, Softmax has higher mean AUC under $C_0$ and $C_1$, while ALN is slightly higher under $C_2$. At $\rho=0.50$, the normalizer differences under $C_0$ and $C_1$ are small. At $\rho=1.00$, ALN has higher mean AUC under all three clipping rules, with the largest ALN $-$ Softmax difference under $C_0$. The relative performance of ALN depends on the budget. Softmax is slightly better at the smallest budget, and ALN is better at the largest budget, where variation across seeds is also largest. The stronger performance of upper-bound gradient blocking under ALN at $\rho=0.50$ and 1.00, together with ALN's advantage at $\rho=1.00$, favors configurations closer to the paper's settings and follows the same direction as SrGS's stronger results against other baselines at moderate and high budgets. ### G4. Selection and retraining Reviewer 6796 (Q4) asked about retaining jointly trained weights, and VAEC (W5/Q3) asked for the performance change at final selection and retraining. We evaluate trained SrGS models on MNIST and Avazu with sampled gates, the final subset, and the subset after retraining, measuring five-seed mean accuracy and cross-entropy on MNIST, AUC on Avazu, and changes between stages. **Methods.** We use the paper's SrGS settings. On MNIST, $A$ uses the selected top-$K$ features with the jointly trained prediction weights. $B$ uses sampled soft gates, averaged over 20 full-test evaluations per seed. $C$ uses the same features as $A$ with fresh prediction weights trained for 50,000 steps. All stages use the same test set and five seeds. We report $B-A$ for sampled gates and $C-A$ for retraining, with positive values indicating higher accuracy or AUC than $A$. Stage $C$ follows the paper's select–retrain reporting protocol. **Results.** Values are mean $\pm$ sample standard deviation across seeds. Accuracy is reported in percent. $$\begin{array}{ccccc} \hline K & \text{Metric} & \begin{array}{c}\text{Selected features} \\\\ A\end{array} & \begin{array}{c}\text{Sampled gates} \\\\ B\end{array} & \begin{array}{c}\text{Retrained} \\\\ C\end{array} \\\\ \hline 20 & \text{Accuracy} & 81.95\pm2.78 & 79.54\pm1.07 & 87.92\pm0.71 \\\\ & \text{Cross-entropy} & 0.5499\pm0.0795 & 0.6218\pm0.0309 & 0.4053\pm0.0186 \\\\[3pt] 50 & \text{Accuracy} & 95.97\pm0.83 & 95.45\pm0.86 & 95.65\pm0.19 \\\\ & \text{Cross-entropy} & 0.1289\pm0.0264 & 0.1455\pm0.0263 & 0.3078\pm0.0121 \\\\[3pt] 100 & \text{Accuracy} & 97.89\pm0.12 & 97.79\pm0.12 & 97.60\pm0.08 \\\\ & \text{Cross-entropy} & 0.0919\pm0.0060 & 0.0953\pm0.0057 & 0.1761\pm0.0172 \\\\ \hline \end{array}$$

General response (3/4)

G4. Selection and retraining

Relative to the selected-feature baseline AA, the mean accuracy changes B−AB-A are −2.41-2.41, −0.52-0.52, and −0.10-0.10 percentage points at K=20,50,100K=20,50,100. The retraining changes C−AC-A are +5.97+5.97, −0.33-0.33, and −0.30-0.30 points. Positive values indicate higher accuracy than AA.

For Avazu, we evaluate the same three stages in 15 runs (ρ=0.25,0.50,1.00\rho=0.25,0.50,1.00, five seeds, 100,000 steps per phase) on the full test set of 3,621,003 examples. AUC is mean ±\pm sample standard deviation across seeds. Differences are seed-paired and reported in units of 10−410^{-4} AUC.

\hline \rho & A & B & C & B-A & C-A \\\\ \hline 0.25 & 0.76536\pm0.00206 & 0.77141\pm0.00019 & 0.77200\pm0.00014 & +60.5 & +66.4 \\\\ 0.50 & 0.77214\pm0.00058 & 0.77261\pm0.00032 & 0.77239\pm0.00029 & +4.7 & +2.5 \\\\ 1.00 & 0.76948\pm0.00128 & 0.76957\pm0.00117 & 0.77075\pm0.00199 & +0.9 & +12.7 \\\\ \hline \end{array}$$ At $\rho=0.25$, the selected subset without retraining loses $60.5\times10^{-4}$ AUC relative to $B$ in all five seeds, and retraining recovers the loss, ending $5.9\times10^{-4}$ above $B$. At this budget no candidate block has $z_i>0.04$, so the greedy scan turns on 16–43 blocks that were almost never active during training. At $\rho=0.50$ and $1.00$, $A$, $B$, and $C$ agree within seed variation. On MNIST, final selection improves mean accuracy at all budgets, consistently across seeds at $K=50$ and 100. Retraining further helps at $K=20$, while retaining jointly trained weights gives higher accuracy and lower cross-entropy at $K=50$ and 100. Compared with sampled gates, retraining improves mean accuracy at $K=20$ and 50 and reduces it by $0.19$ points at $K=100$. On Avazu, retraining is needed at the smallest budget. These results suggest that retraining is most valuable when selection substantially changes the feature combinations seen during training. ### G5. Linear allocation Reviewer AmJf (W2/Q1/Q3) asked whether the learned allocation approaches Theorem 2, and 6796 (Q5) asked how the analysis supports SrGS's training procedure. **Connection to nonlinear soft gates.** The exact guarantees cover linear squared loss with hard gates. In our experience, soft gates usually give slightly better performance, with small differences between gate choices. We use the hard gate result as a reference for soft gate training and the paper's local approximation to connect it to nonlinear models. Using the paper's notation, the second-order expansion in mask coordinates is $$\mathbb{E}_w[\mathcal{L}(w\circ\theta)] \approx \mathcal{L}(z\circ\theta) +\frac{1}{2}\sum_{i,j} \frac{\partial^2\mathcal{L}(z\circ\theta)}{\partial z_i\partial z_j} \operatorname{Cov}(w_i,w_j).$$ For independent hard gates, $\mathbb E[w_i]=z_i$, off-diagonal covariances vanish, and $\operatorname{Var}(w_i)=z_i(1-z_i)$. For soft gates, a corresponding expansion uses their actual mean and covariance. This local approximation requires a smooth loss and a small higher-order remainder, and applies to piecewise-linear networks within fixed activation regions. Linear squared loss makes the hard gate expansion exact. To test Theorem 2's allocation benchmark under its linear hard gate assumptions, we train on normalized Gaussian features with known targets and heterogeneous costs, compute $q^*$ from the learned effective coefficients, and measure checkpoint-wise probability error and relative excess masking variance for runs with and without the added variance gradient. **Methods.** We use 32 Gaussian feature columns normalized to $\\|X_i\\|_2=1$, 2,048 training examples, a noiseless target with known coefficients, costs in $\\{1,2,4,8\\}$ totaling 120, and five seeds at $B=30,60,90$. We use the paper's SrGS settings with hard gate and a gate-to-probability derivative of one. We train with Adam at learning rate $10^{-3}$ and batch size 128 for 20,000 steps, plus a 100,000-step extension, with a fixed budget throughout. Every 250 steps (1,000 in the extension), we hold the current learned $\beta=z\circ\theta$ fixed and compute $$q_i^*=\min\left\\{1,\frac{|\beta_i|}{\sqrt{\lambda c_i}}\right\\}, \qquad\sum_i c_iq_i^*=B.$$ Here $q$ and coefficients $a$ describe alternative allocations with $q\circ a=\beta$, as in the paper, while $z$ denotes the learned probabilities. For $\mathcal{P}_{B,c}=\\{q\in[0,1]^N:\sum_i c_iq_i\le B\\}$, the paper defines $$R(q;\beta)=\sum_i\\|X_i\\|_2^2\beta_i^2\frac{1-q_i}{q_i}, \qquad R^*\_{B,c}(\beta)=\inf_{q\in\mathcal{P}_{B,c}}R(q;\beta).$$ The learned masking penalty is $R(z;\beta)=R_{\mathrm{var}}(z,\theta)$, and $R^*\_{B,c}(\beta)$ is its minimum for the same effective coefficients and budget.

General response (4/4)

G5. Linear allocation

We measure ∣z−q∗∣1\\|z-q^*\\|_1 and the relative penalty gap (R(z;β)−R∗_B,c(β))/R∗_B,c(β)\bigl(R(z;\beta)-R^*\_{B,c}(\beta)\bigr)/R^*\_{B,c}(\beta). We also repeat training with the same data and random streams after adding ∣Xi∣22θi2(1−2zi)\\|X_i\\|_2^2\theta_i^2(1-2z_i) to the gradient with respect to ziz_i, which supplies the missing variance derivative and makes that gradient correct on average for the full expected loss.

Results. After 20,000 steps, the first two error columns report the basic hard gate run (mean ±\pm sample standard deviation across five seeds). The last column gives the mean probability error after adding the variance gradient:

\hline B & \\|z-q^*\\|_1 & \frac{R(z;\beta)-R^*\_{B,c}(\beta)}{R^*\_{B,c}(\beta)} & \begin{array}{c}\ell_1\text{ error after} \\\\ \text{adding variance gradient}\end{array} \\\\ \hline 30 & 1.71\pm0.12 & 0.27\pm0.02 & 0.019 \\\\ 60 & 2.36\pm0.11 & 0.29\pm0.01 & 0.012 \\\\ 90 & 2.57\pm0.19 & 0.58\pm0.02 & 0.0044 \\\\ \hline \end{array}$$ The basic run's penalty is 27%, 29%, and 58% above its minimum, with gaps persisting after 100,000 steps and after $\beta$ stabilizes. Checks using the clipping limits and budget $\sum_i c_iz_i$ preserve the gap. With the added gradient, $R(z;\beta)-R^*\_{B,c}(\beta)$ is approximately $0.006/0.004/0.003$. All 15 basic runs give the same greedy subset when ranked by learned scores or $q^*$. The learned probabilities retain a gap from the optimum for their current $\beta$. The large reduction after adding the variance gradient is consistent with the missing variance derivative contributing to this allocation gap. The agreement in final subsets suggests that the gap affects stochastic training probabilities more strongly than the deployed subset in this setting. The theorem characterizes the expected loss, and convergence of the training updates requires separate analysis. ### G6. Training cost Reviewer VAEC (Q3) requested the per-batch distribution of gate-weighted training cost. After budget annealing, we use MNIST and Avazu training gates and feature costs to measure each example's cost and the batch mean, reporting budget-normalized means, standard deviations, 99th percentiles, and exceedance rates. **Methods.** We use SrGS with the paper's settings on MNIST, with batch size $m=128$ and target budget $B=K$. For example $j$ in batch $b$, define its sampled cost $C_{b,j}$ and the batch mean $\bar C_b$ as $$C_{b,j}=\sum_i c_iw_{b,j,i}, \qquad \bar C_b=\frac1m\sum_{j=1}^{m}C_{b,j},$$ where $w_{b,j,i}$ is the gate used in that training step and $c_i=1$ on MNIST. Costs are recorded every 100 steps across five seeds, with warm-up, annealing, and final-budget phases logged separately. The table uses only the final-budget phase, at training fraction $p\ge0.2$, after annealing has reached $B=K$. All displayed costs are normalized by this fixed target budget. **Results.** $$\begin{array}{cccccc} \hline K & \text{Mean }C_{b,j}/B & \text{SD }C_{b,j}/B & Q_{0.99}(C_{b,j}/B) & \text{SD }\bar C_b/B & P(\bar C_b>B) \\\\ \hline 20 & 1.0103 & 0.1964 & 1.4919 & 0.0182 & 0.708 \\\\ 50 & 1.0037 & 0.1095 & 1.2742 & 0.0112 & 0.619 \\\\ 100 & 0.9961 & 0.0639 & 1.1597 & 0.0104 & 0.383 \\\\ \hline \end{array}$$ The mean allocation scale is close to the budget, and 41.0%, 32.5%, and 18.8% of examples exceed $1.05B$. We separately record unclipped allocation cost, clipped-probability cost, and sampled cost, with warm-up, annealing, and target-budget intervals kept separate. For Avazu, we use the same configuration in 15 runs ($\rho=0.25,0.50,1.00$, five seeds, batch size 1,024), recording 1,024 training examples every 200 steps. Costs are counted in parameters and include the always-on first block of each field. The final-budget phase gives: $$\begin{array}{cccccc} \hline \rho & \text{Mean }C_{b,j}/B & \text{SD }C_{b,j}/B & Q_{0.99}(C_{b,j}/B) & P(C_{b,j}>1.05B) & P(\bar C_b>B) \\\\ \hline 0.25 & 1.0028 & 0.2403 & 1.8629 & 0.246 & 0.668 \\\\ 0.50 & 0.9852 & 0.1699 & 1.5051 & 0.313 & 0.159 \\\\ 1.00 & 0.9822 & 0.1062 & 1.2892 & 0.239 & 0.035 \\\\ \hline \end{array}$$ At $\rho=0.25$, the always-on blocks account for 80% of the budget, and the candidate part of the cost is highly skewed, with median $0.55$ and 99th percentile $5.3$ times the candidate budget. At $\rho=0.50$ and $1.00$, the mean sampled cost is 1.5–1.8% below the budget because 66–169 blocks saturate at the upper clipping bound. Variance from mask sampling explores feature combinations, which we view as central to SrGS's performance. Finite-temperature gates can have $\mathbb E[w_i]$ different from $z_i$, with sampled costs fluctuating around the budget. The deployment subset satisfies the hard budget.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.