- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes Self-regularized Gumbel Sigmoid (SrGS) for feature selection under a resource budget. It assigns one learnable score to each candidate, normalizes the scores into a competitive distribution, converts that distribution into retention probabilities using candidate costs, and trains with stochastic Gumbel-Sigmoid gates. At deployment, it constructs a subset whose total cost does not exceed the budget.
Critical strengths
Practical problem formulation. The paper treats feature costs as heterogeneous, which is more appropriate than a fixed feature count when embedding tables have different vocabulary sizes.
Simple integration. SrGS adds one score per candidate and masks inputs without changing the downstream backbone. The select–retrain evaluation helps assess the selected subset rather than the trained gates alone.
Critical weaknesses
1. Limited evidence for the benefit of ALN. The paper motivates ALN as a remedy for Softmax gradient suppression, but its empirical gain is small and inconsistent. In Table 1, ALN improves accuracy by only about 0.10 percentage points on average across the paired settings, and it performs worse than Softmax under one-stage selection with standard annealing (0.9535 versus 0.9561). The paper does not provide gradient or feature-recovery measurements showing that the proposed optimization problem occurs in practice or that ALN resolves it.
2. The UAS rule is insufficiently justified. UAS chooses a particular subgradient when a score is exactly zero, but this choice is not uniquely implied by the objective. When a feature has positive utility disadvantage, zero is locally preferable, yet the rule still moves its score away from zero and can increase the loss on that step. Moreover, UAS acts only when a score is exactly zero, and the paper provides no isolated ablation establishing how often this happens or whether UAS improves the final result.
3. Reproducibility is limited by the absence of code. The manuscript provides pseudocode and experimental settings but no implementation or repository link. Details such as the custom UAS gradient, asymmetric clipping, budget annealing, preprocessing, and select–retrain procedure are implementation-sensitive. Without code, it is difficult to independently verify the reported comparisons.
4.The ablation results are modest and narrow. Table 1 evaluates only MNIST at . The reported average gains from two-stage selection, slow annealing, and ALN are approximately 0.19, 0.37, and 0.10 percentage points, respectively—small relative to some of the reported variation across seeds. Ablations across additional datasets and budgets would provide stronger evidence that each component contributes reliably.
5. The final greedy selection is not justified by the training objective. At inference, the method ranks candidates by retention probability and accepts each one if it fits the remaining budget. This guarantees feasibility but provides no guarantee of selecting the best subset under heterogeneous costs or interacting feature utilities. The deterministic subset can also differ substantially from the stochastic masks used during training. The paper should quantify this training-to-selection gap and compare the greedy rule with alternative budgeted selection procedures.
Questions for authors
Q1: Line 464 cites “REINFORCE (?)”, and the placeholder also appears in Appendix G. What is the intended reference? Please correct these citations.
Q2: How were the marginal gains of 0.19 and 0.37 percentage points reported for Table 1 calculated? Averaging the displayed factorial contrasts appears to give approximately 0.34 points for two-stage selection and 0.52 points for slow annealing. Please clarify or correct these figures. Could you also provide separate ablations of UAS and the asymmetric clipping rule across multiple datasets and budgets?
Q3: During the actual soft-gate training, what is the per-batch distribution of the gate-weighted cost ? How much performance is lost when converting the learned gates into a hard-budget subset, both before and after retraining? These measurements would help connect the implementation to the idealized hard-gate, linear-model analysis.
Q4: For the strongest baselines, what hyperparameter ranges were searched at each budget, which settings were selected, and how many trials or how much compute was used? This information is needed to assess the fairness of the comparisons.
2 comments
This version has indeed improved readability and informativeness. However, I did not see any superiority over other baselines, typo exists (see line 464) and the code is not released, so I will raise score to weak rejection.
Thank you for your review and helpful comments. We address your concerns and questions below, together with additional experimental results.
The broader normalization and clipping ablation is reported in General response, G1.
The requested feature-recovery measurements are reported in General response, G2.
We also measured score gradients directly in the MNIST ablation. At K=20, about 250 of the 784 Softmax scores end below the lower clipping bound, and their median score-gradient magnitude is (3.7–4.5)×10−7 of that of the retained features. Under ALN, the corresponding gradients remain of order one relative to the retained features. As General response, G1 shows, this difference in gradients does not translate into a consistent accuracy gain.
The ablation is extended to Avazu embedding allocation at three budgets in General response, G3.
We address this point in the General response above.
We plan to release the implementation and configuration files for all reported runs. Completing the release process will take additional time beyond the rebuttal period. We also expect to gain access to the original development materials, which would allow us to check for additional records of hyperparameter selection. We will report any further details that we can verify.
Our recollection is that we generally started from the hyperparameter settings in the original papers. When adjustment was needed, we tried values on a coarse multiplicative scale, with roughly threefold spacing. Complete records of this selection process are unavailable, so we cannot reliably reconstruct the exact search ranges, trial counts, or tuning compute for each method and budget. The current manuscript's Reproducibility and Hyperparameter Details appendix lists the final method-specific configurations. The final evaluation runs used one fixed configuration per method, dataset, and budget across five seeds, with settings largely shared across datasets and budgets. The recorded compute for the published runs is about 5,500 job-hours, including queueing time and both selection and retraining. This figure covers the final evaluation jobs.
The comparison of stochastic gate evaluation, final feature selection, and fresh retraining is reported in General response, G4.
The final greedy scan enforces budget feasibility. The evaluation in General response, G4 measures its behavior on MNIST and Avazu, including a case (25% of the reference embedding budget) where the greedy subset departs from the training masks and retraining is required. Optimality for interacting neural utilities requires further analysis. We will state these limits explicitly. Comparisons with alternative final-selection procedures remain future work.
The per-example and per-batch cost distributions, measured after budget annealing, are reported in General response, G6.
The intended REINFORCE reference is Williams (1992), Simple statistical gradient-following algorithms for connectionist reinforcement learning [3]. Sorry for the typo. The current manuscript includes this citation in the main text and in the appendix's Connection to REINFORCE section.
Your recalculation is correct. Sorry for the typo.
References
[3] Ronald J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 8:229–256, 1992. https://doi.org/10.1007/BF00992696