- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
In this paper, the authors reformulate feature selection as a reward-guided latent denoising process, and develop RGFS (Reward-Guided Feature Selection), its sparse-attention variant RGFS+, Its core components include: (1) a Reward Evaluator (an MLP that predicts downstream task performance from latent representations), (2) a Latent Diffusion Model (LDM), (3) Graph-induced sparse attention for scaling attention-based feature selection
Critical strengths
(1) A novel formulation of feature selection as reward-guided latent diffusion, enabling distribution-level exploration of high-utility subsets rather than optimizing a single latent point (2) Integrating a Latent Diffusion Model (LDM) with a Variational Autoencoder (VAE) and a Reward Evaluator (3) introduce a Graph-Induced Sparse Attention mechanism, denoting the resulting variant as RGFS+
Critical weaknesses
(1) The developed framework RGFS includes VAE, diffusion model, and reward evaluator, which is trained on only 200–300 mask-reward pairs (Appendix C). The "Training Data Expansion" technique maybe merely replicate existing data points without introducing any new feature subsets, and consider no analysis of training data sensitivity, such as table 2 with more compared methods, some discussion about the parameters such as
(2) The relationship , comparison or difference among Gong et al. (2025) "Sculpting Features from Noise: Reward-Guided Hierarchical Diffusion for Task-Optimal Feature Transformation", VTFS.
(3) discuss the role of sequence modal, some typos: long sentence" These results support reward-guided diffusion for feature-mask generation and graph-induced sparse attention for high-dimensional feasibility under the evaluated settings", some parameters such as σ
Questions for authors
(1) Circular dependency undermines and extremely sparse and artificially inflated training data questions generalization (2) the comparison and discussion about the related works (3) the improvement for table 2 with some other methods
3 comments
Response to Reviewer VSFD (2/2)
Q3: Sequence Modeling Role, Notation, Hyperparameters, and Table 2 Baselines
-
Role of Sequence Modeling: Indexing features turns a mask into an -element sequence where positional encodings act as feature-identity embeddings. Self-attention models high-order feature interactions in latent space, while the parallel decoder predicts all inclusion probabilities simultaneously without order sensitivity.
-
Notation and Hyperparameters:
- in Eq. (3) is the VAE diagonal posterior variance (renamed to in the revision); in Eq. (7) is the logistic sigmoid; in Eqs. (13)–(14) is absorbed into .
- Remaining hyperparameters follow Appendix C (, , neighbor count , , ).
-
Expanding Table 2 and Revising Text: We evaluated official STG (Yamada et al., 2020; selecting the top- features ranked by learned gate probabilities) and Concrete Autoencoders (CAE) (Balın et al., 2019; with selector layer size ), tuning over 3 seeds via search-CV selection on
OpenML_1082() andOpenML_1085():Method OpenML_1082Acc (%)OpenML_1085Acc (%)LASSO 68.3 56.2 STG (Yamada et al., 2020; 3 seeds, search CV) 73.3 ± 0.0 54.6 ± 2.2 CAE (Balın et al., 2019; 3 seeds, search CV) 71.7 ± 2.9 65.1 ± 5.1 RGFS+ 73.3 69.5 We will also split the long sentence ("These results support...") in the Abstract and Section 6 and proofread the manuscript thoroughly.
References for Newly Added Baselines:
- Balın, M. F., Abid, A., & Zou, J. (2019). Concrete Autoencoders: Differentiable Feature Selection and Reconstruction. ICML.
- Yamada, Y., Lindenbaum, O., Negahban, S., & Kluger, Y. (2020). Feature Selection using Stochastic Gates. ICML.
Dear Reviewer,
The rebuttal and discussion phase is now in its final stages, with very limited time remaining for further exchanges. We would greatly appreciate your prompt consideration of the authors’ responses and any additional comments you may have before the discussion closes.
Many thanks for your valuable contribution.
Response to Reviewer VSFD (1/2)
We thank Reviewer VSFD for recognizing the novelty of our reward-guided latent diffusion formulation and graph-induced sparse attention, and for the constructive suggestions.
Q1: Training Data Sensitivity, "Training Data Expansion", and Circular Dependency
1. Clarifying "Training Data Expansion" in Appendix C: You are right that permuting feature indices before converting to a fixed-order binary mask yields the exact same mask—as noted in Appendix C (lines 770–772): "permutations of the same subset yield identical masks: this expansion repeats existing mask–reward pairs rather than introducing distinct feature subsets." Inherited from VTFS (Ying et al., 2024), this step acts purely as 26× repeated sampling per epoch over the 253–301 distinct masks (reported approximately as 200–300 in Appendix C). We will rename it "Epoch Update Scaling via Repeated Sampling" and report only the actual distinct mask counts.
2. Sensitivity to Training Collection Size (
K ∈ {50, 100, 150, 301}): We trained RGFS from scratch on random subsamples of K=50,100, and 150 masks (2 subsamples × 3 seeds) alongside the full pool (K=301, 3 seeds). To keep evaluation lightweight without favoring smaller pools, we also imposed a strict double constraint by capping the inference candidate budget at M=K under 5-fold search cross-validation:mean ± std)Note: For the pool baseline, the single mask with the highest collector reward on the search split is selected without accessing the test set and evaluated once on the held-out test split. Because this study caps the inference candidate budget at M=K≤301 (much smaller than the full inference budget in Table 1, where RGFS reaches
55.31and75.36), scores should be compared across K under this restricted budget. Crucially, performance does not collapse as K drops from 301 to 50: even under the double constraint K=M=50 onOpenML_616—where the best collected mask scores only20.63—RGFS reaches48.20 ± 0.72by recombining informative feature patterns across masks, whileGerman Creditremains stable (64.8–67.5, matching the pool baseline within 1 std at this tiny budget M≤301 and rising to75.36under the standard inference budget).3. Addressing Circular Dependency and Out-of-Pool Generalization:
OpenML_616).OpenML_616pool by over 26 points (48.50vs.21.65at K=301, and48.20vs.20.63at K=50).Q2: Comparison Among Gong et al. (2025), VTFS (Ying et al., 2024), and RGFS
We will add the following comparison in Section 2:
In short, DIFFT targets symbolic formula synthesis with intra-chunk autoregressive decoding, whereas RGFS performs fixed-position parallel binary mask generation and introduces O(Nk) correlation-guided sparse attention for high-dimensional scaling.