acceptodds
Under review as a conference paper at ICLR 2027

Reward-Guided Latent Diffusion for Scalable Feature Selection

Abstract

Feature selection (FS) is a cornerstone of high-dimensional data analysis, aimed at identifying informative feature subsets that enhance model generalizability and interpretability. Despite the emergence of latent-space generative feature-selection frameworks, two challenges remain: candidate generation can rely on direct optimization of selected latent points, and dense interaction modeling becomes prohibitively expensive as dimensionality grows. We propose Reward-Guided Feature Selection (RGFS), which reformulates FS as a reward-guided latent denoising process. By integrating a Latent Diffusion Model (LDM) with a Variational Autoencoder (VAE) and a Reward Evaluator, RGFS models the empirical distribution of latent embeddings from collected reward-labeled masks. During inference, evaluator gradients steer independently initialized reverse trajectories toward regions of higher predicted utility. To further scale interaction modeling to ultra-high-dimensional data, we introduce RGFS+, which incorporates Graph-Induced Sparse Attention. RGFS+ restricts each feature to a correlation-defined neighborhood, reducing sparse attention computation and attention-weight storage from to for fixed neighborhood size . Across 14 standard benchmarks, RGFS or RGFS+ achieves the highest or tied-highest reported score. On two additional datasets with approximately 60,000 features, RGFS+ remains runnable when the evaluated dense alternatives encounter out-of-memory errors. These results support reward-guided diffusion for feature-mask generation and graph-induced sparse attention for high-dimensional feasibility under the evaluated settings.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Feature Selection, Data-Centric AI

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_VSFDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, the authors reformulate feature selection as a reward-guided latent denoising process, and develop RGFS (Reward-Guided Feature Selection)​, its sparse-attention variant RGFS+, Its core components include: (1) a Reward Evaluator (an MLP that predicts downstream task performance from latent representations), (2) a Latent Diffusion Model (LDM), (3) Graph-induced sparse attention for scaling attention-based feature selection

Critical strengths

(1) A novel formulation of feature selection as reward-guided latent diffusion, enabling distribution-level exploration of high-utility subsets rather than optimizing a single latent point (2) Integrating a Latent Diffusion Model (LDM) with a Variational Autoencoder (VAE) and a Reward Evaluator (3) introduce a Graph-Induced Sparse Attention mechanism, denoting the resulting variant as RGFS+

Critical weaknesses

(1) The developed framework RGFS includes VAE, diffusion model, and reward evaluator, which is trained on only 200–300 mask-reward pairs (Appendix C). The "Training Data Expansion" technique maybe merely replicate existing data points without introducing any new feature subsets, and consider no analysis of training data sensitivity, such as table 2 with more compared methods, some discussion about the parameters such as
(2) The relationship , comparison or difference among Gong et al. (2025) "Sculpting Features from Noise: Reward-Guided Hierarchical Diffusion for Task-Optimal Feature Transformation", VTFS. (3) discuss the role of sequence modal, some typos: long sentence" These results support reward-guided diffusion for feature-mask generation and graph-induced sparse attention for high-dimensional feasibility under the evaluated settings", some parameters such as σ

Questions for authors

(1) Circular dependency undermines and extremely sparse and artificially inflated training data questions generalization (2) the comparison and discussion about the related works (3) the improvement for table 2 with some other methods

3 comments
AuthorsOpenReview ↗

Response to Reviewer VSFD (1/2)

We thank Reviewer VSFD for recognizing the novelty of our reward-guided latent diffusion formulation and graph-induced sparse attention, and for the constructive suggestions.


Q1: Training Data Sensitivity, "Training Data Expansion", and Circular Dependency

1. Clarifying "Training Data Expansion" in Appendix C: You are right that permuting feature indices before converting to a fixed-order binary mask yields the exact same mask—as noted in Appendix C (lines 770–772): "permutations of the same subset yield identical masks: this expansion repeats existing mask–reward pairs rather than introducing distinct feature subsets." Inherited from VTFS (Ying et al., 2024), this step acts purely as 26×26\times repeated sampling per epoch over the 253–301 distinct masks (reported approximately as 200–300 in Appendix C). We will rename it "Epoch Update Scaling via Repeated Sampling" and report only the actual distinct mask counts.

2. Sensitivity to Training Collection Size (K ∈ {50, 100, 150, 301}): We trained RGFS from scratch on random subsamples of K=50,100,K = 50, 100, and 150150 masks (2 subsamples ×\times 3 seeds) alongside the full pool (K=301K = 301, 3 seeds). To keep evaluation lightweight without favoring smaller pools, we also imposed a strict double constraint by capping the inference candidate budget at M=KM = K under 5-fold search cross-validation:

DatasetPool Size KK (M=KM = K)Top-Reward Mask in KK-Pool (Test)RGFS Test Score (mean ± std)
OpenML_6165020.6348.20 ± 0.72
10021.0849.98 ± 2.29
15024.8248.94 ± 5.86
30121.6548.50 ± 0.00
German Credit5067.3467.53 ± 0.96
10065.5166.30 ± 1.01
15065.5467.21 ± 1.95
30166.0864.79 ± 1.93

Note: For the pool baseline, the single mask with the highest collector reward on the search split is selected without accessing the test set and evaluated once on the held-out test split. Because this study caps the inference candidate budget at M=K≤301M = K \le 301 (much smaller than the full inference budget in Table 1, where RGFS reaches 55.31 and 75.36), scores should be compared across KK under this restricted budget. Crucially, performance does not collapse as KK drops from 301 to 50: even under the double constraint K=M=50K = M = 50 on OpenML_616—where the best collected mask scores only 20.63—RGFS reaches 48.20 ± 0.72 by recombining informative feature patterns across masks, while German Credit remains stable (64.8–67.5, matching the pool baseline within 1 std at this tiny budget M≤301M \le 301 and rising to 75.36 under the standard inference budget).

3. Addressing Circular Dependency and Out-of-Pool Generalization:

  • No Circular Evaluation (EψE_\psi vs. Selection): EψE_\psi only provides latent guidance gradients during denoising. Generated candidates are filtered by 5-fold downstream CV strictly on the 80% search split and evaluated once on the held-out 20% test split (Table 4 further confirms cross-model transfer across five regressors on OpenML_616).
  • Breaking Circular Dependency on Collector Quality (100% Novel Selections): RGFS is not upper-bounded by the suboptimal masks collected by MARLFS. Across our multi-seed novelty check (4 representative benchmarks ×\times 5 sampling seeds = 20 runs), all 20 final selected masks (0/20 in pool) are unseen during training, and 86.6%–100% of all generated candidates per run are novel—consistent with Q1.2 above, where RGFS outperforms the best collected mask in the OpenML_616 pool by over 26 points (48.50 vs. 21.65 at K=301K = 301, and 48.20 vs. 20.63 at K=50K = 50).

Q2: Comparison Among Gong et al. (2025), VTFS (Ying et al., 2024), and RGFS

We will add the following comparison in Section 2:

DimensionVTFS (Ying et al., 2024)DIFFT (Gong et al., 2025)RGFS / RGFS+ (Ours)
TaskFeature SelectionSymbolic Feature TransformationFeature Selection (F∈0,1NF \in \\{0, 1\\}^N)
RepresentationVariable-order feature-index sequenceHierarchical postfix expression chunksFixed-order NN-bit binary mask
Latent SearchSingle-trajectory gradient ascentHierarchical reward-guided diffusionMulti-trajectory reward-guided DDIM
DecoderAutoregressive token-by-tokenSemi-autoregressive (AR within chunks)Fully parallel single-pass over all NN
ComplexityDense O(N2)\mathcal{O}(N^2)Dense over symbolic tokensGraph-sparse O(Nk)\mathcal{O}(Nk) in RGFS+

In short, DIFFT targets symbolic formula synthesis with intra-chunk autoregressive decoding, whereas RGFS performs fixed-position parallel binary mask generation and introduces O(Nk)\mathcal{O}(Nk) correlation-guided sparse attention for high-dimensional scaling.

AuthorsOpenReview ↗

Response to Reviewer VSFD (2/2)

Q3: Sequence Modeling Role, Notation, Hyperparameters, and Table 2 Baselines

  1. Role of Sequence Modeling: Indexing features j∈1,…,Nj \in \\{1, \dots, N\\} turns a mask into an NN-element sequence where positional encodings act as feature-identity embeddings. Self-attention models high-order feature interactions in latent space, while the parallel decoder predicts all NN inclusion probabilities simultaneously without order sensitivity.

  2. Notation and Hyperparameters:

    • σ2\sigma^2 in Eq. (3) is the VAE diagonal posterior variance (renamed to Σϕ\Sigma_\phi in the revision); σ(⋅)\sigma(\cdot) in Eq. (7) is the logistic sigmoid; 1/(2σS2)1/(2\sigma_S^2) in Eqs. (13)–(14) is absorbed into γeff\gamma_{\text{eff}}.
    • Remaining hyperparameters follow Appendix C (α=0.8\alpha = 0.8, β=0.001\beta = 0.001, neighbor count k=5k = 5, Starget=1.0S_{\text{target}} = 1.0, γ=100\gamma = 100).
  3. Expanding Table 2 and Revising Text: We evaluated official STG (Yamada et al., 2020; selecting the top-kk features ranked by learned gate probabilities) and Concrete Autoencoders (CAE) (Balın et al., 2019; with selector layer size kk), tuning k∈50,100,250,500,1000,3000k \in \\{50, 100, 250, 500, 1000, 3000\\} over 3 seeds via search-CV selection on OpenML_1082 (N=59,004N = 59,004) and OpenML_1085 (N=61,359N = 61,359):

    MethodOpenML_1082 Acc (%)OpenML_1085 Acc (%)
    LASSO68.356.2
    STG (Yamada et al., 2020; 3 seeds, search CV)73.3 ± 0.054.6 ± 2.2
    CAE (Balın et al., 2019; 3 seeds, search CV)71.7 ± 2.965.1 ± 5.1
    RGFS+73.369.5

    We will also split the long sentence ("These results support...") in the Abstract and Section 6 and proofread the manuscript thoroughly.


References for Newly Added Baselines:

  • Balın, M. F., Abid, A., & Zou, J. (2019). Concrete Autoencoders: Differentiable Feature Selection and Reconstruction. ICML.
  • Yamada, Y., Lindenbaum, O., Negahban, S., & Kluger, Y. (2020). Feature Selection using Stochastic Gates. ICML.

Dear Reviewer,

The rebuttal and discussion phase is now in its final stages, with very limited time remaining for further exchanges. We would greatly appreciate your prompt consideration of the authors’ responses and any additional comments you may have before the discussion closes.

Many thanks for your valuable contribution.

Reviewer_5YHjOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

RGFS encodes feature masks with a VAE trained jointly with a reward evaluator. A latent diffusion model learns the distribution of the masks MARLFS collected, and evaluator gradients guide sampling. RGFS+ limits attention to a top-kk correlation graph, giving O(Nk)O(Nk) cost. The paper reports best or tied-best scores on 14 small benchmarks and shows RGFS+ running on two datasets with about 60k features.

Critical strengths

Removing reward guidance lowers the score on all four ablation datasets, by up to 18.8 points (Fig. 2).

Critical weaknesses

Table 1 is single-seed and contradicts the paper's own five-seed Table 3: RGFS+ scores 66.01 vs. 56.3±2.256.3\pm2.2 on OpenML_607. On 6 of 14 datasets, RGFS, RGFS+ and VTFS tie exactly, which suggests they all return the best MARLFS mask. There is no best-collected-mask baseline.

The decoder still uses dense N×NN\times N attention (l.216–218), so the O(Nk)O(Nk) claim doesn't cover the whole model. The 60k-feature test sets have about 19 and 32 samples, and LASSO is the only baseline. Differentiable selectors whose cost grows linearly in NN, such as Concrete Autoencoders (Balın et al., 2019) and STG (Yamada et al., 2020), are missing, even though they are known to run at this dimensionality.

Questions for authors

What are the Table 1 results as mean ± std over 5 or more seeds?

What does the best collected mask score, and are the generated masks new?

How many candidates are scored before "select the best"?

How does the decoder fit in memory at N≈60N\approx60k?

How do Concrete Autoencoders and STG compare on the two 60k-feature datasets?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 5YHj (1/3)

We thank Reviewer 5YHj for the detailed feedback and constructive questions. Below we address each concern with multi-seed evaluations, budget-controlled ablations, and new high-dimensional baselines.

Q1: Reconciling Table 1 (66.01) and Table 3 (56.3 ± 2.2) on OpenML_607, and Five-Seed Results

"Table 1 is single-seed and contradicts the paper's own five-seed Table 3: RGFS+ scores 66.01 vs. 56.3 ± 2.2 on OpenML_607. What are the Table 1 results as mean ± std over 5 or more seeds?"

1. Reconciling 66.01 vs. 56.3 ± 2.2 on OpenML_607:

  • Table 3 (56.3 ± 2.2): The Section 5.6 evaluation script inadvertently used an 8-layer, 512-dim configuration (instead of Appendix C's 2-layer, 64-dim configuration at l.776) and restricted inference sampling to ~14 batches.
  • Table 1 (66.01): This score came from the Appendix C 2-layer, 64-dim dense RGFS (use_sparse_mask: False) and was misrecorded in the adjacent RGFS+ column.
  • Five-Seed Verification: Under the Appendix C configuration with 5-fold CV strictly inside the 80% search split (evaluating the held-out 20% test split only once), RGFS achieves 65.46 ± 0.76, while RGFS+ obtains 59.42 ± 2.97 (replacing the 56.3 ± 2.2 in Table 3 from the mismatched 8-layer, 512-dim / ~14-batch setup; RGFS+ trails dense RGFS here because OpenML_607 is a Friedman regression task with independent features, where correlation-graph sparsity restricts higher-order cross-feature interactions). We will correct the column attribution in Table 1 and report 5-seed results for both RGFS and RGFS+ across all benchmarks in the revised paper.

2. Five-Seed mean ± std Results on Four Benchmarks: Table 1's Original–VTFS columns cited single-split values from Ying et al. (2024). To eliminate test-set selection bias within the limited rebuttal timeframe, we evaluated four benchmarks spanning regression (OpenML_607, OpenML_586) and classification (SVMGuide3, SpamBase) over 5 sampling seeds on the 80/20 split (tuning LASSO over 30 λ\lambda values and RFE/mRMR over k ∈ {10%, ..., 90%}; since these baselines are deterministic across sampling seeds on a fixed split, their std is 0 and omitted). All candidate masks—including the Best-Collected mask from the 301-mask MARLFS pool and RGFS candidates—and baseline hyperparameters were selected exclusively via 5-fold CV within the 80% search split; the 20% test split was evaluated only once after selection. We will complete this 5-seed evaluation for both RGFS and RGFS+ across all 14 Table 1 benchmarks in the revision:

DatasetBest-Collected MaskLASSOmRMRRFERGFS (mean ± std)
OpenML_60755.0651.5557.6564.6365.46 ± 0.76
OpenML_58658.0654.9159.1163.2862.51 ± 1.15
SVMGuide377.7977.9079.2678.7381.03 ± 0.97
SpamBase92.0291.5592.0792.0092.20 ± 0.42

Under this strict protocol, RGFS outperforms the Best-Collected Mask, LASSO, and mRMR on all 4 benchmarks, and beats RFE on 3 of 4 (remaining within 0.77 points on OpenML_586, where RFE explicitly tunes kk and selects k=5k=5, whereas RGFS requires no kk tuning).

AuthorsOpenReview ↗

Response to Reviewer 5YHj (2/3)

Q2 & Q3: Best-Collected Baseline, Mask Novelty, and Candidate Count (MM)

"On 6 of 14 datasets, RGFS, RGFS+ and VTFS tie exactly, which suggests they all return the best MARLFS mask. What does the best collected mask score, and are the generated masks new? How many candidates are scored before 'select the best'?"

  • Mask Novelty: Across these 4 benchmarks ×\times 5 sampling seeds (20 runs), 0/20 final selected masks appear in the MARLFS training pool (100% novel), and at least 86.6% of all generated candidates per run are unseen in the pool. Exact ties in Table 1 occurred because distinct masks sharing core informative features yield identical Random Forest predictions when redundant features do not alter tree splits.
  • Scored Candidates (MM) & Budget-Controlled Comparison: The 1,410 steps in Appendix C refer to diffusion denoising steps, not MM. Under our search-only 5-fold CV protocol (inference batch size = 64), a full run generates and evaluates M≈15,100M \approx 15,100 candidates (~236 batches) on the search split before evaluating the single best mask once on the 20% test split. To test whether RGFS's gains merely stem from a larger candidate pool, we slice the first B ∈ {300, 1000, 3000}generated candidates per run (where B=300B=300 matches the 301-mask MARLFS pool) and report the 5-seed mean Test scores:
DatasetBest-Collected (301 masks)RGFS @B=300B=300 (~5 batches)RGFS @B=1kB=1\text{k} (~16 batches)RGFS @B=3kB=3\text{k} (~47 batches)RGFS @M≈15.1kM \approx 15.1\text{k} (~236 batches)
OpenML_60755.0657.7261.4864.8965.46
OpenML_58658.0657.4359.2360.2062.51
SVMGuide377.7979.2079.7479.8381.03
SpamBase92.0291.8892.0992.0492.20

At the pool-matched budget B=300B=300 (5 batches of one-shot generation vs. 301 collected MARLFS masks), RGFS already outperforms Best-Collected on 2/4 datasets (OpenML_607, SVMGuide3) and is within 0.14–0.63 points on the other two. By a modest budget of B=1,000B=1,000 (~16 batches), RGFS outperforms Best-Collected on all 4/4 datasets: the classification tasks (SVMGuide3, SpamBase) largely plateau early (79.74 →\to 81.03 and 92.09 →\to 92.20), whereas on the Friedman regression tasks (OpenML_607, OpenML_586), larger sampling budgets continue to yield substantial gains (+2.14 to +9.83 at B=3kB=3\text{k}; +4.45 to +10.40 at M≈15.1kM \approx 15.1\text{k}).

AuthorsOpenReview ↗

Response to Reviewer 5YHj (3/3)

Q4: Decoder Memory at N≈60kN \approx 60\text{k} and Sparse Decoder Ablation

"The decoder still uses dense N×NN \times N attention (l.216–218), so the O(Nk)\mathcal{O}(Nk) claim doesn't cover the whole model. How does the decoder fit in memory at N≈60kN \approx 60\text{k}?"

We will qualify the Abstract, Section 4.3, and Section 5.1 so O(Nk)\mathcal{O}(Nk) refers strictly to the VAE encoder and diffusion denoiser, while the decoder (invoked once per candidate batch after multi-step denoising at inference) retains full attention in the original architecture. It fits in memory at N≈60kN \approx 60\text{k} for two reasons: (1) PyTorch's chunked scaled_dot_product_attention avoids materializing the dense N×NN \times N attention matrix (4.85 GB at N=20kN=20\text{k} under dim=64), and (2) the 60k runs in Table 2 used a compact parameterization (dim=8, 1 layer, 1 head, batch size = 4) on an H100 80 GB GPU.

To make the entire pipeline O(Nk)\mathcal{O}(Nk), we also implemented a graph-sparse attention decoder (dim=64, 2 layers, 8 heads) and benchmarked decoder-only training (1 fwd+bwd pass, batch size = 8, A100 40 GB; real datasets for N=21–50N=21–50, synthetic correlation graphs with k≤25k \le 25 neighbors for N≥1kN \ge 1\text{k}):

NNOriginal Time / Peak MemSparse Time / Peak MemSpeedup
21–500.006 s / ≤\le 0.01 GB0.008 s / ≤\le 0.02 GB0.75$\times$
1,0000.016 s / 0.11 GB0.014 s / 0.46 GB1.1$\times$
2,0000.044 s / 0.22 GB0.023 s / 0.98 GB1.9$\times$
5,0000.254 s / 0.60 GB0.059 s / 2.67 GB4.3$\times$
10,0000.990 s / 1.40 GB0.107 s / 4.70 GB9.3$\times$
20,0004.052 s / 4.85 GB0.223 s / 10.25 GB18.2$\times$
60,000OOM (>40 GB)0.713 s / 34.46 GBEnables 60k on 40 GB

At moderate N≤20kN \le 20\text{k}, PyTorch's dense SDPA keeps peak memory low (4.85 GB at 20k) via kernel chunking, whereas our gather-based sparse implementation has a higher constant memory factor (10.25 GB) because it explicitly materializes and caches (B,N,k,d)(B, N, k, d) neighbor tensors for the backward pass. However, dense decoding still suffers from O(N2)\mathcal{O}(N^2) compute time (4.052 s vs. 0.223 s at 20k) and super-linear backward-buffer scaling as NN grows (1.40 GB at 10k →\to 4.85 GB at 20k →\to OOM (>40 GB) at 60k), whereas sparse decoding scales linearly as O(Nk)\mathcal{O}(Nk) in both time and memory (0.46 →\to 4.70 →\to 10.25 →\to 34.46 GB across 1k–60k), achieving an 18.2$\times$ speedup at N=20kN=20\text{k} and enabling N=60kN=60\text{k} on a 40 GB GPU. We will include this sparse decoder ablation and complexity qualification in the revision.


Q5: Comparison with STG and Concrete Autoencoders on 60k-Feature Datasets

"The 60k-feature test sets have about 19 and 32 samples, and LASSO is the only baseline. How do Concrete Autoencoders and STG compare on the two 60k-feature datasets?"

We evaluated official STG (Yamada et al., 2020; selecting the top-kk features ranked by learned gate probabilities) and Concrete Autoencoders (CAE) (Balın et al., 2019; with selector layer size kk), tuning k ∈ {50, 100, 250, 500, 1000, 3000} over 3 seeds under strict search-CV selection on OpenML_1082 and OpenML_1085:

MethodOpenML_1082 Acc (%)OpenML_1085 Acc (%)
LASSO (Table 2)68.356.2
STG (3 seeds, search CV)73.3 ± 0.054.6 ± 2.2
CAE (3 seeds, search CV)71.7 ± 2.965.1 ± 5.1
RGFS+ (Table 2)73.369.5
  • Comparison & Small-Sample Behavior: Under search-CV selection (k≤3,000k \le 3,000), RGFS+ (73.3% / 69.5%) matches STG (73.3 ± 0.0%) and outperforms CAE (71.7 ± 2.9%) and LASSO (68.3%) on OpenML_1082, while achieving the highest score on OpenML_1085 (69.5% vs. 65.1 ± 5.1% for CAE, 54.6 ± 2.2% for STG, and 56.2% for LASSO). As the reviewer notes, because these ultra-high-dimensional benchmarks have small sample sizes (~19 and ~32 test samples, where each sample accounts for ~3–5% accuracy), all methods—including differentiable selectors—exhibit discrete accuracy steps and notable variance across seeds (e.g., CAE ±2.9% / ±5.1%; STG dropping to 54.6% on OpenML_1085).
  • Revision: We will add STG and CAE to Table 2 and explicitly discuss these small-sample statistical caveats alongside the scalability analysis in Section 5.3.

References

  • Balın, M. F., Abid, A., & Zou, J. (2019). Concrete Autoencoders: Differentiable Feature Selection and Reconstruction. ICML.
  • Yamada, Y., Lindenbaum, O., Negahban, S., & Kluger, Y. (2020). Feature Selection using Stochastic Gates. ICML.
  • Ying, W., Wang, D., Chen, H., & Fu, Y. (2024). Feature Selection as Deep Sequential Generative Learning. ACM TKDD (arXiv:2403.03838).

Dear Reviewer,

The rebuttal and discussion phase is now in its final stages, with very limited time remaining for further exchanges. We would greatly appreciate your prompt consideration of the authors’ responses and any additional comments you may have before the discussion closes.

Many thanks for your valuable contribution.

Reviewer_Pn2nOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose a feature selection method based on reward-guided latent diffusion in the latent space of a VAE, which is jointly pretrained with a reward evaluator network. The reward evaluator is trained on feature masks preselected using the RL-based MARLFS method. The utility of each selection is evaluated by fitting a random forest and using its performance as the target for the reward evaluator. The method borrows heavily from VTFS, but replaces its latent-space optimization with a diffusion process and its autoregressive decoder with a non-autoregressive one.

This is not the kind of budget-constrained feature selection where one specifies the desired number of features beforehand or tunes the strength of a sparsity penalty, as in Lasso. Instead, the authors optimize for maximum predictive performance, with no constraint on the number of selected features, potentially allowing all features to be selected if doing so improves performance.

Critical strengths

The authors successfully replace the latent-space optimization and autoregressive decoder of VTFS with a latent diffusion process and a parallel decoder, offering a novel solution to feature selection without a constrained feature budget. They demonstrate performance on par with, and sometimes better than, VTFS, and outperform all other baselines in absolute terms when the number of selected features is not taken into account.

Critical weaknesses
  1. The fact that the number of selected features is not constrained by any budget raises the question of how the number of selected features was determined for the baselines. For example, how was Lasso's lambda tuned? If it was fixed at a value that selects significantly fewer features, Lasso would be at a substantial disadvantage. Neither the tuning of baseline hyperparameters nor the resulting numbers of selected features appear to be reported in the paper.

  2. Training data. The jointly trained VAE and reward evaluator, as well as the separately trained diffusion model, are all pretrained on 200–300 masks selected by MARLFS. How sensitive is performance to the number of MARLFS-selected masks? Could simply collecting more masks make up for the performance gains currently provided by diffusion?

Questions for authors
  1. Please explain how the baselines were tuned and report the numbers of features they selected.

  2. Motivation for correlation-based attention masking. It is unclear why restricting features to attend only to other features that correlate with them should be beneficial for learning feature interactions. I understand the motivation to reduce the computational cost of attention from (O(N^2)), but I don't understand why this should improve results beyond simply reducing computational requirements.

  3. Please clarify how dataset size affects performance and whether a larger dataset could remove the need for the diffusion step.

4 comments
AuthorsOpenReview ↗

Response to Reviewer Pn2n (1/2)

We thank Reviewer Pn2n for the positive assessment and constructive comments, and address each question below.


Q1: Baseline Tuning and Number of Selected Features (#Sel)

"Please explain how the baselines were tuned and report the numbers of features they selected."

Table 1 & Table 2 protocol. In Table 1 (following Ying et al., 2024), ranking/wrapper baselines such as RFE (Granitto et al., 2006) and mRMR (Peng et al., 2005) were evaluated at a fixed subset size kk matched to MARLFS (Liu et al., 2019; e.g., 11–31 features on the Friedman benchmarks) rather than tuned per dataset; we will mark those published columns explicitly in the revision. In Table 2, LASSO (Tibshirani, 1996; capped at 5% of total features, with ℓ1\ell_1 regularization zeroing out most coefficients) selected 244 features on OpenML_1082 (out of 59,004) and 293 features on OpenML_1085 (out of 61,359) in our rerun.

Search-CV tuned comparison. We re-ran LASSO, RFE, mRMR, and RGFS over 5 random seeds on the fixed 80/20 split across 4 benchmarks under a unified search-only 5-fold CV protocol (30-point grid for LASSO's λ\lambda; k∈[10k \in [10\\%, 90\\%]of features for RFE/mRMR; tuning and candidate selection solely on the 80% search split, with the held-out 20% test split evaluated once; differences from Table 1 reflect 5-seed averaging and search-only CV selection rather than Table 1's single-seed setup):

Dataset (NN features)RGFS (#Sel)Tuned LASSO (#Sel)Tuned RFE (#Sel)Tuned mRMR (#Sel)
OpenML_607 (50)65.46 ± 0.76 (4.4)51.55 (45.0)64.63 (5.0)57.65 (10.0)
OpenML_586 (25)62.51 ± 1.15 (5.4)54.91 (24.4)63.28 (5.0)59.11 (8.0)
SVMGuide3 (21)81.03 ± 0.97 (11.2)77.90 (11.6)78.73 (15.6)79.26 (15.4)
SpamBase (57)92.20 ± 0.42 (38.4)91.55 (50.8)92.00 (46.8)92.07 (43.4)

All entries report 5-seed means of test scores and selected feature counts (#Sel), with ± std shown for RGFS due to stochastic diffusion sampling. For Tuned LASSO, Tuned RFE, and Tuned mRMR, varying the random seed during candidate evaluation causes only minor fluctuations in #Sel on a few borderline features while leaving test performance unchanged (hence ± std is omitted).

Notably, RGFS selects fewer features than Tuned LASSO across all 4 datasets (and fewer than all three tuned baselines on SVMGuide3 and SpamBase) while achieving higher accuracy—confirming that its gains do not come from selecting larger subsets. On the non-linear Friedman benchmarks (OpenML_586, OpenML_607, where the target depends on 5 true features), grid-searching kk allows RFE to recover k=5k=5 (63.28 and 64.63). By contrast, Tuned LASSO retains far more features (24.4–45.0) yet scores substantially lower (51.55–54.91), showing that its gap stems from linear model misspecification rather than an overly sparse λ\lambda. Without specifying or grid-searching kk, RGFS compresses the collected MARLFS subsets (11–31 features in Table 1's protocol) down to 4.4–5.4 features and achieves the best test score on 3/4 benchmarks (trailing Tuned RFE by only 0.77 on OpenML_586), whereas LASSO and mRMR trail RGFS by up to 13.9 and 7.8 points.


Q2: Motivation for Correlation-Based Attention Masking

"I understand the motivation to reduce the computational cost of attention from O(N2)O(N^2), but I don't understand why this should improve results beyond simply reducing computational requirements."

We agree that correlation-based sparsification can miss certain feature interactions; its primary motivation in RGFS+ is computational scalability (O(N2)→O(Nk)\mathcal{O}(N^2) \to \mathcal{O}(Nk)) rather than a universal accuracy improvement over dense RGFS. Specifically, RGFS and RGFS+ are designed to explicitly address the trade-off between interaction coverage and computational efficiency across feature scales: RGFS preserves dense interaction modeling when dimensionality permits and accuracy is prioritized, whereas RGFS+ provides a scalable option for high-dimensional settings. As noted in Section 5.2, top-kk correlation neighborhoods act as a structural inductive bias—occasionally regularizing noisy small-sample estimates, but potentially losing accuracy when synergistic features have low marginal correlation—so the degree of sparsification should be chosen according to data characteristics and compute budget. We will revise the Introduction and Section 4.3 to state the applicable regimes and limitations of both designs explicitly.

AuthorsOpenReview ↗

Response to Reviewer Pn2n (2/2)

Q3: Sensitivity to Collection Dataset Size (KK) and Role of Diffusion vs. More Collection

"Please clarify how dataset size affects performance and whether a larger dataset could remove the need for the diffusion step."

Sensitivity to collection size (K∈50,100,150,301K \in \\{50, 100, 150, 301\\}). On OpenML_616 and German Credit, we subsampled K∈50,100,150K \in \\{50, 100, 150\\} masks from the MARLFS pool (6 runs per KK: 2 independent subsamples ×\times 3 sampling seeds) alongside the full K=301K = 301 pool (3 sampling seeds), and retrained RGFS from scratch. To keep evaluation lightweight without favoring smaller pools, we generated only M=KM = K candidates during inference (much smaller than the full inference budget used in Table 1, where RGFS reaches 55.31 on OpenML_616 and 75.36 on German Credit), and compared against the best mask in each KK-pool (selected by collector reward on the search split):

DatasetCollection Size KK (M=KM = K)Best Mask in KK-Pool (Test Score)RGFS Test Score (Mean ± Std)
OpenML_6165020.6348.20 ± 0.72
10021.0849.98 ± 2.29
15024.8248.94 ± 5.86
301 (Full)21.6548.50 ± 0.00
German Credit5067.3467.53 ± 0.96
10065.5166.30 ± 1.01
15065.5467.21 ± 1.95
301 (Full)66.0864.79 ± 1.93

Even at K=M=50K=M=50, where the best collected mask on OpenML_616 scores 20.63, RGFS reaches 48.20 ± 0.72, matching the full-pool model (48.50 ± 0.00). On German Credit, where the collected pool is already near 65.5–67.3, RGFS remains stable across KK (64.79–67.53, matching the pool baseline within 1 std at this restricted budget M≤301M \le 301 and rising to 75.36 under the standard inference budget).

Why collecting more masks does not remove the need for diffusion. Expanding the MARLFS pool 6$\times$ (K=50→301K=50 \to 301) yields little improvement in the best collected mask (20.63–24.82 on OpenML_616; 65.51–67.34 on German Credit), as generating each candidate requires an online multi-agent RL step that tends to stall on hard subsets. By contrast, once RGFS learns a reward-aligned latent space from 50–300 offline samples, reward-guided reverse diffusion explores novel combinations beyond the collected pool; disabling reward guidance (γ=0\gamma=0) lowers performance across all four evaluated datasets in Figure 2 (by up to 18.80 points on OpenML_616).


References

  • Granitto, P. M., Furlanello, C., Biasioli, F., & Gasperi, F. (2006). Recursive feature elimination with random forest for PTR-MS analysis of agroindustrial products. Chemometrics and Intelligent Laboratory Systems, 83(2), 83–90.
  • Liu, K., Fu, Y., Wang, P., Wu, L., Bo, R., & Li, X. (2019). Automating feature subspace exploration via multi-agent reinforcement learning. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD), pp. 207–215.
  • Peng, H., Long, F., & Ding, C. (2005). Feature selection based on mutual information criteria of max-dependency, max-relevance, and min-redundancy. IEEE Transactions on Pattern Analysis and Machine Intelligence, 27(8), 1226–1238.
  • Tibshirani, R. (1996). Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series B (Methodological), 58(1), 267–288.
  • Ying, W., Wang, D., Chen, H., & Fu, Y. (2024). Feature selection as deep sequential generative learning. ACM Transactions on Knowledge Discovery from Data, 18(9), 1–21.

Dear Reviewer,

The rebuttal and discussion phase is now in its final stages, with very limited time remaining for further exchanges. We would greatly appreciate your prompt consideration of the authors’ responses and any additional comments you may have before the discussion closes.

Many thanks for your valuable contribution.

The newly tuned baselines reveal and partially correct a serious weakness in the original evaluation: RFE and mRMR were evaluated at predetermined feature cardinalities, whereas RGFS was free to optimize subset cardinality. Given that the paper considers unconstrained predictive-performance maximization rather than fixed-budget feature selection, this appears to place the baselines at a substantial disadvantage. While the new experiments improve the fairness of the evaluation, they also substantially reduce the reported advantage over RFE, with tuned RFE even outperforming RGFS on one dataset. I therefore see these results primarily as correcting an issue with the original evaluation, rather than strengthening the empirical case for RGFS.

The collection-size experiments also do not fully address my original question. The authors only evaluate smaller subsets (50, 100, 150) of their existing 301-mask collection, rather than expanding the collection beyond 301 masks. This demonstrates that RGFS can perform well with fewer training masks, but does not establish that further MARLFS exploration would fail to improve performance or that additional data collection could not reduce the need for diffusion. The claim that MARLFS exploration tends to stall is therefore not supported by the reported experiment.

Nevertheless, I still find the proposed approach conceptually interesting. Reward-guided diffusion in a learned latent space, followed by decoding into feature-selection masks, is an interesting alternative to conventional feature-selection methods. The additional experiments suggest that RGFS remains competitive with properly tuned baselines, although its performance advantage is substantially smaller than originally reported. Given these considerations, I will maintain my original weak accept score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal for Area Chairs

Dear Area Chairs,

We thank the reviewers for their constructive feedback. Our paper introduces RGFS / RGFS+, a reward-guided latent diffusion framework for feature selection with parallel decoding and graph-sparse attention. It (1) synthesizes unseen, higher-performing subsets beyond upstream search heuristics, (2) selects compact subsets without pre-specifying or tuning subset size kk, and (3) scales to ultra-high dimensions (N≈60kN \approx 60\text{k} features) on a single GPU where dense attention runs out of memory.

To address every reviewer concern (5YHj Q1–Q5, Pn2n Q1–Q3, VSFD Q1–Q3), we ran five sets of new experiments (detailed in our individual responses). Below are the key questions and findings:

1. Do the gains and compactness hold under strict multi-seed CV against tuned classical selectors? (5YHj Q1, Pn2n Q1)

Under a strict 5-seed protocol (all tuning/selection via 5-fold CV in the 80% search split; 20% test evaluated once), RGFS outperforms the best search-collected mask (+0.18 to +10.40 points), Tuned LASSO (+0.65 to +13.91), and Tuned mRMR (+0.13 to +7.81) on 4/4 diagnostic benchmarks, and beats Tuned RFE on 3/4. Without tuning kk, RGFS selects fewer features (#Sel) than Tuned LASSO on 4/4 datasets (4.4–38.4 vs. 11.6–50.8). We also resolved the OpenML_607 Table 1 vs. Table 3 discrepancy (an unaligned 8-layer script in Table 3 and a misrecorded dense/sparse flag in Table 1). Under the unified setup, RGFS (65.46 ± 0.76) and RGFS+ (59.42 ± 2.97) both surpass Best-Collected (55.06), LASSO (51.55), and mRMR (57.65). This clarifies a regime boundary: dense RGFS excels on statistically independent features (OpenML_607 Friedman data, where weak marginal correlations omit graph edges), whereas sparse RGFS+ targets high-dimensional correlated features. We are completing 5-seed runs for all 14 benchmarks for the revision.

2. Does diffusion generate unseen masks (without circularity) and win at a matched budget? (5YHj Q2–Q3, VSFD Q1.3)

The reward predictor EψE_{\psi} guides denoising only; final masks are chosen by true 5-fold search-CV (no circularity or test leakage). Across 20 runs (4 datasets ×\times 5 seeds), 100% (20/20) of selected final masks and 86.6%–100% of generated candidates are entirely novel (absent from the 301-mask MARLFS training pool). To test whether generating M≈15.1kM \approx 15.1\text{k} candidates gives an unfair budget advantage, we restricted RGFS to its first B=300B=300 candidates (matching the 301-mask pool, $\approx$5 batches): it already beats Best-Collected on 2/4 datasets (within 0.14–0.63 elsewhere) and wins on all 4/4 datasets by B=1kB=1\text{k} ($\approx$16 batches).

3. Why use reward-guided diffusion instead of running upstream search longer? (VSFD Q1.2–Q1.3, Pn2n Q3)

Our pool-size (KK) study contrasts search stagnation with generative extrapolation. On OpenML_616, expanding the upstream MARLFS pool 6$\times$ (K=50→301K=50 \to 301) barely moves Best-Collected (20.63 →\to 21.65). In contrast, training RGFS from scratch on K=50K=50 masks and sampling M=50M=50 candidates (100 total evaluations, 3$\times$ fewer than the 301-mask baseline) reaches 48.20 ± 0.72, matching our full-pool model (48.50). Removing reward guidance (γ=0\gamma=0) drops scores by up to 18.80 points across ablation datasets, showing that reward-guided denoising extrapolates into higher-reward regions that heuristic search misses.

4. How does RGFS+ compare to modern deep selectors at N≈60kN \approx 60\text{k}? (5YHj Q5, VSFD Q3)

Evaluated against modern neural baselines (STG, CAE) and LASSO using official implementations (tuning k∈50,…,3000k \in \\{50, \dots, 3000\\} via search-CV over 3 seeds), RGFS+ matches the strongest baseline on OpenML_1082 (N=59kN=59\text{k}: 73.3%, tying STG at 73.3% and beating CAE at 71.7% and LASSO at 68.3%) and achieves the highest accuracy on OpenML_1085 (N=61kN=61\text{k}: 69.5% vs. 65.1% CAE, 56.2% LASSO, and 54.6% STG), without tuning kk.

5. What are the complexity, speedup, and memory profile of the graph-sparse decoder? (5YHj Q4, VSFD Q1.1/Q2/Q3, Pn2n Q2)

We restricted the main-text O(Nk)\mathcal{O}(Nk) complexity claim strictly to the encoder and denoiser and clarified Table 2's hardware settings. In controlled decoder-only benchmarks on an A100 (40 GB), our graph-sparse attention decoder yields an 18.2$\times$ speedup at N=20kN=20\text{k} (4.052 s →\to 0.223 s) and runs at N=60kN=60\text{k} in 0.713 s (34.46 GB) where the dense decoder runs out of memory (>40 GB). To clarify positioning against concurrent diffusion selectors, we added a comparison table contrasting VTFS, DIFFT (Gong et al., 2025), and RGFS(+), and unified all notation and terminology.

We thank the Area Chairs and reviewers for their time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.