acceptodds
Under review as a conference paper at ICLR 2027

End-to-End Differentiable Feature Selection for Deep Reinforcement Learning

Abstract

Lightweight reinforcement learning (RL) agents must remove unnecessary raw observation features without sacrificing control performance. Differentiable feature gates are a natural approach, but their standard theory assumes a fixed dataset, as in supervised learning. In RL that assumption breaks down: the gate reshapes the policy, the policy reshapes its trajectories, and those trajectories drive gate updates. We systematically study end-to-end differentiable feature selection under this policy-dependent feedback, comparing four gate mechanisms on a live commercial racing game with a -dimensional observation across two key design choices: stochastic versus deterministic gating, and expected- versus KL-to-prior regularization. Sweeping the number of selected features, the best mechanism, Hard Concrete, removes of the inputs while cutting completion time across twelve maps from to seconds, a improvement. Gate masks, however, reveal which features are selected, not how strongly the trained policy relies on them. We introduce GateKL, which toggles one gate at inference—closing an open gate or opening a closed one while keeping the policy parameters and all other learned gates fixed—and measures the resulting change in the action distribution using KL divergence. Across all gates and twelve maps, gate values learned by Hard Concrete and Stochastic Gates closely reflect policy reliance, whereas those learned by Concrete Dropout and Interpretable Feature Extractor carry little such information. These findings distinguish sparse selection from policy reliance and show why gate masks alone are insufficient to assess learned feature selection.

Primary area:
reinforcement learning
Keywords:
differentiable feature selection, deep reinforcement learning

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_HYQMOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies end-to-end differentiable feature selection in deep reinforcement learning. The authors introduce learnable gates on raw observations and compare four existing gating mechanisms, including Hard Concrete (HC), Stochastic Gates (STG), Concrete Dropout (CD), and Interpretable Feature Extractor (IFE). The experiments are conducted on a commercial racing game with 4,244-dimensional observations. The authors find that HC performs best among the evaluated methods, achieving a 9.1% reduction in completion time when retaining approximately 30% of the input features. They also introduce GateKL, a metric that measures how much the policy's action distribution changes when an individual feature gate is toggled. Based on this metric, they show that HC and STG have better alignment between learned gate values and policy sensitivity.

Critical strengths
  • Reasonably extensive experiments within the chosen environment. The authors evaluate four gating mechanisms across twelve racing maps, multiple seeds, and different feature budgets. The performance differences between HC and the other methods are interesting.

  • An additional perspective on learned feature gates. GateKL provides a way to analyze whether learned gate values reflect the sensitivity of the trained policy to individual input features. This is a useful diagnostic beyond simply counting the number of selected features.

Critical weaknesses
  • The motivation for explicit feature selection is not sufficiently established. The paper does not explain why an additional feature selection mechanism is necessary for a deep RL policy. Neural networks can already learn to suppress irrelevant input features through their internal representations. While explicit feature selection may offer advantages in sample efficiency, generalization, or computational cost, none of these benefits is convincingly established in the current experiments. In addition, the authors emphasize lightweight RL agents, but removing raw observation dimensions does not necessarily translate into meaningful computational savings, particularly when the remaining network architecture is unchanged.

  • The methodological novelty is limited, and several important baselines are missing. The main approach consists of inserting existing differentiable gating mechanisms before the policy encoder and optimizing them jointly with the PPO objective. The four gating mechanisms are largely adopted from prior work, while the RL-specific policy-data feedback is primarily discussed conceptually rather than addressed through a new optimization method. More importantly, the experimental comparisons are restricted to different gating mechanisms. Missing baselines include random feature selection at matched sparsity, input dropout, conventional regularization, and input-layer pruning. Without these comparisons, it is difficult to determine whether the observed improvements are attributable to meaningful feature selection or simply to regularization and reduced input dimensionality.

  • The reported performance improvement does not clearly establish the benefit of end-to-end feature selection. The headline 9.1% improvement in Figure 4 is obtained by extracting feature subsets from intermediate checkpoints and retraining a fresh policy using those features. This is different from directly evaluating the end-to-end trained gated policy. In fact, Table 3 reports that the converged HC model achieves performance close to, but slightly worse than, the no-gating baseline (34.5 versus 34.3 seconds). Although these experiments use different evaluation protocols, the distinction is important and should be made much clearer. I would like to see a more direct evaluation of whether jointly learned gates improve the final policy, without relying on an additional retraining stage.

  • GateKL measures policy sensitivity rather than actual feature importance. I am not fully convinced by the interpretation of GateKL. The proposed metric measures the change in the action distribution when a gate is toggled, but a large change does not necessarily mean that the corresponding feature is beneficial or necessary for achieving high returns. A policy could rely heavily on a spurious or even harmful feature.

  • The RL-specific claims are not adequately supported by the experiments. The paper repeatedly emphasizes that feature gates influence the policy, which in turn changes the state distribution used to update the gates. This is a reasonable distinction from supervised learning, but the experiments do not directly isolate or investigate this feedback mechanism.

Questions for authors
  1. What is the main advantage of explicit feature selection over implicit feature learning in the policy encoder? Could the authors compare against random feature selection, input dropout, and standard regularization under matched feature budgets?

  2. Could the authors clarify the difference between the results in Figure 4 and Table 3? In particular, what performance improvement can be achieved by the end-to-end trained policy without retraining on the selected features?

  3. To what extent is GateKL alignment influenced by the magnitude of the learned gates themselves? Could the authors validate feature importance using return-based ablation or another metric less directly coupled to gate values?

  4. Can the authors provide experiments that directly investigate policy-data feedback, rather than merely demonstrating performance differences between existing gating mechanisms?

2 comments

Q1: What is the advantage of explicit selection over implicit learning, and do the computational savings materialize?

The backbone architecture is unchanged, so a drop from 4,244 to 1,270 does not by itself imply faster inference. What feature removal actually affects is the input encoding path, and we counted it layer by layer. Take HC: after compaction, encoders that are fully closed are removed whole, and partially closed ones lose the first-layer columns of their closed dimensions. Layer by layer:

PathwayDimsOpenNo gateSTG (756)HC (664)
Self state5009.3%0.180.060.06
Surrounding characters72032.2%0.630.500.49
Interactable objects50067.3%0.620.580.57
NPCs5000%0.6200
Scene props30027.0%0.200.170.17
Task goals5009.1%0.620.160.16
Task metadata2007.3%0.100.050.05
Visual block (32×32 raycast)1,0240%3.0300
Total4,2445.991.531.50
Reduction−74.5%−74.9%

Units are MFLOPs per frame, computed at the converged configuration of Table 3 rather than the k=1270 sweep point; STG uses 756 dimensions and HC is scaled to 664 by the same closure ratios; rows are rounded.

The temporal core and policy heads are unchanged and are not included above, so 74.5% and 74.9% cover the feature path only, not the whole network. End-to-end latency depends on the inference stack and batching.

Q2: Could the authors clarify the difference between the results in Figure 4 and Table 3?

The difference between Figure 4 and Table 3 is in common response Q4. To sum up, each answers a different question: Figure 4 takes a feature subset, retrains from scratch on it, and asks whether the subset is usable; at k=1270 all twelve maps beat the full-feature reference, 34.0 s against 37.4 s on average; Table 3 trains gates and policy jointly to convergence and evaluates directly, without retraining.

Q3: Is GateKL confounded by gate magnitude, and can return-based ablation validate it?

It is affected, and this follows from the definition: toggling a gate removes z_u·x_u, so a larger gate removes more input. We do not treat GateKL as feature importance independent of the gate configuration.

It is defined this way because it answers the deployment question: under this trained gating, what happens to the policy if a channel is closed? At deployment the policy sees z_u·x_u. A gate value is a judgment learned by the end-to-end gating itself; a large z means the mechanism thinks the feature should stay. GateKL measures how the policy changes after the toggle. Whether the two agree can be compared directly: what the mechanism calls important should change the policy when toggled, and what it calls closed should not respond. Agreement means the mechanism learned well, disagreement means it learned poorly; under the same setting the four mechanisms' AUC ranges from 0.96 to 0.59, and that is the difference.

Validating feature importance with task return is another question: return-based ablation removes a feature and measures how much the episode return drops, a line that does not pass through GateKL. We did not run this experiment.

Q4: Can policy-data feedback be tested directly?

Gates and policy are not trained separately: one objective, gradients update policy parameters and gate parameters together, and the data comes from the current gated policy. Appendix E writes this loop as three relations: policy parameters are a function of gate history, trajectories are sampled from the current policy, and gate gradients are estimated on those trajectories.

In supervised learning the design matrix has nothing to do with the gates; in RL the gate decides what the policy sees, the policy decides which states are collected, and those states then decide the gate gradient. The gate changes the data, and the data change the gate. The consequence is that once a gate closes, the incentive in that direction shrinks and nothing pushes it open again: hard closure locks easily, soft closure locks less.

Thanks for the authors' responses. However, I found it's hard to follow the proposed evidence logics, and how they support the paper's claims. Meanwhile, several core concerns in the listed weaknesses like limited novelty, insufficient baselines, and limited end-to-end performance gain are neglected. Considering these concerns are not resolved and this work is not evaluated on common RL benchmarks, I would prefer to keep my score.

Reviewer_PzfQOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies end-to-end differentiable feature selection in deep RL by applying four existing gating methods (STG, HC, CD, and IFE) to RL. The authors introduce GateKL, which measures changes in the policy's action distribution when individual feature gates are toggled. The experiments show that HC achieves the highest alignment with GateKL among the four methods.

Critical strengths

The experiments compare multiple gating methods under the same RL training framework and examine both feature reduction and control performance.

Critical weaknesses
  1. The methodological novelty is limited. The paper applies existing differentiable feature selection methods to deep RL and compares their performance, rather than proposing a new feature selection algorithm. The main addition is GateKL, an inference-time measure of changes in the policy's action distribution when individual gates are toggled.
  2. The usefulness of GateKL is not demonstrated. The reported AUC only measures how well gate values predict changes in the policy's action distribution. It does not show whether the selected features are useful for the task or whether higher alignment leads to better control performance. GateKL is also not used to improve feature selection or policy training, leaving its practical role unclear.
Questions for authors
  1. What is the main contribution beyond applying existing differentiable feature selection methods to deep RL?
  2. Does GateKL alignment correlate with feature selection quality or task performance? How can GateKL be used in practice?
  3. The 9.1% improvement in Figure 4 is obtained by retraining policies on selected feature subsets, while Table 3 reports no improvement over the full-feature baseline for the jointly trained HC policy. How should these two results be interpreted in assessing the effectiveness of end-to-end feature selection?
2 comments

Q1: What is the main contribution beyond applying existing differentiable feature selection methods to deep RL?

There are two contributions. First, the four differentiable gate mechanisms are placed inside an RL policy and compared under one PPO objective and one sparsity budget; this had not been done systematically. Second, we give a measurable criterion, GateKL with AUC, that separates how much is selected from what the policy still relies on. The first is empirical work and the second a diagnostic tool; we do not claim a new gating algorithm.

Q2: Does GateKL alignment correlate with feature selection quality or task performance? How can GateKL be used in practice?

Please refer to common response Q3.

Q3: How should the 9.1% in Figure 4 and the Table 3 results be interpreted together?

The difference between Figure 4 and Table 3 is in common response Q4. To sum up, each answers a different question: Figure 4 takes a feature subset, retrains from scratch on it, and asks whether the subset is usable; at k=1270 all twelve maps beat the full-feature reference, 34.0 s against 37.4 s on average; Table 3 trains gates and policy jointly to convergence and evaluates directly, without retraining.

Thank you for the response. The distinction between the retraining results in Figure 4 and the joint training results in Table 3 is now explicit. However, my question concerned the benefit of joint training itself. The correction identifies the 34.3 s baseline as the configuration with 3220 inputs, but its implications for the comparisons and conclusions in Table 3 remain unresolved.

The response on GateKL does not fully address my concern. The agreement between the AUC and completion time rankings across four methods suggests an association. However, it remains unclear whether higher GateKL identifies features that are more important for task performance, and how the metric would inform feature selection in practice.

Given these remaining concerns, I will maintain my overall rating.

Reviewer_bRw5OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies differentiable feature gates for RL, which learn a structured representation from images. The authors consider four different gate samplers (STG, HC, CD, IFE) and propose GateKL, a mechanism to determine which gates to toggle at test time. They compare these different gate samplers in a racing game environment and perform several analyses to understand their effect.

Critical strengths
  • The paper contains relatively extensive analyses of different feature learning methods.
Critical weaknesses
  • The paper is poorly written (unfortunately, a large portion of the text seems AI-generated to me). Due to the subpar quality of the writing, it was generally difficult to read and understand the paper. For example, the authors say "We study end-to-end differentiable feature selection inside a deep RL policy, a problem that has not been systematically studied before." in the Introduction, but it's unclear to me why this is an important problem. Why do we want to have a separate feature gate learner? From the main results, the gain from this additional mechanism seems marginal as well.
  • The authors say that "a systematic study of end-to-end differentiable feature selection in deep RL" is one of the main contributions. However, the experiments are done only on one car racing game domain. This makes it unclear how the results in the paper transfer to other RL tasks. It's also unclear how the feature learning method considered in this work (as well as GateKL) compares to previous, more standard RL representation learning approaches.

In general, while (an improved version of) this paper might possibly be an interesting workshop paper, I don't think it meets the bar of ICLR in terms of both contributions and results.

Questions for authors

I don't have outstanding questions.

2 comments

W: Why is a separate feature gate learner needed, and what is new compared with standard RL representation learning approaches?

First of all, there is no separate feature gate learner. A gate is a per-dimension scalar multiplier in the policy forward pass; it shares one PPO objective and one set of on-policy data with the policy, and there is no second optimizer. Reading this as "an extra feature gate learner outside the policy" makes the rest of the judgment go wrong.

In SL this approach is established and widely used: the gate trains end-to-end with the main model, and sparsity pressure closes most inputs without losing accuracy. In RL this is blank. The difference is the data: SL has a fixed validation set, so the selection criterion can be built on the same data; on-policy RL generates data from the current policy, and selecting features with a frozen policy amounts to selecting on the distribution the policy is about to leave, so the gates must stay in the training loop.

What this paper does is compare the design space at this position systematically: one PPO objective, one sparsity budget, four gate mechanisms side by side, plus GateKL as a criterion for whether gate values match what the policy actually relies on.

Only two works studied the input pruning in RL: Grooten et al. [1] use a discrete prune-and-regrow binary mask for input-layer filtering, and the mask is not differentiable; Pham and Cangelosi's [2] gate targets interpretability of vision policies, a single method with no comparison across mechanisms. The rest either build on linear or fixed feature bases or use RL to select features for an external supervised model.

These points are answered in the common response:

  • the contribution → Common-1 Q1;
  • the separate feature gate learner and its motivation → Common-1 Q2;
  • the gain seems marginal → Common-1 Q4;
  • the single car racing game domain → Common-2 Q5;
  • comparison with standard RL representation learning approaches → Common-2 Q6.

References

[1] B. Grooten, et al. Automatic Noise Filtering with Dynamic Sparse Training in Deep Reinforcement Learning. AAMAS 2023. arXiv:2302.06548.

[2] T. Pham, A. Cangelosi. Pay Attention to What and Where? Interpretable Feature Extractor in Vision-based Deep Reinforcement Learning. arXiv:2504.10071, 2025.

Thanks for the response. While I appreciate the authors' clarification, it's still unclear to me why we need explicit feature selection (related to Q2), especially given the seemingly marginal performance/compute gain on a single domain. If the main benefit comes from reduced computation, I would have expected comparisons with other inference optimization approaches (e.g., baselines using quantization, pruning, etc.). My concerns regarding the contributions, writing quality, and results (comparisons) remain mostly unaddressed even after reading both the individual and common responses, so I'd like to maintain my original rating this time.

Reviewer_fPM4OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper provides a systematic study of end-to-end differentiable feature selection in deep reinforcement learning. Importantly, the authors formulate feature selection and policy learning as a joint optimization problem. Namely, the RL agent learns to perform the task while simultaneously identifying which raw observation features can be removed. Their results are promising, substantial feature reduction can be achieved while maintaining or improving control performance.

Critical strengths
  • The authors explicitly position the problem as raw observation selection jointly trained with a nonlinear RL policy.
  • The evaluations are quite substantial in terms of computational scale and the number of evaluation runs, although they are limited to a single racing environment across 12 maps.
Critical weaknesses
  • The empirical evaluations mostly focus on four differentiable gating mechanisms. Could the authors compare against other feature-selection or input-pruning baselines to better establish the advantage of the proposed end-to-end approach?
Questions for authors
  1. What exactly are the 4,244 input dimensions in the observation space?
  2. To what extent do you expect the method to generalize beyond the racing environment, given that the experiments are limited to 12 maps
1 comment

Q1: What exactly are the 4,244 observation dimensions?

The composition of the 4,244 dimensions:

TypeTotalSelfother playersobjectsNPCsScene propsTask goalsTask metasVisual
4,2445009×8010×5010×5010×3010×502001,024

*The visual part is a 32×32 single-channel depth map.

Each scalar dimension has its own gate, not one gate per entity, because dimensions within the same entity differ in how removable they are and entity-level granularity cannot express that difference.

Q2: How far does the method generalize beyond the racing environment?

This environment is a live commercial game, not a benchmark built for the paper. It satisfies three conditions that make the check meaningful: the observation is high-dimensional; there is redundancy to remove, as the visual block being closed by both mechanisms shows; and completion time is an objective continuous indicator. Our claims cover this kind of high-dimensional, entity-structured raw observation with continuous control, not all of RL.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Common response (1/2): contribution, motivation, GateKL and protocols

Q1: The paper applies four existing gates to RL; is the contribution sufficient, and is the "not systematically studied" claim justified? (bkWS W2 / PfZQ W1 / HYOM W2)

Differentiable gates for feature selection are established in supervised learning: STG, HC and CD train end-to-end with the model, and sparsity pressure closes most inputs without losing accuracy. That is closed in SL.

RL has no corresponding systematic study. The difference is the data: in SL, closing a gate leaves the remaining data and the evaluation criterion unchanged; in RL the gate changes what the policy sees, the policy changes the states it collects, and those states then determine the gate gradient. That feedback does not exist in SL, that comparison had not been done, and this paper fills that gap.

Two things specifically. First, the four gate mechanisms are placed inside one RL policy and compared under one PPO objective and one sparsity budget, looking at their closure patterns, completion time and GateKL response. Second, we give a measurable criterion, GateKL with AUC, that asks whether learned gate values rank gates by how much toggling them changes the action distribution. The first is empirical work and the second a diagnostic tool; this paper does not claim a new gating algorithm.

Q2: Neural networks can already suppress irrelevant inputs internally; why is explicit selection on the raw observation needed? (bkWS W1 / HYOM W1)

Explicit feature selection is, by design, pruning: inputs that are not used are removed from the observation, so training samples and transmits less and deployment computes less. Under the current gate distribution, the input path in online deployment is estimated to need about 75% less compute (74.5% for STG, 74.9% for HC). The process is lossless: on the twelve maps, retaining about thirty percent of the input dimensions brings completion time from 37.4 s to 34.0 s, and bot capability is not sacrificed.

Online game bots are especially sensitive to this. The user base is large and many bots run at the same time, so per-inference compute multiplied by call volume is machine cost. Cutting the input path to about a quarter lowers this part of the machine cost by the same proportion in large-scale deployment.

The approach is pruning, not multiplying inputs by zero and computing as before. Once the unused raw features are removed, they are no longer sampled, transmitted or computed. By contrast, an internal representation can suppress irrelevant inputs, but the interface is unchanged: all 4,244 dimensions are still sampled, transmitted and fed into the first layer. Pruning changes exactly this interface.

Q3: GateKL only measures action-distribution changes when a gate is toggled; does it show that features are useful for the task, or that better alignment yields better control? (PfZQ W2 / HYOM W4)

GateKL is a diagnostic. AUC answers only one question: whether gate values can rank gates by how much toggling them changes the action distribution. It does not answer whether a feature is necessary for the task, nor which mechanism is faster; Section 4.4 says this. It is not a training objective, and it is not claimed to improve training.

That does not mean it is unrelated to the task. Gates are driven by the PPO objective, and the same reward updates both the policy and the gates; a feature is kept because closing it under the current policy would make the objective worse. Three mutually independent indicators give the same ordering. AUC in Figure 6: HC 0.96, STG 0.85, CD 0.69, IFE 0.59. Training return over six maps in Figure 5: HC highest, STG next. Converged completion time in Table 3: 34.5, 34.9, 38.3, 38.8 s. Counting retained features alone does not show this difference.

Q4: The 9.1% in Figure 4 comes from retraining on selected feature subsets, while the jointly trained policy in Table 3 is on par with the baseline; how should the two be interpreted? (PfZQ Q3 / bkWS)

The two sets of numbers come from two protocols and answer two different questions.

Figure 4 sweeps the number of retained features k. When the gates retain about k features, that feature set is taken and a fresh policy is retrained from scratch for 24 hours on it. At k=1270, where 29.9% of features are retained, the twelve-map average falls from 37.4 s to 34.0 s, 9.1% less, and all twelve maps are faster. Each k is an independent retraining; adjacent points are not consecutive states of one trajectory.

Table 3 is joint training: gates and policy are trained together to convergence and evaluated directly, without retraining. HC averages 34.5±0.29 s over the twelve maps. The 34.3±0.30 s baseline entry in the appendix table is in fact the 3,220-dimension configuration without the 32×32 visual block; it is a typo and will be updated.

So the 9.1% belongs to the retraining protocol of Figure 4 and cannot be read directly as an end-to-end gain of the jointly trained policy.

AuthorsOpenReview ↗

Common response (2/2): contribution, motivation, GateKL and protocols

Q5: The evaluation covers one racing environment and twelve maps; to what extent do the conclusions transfer to other RL tasks?

There is one environment, this is not a benchmark built for the paper but a live commercial game, with a 4,244-dimensional observation, twelve maps and three seeds per map, and the policy is trained and evaluated under real deployment constraints. In this environment, after removing 70.1% of the input dimensions, the policy retrained on the selected features is faster on all twelve maps, from 37.4 s to 34.0 s on average.

This does not mean the conclusions transfer to other tasks. The current claims cover only this kind of high-dimensional raw observation and the corresponding control tasks.

Q6: Missing baselines (random matched-sparsity selection, input dropout, standard regularization, input-layer pruning) leave it unclear whether the gains come from feature selection or from regularization and dimensionality reduction.

All four reviewers mention baselines. The comparisons in the paper are the full-feature reference and the architecture-level ablation of the visual block. The random-mask comparison separates "the subset the gate picks" from "any subset of the same size"; the design is to keep the same number of raw features uniformly at random, retrain the policy with the same procedure, and compare completion time. Thank you for your suggestion; we will take it into consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.