- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies end-to-end differentiable feature selection in deep reinforcement learning. The authors introduce learnable gates on raw observations and compare four existing gating mechanisms, including Hard Concrete (HC), Stochastic Gates (STG), Concrete Dropout (CD), and Interpretable Feature Extractor (IFE). The experiments are conducted on a commercial racing game with 4,244-dimensional observations. The authors find that HC performs best among the evaluated methods, achieving a 9.1% reduction in completion time when retaining approximately 30% of the input features. They also introduce GateKL, a metric that measures how much the policy's action distribution changes when an individual feature gate is toggled. Based on this metric, they show that HC and STG have better alignment between learned gate values and policy sensitivity.
Critical strengths
-
Reasonably extensive experiments within the chosen environment. The authors evaluate four gating mechanisms across twelve racing maps, multiple seeds, and different feature budgets. The performance differences between HC and the other methods are interesting.
-
An additional perspective on learned feature gates. GateKL provides a way to analyze whether learned gate values reflect the sensitivity of the trained policy to individual input features. This is a useful diagnostic beyond simply counting the number of selected features.
Critical weaknesses
-
The motivation for explicit feature selection is not sufficiently established. The paper does not explain why an additional feature selection mechanism is necessary for a deep RL policy. Neural networks can already learn to suppress irrelevant input features through their internal representations. While explicit feature selection may offer advantages in sample efficiency, generalization, or computational cost, none of these benefits is convincingly established in the current experiments. In addition, the authors emphasize lightweight RL agents, but removing raw observation dimensions does not necessarily translate into meaningful computational savings, particularly when the remaining network architecture is unchanged.
-
The methodological novelty is limited, and several important baselines are missing. The main approach consists of inserting existing differentiable gating mechanisms before the policy encoder and optimizing them jointly with the PPO objective. The four gating mechanisms are largely adopted from prior work, while the RL-specific policy-data feedback is primarily discussed conceptually rather than addressed through a new optimization method. More importantly, the experimental comparisons are restricted to different gating mechanisms. Missing baselines include random feature selection at matched sparsity, input dropout, conventional regularization, and input-layer pruning. Without these comparisons, it is difficult to determine whether the observed improvements are attributable to meaningful feature selection or simply to regularization and reduced input dimensionality.
-
The reported performance improvement does not clearly establish the benefit of end-to-end feature selection. The headline 9.1% improvement in Figure 4 is obtained by extracting feature subsets from intermediate checkpoints and retraining a fresh policy using those features. This is different from directly evaluating the end-to-end trained gated policy. In fact, Table 3 reports that the converged HC model achieves performance close to, but slightly worse than, the no-gating baseline (34.5 versus 34.3 seconds). Although these experiments use different evaluation protocols, the distinction is important and should be made much clearer. I would like to see a more direct evaluation of whether jointly learned gates improve the final policy, without relying on an additional retraining stage.
-
GateKL measures policy sensitivity rather than actual feature importance. I am not fully convinced by the interpretation of GateKL. The proposed metric measures the change in the action distribution when a gate is toggled, but a large change does not necessarily mean that the corresponding feature is beneficial or necessary for achieving high returns. A policy could rely heavily on a spurious or even harmful feature.
-
The RL-specific claims are not adequately supported by the experiments. The paper repeatedly emphasizes that feature gates influence the policy, which in turn changes the state distribution used to update the gates. This is a reasonable distinction from supervised learning, but the experiments do not directly isolate or investigate this feedback mechanism.
Questions for authors
-
What is the main advantage of explicit feature selection over implicit feature learning in the policy encoder? Could the authors compare against random feature selection, input dropout, and standard regularization under matched feature budgets?
-
Could the authors clarify the difference between the results in Figure 4 and Table 3? In particular, what performance improvement can be achieved by the end-to-end trained policy without retraining on the selected features?
-
To what extent is GateKL alignment influenced by the magnitude of the learned gates themselves? Could the authors validate feature importance using return-based ablation or another metric less directly coupled to gate values?
-
Can the authors provide experiments that directly investigate policy-data feedback, rather than merely demonstrating performance differences between existing gating mechanisms?
2 comments
Thanks for the authors' responses. However, I found it's hard to follow the proposed evidence logics, and how they support the paper's claims. Meanwhile, several core concerns in the listed weaknesses like limited novelty, insufficient baselines, and limited end-to-end performance gain are neglected. Considering these concerns are not resolved and this work is not evaluated on common RL benchmarks, I would prefer to keep my score.
The backbone architecture is unchanged, so a drop from 4,244 to 1,270 does not by itself imply faster inference. What feature removal actually affects is the input encoding path, and we counted it layer by layer. Take HC: after compaction, encoders that are fully closed are removed whole, and partially closed ones lose the first-layer columns of their closed dimensions. Layer by layer:
Units are MFLOPs per frame, computed at the converged configuration of Table 3 rather than the k=1270 sweep point; STG uses 756 dimensions and HC is scaled to 664 by the same closure ratios; rows are rounded.
The temporal core and policy heads are unchanged and are not included above, so 74.5% and 74.9% cover the feature path only, not the whole network. End-to-end latency depends on the inference stack and batching.
The difference between Figure 4 and Table 3 is in common response Q4. To sum up, each answers a different question: Figure 4 takes a feature subset, retrains from scratch on it, and asks whether the subset is usable; at k=1270 all twelve maps beat the full-feature reference, 34.0 s against 37.4 s on average; Table 3 trains gates and policy jointly to convergence and evaluates directly, without retraining.
It is affected, and this follows from the definition: toggling a gate removes
z_u·x_u, so a larger gate removes more input. We do not treat GateKL as feature importance independent of the gate configuration.It is defined this way because it answers the deployment question: under this trained gating, what happens to the policy if a channel is closed? At deployment the policy sees
z_u·x_u. A gate value is a judgment learned by the end-to-end gating itself; a largezmeans the mechanism thinks the feature should stay. GateKL measures how the policy changes after the toggle. Whether the two agree can be compared directly: what the mechanism calls important should change the policy when toggled, and what it calls closed should not respond. Agreement means the mechanism learned well, disagreement means it learned poorly; under the same setting the four mechanisms' AUC ranges from 0.96 to 0.59, and that is the difference.Validating feature importance with task return is another question: return-based ablation removes a feature and measures how much the episode return drops, a line that does not pass through GateKL. We did not run this experiment.
Gates and policy are not trained separately: one objective, gradients update policy parameters and gate parameters together, and the data comes from the current gated policy. Appendix E writes this loop as three relations: policy parameters are a function of gate history, trajectories are sampled from the current policy, and gate gradients are estimated on those trajectories.
In supervised learning the design matrix has nothing to do with the gates; in RL the gate decides what the policy sees, the policy decides which states are collected, and those states then decide the gate gradient. The gate changes the data, and the data change the gate. The consequence is that once a gate closes, the incentive in that direction shrinks and nothing pushes it open again: hard closure locks easily, soft closure locks less.