- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies non-myopic active feature acquisition (AFA), where a policy sequentially selects costly features and decides when to stop and predict. The authors propose NM-PPG, which uses Gumbel-Softmax relaxation and straight-through estimation to enable pathwise gradients through complete feature-acquisition trajectories. The method further introduces survival-weighted stopping to support differentiable optimization of acquisition decisions and stopping behavior.
The paper provides theoretical results on relaxation consistency, gradient computation complexity, and an AFA-specific upper bound on gradient variance, which motivates staged temperature sharpening. Experiments on 12 synthetic and real-world datasets compare NM-PPG against seven baselines under both hard- and soft-budget settings. The proposed method achieves the highest overall average cost-performance AUC in both settings, although its advantage over the strongest non-myopic RL baseline is relatively modest.
Critical strengths
-
Meaningful methodological contribution. The paper presents a reasonable approach to non-myopic AFA by enabling pathwise gradients through full acquisition trajectories, including differentiable stopping decisions.
-
Solid theoretical and empirical evaluation. The paper provides theoretical analyses of gradient variance and computational complexity, along with extensive experiments across 12 datasets and seven baselines.
Critical weaknesses
-
Insufficient isolation of the core contribution. The method combines several established techniques, but the benefit of full-trajectory pathwise gradients is not clearly isolated. A controlled comparison between one-step and full-trajectory gradients would strengthen the contribution. Moreover, the straight-through estimator introduces bias, while the theoretical analysis primarily focuses on variance.
-
Limited practical gains relative to complexity. NM-PPG achieves only modest improvements over the strongest non-myopic RL baseline (GSMRL), despite introducing several additional training mechanisms. The ablations also suggest that some components are not consistently necessary. It remains unclear whether the added complexity is justified by the empirical gains.
Questions for authors
My main concern is whether NM-PPG offers a meaningful advantage over existing non-myopic RL methods such as GSMRL (PPO). The overall performance improvements are relatively modest, and the training time is not consistently lower. Moreover, the proposed straight-through estimator introduces bias while reducing gradient variance.
Could the authors clarify what fundamental limitation of conventional RL is addressed by NM-PPG, and under what conditions the proposed approach is expected to perform better? In particular, is there evidence that the pathwise formulation provides meaningful advantages in training stability, sample efficiency, or optimization quality?
A convincing clarification of these advantages would substantially strengthen my assessment of the paper.
2 comments
Thank you for the clarification. The response helps me better appreciate the motivation and design of NM-PPG. I particularly appreciate the simple and direct idea of exploiting the differentiable structure of the feature acquisition process to enable pathwise gradient optimization over complete trajectories, rather than relying solely on sampled returns as in conventional RL methods. The results on MNIST and Fashion-MNIST also provide encouraging evidence of its advantages over PPO in high-dimensional settings.
While the bias introduced by the straight-through estimator and the necessity of some additional components remain interesting aspects for further investigation, I do not consider them fundamental issues that undermine the contribution.
Overall, I find the proposed method simple, intuitive, and meaningful. The authors' clarification has addressed my main concern regarding its advantages over existing RL methods, and I am willing to raise my rating from Weak Reject to Weak Accept.
Response to reviewer eKT5
We thank the reviewer for the insightful feedback. All new results are included in Appendix G of updated PDF.
See response to question Q3 for Reviewer c1k6 regarding GSMRL
Generic RL methods, e.g. value based methods like DQN or methods that compute policy gradients based on the score function like PPO, infer long-term action effects from sampled returns or learned value estimates. This can lead to large variance, and learning can become unstable, in particular in larger state spaces. There are a number of works in the RL literature that instead propose to compute policy gradients where gradients are propagated directly through the transition dynamics of the MDP (sometimes referred to as pathwise gradients). This provides direct temporal credit assignment without a learned critic. I.e. it provides richer local gradient information at every step of the acquisition process, which reduces variance of the gradient estimator and often makes learning more stable. However, this is only possible if the full dynamics of the MDP is differentiable. In this paper, we propose a continuous relaxation of the AFA acquisition process such that pathwise gradients can be utilized in the context of AFA. This is mainly motivated by the fact that existing non-myopic methods that use generic RL are often unstable, because the state space of the AFA MDP is very large (exponential in number of features).
This is also supported by our experimental evaluation. Our method (NM-PPG), clearly outperforms the RL baselines (OL which uses DQN and GSMRL which uses PPO) on higher-dimensional datasets like MNIST and Fashion-MNIST.
In the paper, we emphasize that our method NM-PPG is the only method which performs consistently well across many different settings, including: soft-budget, hard-budget, datasets with myopic structure, datasets with non-myopic structure, and high-dimensional datasets where the state space is very large. Existing baselines struggle in at least one of these scenarios.
See response to question Q4 for Reviewer c1k6.