- Rating
- 4(+1)
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
The paper addresses limitations of traditional greedy CMI-based AFA policy optimization by constructing an objective that incorporates unseen features, where the number of included features is controlled by a regularizer. This novel objective can be interpreted as an additional lookahead term that captures interactions between the candidate feature and other unobserved features. The proposed method is experimentally validated to assess whether it can capture synergistic information on synthetic tasks, and real-world performance is also validated where gamma-CMI provides reliable gains relative to both generative and amortized approaches.
Critical strengths
-
The authors propose a practical method without significant implementation burden for incorporating lookahead signal in the AFA optimization that extends current CMI approaches
-
The authors provide interesting and novel theoretical insights about -CMI by decomposing the objective into a sum of all cardinality subsets of feature interactions, and also provides insights to connections to alternative objectives of feature acquisition such as reinforcement learning and Shapley values.
-
The paper is clearly written and well organized. For example, the modification to existing methods is clearly delineated in Algorithm 1, and the results are organized into key takeaways and insights.
Critical weaknesses
-
Reliance on a conditional generative model for unseen features given the observed features leads to a potential tradeoff in performance as is increased and the model approximation error accumulates. However, this tradeoff is characterized both theoretically and empirically.
-
Evaluating each feature candidate requires approximating expectations over M mask draws by performing forward passes for and . While computationally intensive and difficult to scale in feature dimensionality, this is a standard limitation of generative frameworks that does not undermine the core conceptual contributions. However, it would be helpful to include a brief discussion of this practical limitation in the text.
Questions for authors
Clarification requests and missing details:
-
How was -CMI integrated into non-generative approaches (such as DIME and GDFS)? Specifically, given that -CMI generates hypothetical contexts of imputed unobserved features across varying cardinalities, how are these aggregated via the policy or reward model to yield a score for each candidate or select the optimal acquisition at each step?
-
Was used across all real-world experiments? If so, this number of evaluated trajectories appears quite low for reliably uncovering complex feature interactions. Could increasing reduce estimation variance and thereby broaden or stabilize the tolerance for ?
-
For the real world experiments, is it possible to provide the generative model error (i.e. average log loss) across the methods (or even just VAE)? It may be useful to see if the benefits of lookahead and is correlated with the ability to model the conditional feature distributions in Table 1/2 and Appendix I.
-
Additionally, would it be possible to provide the full performance curves (e.g., log loss, AUROC, or accuracy) across each acquisition step for the real-world experiments? While the conceptual contributions and empirical gains are already substantial, seeing these curves would provide insight into whether feature interactions are both helpful and captured early in the acquisition trajectory on real-world settings. My sense is that interactions may not always be helpful depending on the dataset and error of the generative model.
6 comments
[Q2] Whether = 5 Was Used Consistently Across All Experiments
We used = 5 in all experiments involving both synthetic and real-world datasets.
Also, we would like to clarify a potentially misleading description of mask sampling in our original manuscript. In the actual implementation, all samples generated by the generative model do not share the same set of masks. Instead, masks are independently sampled for each generated sample. Therefore, if denotes the number of samples produced by the generative model, up to distinct masks can be sampled. Since masks are sampled independently for each generated sample, even = 1 allows up to distinct masks to be evaluated. Compared with sharing a single mask across all generated samples, this enables the method to consider a wider variety of hypothetical future contexts. To avoid further confusion, we have revised the pseudocode and clarified this point in the manuscript. Notably, even with = 1, -CMI can improve acquisition performance while incurring virtually the same computational cost as Greedy CMI.
Regarding the choice of , we considered and . The resulting average standard deviations in performance for each are summarized below.
| Task | ||||
|---|---|---|---|---|
| Overall (12 datasets) | 0.005743 | 0.005159 | 0.005096 | 0.004528 |
We also report the average performance differences relative to = 5 below.
| Task | ||||
|---|---|---|---|---|
| Overall (12 datasets) | −0.277% | 0.000% (baseline) | +0.074% | +0.109% |
We additionally measured the wall-clock time overhead for each value of , as shown below.
| Task | ||||
|---|---|---|---|---|
| Overall (12 datasets) |
The wall-clock time overhead starts at a level comparable to Greedy CMI when and gradually increases with . We selected by considering the trade-off between Monte Carlo estimation accuracy and increased wall-clock time overhead.
Also, in Table J.1 of the original manuscript, we labeled Greedy as . However, since Greedy does not consider diverse future contexts, we have refined the notation for clarity in the revised manuscript.
[Q3] Conditional Distribution Quality and Lookahead Performance
To examine how the quality of the conditional distributions learned by the generator relates to lookahead performance, we have added new experiments using VAE on all 12 real-world datasets to Appendix K.3 of the revised manuscript (Figures K.3 and K.4). Within each model repeat, we held the predictor fixed and varied only the generator, comparing the existing saved checkpoint (the reference) with generator checkpoints from epochs 1, 5, and 120 (the final epoch).
In the additional figures, dashed lines represent the reference, while solid lines represent the checkpoints from different epochs. The legends report the Energy Score (ES), a sample-based multivariate scoring rule (Gneiting and Raftery, 2007, Section 4.3). The energy score is computed as , where and are independent samples from the conditional generative distribution , is the true unobserved feature vector, and is the distance used for evaluation. Lower ES indicates better conditional predictive quality. ES is evaluated on the same validation examples using identical observation masks across all generator checkpoints. The horizontal axis shows , and the vertical axis shows test performance averaged over all acquisition steps, consistent with Figures I.2–I.3. Each curve is averaged over five model repeats.
Across several datasets, the overall pattern of performance as a function of is preserved across generator checkpoints. On Bank Marketing, each evaluated checkpoint benefits from using a positive value of rather than , whereas on MNIST, performance consistently declines beyond small values of . Although the broad trends across are often preserved, the magnitude of the performance change varies across generator checkpoints. On Kin8nm and White Wine Quality, early checkpoints with higher ES exhibit relatively worse performance at large . On Parkinson, checkpoints with higher ES also show smaller improvements from lookahead.
Conversely, on Bank Marketing and MiniBooNE, improvements over are maintained, or even become larger in some cases, for checkpoints with higher ES. Thus, across the datasets and checkpoints examined, we did not observe a consistent monotonic relationship in which greater generative error invariably leads to smaller lookahead gains.
These results suggest that the ability to model conditional distributions can affect the effectiveness of lookahead, but the measured average ES alone is insufficient to explain the favorable range of or the magnitude of the performance gains. Interaction structures, including synergy and redundancy, may also shape the information that lookahead can exploit. Consequently, improved generative model quality need not translate into greater lookahead gains. For example, even when generative model error decreases, additional gains may remain limited if there is little room to improve the acquisition order by considering future contexts.
Reference
Gneiting, T., & Raftery, A. E. (2007). Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102(477), 359–378.
We sincerely thank the reviewer for the positive evaluation of our paper. We have revised several parts of the manuscript to address the points raised, with the changes highlighted in blue for ease of reference. Below, we provide detailed responses to the reviewer's comments and questions.
[W1 & W2] Discussion on Generative Models
As the reviewer pointed out, the trade-off between γ and predictive performance, as well as the increased computational cost and scalability challenges as feature dimensionality grows, are fundamental limitations associated with the use of generative models. We have added further discussion of these limitations in Appendix B and referred to it in Section 3 of the main text. We thank the reviewer for encouraging us to elaborate on these practical considerations.
[Q1] Application of γ-CMI to DIME and GDFS
For methods that do not use generative models, such as DIME and GDFS, the currently observed feature values and observation mask are provided as inputs to compute candidate-specific acquisition scores, and the feature with the highest score is selected for acquisition. To apply γ-CMI to these methods, we first obtain K samples generated by a VAE conditioned on the current observations. We then construct hypothetical future contexts for each generated sample by independently sampling M=5 future masks, where each unobserved feature is included with probability γ. When evaluating a candidate feature, that feature is excluded from the corresponding context. For each context, we compute either DIME's value-based score or the raw logits produced by GDFS's selector, and average these scores over the generated samples and future masks. Finally, we acquire the currently unobserved candidate feature with the highest average score, thereby incorporating hypothetical future observations into the acquisition decision.
However, directly introducing VAE-generated inputs may cause out-of-distribution (OOD) issues for DIME and GDFS, as these models were not originally trained on such inputs. To mitigate this issue, we additionally performed fine-tuning to align the models with the generated inputs. Since these experiments were intended as an exploratory investigation of whether our approach could also be applied to CMI-based methods that do not use generative models, we exceptionally performed fine-tuning in this setting, unlike the experiments with generative-model-based methods. Extending γ-CMI to these methods without introducing an additional VAE remains an important direction for future research. Furthermore, to distinguish whether the performance improvements of these two models were simply attributable to fine-tuning, we conducted the following additional experiments. Please refer to [W5 & Q3] in our response to Reviewer s3A4 for additional experiments.