acceptodds
Under review as a conference paper at ICLR 2027

Active Feature Acquisition from Incomplete Training Data

Abstract

In many prediction tasks, acquiring all features can be a prohibitively expensive or outright impossible task. Further, in many cases a static subset of features may not be enough to solve the problem sufficiently across various instances. Active Feature Acquisition (AFA) addresses these problems by formalizing the trade-off between feature cost and predictive performance during sequential feature selection. However, prior AFA work largely assumes access to complete training data, an assumption that is often violated in practice. Here we study AFA with Incomplete Training Data (\AFAITD), showing that under missing completely at random (MCAR) data, one-step acquisition values remain unchanged, whereas multi-step values can decrease. We analyze three approaches to learning from incomplete data: aliasing, filtering, and generative restoration. We show that filtering can require a number of training instances scaling exponentially with the dimension, whereas generative restoration scales exponentially with the acquisition budget. We empirically test our theory on a controlled experiment and across common AFA datasets and find that missingness mainly damages methods that exploit multi-step acquisitions and that generative restoration is able to recover lost performance in many experiments.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
cost-sensitive decison-making, missing data, active feature acquisition

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_MyZwOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper formulates the AFA-ITD problem as a Partially Observable Markov Decision Process and discusses three solutions for the missingness problem: aliasing, filtering, and generative restoration.

Critical strengths
  1. This paper formalizes the incomplete training data problem in AFA settings and establishes a theoretical analysis of data requirements for different solutions
  2. This paper includes both a synthetic controlled study and real AFA benchmarks with diverse missingness setups.
Critical weaknesses
  1. The paper theoretically analyses the data requirements under MCAR with uniform probabilities, without addressing MAR or MNAR. While useful for understanding the role of b and d, the theorem could potentially be extended to more general settings with feature-specific missingness rates or MAR propensities.

  2. The experimental results (Figure 4) of the AFA benchmark are poorly analyzed. Please see the "Questions" section.

Questions for authors

Q1. For line 428, you mention that restoration can perform worse than restricted training because "missingness can remove redundant or noisy features, making the problem easier.” According to Table 1 (line 1067), CUBE-NM has the lowest ωroute score, indicating that many distinct routes yield relatively high gains, so missingness helps simplify the problem. However, for a dataset with a high ωroute score (eg, CUBE), why does restoration still hurt performance for OL, ODIN, and DIME? Instead of missingness simplifying the problem, could the performance be worse because of the approximation error from the PVAE generative model?

Q2. Following the discussion of Q1, if we increase the training dataset, would restoration always outperform restricted training?

Q3. Figure 4 shows that, for OL, JAVA, and ODIN, aliasing approaches outperform filtering approaches on some tasks in both Restricted-Action Training and Generative Restoration. How would you explain this issue? Does that hold for other missingness patterns and missingness ratios?

Reviewer_3cMQOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies active feature acquisition (AFA) when training instances have missing features but test instances are complete (AFA-ITD). Since missing features cannot be acquired during training, the training MDP has a smaller action space than the evaluation MDP. Proposition 1 shows that under MCAR this leaves one-step values unchanged but can lower multi-step values, so non-myopic methods are the ones at risk. The paper then compares three tabular approaches: aliasing (ignore the mask), filtering (condition on the mask, so only complete instances inform the evaluation values), and generative restoration (estimate the missing conditionals from all data). Theorem 1 shows that filtering needs a number of samples exponential in the number of features d, while restoration needs a number exponential only in the budget b. Experiments include a synthetic shortcut problem and six AFA methods on eight AFABench datasets, where restoration is implemented as PVAE imputation of the training set.

Critical strengths
  • The problem is practically relevant, since records such as medical data often contain only the tests that were actually ordered.
  • The shortcut example (Example 1) clearly shows why multi-step values break under missingness and why aliasing converges to the wrong policy.
  • The empirical study is broad, covering six methods, eight datasets, and four missingness mechanisms, and code is provided.
Critical weaknesses

1. The theory is close to classical missing-data results. Proposition 1 follows directly from the training action set being a subset of the evaluation action set, and the proof is a short induction. Similarly, filtering versus restoration in Theorem 1 is essentially complete-case versus available-case estimation under MCAR. It is well known that complete cases occur with probability (1-p)^d, while a conditional over a few variables can be estimated from every row where those variables are observed (Little & Rubin, 2002, which the paper cites). Theorem 1 therefore plugs this standard fact into a tabular regret bound. The proof looks correct, but I do not see a new technical idea in it.

2. The lower bound is weak. Proposition 2 only says that filtering fails when there is no complete instance at all. It does not depend on ε, and there is no lower bound for restoration. As a result, the "exponential in d versus exponential in b" claim compares two upper bounds plus a trivial necessity condition. Moreover, the K^b factor appears in both bounds, so restoration is also exponential in b for reasons unrelated to missingness.

3. The predictor is trained on complete data. The predictor is trained before missingness is imposed and is kept fixed across treatments (L412-413). In the AFA-ITD setting, however, a practitioner would not have complete data to train it. This choice removes what is likely the main source of damage in practice, so the experiments isolate only the policy-learning part under an unrealistic advantage. A version where the predictor is also trained on incomplete or restored data is needed to support the practical claims.

4. Simple imputation baselines are missing. PVAE restoration is the only remedy tested. Yet Janisch et al. (2020), which the paper cites as the closest prior work, already compared mean and MICE imputation in this setting. Without mean, MICE, kNN, or multiple-draw imputation, it is unclear whether the gains come from the generative model or from imputation in general. In addition, the PVAE in EDDI (Ma et al., 2019) was built to train on partially observed data, which makes it a direct precedent for restoration and weakens the claim that prior AFA work assumes complete training data.

5. The theory covers less than the experiments. All results assume MCAR, discrete features, a fixed predictor, and a hard budget. The experiments, however, also include MAR and two MNAR settings (Fig. 6), which receive no analysis and are summarized only as showing "similar patterns." For instance, it is unclear whether label-conditioned imputation is even consistent under self-masking MNAR.

Questions for authors
  1. How does PVAE restoration compare with mean, MICE, and multiple-draw imputation?
  2. What happens when the predictor is also trained on incomplete or restored data?
  3. Can Theorem 1 be extended to a learned generative model, for example with a term for its error in p(xa∣xS)p(x_a | x_S)?
  4. In how many method-dataset cells of Fig. 4 is the restoration gain significantly positive, and in how many is it significantly negative?
  5. What fraction of training instances is complete in each real dataset at p = 0.7?
Reviewer_XaaYOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper analyzes the effect of missing training data on policy training for active feature acquisition. It shows that missingness during training may hurt action valuation over multiple steps (but it is also shown that one-step valuation is not affected) and that generative restoration can restore the valuation when the missingness mechanism follows MCAR, with data requirements that depend exponentially on the budget. The synthetic experimental results clearly support the argument, although the real-world experiments show mixed results.

Critical strengths
  1. The idea is clearly motivated and supported by the theory.

  2. The evaluation covers several acquisition methods and missingness mechanisms.

Critical weaknesses
  1. The results of the real-world experiments are mixed, and generative restoration does not consistently improve performance across methods and datasets.

  2. The predictor is trained on complete data before missingness is introduced. Therefore, the experiments do not evaluate the setting where both the predictor and the acquisition policy must be learned from incomplete data.

Questions for authors
  1. Can you examine the relationship between imputation quality and the amount of downstream performance recovered? Specifically, could you repeat the experiments with at least two additional imputation methods and report both their imputation errors and the resulting AFA performance? Alternatively, could you save intermediate checkpoints during PVAE training and report their imputation errors and corresponding downstream performance? The artificially masked values could provide the ground truth for measuring imputation errors.

  2. I think an assumption is missing in Example 1. The proof of Proposition 2 relies on the context and branch features being independent, uniformly distributed binary variables, but Example 1 only explicitly specifies the distribution of the additional noise features. Could you clarify whether these assumptions are intended and state them explicitly?

  3. I understand that the predictor is trained before missingness is introduced to isolate its effect on acquisition learning. However, could you include an additional experiment where the predictor is also trained on incomplete data? This would help assess whether the findings extend to the practical setting motivating the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.