- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper formulates the AFA-ITD problem as a Partially Observable Markov Decision Process and discusses three solutions for the missingness problem: aliasing, filtering, and generative restoration.
Critical strengths
- This paper formalizes the incomplete training data problem in AFA settings and establishes a theoretical analysis of data requirements for different solutions
- This paper includes both a synthetic controlled study and real AFA benchmarks with diverse missingness setups.
Critical weaknesses
-
The paper theoretically analyses the data requirements under MCAR with uniform probabilities, without addressing MAR or MNAR. While useful for understanding the role of b and d, the theorem could potentially be extended to more general settings with feature-specific missingness rates or MAR propensities.
-
The experimental results (Figure 4) of the AFA benchmark are poorly analyzed. Please see the "Questions" section.
Questions for authors
Q1. For line 428, you mention that restoration can perform worse than restricted training because "missingness can remove redundant or noisy features, making the problem easier.” According to Table 1 (line 1067), CUBE-NM has the lowest ωroute score, indicating that many distinct routes yield relatively high gains, so missingness helps simplify the problem. However, for a dataset with a high ωroute score (eg, CUBE), why does restoration still hurt performance for OL, ODIN, and DIME? Instead of missingness simplifying the problem, could the performance be worse because of the approximation error from the PVAE generative model?
Q2. Following the discussion of Q1, if we increase the training dataset, would restoration always outperform restricted training?
Q3. Figure 4 shows that, for OL, JAVA, and ODIN, aliasing approaches outperform filtering approaches on some tasks in both Restricted-Action Training and Generative Restoration. How would you explain this issue? Does that hold for other missingness patterns and missingness ratios?