- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies learning under feature missingness without relying on imputation. Building on masked ERM, the paper argues that matching train-test mask distributions alone is insufficient, since prediction sensitivity to additional feature deletions can degrade both robustness under stochastic missingness and the fidelity of Shapley-style feature attributions. Under the assumption that missingness patterns are correlated with target labels, the authors propose MaDRO, a transport cost DRO framework defined around a chosen base masking distribution and a cost-penalized monotone feature deletion process. There are several related works and clarifications that should be discussed further (see Weaknesses and Questions).
Critical strengths
- The supplementary material is thorough and contains many additional analyses. e.g., comprehensive, covering different training setups, masking mechanisms, and hyperparameter sweeps.
- The paper tackles a practically relevant problem that is likely to be of interest to the ICLR community (robustness to missing features at test time without relying on feature imputation).
Critical weaknesses
-
Main message is unclear. The central contribution is somewhat difficult to identify. The main results are around budget comparisons but robustness is emphasized through out the paper. It is unclear which of these is the primary motivation and if the results are showing the robustness or even outperforming baselines. Also, the discussion around training budget and the use of only 3,000 updates appears important in the paper, but the rationale and practical significance are not sufficiently explained.
-
Limited baseline comparisons and motivation behind DRO. Main results include comparisons between the variants of the proposed method alongside relatively few external baselines (SAdD, HGB, Dropout) in the main paper. The evaluation would be stronger with comparisons against a broader set of existing approaches, especially methods that explicitly handle missing features or provide feature attributions. In addition, the implementation details of the dropout are not described, making it difficult to assess the fairness.
-
What is the key conceptual advantage gained from the DRO perspective compared to directly optimizing a masked ERM objective? Could a similar idea be incorporated into masked ERM while retaining the same robustness goals? More generally, if the key contribution is the cost-penalized feature deletion mechanism, alternative optimization formulations could achieve similar effects.
-
If computational efficiency is an important part of the paper, it would be useful to discuss connections to efficient Shapley-value approximation methods (e.g., FastSHAP and related approaches). Such methods may provide a natural baseline or point of comparison for the attribution-fidelity motivation.
-
Performance improvements are not consistently convincing. The empirical results do not consistently demonstrate clear improvements over the baselines. For example, Tables 1 and 2 do not show consistently superior performance compared to competing methods. The paper would benefit from a clearer explanation of when and why MaDRO is expected to outperform competing approaches.
-
- Addition to above, there is limited discussion of dataset-specific behavior. Although vision tasks were mentioned in the main manuscript multiple times, the paper presents results on non-imaging datasets. Given that these results appear in supplementary materials and that Table 5 shows limited gains in several cases, additional discussion is necessary to explain where MaDRO succeeds, where it does not, and what characteristics of the datasets drive these outcomes.
-
For the robustness claim, having different masking rules and their results/discussion should be reported in the main manuscript. Important vs unimportant feature removal experiments also will beneficial.
(minor) The section on theoretical properties could likely be shortened and moved to the appendix. More space could instead be devoted to above weaknesses.. (minor) Intro section appears to be missing a citation to SAdD.
Questions for authors
Most of my questions are reflected in the weaknesses above. In particular, I would appreciate clarification regarding (i) the necessity of the DRO formulation over masked ERM and their potential variants w/o DRO, (ii) the role of computational efficiency and the training budget in the paper's motivation, and (iii) the interpretation of the empirical results where the proposed method does not substantially outperform existing baselines.