- Rating
- 4(+1)
- Confidence
- 3(+1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes a missingness-aware weights calibration for conformal predictions under a multi-site, multi-group environment, and aims to guarantee conditional coverage for subgroups. The paper's key focus is not a novel model but instead weights calibration for weights learned from existing methods in general, if I understand correctly. This paper is overall suitable for publishing in a place like ICLR, but requires some revisions specified below for clarity on assumptions and scope.
Critical strengths
The paper is overall well-written (with some minor clarifications still needed). Some plain language illustrations are included in the main text, which is great for readability. The paper's focus is also very important and interesting, although some knowledge gaps related to missingness assumption require some clarifications and potential further investigations. Presentation in the main text is clear, with a good amount of theoretical results and excellent and highly practical numerical studies. The method has an advantage that it is pretty natural and intuitive, which has a good potential for practice. The use of notation in different weights is quite helpful for understanding.
Critical weaknesses
-
The paper considered missingness may vary across different sites. However, there doesn't seem to be an explicit statement on the missingness assumption. Is the working addressing missing at random (MAR) or missing not at random (MNAR) for each variable. The latter is usually a much harder problem as it means the missingness depends on unknown information. Could the author(s) clarify what the level of missingness their framework can tackle? This could potentially broaden the audience of this paper.
-
The following work is related in "Distribution Shift" literature part. They addressed missing outcomes prediction under a missing-at-random (MAR) assumption. However, this work considers only missingness on the outcome, but the current paper considers missingness on all variables.
Liu, Yi, et al. "Multi-source conformal inference under distribution shift." Proceedings of machine learning research 235 (2024): 31344.
-
The outcome is denoted by but is used for probability density notation, should it be there? Does include covariates, treatment, etc. or just multi-variate outcomes? The notation introduction part is a little bit confusing. Does the paper only address binary outcome? Or the outcome here means a summary of multiple outcomes based on a learner or transformation ? I don't exactly know what did they mean when they were first introduced.
-
The main text misses some details on what models used for getting the nonconformity score, while only some technical details about the weights are presented. It is unclear what does one need to obtain these weights, and whether they depend on some hard missingness assumptions. Or is the weight-calibration method assumption and model-free?
Questions for authors
None. See some questions come with the weakness above.
Thank you for your response and the additional experimental results. However, the issues of the XGBoost ranking reversal and unstable MLP attribution, as well as the limited interpretation due to small-cell sampling noise, remain. I have decided to keep my original score.