acceptodds
Under review as a conference paper at ICLR 2027

Missingness-Aware Conformal Prediction Under Cross-Hospital Distribution Shift

Abstract

Clinical measurements are recorded for some patients but not others, at rates that differ across hospitals, and marginal conformal coverage does not ensure coverage within groups defined by missingness. We propose a missingness-aware conformal calibration procedure for mortality prediction under cross-hospital distribution shift. It selects a measurement on an independent sample, groups patients by whether that measurement is recorded, and applies Mondrian calibration within each group, so no calibration outcome is reused. We evaluate the procedure across hospitals in eICU and across care units within one MIMIC-IV hospital, using three predictors. Relative to pooled calibration, it reduces the average worst-group coverage gap on its selected groups in all six settings, with a median reduction of 1.9 percentage points; paired site-bootstrap intervals exclude zero in five. These gains do not extend uniformly. Calibration by predicted risk achieves smaller gaps on a broader panel of missingness groups, and when eICU hospitals are evaluated separately, the gain shrinks for all three predictors and reverses in sign for one. We explain this discrepancy with a hospital-level decomposition. Pooling reweights hospitals through a covariance between group shares and coverage errors, and lets errors of opposite sign cancel: weighting explains the reversal, and cancellation accounts for most of the attenuation for the other two predictors. Constructed population distributions show that pooled and within-hospital evaluations can rank calibration methods oppositely even without sampling noise. Pooled improvement alone therefore cannot establish better coverage within hospitals, even when the calibration groups are fixed.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
conformal prediction, missing data, distribution shift, subgroup coverage, clinical prediction

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_yfbhOpenReview ↗
Rating
4(+1)
Confidence
3(+1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a missingness-aware weights calibration for conformal predictions under a multi-site, multi-group environment, and aims to guarantee conditional coverage for subgroups. The paper's key focus is not a novel model but instead weights calibration for weights learned from existing methods in general, if I understand correctly. This paper is overall suitable for publishing in a place like ICLR, but requires some revisions specified below for clarity on assumptions and scope.

Critical strengths

The paper is overall well-written (with some minor clarifications still needed). Some plain language illustrations are included in the main text, which is great for readability. The paper's focus is also very important and interesting, although some knowledge gaps related to missingness assumption require some clarifications and potential further investigations. Presentation in the main text is clear, with a good amount of theoretical results and excellent and highly practical numerical studies. The method has an advantage that it is pretty natural and intuitive, which has a good potential for practice. The use of notation A,D,BA, D, B in different weights is quite helpful for understanding.

Critical weaknesses
  1. The paper considered missingness may vary across different sites. However, there doesn't seem to be an explicit statement on the missingness assumption. Is the working addressing missing at random (MAR) or missing not at random (MNAR) for each variable. The latter is usually a much harder problem as it means the missingness depends on unknown information. Could the author(s) clarify what the level of missingness their framework can tackle? This could potentially broaden the audience of this paper.

  2. The following work is related in "Distribution Shift" literature part. They addressed missing outcomes prediction under a missing-at-random (MAR) assumption. However, this work considers only missingness on the outcome, but the current paper considers missingness on all variables.

Liu, Yi, et al. "Multi-source conformal inference under distribution shift." Proceedings of machine learning research 235 (2024): 31344.

  1. The outcome is denoted by YY but f(Z)f(Z) is used for probability density notation, should it be f(Y)f(Y) there? Does XX include covariates, treatment, etc. or just multi-variate outcomes? The notation introduction part is a little bit confusing. Does the paper only address binary outcome? Or the outcome YY here means a summary of multiple outcomes XX based on a learner or transformation ff? I don't exactly know what did they mean when they were first introduced.

  2. The main text misses some details on what models used for getting the nonconformity score, while only some technical details about the weights are presented. It is unclear what does one need to obtain these weights, and whether they depend on some hard missingness assumptions. Or is the weight-calibration method assumption and model-free?

Questions for authors

None. See some questions come with the weakness above.

Reviewer_VdVgOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper selects one measurement on an independent set of sites, partitions patients according to whether that measurement is missing, and performs Mondrian calibration within the two groups. The selector favors large between-site variation in missingness rates. The clinical evaluation uses three mortality predictors across eICU hospitals and MIMIC-IV care units, comparing pooled calibration, alternative groupings, mask weighting, and conditional conformal prediction. I view the work primarily as a careful analysis of evaluation targets and aggregation, rather than a new method that solves cross-hospital calibration. Under that interpretation, it offers a useful contribution, although the selector's rationale and the robustness of the empirical effect sizes deserve further clarification.

Critical strengths

S1. The aggregation analysis addresses an important evaluation failure and connects theory to actual data. Proposition 1 identifies both the change in hospital weights induced by group pooling and the inequalities among aggregation orders. Appendix C then constructs population distributions with exact known coverages, separating a genuine aggregation effect from finite-sample maxima. The eICU decomposition links the XGBoost point-estimate reversal to reweighting and most LR/MLP attenuation to cancellation. These results support a concrete lesson for multi-site uncertainty evaluation, even though the algebra itself is elementary.

S2. The empirical reporting is thorough and appropriately self-critical. Site-disjoint roles, shared predictions and test patients, two- versus three-risk-group controls, broader missingness audits, group-size sensitivity, set composition, label coverage, and probability transformations reduce several common interpretive confounds. The paper openly reports when its proposed method loses, states that the clinical reversal is not statistically established, and distinguishes MIMIC-IV care-unit shift from independent hospital replication. In particular, the outcome-coverage and empty-set results prevent a smaller average set size from being mistaken for clinical usefulness.

Critical weaknesses

W1. The calibration procedure is modest, and its selector is not aligned with the error mechanism analyzed by the paper. Selecting a binary mask partition independently and applying Mondrian calibration preserves a standard guarantee under exchangeability. Missingness-conditional conformal prediction has already been studied, including Zaffran et al. (2023). The manuscript acknowledges that work and appropriately explains why its input-augmentation method is outside the fixed-prediction benchmark. The novelty here therefore lies mainly in the cross-site evaluation analysis.

The maximum-minus-minimum missingness-rate criterion is not connected to the score-distribution discrepancy or within-group deployment shift in Appendix B. The appendix explicitly shows that rate variation need not imply a large pooling error. Moreover, a range can be sensitive to an extreme selection site and unequal site sample sizes. Empirically, risk terciles have smaller selected gaps in five of six settings and smaller panel gaps in all six, while the random or lactate controls can beat the proposed selector on eICU. Appendix G adds useful evidence that a limited learned mask correction often does not improve on risk, but it does not validate the original selection criterion.

I do not regard failure to outperform every comparator as a reason to reject an informative diagnostic paper. However, the proposed method should be presented as a transparent probe for studying aggregation, rather than as the principal advance in handling hospital shift. The paper would be stronger if it showed when selection by missingness-rate variation is useful, or demonstrated that the central aggregation findings persist under several independent, prespecified partitions rather than relying primarily on the selector's changing partition.

W2. The empirical magnitudes and decomposition uncertainty remain conditional on a fairly narrow fitted-model experiment. The hospital bootstrap is a considerable improvement over treating overlapping assignments as independent. Nevertheless, it conditions on fitted predictors, selected partitions, and calibration thresholds. The decomposition itself is reported mainly as means over assignments without uncertainty for the weighting and cancellation contributions. It is an exact decomposition of observed quantities, but that alone does not establish that the same empirical mechanism dominates after refitting or under a new site assignment.

This matters because the effects are small and sometimes unstable. The unique-hospital XGBoost reversal is about +0.23 percentage points, its interval includes zero, and all adjusted eICU within-hospital intervals include zero (Section 5.5). Appendix D also shows that the small MLP within-hospital difference changes sign with the training cap. The manuscript acknowledges these facts, so a significant population reversal should not be demanded or presumed. Still, a limited refitting sensitivity study would help distinguish a robust evaluation lesson from a particular numerical decomposition.

The observed panel and within-site absolute maxima are additionally affected by finite-test noise. Appendix E.1 recognizes this, and the group-size restrictions in Appendix D.2 strengthen the risk-versus-missingness comparison, but neither fully quantifies uncertainty in the hospital-level decomposition. The most defensible claim is that the proposed mechanisms are possible in population and explain the arithmetic of the observed comparisons; broader claims about their prevalence or effect size across hospitals require additional evidence.

Questions for authors

Q1 — W1, framing and selector justification. Is the principal contribution the missingness selector or the aggregation diagnostic? Can you identify a score-based condition under which the range-of-rates selector should help, or report its stability after removing an extreme selection site? A negative result would also be informative if the method is explicitly framed as an audit probe.

Q2 — W1, generality of the main insight. Do the weighting-versus-cancellation conclusions persist for prespecified lactate groups, binary risk groups, and random eligible mask groups under the same site assignments? A compact matched analysis would show whether the empirical mechanisms extend beyond the proposed selector without requiring a new calibration method.

Q3 — W2, robustness of attribution. Can you give paired hospital-bootstrap uncertainty for the decomposition contributions and repeat a limited subset of comparisons with independently refitted models, selection rules, and calibration thresholds? Which conclusions about dominant mechanisms remain stable, even where the sign of the small within-site improvement does not?

1 comment

Thank you for your response and the additional experimental results. However, the issues of the XGBoost ranking reversal and unstable MLP attribution, as well as the limited interpretation due to small-cell sampling noise, remain. I have decided to keep my original score.

Reviewer_WfqbOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates a relatively mature research area, missingness-aware conformal prediction under cross-hospital distribution shift, but still finds its own research angle. Considering missing data, the authors propose a calibration procedure that first selects the medical variable with the largest variation in missingness rates across hospitals and then applies Mondrian conformal calibration separately to patients with and without that measurement. The authors also analyze how changes in hospital weights and cancellation of coverage errors affect evaluation results. Experiments on the eICU and MIMIC-IV datasets using three prediction models show that the proposed method improves coverage for the selected missingness groups.

Critical strengths
  1. Research Value of the Problem and Analysis

Using clinical medicine as the research setting has significant research and practical value, and cross-hospital distribution shift and missingness are real-world problems. Moreover, not only the method but also the analysis is valuable. Although the mathematical decomposition itself is relatively basic, the authors not only point out that pooled evaluation can be misleading, but also demonstrate the possibility of ranking reversal through a constructed example and use the weighting and cancellation decomposition to explain differences in evaluation results in the clinical experiments.

  1. Rigorous and Transparent Experimental Design

The authors use separate data for model training, group selection, calibration, and testing to avoid obvious data leakage. They also compare multiple calibration baselines and openly report that the proposed method's advantages diminish when evaluated on broader missingness groups and within individual hospitals, rather than presenting only favorable results.

Critical weaknesses

1.Limited Novelty and Lack of Theoretical Support

Although the paper claims multiple methodological and theoretical contributions, its actual novelty appears limited. Grouping patients by missingness status and applying Mondrian calibration have already been studied. The main methodological novelty seems to be selecting the variable with the largest variation in missingness rates across hospitals. However, the theoretical analysis does not address this new design. Instead, it focuses on general properties of groupwise calibration and how hospital weighting and error cancellation affect pooled evaluation. While these analyses have value, they do not explain why the proposed variable selection strategy should improve coverage. Thus, the theoretical contributions do not directly support the main methodological novelty.

  1. Insufficient Attention to Writing and Formatting

The paper has several issues with writing conventions and formatting, including: The page header is missing the standard “Under review as a conference paper at ICLR 2027” text provided by the official ICLR 2027 template. Page 9 contains a large amount of blank space, while some appendix content could have been moved into the main text.

Other similar issues.

These issues collectively suggest insufficient attention to the overall writing organization, formatting requirements, and final proofreading of the paper, leaving room for improvement.

Questions for authors
  1. The authors should provide a theoretical explanation for the proposed core method, particularly why selecting the variable with the largest variation in missingness rates across hospitals can improve coverage.

  2. The authors should thoroughly review the writing and formatting issues throughout the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.