- Rating
- 3
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Disclaimer: I lack the expertise in conformal prediction to assess the theoretical results, and I have indicated this to the ACs. This summary reflects my understanding of the paper’s aims and empirical findings.
The paper addresses conformal calibration when data are not IID but clustered by source, as in life-science settings where data come from different hospitals or donors. The authors argue that clustering has little effect on marginal coverage but can substantially affect conditional coverage, because score distributions differ between clusters. They attribute this to tail dependency from within-cluster thresholding rather than intraclass correlation. They show theoretically that conformal prediction and conformal risk control cannot guarantee PAC coverage when there are too few clusters, which motivates a cluster floor. The experiments use two multi-site/multi-donor datasets, one neuroimaging and one single-cell. The proposed threshold-level correction is the only PAC-compliant method on the neuroimaging data and the best-suited method on the single-cell data.
Critical strengths
The problem is highly relevant: clustered, non-IID data are common in medical and life-science applications, where calibration matters.
The work could guide both experimental design and the evaluation of conformal prediction methods, in medical tasks and beyond.
It contributes to trust in predictions and to the reliability of ML methods for such tasks.
Critical weaknesses
I lack the expertise in conformal prediction to assess the correctness of the propositions, theorems and proofs, and I have indicated this to the ACs. My assessment therefore covers only the relevance of the problem.
Questions for authors
None.
3 comments
Thank you for the response.
Thanks for the clarifications and additional results. As stated previously I have a hard time assessing the validity of the main claims. But the relevance of the problems and the additional experiments supporting results makes me inclined to keep the weak accept score (but very low confidence of 1).
Thank you for your feedback
Thank you for reviewing our additional experiments and for highlighting the practical value of reliable multi-site calibration in biomedical studies.
Response to Reviewer HmWC
We thank Reviewer HmWC for the supportive evaluation and for stressing the practical importance of trustworthy calibration in multi-site medical and life-science studies.
Protection for individual hospitals and donors (new §5, Proposition 3). Average coverage can hide poor performance at specific centers, so we added an exact finite-sample site-level calibration rule that certifies coverage for at least a 1−γ fraction of new sites. Across the neuroimaging, single-cell and census benchmarks, it reduces the share of under-covered sites or donors from 27–46% under standard i.i.d. calibration to 0–7.5% (Figure 3, Tables 2–3, 11).
Comparison with published hierarchical methods (§7, Figure 3). We added five baselines from Dunn et al. (2023) and Lee et al. (2025) and a third benchmark, 51-state U.S. Census data, and we report both high-probability pooled validity and per-site coverage. Our two procedures are the only methods that meet the PAC target with finite prediction intervals in all six real-data settings, including when coverage is computed exactly against the real site populations (Appendix K).
Study-design guidance and assumption checks (§3.1, §5, Appendix G). Following your point on experimental design, the paper gives sample-size rules: about b≥logδ/log(1−α) clusters (22 hospitals or donors at 90% coverage and 90% confidence) suffice for a pooled PAC guarantee without assumptions on within-site dependence, and about b≥logδ/log(1−γ) sites suffice for site-level guarantees. Cross-fitted residual diagnostics (Appendix G) let practitioners check between-site heterogeneity and cohort range before deployment.
Readability. We rewrote the abstract in plain language, added a workflow diagram (Figure 1), and included a concrete neuroimaging running example after the main results.