- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper studies uncertainty quantification for federated multi-site rs-fMRI diagnosis. The authors argue that existing federated conformal prediction (FCP) methods (a) pool source sites by sample size and (b) measure source-target similarity on whole-brain representations that mix disease signal with scanner effects, and that both problems hide coverage failures on rare disorder subtypes.
Critical strengths
-
The paper provides a reasonably structured conformal prediction framework, distinguishing coverage validity from efficiency. The matched-coverage comparison and subtype-conditional evaluation are relevant to the stated problem.
-
Experiments across two datasets and multiple federated backbones suggest improved prediction-set efficiency and subtype-conditional coverage.
Critical weaknesses
-
Clinical utility remains unsubstantiated. Although the paper emphasizes uncertainty-aware diagnosis and the clinical relevance of empty prediction sets, neither claim is experimentally validated. There is no risk-coverage analysis, AURC, or assessment of whether empty sets identify incorrect or atypical cases. The absence of backbone diagnostic accuracy further limits interpretation. Coverage and average set size alone are insufficient to establish meaningful clinical benefits.
-
The theoretical guarantees rely on insufficiently validated assumptions. Assumption 2 requires invariant subtype-conditional label distributions across sites, which is particularly questionable under the scanner and phenotypic heterogeneity motivating the study. No site-specific coverage analysis, calibration assessment, or assumption-violation experiments are provided. Consequently, the claimed robustness to heterogeneity is not adequately supported.
-
Experimental validation is insufficient. The evaluation involves relatively few clinical sites, without systematic sensitivity analysis for key hyperparameters or the number of collaborating sites. The reliability and provenance of ADHD/ASD subtype labels are also unclear, despite subtype stratification being central to the method. Furthermore, the matched-coverage comparison uses a single operating point rather than a comprehensive coverage–efficiency analysis.
Questions for authors
- How robust are the theoretical guarantees and empirical coverage under site-specific distribution shifts or violations of Assumption 2?
- Can the authors provide per-site coverage, subtype distributions, hyperparameter sensitivity, and complete coverage–set-size curves across different significance levels?