acceptodds
Withdrawn from ICLR 2027

Federated Conformal Prediction for Reliable Brain Disorder Subtyping

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Federated Learning (FL) enables multi-site Functional Magnetic Resonance Imaging (fMRI) analyses to jointly learn shared brain connectivity patterns associated with neurological disorders while adapting to site-specific phenotypic and scanner variations. However, these frameworks often overlook uncertainty quantification for individual model predictions. In neuroimaging, models must alert clinicians when diagnostic predictions are uncertain or have low confidence. This becomes especially important for rare brain disorder subtypes, where limited training data can lead to unreliable yet overconfident predictions masked by high overall model accuracy. To address these issues, we propose BrainFCP (Brain Federated Conformal Prediction), a novel framework that applies Conformal Prediction (CP) in a federated multi-site fMRI setting to provide site-wise and subtype-specific uncertainty estimates with theoretical guarantees. We extensively evaluated our framework on two neurodevelopmental disorders: Attention-Deficit Hyperactivity Disorder (ADHD) and Autism Spectrum Disorder (ASD), using the ADHD-200 and ABIDE-I resting-state fMRI datasets, respectively. Our results demonstrate BrainFCP’s ability to construct valid personalized subtype-level prediction sets for each site in a federated setting.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Federated Learning, Uncertainty Estimation, Functional MRI

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_ZrdUOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies uncertainty quantification for federated multi-site rs-fMRI diagnosis. The authors argue that existing federated conformal prediction (FCP) methods (a) pool source sites by sample size and (b) measure source-target similarity on whole-brain representations that mix disease signal with scanner effects, and that both problems hide coverage failures on rare disorder subtypes.

Critical strengths
  1. The paper provides a reasonably structured conformal prediction framework, distinguishing coverage validity from efficiency. The matched-coverage comparison and subtype-conditional evaluation are relevant to the stated problem.

  2. Experiments across two datasets and multiple federated backbones suggest improved prediction-set efficiency and subtype-conditional coverage.

Critical weaknesses
  1. Clinical utility remains unsubstantiated. Although the paper emphasizes uncertainty-aware diagnosis and the clinical relevance of empty prediction sets, neither claim is experimentally validated. There is no risk-coverage analysis, AURC, or assessment of whether empty sets identify incorrect or atypical cases. The absence of backbone diagnostic accuracy further limits interpretation. Coverage and average set size alone are insufficient to establish meaningful clinical benefits.

  2. The theoretical guarantees rely on insufficiently validated assumptions. Assumption 2 requires invariant subtype-conditional label distributions across sites, which is particularly questionable under the scanner and phenotypic heterogeneity motivating the study. No site-specific coverage analysis, calibration assessment, or assumption-violation experiments are provided. Consequently, the claimed robustness to heterogeneity is not adequately supported.

  3. Experimental validation is insufficient. The evaluation involves relatively few clinical sites, without systematic sensitivity analysis for key hyperparameters or the number of collaborating sites. The reliability and provenance of ADHD/ASD subtype labels are also unclear, despite subtype stratification being central to the method. Furthermore, the matched-coverage comparison uses a single operating point rather than a comprehensive coverage–efficiency analysis.

Questions for authors
  1. How robust are the theoretical guarantees and empirical coverage under site-specific distribution shifts or violations of Assumption 2?
  2. Can the authors provide per-site coverage, subtype distributions, hyperparameter sensitivity, and complete coverage–set-size curves across different significance levels?
Reviewer_caVwOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

BrainFCP combines biological-representation density ratios, personalized source mixtures, and target-site subtype-stratified conformal calibration. ADHD-200 and ABIDE-I experiments report smaller prediction sets and improved rare-subtype coverage across federated backbones.

Critical strengths
  1. Rare-subtype coverage addresses a meaningful weakness of marginal uncertainty guarantees.
  2. Five federated backbones, matched-coverage comparisons, and component ablations provide useful empirical breadth.
Critical weaknesses
  1. Proposition 2 is false as stated: interpolating quantile thresholds does not interpolate coverage. Even with population quantiles, subtype/pooled scores taking value 1 with probabilities 0.8/0.04 and otherwise 0 give coverage 0.20 at alpha=0.2 and lambda=0.5, below Eq. (11)'s claimed bound of 0.42. This invalidates the proposed pooling guarantee.
  2. The rare-sample procedure is undefined. At alpha=0.2, Eq. (7) requests a nonexistent order statistic for n_c<=3; Eq. (10) does not resolve this. Figure 1 includes zero- and one-subject subtype-site cells, but handling of these cases and per-split counts are unreported.
  3. Core algorithmic steps are unspecified: V_i is described only as a score's position, and neither the biological/scanner decomposition loss nor the mixture-network training objective is defined (Sections 3.1-3.3). The supplied description is insufficient to reproduce the method.
Questions for authors
  1. Can Proposition 2 be replaced with a valid bound or a provably valid pooling rule?
  2. How are n_c=0 and n_c<4 handled, and what are the per-site/subtype calibration and test counts?
  3. What are the exact score transformation, training objectives, and independent data partitions used to fit them?
Reviewer_TRWrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

BrainFCP combines federated representation learning with subtype-conditional conformal prediction for brain disorder subtyping across sites. It uses estimated density ratios to transfer calibration information and shrinks subtype-specific thresholds toward a pooled threshold when calibration samples are scarce. Experiments evaluate prediction-set coverage and size on multisite neuroimaging data. The clinical motivation is relevant that uncertainty estimates should remain useful for small diagnostic subgroups rather than only for the overall population.

Critical strengths
  1. The paper addresses a useful problem. Marginal coverage can hide poor performance on rare subtypes, so evaluating subtype-level reliability in a federated setting is well motivated.
  2. An appealing aspect is the explicit separation between validity and efficiency. The finite-sample coverage guarantee is provided by target-site subtype-wise calibration, while source weighting and density-ratio estimation are used to improve prediction-set sharpness. This separation makes the framework conceptually clean and allows it to be applied on top of different FL backbones.
  3. Comprehensive empirical evaluation. The paper evaluates BrainFCP on two multi-site rs-fMRI datasets, ADHD-200 and ABIDE-I, using five FL backbones and four federated conformal prediction baselines. The matched-coverage experiment is particularly useful because it shows that the smaller prediction sets are not simply caused by accepting lower marginal coverage.
Critical weaknesses
  1. The paper combines federated learning, density-ratio weighting, subtype-conditional calibration, and threshold shrinkage. These components have existing counterparts, but the paper does not clearly identify which new technical contribution goes beyond their combination. In particular, Eq. (10) closely resembles existing interpolation between group-specific and pooled thresholds. Making the interpolation coefficient depend on subtype sample size does not, by itself, establish a substantial methodological advance. The authors need to explain what this change enables and provide evidence that it performs better than existing interpolation rules. Applying the framework to brain disorder subtyping is useful, but the application alone does not resolve this concern.

  2. The coverage guarantee does not clearly apply to the final algorithm. Theorem 2 describes subtype-conditional calibration, but Eq. (10) then changes the threshold through shrinkage. The resulting threshold need not retain the original subtype coverage guarantee. Proposition 2 therefore needs a complete proof, including its assumptions and the effect of using estimated density ratios. The paper should also state how thresholds are computed for subtypes with very few or no calibration samples. These details are necessary to determine whether the claimed reliability holds for the method used in the experiments.

  3. Several recent methods are missing from the experimental comparison, such as [1]. Besides, a target-site-only Mondrian conformal baseline is also needed to determine whether cross-site calibration provides a benefit. [1] Wen H, Simeone O, Xing H. Efficient federated conformal prediction with group-conditional guarantees. UAI 2026

  4. Two core methodological components are insufficiently specified and validated. The paper assumes a decomposition hψ(x)=(zb,zsc)h_\psi(x)=(z_b,z_{sc}), referring to zbz_b as the biological representation and zscz_{sc} as scanner-specific, but the manuscript does not describe an explicit disentanglement objective that prevents scanner/site information from remaining in zbz_b. Using only zbz_b for classification does not by itself establish scanner invariance. Similarly, the subtype-aware source mixture is defined through a learnable network GϕG_\phi, but its training objective is not clearly specified. The uncertainty term ukc=Var[log⁡r^b]u_k^c=\mathrm{Var}[\log\hat r_b] is described as a posterior variance without explaining what posterior model is used or how this quantity is estimated. In addition, Eq. (2) directly identifies domain-classifier odds with a density ratio, but the required correction for target/source sampling priors is not discussed. These omissions affect both reproducibility and the credibility of two of the paper's main technical contributions.

Questions for authors
  1. Can you provide a proof of Proposition 2 for the threshold in Eq. (10)? Please specify whether it guarantees each subtype’s coverage, marginal coverage, or an approximate bound, and state how density-ratio estimation affects that result. This would directly affect my assessment of soundness.
  2. How are the training, validation, and calibration sets separated at each site, and how many calibration subjects belong to each subtype? What threshold is used when a subtype has zero or very few calibration subjects? Please also define V in Eqs. (7)–(8) and explain how its parameters are selected without using test labels.
  3. How does BrainFCP compare with target-site-only Mondrian calibration and a matched interpolation baseline using the same scores, data split, and nominal coverage? Please report subtype coverage with uncertainty intervals alongside set size. These comparisons would clarify whether the proposed weighting and shrinkage account for the improvement.
Reviewer_dyPpOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper describes a federated method to produce conformal prediction set for a fMRI based classification task. The task is to predict the classification of a target client based on a set of given source clients without aggregating their data to a single server. The main insights are:

  1. Each subtype (class) needs to have its own alibration threshold so that the rare classes are represented appropriately.
  2. Split the data representation in to biological and scanner dependent features and the source-target matching uses biological features only.
  3. When matching the source and target distributions, weight them using a learned network and not using the number of samples.
Critical strengths
  • The problem in the paper has an interesting angle for federated computation.
  • The aspect of conformal prediction is interesting in this context.
Critical weaknesses
  • The paper could have been written better to improve readability. The method section may benefit from explaining the algorithm from a high level and then go into detail. Some of the concepts need more explanations to be appreciated such as marginal coverage.
  • The paper describes a slightly different federated setting than typical FL papers where the goal is to predict samples in a target client rather than held out data at the source client sites. It'll be great to have some justification of why is this the case and what happens if this is evaluated on the same source clients.
  • The datasets are fairly small and the numnber of classes are also small compared to the set sizes reported in the results. I understand this might be a limitation of the datasets available in the domain, but can this method be generalized and applied to other datasets?
Questions for authors

As above

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.