acceptodds
Under review as a conference paper at ICLR 2027

xHetMed-FL: Explainable Federated Shared Representation Learning Across Heterogeneous Medical Imaging Modalities

Abstract

Medical images are usually stored in different hospitals and with different imaging modalities and are often subject to different storage formats, privacy concerns, modality-specific characteristics and different security restrictions, which hinders centralized learning. Current federated learning methods usually assume a uniform class or learning goal across clients and the raw patient data cannot be shared. To address these challenges, we suggest a multimodal federated learning scheme, xHetMed-FL, using 2D/3D CNNs to create a shared 128-D latent representation while federating only a few projection parameters. The proposed framework is organized around four heterogeneous clients representing SPECT, Chest X-ray, MRI, and CT+PET imaging. Rather than forcing these modalities into a common input architecture, each client retains a modality-specific preprocessing pipeline and encoder. The SPECT client uses verified pathological and non-pathological labels for supervised disease classification, whereas the Chest X-ray, MRI, and CT+PET clients learn representations without disease labels through a cosine-similarity-based self-supervised objective. Only shared projection is communicated and aggregated by the federated server, while the modality-specific encoders, client-specific components, prediction heads, and Batch Normalization parameters remain local. To address optimization under heterogeneous client conditions, FedProx is incorporated into local training, providing proximal regularization with respect to the global shared parameters, while FedBN preserves client-specific Batch Normalization statistics. The obtained SPECT client's validation accuracy, F1 score, and ROC-AUC were 82.35%, 82.86%, and 91.77%, respectively. The cosine similarity in the Chest X-ray, MRI, and CT+PET clients was found to be 1.0000, 1.0000, and 0.9992, respectively, while the variance in their latent representations was 0.3185, 0.3168, and 0.2965, respectively. Grad-CAM is used to interpret the trained network at the post-training stage, providing modality-specific explanations while preserving the heterogeneous local components. Overall, the proposed framework shows the viability of learning shared representations from heterogeneous medical imaging modalities with different learning goals while keeping modality-specific components and keeping the models interpretable.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Multimodal Representation Learning, Heterogeneous Federated Learning, Self-Supervised Learning, Medical Imaging, Shared Latent Representations, Privacy-Preserving Learning, Explainable AI

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_US9pOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This work focuses on the significant heterogeneity in medical image federated learning, where different clients may vary in data modalities, model architectures, and learning objectives, making conventional federated learning difficult to apply through direct model sharing and aggregation. To address this issue, the authors propose xHetMed-FL, which preserves client-specific modeling while establishing a unified representation interface shared across clients, enabling clients with different modalities, network architectures, and supervision settings to collaboratively learn within a common federated framework.

Critical strengths
  1. The paper addresses a practically relevant and challenging federated learning setting, where heterogeneity exists simultaneously in data modalities, model architectures, and learning objectives across clients.
  2. The proposed framework is flexible in accommodating heterogeneous clients, allowing each client to retain its modality-specific preprocessing pipeline, encoder architecture, and learning objective rather than enforcing a unified local model.
  3. The framework provides a simple and modular design for heterogeneous federated learning, separating client-specific components from the federated component and thereby avoiding direct aggregation of structurally incompatible local models.
  4. The paper considers both supervised and self-supervised clients within the same federated framework, reflecting a realistic medical imaging scenario in which annotation availability may differ across institutions and modalities.
  5. The paper includes post-hoc explainability analysis for different types of clients, using modality-specific visualization and quantitative attribution statistics to provide additional analysis of the learned models and representations.
Critical weaknesses
  1. The validity of the shared representation is insufficiently verified because FedProx only constrains the shared projection parameters to remain close across clients, while parameter proximity does not demonstrate that the projection performs the same function or encodes semantically consistent knowledge across different modalities, architectures, and learning objectives.
  2. The paper does not demonstrate that federated collaboration itself improves performance because no local-only or non-federated baseline is provided.
  3. Comparisons with representative heterogeneous federated learning methods are missing, making it difficult to assess the practical advantage of xHetMed-FL over existing approaches. For example,

[1] FedRolex: Model-Heterogeneous Federated Learning with Rolling Sub-Model Extraction (NeurIPS 2022); [2] MH-pFLID: Model Heterogeneous Personalized Federated Learning via Injection and Distillation for Medical Data Analysis (ICML 2024); [3] Bridging the Generalization Gap of Heterogeneous Federated Clients Using Generative Models (ICLR 2026).

Questions for authors
  1. How do the authors demonstrate that the shared projection learns semantically consistent and transferable knowledge across heterogeneous clients?
  2. Does federated collaboration actually improve each client compared with local-only or non-federated training?
  3. How much does the shared projection itself contribute compared with independently trained client-specific projections?
  4. How does xHetMed-FL compare with representative existing federated and heterogeneous federated learning methods? Does the shared projection consistently benefit all clients, or does federated aggregation cause negative transfer for some clients?
Reviewer_ZmuZOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes xHetMed-FL for federated representation learning across SPECT, chest X-ray, MRI, and CT+PET clients. Modality-specific 2D/3D encoders remain local, while a shared 128-dimensional projection is aggregated using sample-based weights, with FedProx regularization and local batch normalization. SPECT uses supervised binary classification, and the other clients optimize a cosine-similarity objective. A four-client, ten-round experiment reports 82.35% SPECT validation accuracy and 0.9177 ROC-AUC, near-unit cosine similarities for the self-supervised clients, and post-training Grad-CAM visualizations.

Critical strengths

The selective sharing design provides a common projection interface for 2D and 3D encoders while retaining modality-specific components and normalization locally. This is a sensible architectural starting point for accommodating heterogeneous inputs.

Critical weaknesses
  1. The rationale and expected benefit of cross-modal collaboration are insufficiently established. Sections 3.1–3.5 define the local objectives, but do not identify the transferable information connecting the four datasets or the task expected to benefit from their collaboration. Matching feature dimensions and sharing projection parameters do not establish semantic correspondence across modality-specific encoders. The near-unit cosine similarities in Table 5 and Appendix K measure agreement between views within each client, providing no direct evidence of cross-modal alignment or useful transfer. Consequently, the central claim of learning a useful shared representation across these modalities remains unsupported.

  2. The experimental design does not isolate the value of federation or the proposed components. Tables 4–6 and Appendix K report only the proposed configuration and its evolution across communication rounds. The submission provides no matched local-only comparison, no appropriate federated baseline comparison, and no client or component ablations to identify the contributions of the auxiliary modalities, projection sharing, or FedProx. Round-wise changes also include additional local training and cannot establish a collaboration benefit. The reported validation performance therefore leaves the central questions of whether federation helps and which design choices matter unanswered.

Questions for authors
  1. What transferable information is expected to connect these four datasets, and which task is intended to benefit? Please explain what evidence supports cross-modal alignment or useful transfer beyond the reported within-client cosine similarities.

  2. How does SPECT performance compare with matched local-only training and an appropriate parameter-sharing federated baseline? Please isolate the effects of auxiliary-client participation and FedProx through targeted ablations under comparable training budgets.

Reviewer_yED3OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes xHetMed-FL, a federated learning framework for heterogeneous medical imaging modalities. It employs modality-specific encoders and a shared 128-dimensional projection to enable knowledge sharing across clients with different architectures and learning objectives

Critical strengths
  1. The paper explores an interesting and practically relevant setting in federated medical imaging, where clients have different imaging modalities, model architectures, and learning objectives.
  2. The proposed framework allows heterogeneous clients to collaborate by sharing only projection parameters while preserving modality-specific components locally.
Critical weaknesses
  1. The motivation for using self-supervised learning on datasets with available annotations is unclear. More importantly, the learned representations are not evaluated on downstream tasks, making it difficult to assess whether they capture meaningful semantic information.

  2. High cosine similarity between two views of the same sample only demonstrates positive-pair alignment, without ensuring representation discriminability, intra-class compactness, or inter-class separability. Moreover, Appendix Figure 5 appears to highlight background regions rather than meaningful anatomical structures, raising concerns about the semantic quality of the learned representations.

  3. The proposed cosine similarity loss lacks explicit mechanisms to prevent representation collapse. More informative objectives, such as AE/MAE-based reconstruction or other self-supervised methods with anti-collapse mechanisms, should be considered. The authors should justify their choice of objective and provide comparisons with these alternatives.

  4. The contributions of key components, including the shared projection, FedProx, and FedBN, are not individually evaluated, making it difficult to assess their effectiveness.

  5. The paper lacks sensitivity analyses of key hyperparameters, such as the latent dimension and FedProx coefficient. Moreover, experiments are conducted with only one random seed, without reporting standard deviations or statistical significance.

Questions for authors

While the paper explores an interesting setting for heterogeneous medical imaging, I have two fundamental concerns. First, it is unclear whether sharing a common projection across substantially different modalities and learning objectives leads to meaningful knowledge transfer. Second, the necessity of federated learning is insufficiently justified, as the paper does not demonstrate any clear benefit of federated collaboration over independent local training. Together with the methodological and experimental weaknesses discussed above, these concerns undermine the paper's core contributions. Therefore, I recommend rejection.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.