Adaptive Decoupled Augmentation with Prototype Tuning for Multimodal Federated Learning with Missing Modalitie

Automated medical report generation has the potential to improve radiologists' productivity. Recently, large language models have advanced report generation through their strong language modeling and generation capabilities. However, training these models in real-world scenarios encounters two primary obstacles: privacy requirements that restrict data centralization and the prevalence of incomplete image–report pairs across institutions. Although federated learning enables collaborative training without centralizing patient records, learning from clients with missing modalities requires appropriate local supervision and effective integration of heterogeneous updates. To address these challenges, we propose Adapt-MFL, an Adaptive Decoupled Augmentation with Prototype Tuning for Multimodal Federated Learning under missing modality conditions. We first introduce a trainable prototype dictionary as a semantic bridge between modalities. The shared prototypes are optimized through cross-modal assignment alignment on paired clients and provide learned semantic anchors for visual adaptation on image-only clients. On text-only clients, local prototype copies are further optimized jointly with decoder adapters, serving as adaptive surrogate prefixes for report generation. Additionally, we devise a modular decoupled aggregation scheme that separately integrates visual and decoder updates from their contributing clients, with dynamic weighting based on feature–prototype similarity. The global prototype dictionary is updated exclusively by paired clients, separating local text-prefix adaptation from shared cross-modal anchor learning. Finally, we apply low-rank adaptation to the visual encoder, cross-modal projection, and language decoder to enable parameter-efficient fine-tuning. Extensive experiments conducted on two publicly available medical datasets, IU X-Ray and MIMIC-CXR, demonstrate that the proposed method achieves superior performance compared to state-of-the-art frameworks.