acceptodds
Under review as a conference paper at ICLR 2027

Adaptive Decoupled Augmentation with Prototype Tuning for Multimodal Federated Learning with Missing Modalitie

Abstract

Automated medical report generation has the potential to improve radiologists' productivity. Recently, large language models have advanced report generation through their strong language modeling and generation capabilities. However, training these models in real-world scenarios encounters two primary obstacles: privacy requirements that restrict data centralization and the prevalence of incomplete image–report pairs across institutions. Although federated learning enables collaborative training without centralizing patient records, learning from clients with missing modalities requires appropriate local supervision and effective integration of heterogeneous updates. To address these challenges, we propose Adapt-MFL, an Adaptive Decoupled Augmentation with Prototype Tuning for Multimodal Federated Learning under missing modality conditions. We first introduce a trainable prototype dictionary as a semantic bridge between modalities. The shared prototypes are optimized through cross-modal assignment alignment on paired clients and provide learned semantic anchors for visual adaptation on image-only clients. On text-only clients, local prototype copies are further optimized jointly with decoder adapters, serving as adaptive surrogate prefixes for report generation. Additionally, we devise a modular decoupled aggregation scheme that separately integrates visual and decoder updates from their contributing clients, with dynamic weighting based on feature–prototype similarity. The global prototype dictionary is updated exclusively by paired clients, separating local text-prefix adaptation from shared cross-modal anchor learning. Finally, we apply low-rank adaptation to the visual encoder, cross-modal projection, and language decoder to enable parameter-efficient fine-tuning. Extensive experiments conducted on two publicly available medical datasets, IU X-Ray and MIMIC-CXR, demonstrate that the proposed method achieves superior performance compared to state-of-the-art frameworks.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
multimodal federated learning · medical report generation · missing modality· large language model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_4czxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Adapt-MFL, a prototype-guided federated learning framework for medical report generation under missing modality settings. The proposed framework introduces a learnable prototype dictionary as a shared semantic bridge across heterogeneous clients. Paired clients optimize cross-modal prototype alignment, image-only clients use fixed global prototypes for visual adaptation, and text-only clients optimize local prototype copies together with decoder adapters to provide surrogate visual prefixes for report generation. Experiments train and test on MIMIC-CXR and evaluate zero-shot on IU X-Ray, reporting gains in higher-order BLEU and ROUGE-L.

Critical strengths

1 The paper studies a challenging federated medical learning scenario where institutions have heterogeneous modality availability. This is more realistic than conventional federated learning assumptions requiring all clients to contain complete multimodal pairs.

2 The design assigns clear roles: each client type updates only the modules it can supervise, and the global prototypes are learned from paired data alone, which keeps text-side drift out of the shared anchors.

Critical weaknesses

1 The main concern is that the proposed framework appears to combine several existing ideas rather than introducing a fundamentally new learning principle. Prototype-based federated learning has been explored previously, and parameter-efficient federated adaptation has also been widely studied. While adapting these ideas to medical report generation with missing modalities is meaningful, the paper does not clearly identify what fundamental algorithmic insight differentiates Adapt-MFL from prior prototype-based multimodal federated learning methods.

2 The main results are hard to trust or compare. On IU X-Ray, the proposed method's F1 exceeds both its precision and recall, and the F1 values for PromptMRG and FedMRG exceed the mean of their precision and recall, which is not possible under standard averaging.

3 The experimental section demonstrates overall improvements, but it does not sufficiently validate why the proposed components are necessary. The current ablation only removes prototype tuning or decoupled aggregation as a whole, which does not isolate the contribution of individual design choices.

Questions for authors

1 What is the main algorithmic contribution of Adapt-MFL beyond existing prototype-based federated learning and missing-modality multimodal learning methods? Which specific component should be considered the fundamental new learning idea?

2 What semantic information do the learned prototypes capture?

3 Can the authors separate the effect of: modality-specific aggregation, dynamic prototype similarity weighting, and prototype dictionary learning? The current ablation does not fully isolate these contributions.

4 What averaging is used for the clinical metrics, and how can F1 on IU X-Ray exceed both precision and recall?

5 Figure 1 shows orthogonal and entropy regularization, Hungarian matching, and a contrastive loss that never appear in the text. Are these part of the final method?

2 comments
AuthorsOpenReview ↗

Response to Reviewer 4czx

We thank Reviewer 4czx for recognizing the practical motivation of heterogeneous-modality federated learning and the client role separation that keeps text-side drift out of global anchors. Below we respond to each question:

W1 & Q1: Fundamental algorithmic contribution beyond prototype FL and PEFT FL.
TL;DR: Encoder-only prototype FL (ProtoAlign-Enc, Table 2, p. 9) breaks down on generative MLLMs (36.97% Abn-12 F1, 23.30% "no acute" collapse) because autoregressive decoders can ignore encoder anchors; our contribution is an anti-bypass encoder-to-decoder prototype bridge paired with confidence-weighted decoupled aggregation (+4.97 Abn-12 F1, -23.4% relative boilerplate).

Earlier prototype FL methods (FedProto, PmcmFL, MFCPL) align encoder representations for discriminative classification. When implemented on Qwen3-VL-8B report generation with the same bipolar dictionary and unimodal losses (ProtoAlign-Enc, Sections 4.1–4.2, Tables 1–2, pp. 8–9), the autoregressive decoder ignores encoder anchors and falls back on raw patches and language priors, collapsing to "no acute" templates in 23.30% of test cases (599 FP No Finding, 36.97% Abn-12 F1). Our design addresses this decoder bypass through three coupled parts:

  1. A polarity-explicit bipolar dictionary (14×2×2×25614\times 2\times 2\times 256) with spatial visual queries and shifted-bigram text queries (Section 3.2, pp. 4–5);
  2. Dual encoder-to-decoder conditioning (14 continuous prefix slots + 96 calibrated diagnostic text slots) enforced against visual bypass via stochastic visual masking (pdropp_{\mathrm{drop}}) and counterfactual prefix-swapping margin regularization (Lcf\mathcal{L}_{\mathrm{cf}}, Section 3.3, pp. 5–6); and
  3. Pathway-specific compatibility aggregation (Section 3.4, Eqs. (8)–(11), p. 7) that keeps Ctxt\mathcal{C}_{\mathrm{txt}} dictionaries local.

W2 & Q4: Averaging method and standardized IU X-Ray F1.
TL;DR: Both MIMIC-CXR and IU X-Ray (N=3,748N=3{,}748) use official CheXpert corpus micro-averaging (Pre=TPTP+FP\mathrm{Pre}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP}}, Rec=TPTP+FN\mathrm{Rec}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}}, F1=2PRP+R\mathrm{F1}=\frac{2PR}{P+R}) and unweighted Macro-F1; every F1 value falls between Precision and Recall (Table 9, p. 16).

We fixed the earlier IU X-Ray metric calculation issue in Appendix A.6 / Table 9 (pp. 15–16):

  • Every reported F1 falls cleanly between Precision and Recall (e.g., Adapt-MFL: 17.24% Pre, 17.45% Rec, 17.34% F1; PromptMRG: 17.01% / 15.60% / 16.27%; FedMRG: 21.53% / 18.40% / 19.84%; full IU X-Ray table appears under Reviewer X2kD W4).
  • On IU X-Ray, Adapt-MFL achieves the top missing-modality Abn-12 F1 (17.34%) and Macro-F1 (10.94% / 15.65%), above pooled PromptMRG (16.27%) and ProtoAlign-Enc (16.42%). Notice also that paired MIMIC baselines PromptMRG and FedMRG show substantial cross-hospital boilerplate inflation (54.94% and 71.34%), whereas Adapt-MFL reduces boilerplate to 57.28% (-11.37 pts vs. ProtoAlign-Enc, -14.06 pts vs. FedMRG).

W3 & Q3: Separating modality-specific aggregation, dynamic αcr\alpha_c^r, and prototype learning.
TL;DR: Tables 2, 3, and 10 (pp. 9, 10, 17) isolate all three axes under identical settings, showing distinct contributions from each mechanism.

Axis Isolated (Tables 2, 3, 10)Controlled Variant vs. Adapt-MFL (Full)Observed Clinical Impact (pp. 9, 10, 17)
1. Modality AggregationPaired-Only, w/o Text-Only, w/o Image-Only, Hard ImageAdding Cimg+Ctxt\mathcal{C}_{\mathrm{img}}+\mathcal{C}_{\mathrm{txt}} lifts All-14 Macro (29.53% → 30.36%), Rare-6 (11.05% → 12.73%), FINDINGS% (85.5% → 87.3%), and cuts prior comparisons (40.2% → 37.0%); Hard Image collapses recall (23.93%)
2. Dynamic αcr\alpha_c^r Weightingw/o Compat. Weighting (αc=1\alpha_c=1)Disabling αcr\alpha_c^r drops Abn-12 F1 (41.94% → 41.05%), Abn-12 Macro (26.52% → 25.48%), and Rare-6 (12.73% → 11.73%)
3. Prototype & Bridge LearningProtoAlign-Enc (Table 2), w/o Verbal, w/o Div., w/o Align., w/o VisMask, w/o CFEncoder-only alignment leaves Abn-12 F1 at 36.97% (23.30% "No Acute"); removing verbal cues cuts Abn-12 Macro (25.14%); removing λdiv/λalign/pdrop/λcf\lambda_{\mathrm{div}}/\lambda_{\mathrm{align}}/p_{\mathrm{drop}}/\lambda_{\mathrm{cf}} erodes Abn-12 Macro (-0.90 to -1.24) and Rare-6 (-1.69 to -2.17)

Q2: What semantic information do the learned prototypes capture?
TL;DR: Zero-training probes (Appendix A.5 / Table 8, p. 15) verify polarity separation (14/14), affirmed-vs.-negated query discrimination (12/12), direct visual pathology AUROC (0.8357 Effusion, 0.7841 Edema), and counterfactual NLL sensitivity.

Specifically, Table 8 (p. 15) establishes four semantic properties:

  1. Opposite polarity centroids separate across all 14/14 findings (mean cosine -0.3622), while cross-disease axes stay near-orthogonal (∣cos⁡∣=0.2085\lvert\cos\rvert=0.2085) and within-polarity subtypes stay decorrelated (-0.0221).
  2. Shifted-bigram queries distinguish affirmed (0.7611) from negated (0.2270) statements across all 12/12 abnormal pathologies (+0.5342 mean probability margin).
  3. Prior to LLM decoding, visual prototype probabilities pl(zv)p_l(z_v) directly reach 0.8357 AUROC on Pleural Effusion, 0.7841 on Edema, 0.7778 on No Finding, and 0.6929 All-14 Macro-AUROC.
  4. Swapping study bridges increases salient-token NLL both unmasked (+0.0030) and visually masked (+0.0057), confirming decoder reliance on the bridge state.

Q5: Figure 1 alignment with main text.
TL;DR: Figure 1 (p. 4) now aligns box-by-box—across all modules, losses, and upload paths—with Sections 3.2–3.4 (pp. 4–7).

Thank you for pointing out the earlier mismatch between the overview diagram and main text. We updated Figure 1 (p. 4) to remove schematic elements absent from the final formulation (Hungarian matching and generic entropy terms), aligning every module, loss label (KL+NCE / Sup.+Div., Weighted NLL, Counterfactual Margin, Soft-Teacher KL + Prior Reg.), local dictionary isolation box, and upload path (ΔΦvis,ΔΘproto,ΔΦllm\Delta\Phi_{\mathrm{vis}}, \Delta\Theta_{\mathrm{proto}}, \Delta\Phi_{\mathrm{llm}}) with Section 3.

We thank Reviewer 4czx again for these helpful comments on positioning, metric verification, ablation isolation, and figure consistency, and kindly invite you to reconsider your Rating scores if these updates resolve your concerns.

Reviewer_Z2BYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates how to train medical report generation models in federated learning settings where participating institutions have incomplete image–report data. To address this challenge, the authors propose Adapt-MFL, a federated LLM framework specifically designed for medical report generation under missing-modality conditions, enabling collaborative learning across institutions with paired image–report data, image-only data, or report-only data.

Critical strengths

The paper demonstrates a commendably cautious approach to acknowledging its limitations.

Critical weaknesses

1.Limited realism of the assumed clinical setting. In routine clinical practice, medical images and their corresponding reports are generally generated and stored as paired records for the same examination. Therefore, the assumption that an entire institution contributes only images or only reports may not adequately reflect typical clinical workflows.

2.Insufficient analysis of the proposed method. The paper introduces multiple loss terms, but does not sufficiently demonstrate whether each loss achieves its intended objective or clarify its individual contribution to the overall performance. Furthermore, the effects of key hyperparameters on model performance are not systematically investigated. More comprehensive ablation studies and hyperparameter sensitivity analyses are needed to validate the proposed design choices.

Questions for authors

Please refer to the weaknesses listed above.

2 comments
AuthorsOpenReview ↗

Response to Reviewer Z2BY

We thank Reviewer Z2BY for noting our discussion of limitations (Section 5, p. 10) and for raising two important questions on clinical workflow realism and ablation/hyperparameter validation.

W1: Realism of institutional modality separation.
TL;DR: Although radiology exams originate as paired studies, export compliance, PHI scrubbing lag, legacy RIS/NLP archives, and hospital governance often separate image-only (Cimg\mathcal{C}_{\mathrm{img}}) and report-only (Ctxt\mathcal{C}_{\mathrm{txt}}) datasets across sites; incorporating both outperforms paired-only federation (Tables 3 & 10, pp. 10 & 17).

As detailed in Sections 1 (p. 1) and 5 (p. 10), modality separation arises in real clinical deployments for three practical reasons:

  1. Asymmetric export & PHI approval: PACS radiographs can be de-identified by stripping DICOM headers and cleared for research export well before unstructured free-text reports complete PHI scrubbing.
  2. Legacy RIS/NLP & imaging archives: Historical NLP report cohorts often lack linked DICOM pixel files, whereas screening or outpatient imaging archives frequently lack structured reports.
  3. Logical federation across sub-cohorts: Within multi-hospital consortia, sub-cohorts under single-modality governance naturally map onto logical Cimg\mathcal{C}_{\mathrm{img}} and Ctxt\mathcal{C}_{\mathrm{txt}} clients.

Empirically (Tables 3 & 10, pp. 10 & 17), adding Cimg\mathcal{C}_{\mathrm{img}} and Ctxt\mathcal{C}_{\mathrm{txt}} improves over Paired-Only (Cfull\mathcal{C}_{\mathrm{full}} only) on All-14 Macro-F1 (30.36% vs. 29.53%), Abn-12 Macro-F1 (26.52% vs. 25.67%), Rare-6 (12.73% vs. 11.05%), FINDINGS structural completeness (87.3% vs. 85.5%), and prior-comparison suppression (37.0% vs. 40.2%).

W2: Individual loss ablations & hyperparameter sensitivity.
TL;DR: We added a controlled 10-variant ablation suite (Tables 3 & 10, pp. 10 & 17) along with a γtxt=0.1,1.0\gamma_{\mathrm{txt}} = 0.1, 1.0 sensitivity study (Table 11, p. 17), showing that each component helps maintain balanced rare-disease detection while curbing normal-template collapse and frequent-class bias.

Under unified cap=2.0 settings (Section 4.3, pp. 9–10; Appendix A.7, pp. 17–18), each loss and design choice has a distinct effect:

Variant (Tables 3, 10, 11; pp. 10 & 17)Abn-12 Pre / Rec / F1 ↑Abn-12 Macro ↑All-14 Macro ↑Rare-6 ↑"No Ac."% ↓Prior Comp.% ↓Fracture F1 ↑
Adapt-MFL (Full, γtxt=0.1\gamma_{\mathrm{txt}}=0.1)51.49 / 35.37 / 41.9426.5230.3612.7317.837.011.1
Hard Image (Cimg\mathcal{C}_{\mathrm{img}} hard)51.36 / 23.93 / 32.6521.3425.2811.0030.032.16.7
w/o Verbal Prompts (Mprompt=0M_{\mathrm{prompt}}=0)49.84 / 34.52 / 40.7925.1429.2310.9519.137.92.7
w/o Compat. Weighting (αc=1\alpha_c=1)50.35 / 34.65 / 41.0525.4829.3911.7318.535.96.3
w/o Visual Masking (pdrop=0p_{\mathrm{drop}}=0)50.87 / 35.13 / 41.5625.2829.2810.5618.037.33.8
w/o Proto. Diversity (λdiv=0\lambda_{\mathrm{div}}=0)50.86 / 35.00 / 41.4625.4329.3811.0418.937.15.2
w/o Cross-Modal Align (λalign=0\lambda_{\mathrm{align}}=0)50.95 / 35.08 / 41.5525.6229.5610.8619.237.55.0
w/o Image-Only (skip Cimg\mathcal{C}_{\mathrm{img}})51.64 / 35.62 / 42.1625.6429.6010.5818.137.01.3
w/o Counterfactual (λcf=0\lambda_{\mathrm{cf}}=0)51.27 / 35.79 / 42.1525.6129.4911.0118.137.31.3
Unscaled Text (γtxt=1.0\gamma_{\mathrm{txt}}=1.0)51.46 / 36.01 / 42.3725.8629.8611.6718.626.86.3

Three main findings emerge across Tables 3, 10, and 11 (pp. 10 & 17):

  • Soft-teacher routing & αcr\alpha_c^r prevent collapse: Hard pseudo-labeling (Hard Image) drops recall from 35.37% to 23.93% and nearly doubles false-positive No Finding (487 → 962; "No Acute" 29.96%). Disabling αcr\alpha_c^r reduces Abn-12 Macro-F1 by 1.04 points (to 25.48%).
  • Verbal slots, diversity, alignment, and masking protect balanced detection: Removing verbal prompt slots (Mprompt=0M_{\mathrm{prompt}}=0) yields the weakest soft-routing scores (40.79% F1, 25.14% Macro-F1, 10.95% Rare-6). Turning off λdiv\lambda_{\mathrm{div}}, λalign\lambda_{\mathrm{align}}, or pdropp_{\mathrm{drop}} lowers Abn-12 Macro-F1 (25.28%–25.62%) and Rare-6 (10.56%–11.04%).
  • Prevalence-weighted Micro-F1 vs. rare-finding sensitivity: Skipping Cimg\mathcal{C}_{\mathrm{img}} or Lcf\mathcal{L}_{\mathrm{cf}} slightly raises Micro-F1 through frequent comorbidities (Edema, Cardiomegaly) but drops Fracture nearly ninefold (11.1% → 1.3%) and lowers Rare-6 (10.58% / 11.01%). Similarly, unscaled γtxt=1.0\gamma_{\mathrm{txt}}=1.0 (Table 11, p. 17) improves BLEU-1 (37.01%) and reduces prior comparisons (26.8%, 423 FP No Finding), but lowers ROUGE-L (31.96% → 31.79%), increases "No Acute" boilerplate (18.57%), lowers Abn-12 / All-14 / Rare-6 Macro-F1 (25.86 / 29.86 / 11.67), and degrades F1 on 8 of 12 pathologies (Fracture -4.8, Pleural Other -4.4, Pneumonia -2.7, Consolidation -2.3), making γtxt=0.1\gamma_{\mathrm{txt}}=0.1 a better overall clinical operating point.

We hope this clinical context and full ablation/sensitivity analysis address your Soundness concerns, and we would be happy to answer any follow-up questions. If resolved, we kindly ask you to reconsider your rating.

Reviewer_X2kDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes Adapt-MFL, which learns a shared prototype dictionary across paired clients by aligning the image and report prototype assignment distributions. Image-only clients anchor visual features to their nearest prototypes, whereas text-only clients use a retrieved prototype as a surrogate visual prefix for training the decoder. The server aggregates LoRA updates only over clients that trained each block, weighting unimodal clients by feature–prototype similarity. Experiments on MIMIC-CXR and zero-shot IU X-Ray demonstrate the effectiveness of the proposed method.

Critical strengths
  • Practically relevant missing-modality federated setting.

  • Clear role assignment for each client type.

  • The authors discuss limitations and metric caveats.

Critical weaknesses
  • I think the prototype bottleneck is too coarse. Each report is reduced to an argmax over only K=16 prototypes, and zyz_{y} is an order-invariant mean of frozen token embeddings that barely separates effusion from no effusion, so multi-label findings collapse into a 16-way code.

  • Text-only clients do not learn image-conditioned generation, so I think the authors' claim about that seems unreasonable. Their prefix is a single prototype retrieved directly from the target report. In contrast, paired training and inference feed multi-token visual inputs, so Eq. 14 averages decoder updates learned on two different input distributions.

  • Technical novelty is limited, the prototype bridge closely follows FedProto and MFCPL. In addition, in the aggregation step, the authors used standard FedAvg across participating clients.

  • Some experiments seem unfair. Adapt-MFL uses Qwen3-VL-8B while baselines use older backbones. In addition, IU X-Ray F1 exceeds both precision and recall, this makes it difficult to ensure the proposed method's superiority.

Questions for authors

Please see Weaknesses section.

4 comments
AuthorsOpenReview ↗

Response to Reviewer X2kD

We thank Reviewer X2kD for the detailed feedback and for recognizing our missing-modality federated setting, client role separation, and discussion of limitations and metric caveats.

W1: Prototype bottleneck expressiveness & order-invariant text mean.
TL;DR: Refining our prototype bridge with a 14×2×214\times 2\times 2 bipolar soft dictionary, spatial visual queries, and a shifted bigram layer addresses multi-label concurrency and clinical negation (Section 3.2, Eqs. (1)–(3), pp. 4–5; verified in Appendix A.5 / Table 8, pp. 14–15).

We agree that coarse pooling can blur concurrent findings and negation cues. Within the Adapt-MFL bridge, we introduced two targeted refinements (Section 3.2, Eqs. (1)–(3), pp. 4–5):

  1. Finding-level bipolar soft dictionary: Rather than compressing a study through a coarse argmax code, the dictionary allocates explicit negative (s=0s=0) and positive (s=1s=1) subspaces (M=2M=2 subtypes each, dp=256d_p=256) per CheXpert observation, blended via continuous soft gates glg_l and patient-specific residuals (Eq. (2), p. 5).
  2. Spatial & bigram-contextualized queries: To keep localized findings and negation modifiers coupled, 1414 normalized spatial queries attend over projected image patches V(x)V(x), while frozen report token embeddings pass through a lightweight shifted bigram layer, h~t=LN⁡(ht+0.5Wbiht−1)\tilde{h}_t = \operatorname{LN}(h_t + 0.5 W_{\mathrm{bi}} h_{t-1}), keeping negation cues (e.g., "no effusion") bound to the target pathology instead of being washed out by order-invariant pooling.
  3. Quantitative probe results (Table 8, p. 15): Zero-training probes verify polarity geometry, negation sensitivity, visual discriminability, and decoder reliance on the bridge:
Probe (Appendix A.5 / Table 8, p. 15)Measured Result in Revised Adapt-MFL
Dictionary Geometry (14×2×2×25614\times 2\times 2\times 256)Mean Pos–Neg centroid cosine: -0.3622 (14/14 negative, up to -0.4668 on Consolidation); Cross-disease ∣cos⁡∣\lvert\cos\rvert: 0.2085; Within-polarity subtype cosine: -0.0221
Affirmed vs. Negated Text Probe (Abn-12)Affirmed ("Moderate X is present"): 0.7611 vs. Negated ("No X is seen"): 0.2270 (Δ=+0.5342\Delta = \mathbf{+0.5342}, 12/12 separated; Pleural Effusion: 0.8660 vs. 0.1145)
Direct Visual Prototype AUROC (N=512N=512)All-14 Macro-AUROC: 0.6929, Abn-12: 0.6722 (Support Devices 0.8563, Pleural Effusion 0.8357, Edema 0.7841, No Finding 0.7778)
Bridge Counterfactual NLL (N=128N=128)Swapping prefixes raises salient-token NLL: +0.0030 unmasked (1.4051 → 1.4081), +0.0057 visually masked (1.4163 → 1.4220)

W2: Text-only client distribution mismatch, target leakage, and empirical contribution.
TL;DR: Paired and text-only sites feed the same bridge format (14 prefix + 96 prompt slots) with Gaussian logit perturbation, strict report filtering, local dictionary isolation, and γtxt=0.1\gamma_{\mathrm{txt}}=0.1 scaling (Eq. (11), p. 7), yielding net clinical and safety gains over skipping Ctxt\mathcal{C}_{\mathrm{txt}} (Tables 3 & 10, pp. 10 & 17).

We clarified both the architectural safeguards (Sections 3.3–3.4, pp. 6–7) and the isolated contribution of Ctxt\mathcal{C}_{\mathrm{txt}}:

  1. Unified bridge interface & noise regularization: Paired and text-only training both feed 14 continuous prefix tokens and 96 diagnostic prompt tokens into the decoder, while Ctxt\mathcal{C}_{\mathrm{txt}} adds Gaussian logit noise (ϵ∼N(0,0.252)\epsilon \sim \mathcal{N}(0, 0.25^2)) before generating pseudo-visual prefixes without mixing ground-truth labels.
  2. Quality filtering, local dictionary isolation, and scaling: Ctxt\mathcal{C}_{\mathrm{txt}} keeps only reports containing an explicit FINDINGS section and no prior-exam comparison phrasing, discards its locally adapted dictionary PcP_c after each local update (never uploaded to the server), and down-scales decoder increments via γtxt=0.1\gamma_{\mathrm{txt}}=0.1 (Eq. (11), p. 7).
  3. Controlled ablation of Ctxt\mathcal{C}_{\mathrm{txt}} (Tables 3 & 10, pp. 10 & 17): Comparing full Adapt-MFL against w/o Text-Only Clients (skip Ctxt\mathcal{C}_{\mathrm{txt}}) and Paired-Only (Cfull\mathcal{C}_{\mathrm{full}} only) under the same setup shows the net effect of filtered text-only clients:
Metric on MIMIC-CXR (N=2,730N=2{,}730)Paired-Only (Cfull\mathcal{C}_{\mathrm{full}})w/o Text-Only (skip Ctxt\mathcal{C}_{\mathrm{txt}})Adapt-MFL (Full)Net Effect of Adding Ctxt\mathcal{C}_{\mathrm{txt}}
BLEU-1 / BLEU-4 ↑35.37 / 13.4935.23 / 13.3435.70 / 13.37Higher fluency (+0.47 B1, +0.03 B4)
FINDINGS Completion ↑85.5%85.9%87.3%Stronger structure (+1.4 pts vs. w/o Ctxt\mathcal{C}_{\mathrm{txt}})
Prior Comparison Leakage ↓40.2%40.7%37.0%Lower hallucination (-3.7 pts)
"No Acute" Boilerplate ↓ / FP No Find. ↓17.8% / 48118.3% / 50617.8% / 487Fewer false-normal collapses (-19 FP NF)
Abn-12 Pre / Rec / Micro-F1 ↑50.86 / 35.37 / 41.7351.15 / 34.96 / 41.5451.49 / 35.37 / 41.94All 3 Abn-12 micro metrics improve
All-14 Micro-F1 / Macro-F1 ↑47.76 / 29.5347.72 / 30.3547.90 / 30.36Best overall All-14 Micro & Macro-F1
Fracture / Enl. CM / Pl. Other F1 ↑5.2 / 21.7 / 2.47.6 / 20.6 / 4.611.1 / 23.0 / 6.8Gains on hard structural findings

Full Adapt-MFL also improves over w/o Text-Only on Edema (44.0% vs. 41.8%) and No Finding (29.6% vs. 27.2%). While w/o Text-Only is slightly higher on Abn-12 Macro-F1 (26.61% vs. 26.52%, +0.09) and Rare-6 (13.02% vs. 12.73%, +0.29, mainly from Pneumothorax and Consolidation), adding quality-filtered Ctxt\mathcal{C}_{\mathrm{txt}} brings consistent gains in report structure, prior-comparison suppression, precision/recall, key visual findings, and top All-14 Macro-F1.

W3: Novelty beyond FedProto/MFCPL & modular vs. FedAvg aggregation.
TL;DR: Encoder-only prototype FL (ProtoAlign-Enc) still collapses to "no acute" templates (23.30%, 36.97% Abn-12 F1) on generative MLLMs; extending prototypes into an anti-bypass decoder bridge plus block-wise compatibility aggregation (Eqs. (8)–(11), p. 7) lifts Abn-12 F1 by +4.97.

  1. Why encoder-only prototype FL fails on generative MLLMs (Section 4.1–4.2, Tables 1–2, pp. 8–9): Because FedProto and MFCPL target classification rather than conditional sequence generation, we built a controlled Qwen3-VL-8B encoder-only baseline (ProtoAlign-Enc) with the same bipolar dictionary, soft-teacher distillation, text filtering, and decoupled aggregation, but without decoder conditioning. Even with aligned encoder prototypes, the autoregressive decoder ignores encoder anchors and falls back to "no acute" boilerplate in 23.30% of test studies (599 FP No Finding, 36.97% Abn-12 F1). Adding our anti-bypass encoder-to-decoder bridge raises Abn-12 F1 by +4.97 (41.94%), Abn-12 Macro-F1 by +3.75 (26.52%), and BLEU-4 by +1.42 (13.37%), while reducing boilerplate by 23.4% relative (17.84%, 487 FP No Finding).
  2. Modular confidence-weighted aggregation (Section 3.4, Eqs. (8)–(11), p. 7): Server aggregation differs from uniform FedAvg: Θproto\Theta_{\mathrm{proto}} aggregates over Cfull\mathcal{C}_{\mathrm{full}} (Eq. (9)), Φvis\Phi_{\mathrm{vis}} over Cfull∪Cimg\mathcal{C}_{\mathrm{full}} \cup \mathcal{C}_{\mathrm{img}} (Eq. (10)), and Φllm\Phi_{\mathrm{llm}} over Cfull∪Ctxt\mathcal{C}_{\mathrm{full}} \cup \mathcal{C}_{\mathrm{txt}} (Eq. (11)), weighted by prototype confidence margins αcr\alpha_c^r (Eq. (8)) and γtxt\gamma_{\mathrm{txt}}. Setting uniform weights (αc=1\alpha_c=1 in w/o Compat. Weighting, Table 3, p. 10) reduces Abn-12 F1 (41.94% → 41.05%) and Abn-12 Macro-F1 (26.52% → 25.48%).

W4: Unified backbone fairness & IU X-Ray metric calculation.
TL;DR: All seven systems are evaluated under the same Qwen3-VL-8B-Instruct backbone (Table 1, p. 8), and corpus-level CheXpert micro-averaging resolves the IU X-Ray F1 discrepancy (Table 9, p. 16).

  1. Unified baseline setup (Section 4.1, Table 1, pp. 7–8): All seven methods (Centralized, PromptMRG, FedAvg, FedMRG, FedMSplit, ProtoAlign-Enc, and Adapt-MFL) use the same Qwen3-VL-8B-Instruct (bfloat16, LoRA rL=32,αL=32r_{\mathrm{L}}=32, \alpha_{\mathrm{L}}=32) backbone, image resolution (448 px448\text{ px}), indication-aware prompt, and patient-level client split.
  2. Standardized IU X-Ray (N=3,748N=3{,}748) zero-shot evaluation (Appendix A.6 / Table 9, pp. 15–16): Computing corpus micro-precision, micro-recall, and micro-F1 (2PRP+R\frac{2PR}{P+R}) inside the official CheXpert container guarantees that every system's F1 falls between Precision and Recall:
IU X-Ray Zero-Shot (N=3,748N=3{,}748, Table 9, p. 16)Words"No Acute"% ↓Abn-12 Pre / Rec / F1 (Macro) ↑All-14 Micro (Macro) ↑
Centralized (Cfull\mathcal{C}_{\mathrm{full}} Pooled)38.852.9322.92 / 16.50 / 19.19 (11.32)35.43 (15.58)
PromptMRG (Cfull\mathcal{C}_{\mathrm{full}} Pooled)46.554.9417.01 / 15.60 / 16.27 (11.62)33.46 (16.10)
FedAvg (Cfull\mathcal{C}_{\mathrm{full}} only)41.358.2219.84 / 19.91 / 19.87 (11.77)32.83 (15.76)
FedMRG (Cfull\mathcal{C}_{\mathrm{full}} only)47.171.3421.53 / 18.40 / 19.84 (11.80)35.00 (16.53)
FedMSplit (Cfull+Cimg+Ctxt\mathcal{C}_{\mathrm{full}}+\mathcal{C}_{\mathrm{img}}+\mathcal{C}_{\mathrm{txt}})45.856.9114.44 / 17.88 / 15.98 (9.94)30.28 (14.56)
ProtoAlign-Enc (Cfull+Cimg+Ctxt\mathcal{C}_{\mathrm{full}}+\mathcal{C}_{\mathrm{img}}+\mathcal{C}_{\mathrm{txt}})40.768.6520.99 / 13.49 / 16.42 (10.76)36.54 (15.42)
Adapt-MFL (Cfull+Cimg+Ctxt\mathcal{C}_{\mathrm{full}}+\mathcal{C}_{\mathrm{img}}+\mathcal{C}_{\mathrm{txt}})45.757.2817.24 / 17.45 / 17.34 (10.94)32.57 (15.65)

Adapt-MFL achieves the top missing-modality Abn-12 F1 (17.34%), outperforming ProtoAlign-Enc (16.42%), FedMSplit (15.98%), and pooled PromptMRG (16.27%). It also leads missing-modality methods on Abn-12 / All-14 Macro-F1 (10.94% / 15.65%), while lowering "no acute" boilerplate by 11.37 percentage points relative to ProtoAlign-Enc and 14.06 points relative to FedMRG.

Summary of Presentation Improvements:
To make the revised manuscript easier to follow, we: (1) aligned Figure 1 (p. 4) with Sections 3.2–3.4 (pp. 4–7), removing unimplemented schematic blocks so every loss and upload path matches the text; (2) consolidated all hyperparameter and protocol definitions in Section 4.1 (p. 7); (3) added page-indexed diagnostic tables (Tables 3–11, pp. 10–17), including client aggregation shares (Table 6, p. 14) and qualitative examples (Table 7, p. 15); and (4) corrected IU X-Ray clinical averaging (Table 9, p. 16).

We hope these experiments and presentation fixes address your concerns, and we would be glad to clarify any remaining details. If satisfied, we kindly invite you to reconsider your Presentation, Soundness, and Overall Rating scores.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers

We thank Reviewers X2kD, Z2BY, and 4czx for their careful reading and constructive feedback. We appreciate their positive remarks on the clinical motivation of missing-modality federated learning, the parameter-role separation across paired, image-only, and text-only clients, and our discussion of limitations and metric caveats. In the revised manuscript (key additions highlighted in light blue), we address the issues raised through focused refinements within our Adapt-MFL framework:

  1. Unified-Backbone Benchmark & Standardized Clinical Averaging (Tables 1, 2, 4, 5, 9; pp. 8–9, 13–14, 16): To remove backbone or preprocessing confounding, all seven compared systems (Centralized, PromptMRG, FedAvg, FedMRG, FedMSplit, encoder-only ProtoAlign-Enc, and Adapt-MFL) now share the same Qwen3-VL-8B-Instruct (bfloat16, LoRA rL=32r_{\mathrm{L}}=32, 448 px448\text{ px}) configuration. All MIMIC-CXR (N=2,730N=2{,}730) and external zero-shot IU X-Ray (N=3,748N=3{,}748 frontal studies) evaluations compute corpus micro-precision, micro-recall, micro-F1 (2PRP+R\frac{2PR}{P+R}), and unweighted macro-F1 inside the official CheXpert labeler container, resolving the earlier IU X-Ray averaging anomaly.
  2. Polarity-Aware Dictionary Refinement & Semantic Probes (Section 3.2, pp. 4–5; Appendix A.5 / Table 8, pp. 14–15): Within the Adapt-MFL bridge, we structured the prototype dictionary as a finding-level 14×2×2×25614\times 2\times 2\times 256 bipolar dictionary with spatial visual queries and a lightweight shifted bigram text layer to preserve clinical negation, and added zero-training semantic, negation, visual-AUROC, and counterfactual NLL probes.
  3. Controlled 10-Variant Ablations & γtxt\gamma_{\mathrm{txt}} Sensitivity (Section 4.3 / Table 3, pp. 9–10; Appendix A.7 / Tables 10–11, pp. 17–18): Under unified cap=2.0 settings, we isolated each unimodal group, compatibility weight (αcr\alpha_c^r), diagnostic prompt cue (MpromptM_{\mathrm{prompt}}), visual masking (pdropp_{\mathrm{drop}}), loss term (Lcf,Ldiv,Lalign\mathcal{L}_{\mathrm{cf}}, \mathcal{L}_{\mathrm{div}}, \mathcal{L}_{\mathrm{align}}), and text-decoder scale (γtxt=0.1,1.0\gamma_{\mathrm{txt}} = 0.1, 1.0), tracking class-balanced Abn-12 / All-14 / Rare-6 Macro-F1, "No Acute" template collapse, and ungrounded prior-comparison leakage.
  4. Figure 1 & Section 3 Alignment (p. 4, pp. 4–7; Appendix A.3, pp. 13–14): We updated Figure 1 to match Sections 3.2–3.4 and documented block-wise LoRA communication along with local isolation of text-only dictionaries.

Revision Map (Quick Index for AC & Reviewers)

  • Method & Figure Alignment: Clarified Sections 3.2–3.4 (pp. 4–7, Eqs. (1)–(11)) and updated Figure 1 (p. 4) so every module block, training objective, and parameter upload path matches the text.
  • Unified-Backbone Controlled Verification & Standardized Evaluation: Evaluated all seven compared systems under the same Qwen3-VL-8B-Instruct (bfloat16, LoRA rL=32r_{\mathrm{L}}=32, 448 px448\text{ px}) backbone with standardized corpus CheXpert evaluation; updated Sections 4.1–4.2 (pp. 7–9), Tables 1–2 (pp. 8–9), and Appendices A.1–A.3 (Tables 4–6, pp. 13–14).
  • Controlled 10-Variant Component Ablations & γtxt\gamma_{\mathrm{txt}} Sensitivity: Isolated every unimodal client group, bridge cue, loss term, compatibility weight, and text-decoder scaling factor (γtxt=0.1,1.0\gamma_{\mathrm{txt}} = 0.1, 1.0) under unified settings (Section 4.3 / Table 3, pp. 9–10; Appendix A.7 / Tables 10–11, pp. 17–18).
  • Prototype Semantic, Negation & Counterfactual Probes: Added zero-training probes testing bipolar centroid separation, shifted-bigram negation discrimination, visual AUROC, and masked bridge sensitivity (Appendix A.5 / Table 8, pp. 14–15).
  • Standardized Zero-Shot IU X-Ray Evaluation (N=3,748N=3{,}748): Re-evaluated all seven systems under official CheXpert corpus micro/macro averaging with cross-hospital prevalence-shift analysis (Appendix A.6 / Table 9, pp. 15–16).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.