- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Adapt-MFL, a prototype-guided federated learning framework for medical report generation under missing modality settings. The proposed framework introduces a learnable prototype dictionary as a shared semantic bridge across heterogeneous clients. Paired clients optimize cross-modal prototype alignment, image-only clients use fixed global prototypes for visual adaptation, and text-only clients optimize local prototype copies together with decoder adapters to provide surrogate visual prefixes for report generation. Experiments train and test on MIMIC-CXR and evaluate zero-shot on IU X-Ray, reporting gains in higher-order BLEU and ROUGE-L.
Critical strengths
1 The paper studies a challenging federated medical learning scenario where institutions have heterogeneous modality availability. This is more realistic than conventional federated learning assumptions requiring all clients to contain complete multimodal pairs.
2 The design assigns clear roles: each client type updates only the modules it can supervise, and the global prototypes are learned from paired data alone, which keeps text-side drift out of the shared anchors.
Critical weaknesses
1 The main concern is that the proposed framework appears to combine several existing ideas rather than introducing a fundamentally new learning principle. Prototype-based federated learning has been explored previously, and parameter-efficient federated adaptation has also been widely studied. While adapting these ideas to medical report generation with missing modalities is meaningful, the paper does not clearly identify what fundamental algorithmic insight differentiates Adapt-MFL from prior prototype-based multimodal federated learning methods.
2 The main results are hard to trust or compare. On IU X-Ray, the proposed method's F1 exceeds both its precision and recall, and the F1 values for PromptMRG and FedMRG exceed the mean of their precision and recall, which is not possible under standard averaging.
3 The experimental section demonstrates overall improvements, but it does not sufficiently validate why the proposed components are necessary. The current ablation only removes prototype tuning or decoupled aggregation as a whole, which does not isolate the contribution of individual design choices.
Questions for authors
1 What is the main algorithmic contribution of Adapt-MFL beyond existing prototype-based federated learning and missing-modality multimodal learning methods? Which specific component should be considered the fundamental new learning idea?
2 What semantic information do the learned prototypes capture?
3 Can the authors separate the effect of: modality-specific aggregation, dynamic prototype similarity weighting, and prototype dictionary learning? The current ablation does not fully isolate these contributions.
4 What averaging is used for the clinical metrics, and how can F1 on IU X-Ray exceed both precision and recall?
5 Figure 1 shows orthogonal and entropy regularization, Hungarian matching, and a contrastive loss that never appear in the text. Are these part of the final method?
2 comments
W3 & Q3: Separating modality-specific aggregation, dynamic , and prototype learning.
TL;DR: Tables 2, 3, and 10 (pp. 9, 10, 17) isolate all three axes under identical settings, showing distinct contributions from each mechanism.
| Axis Isolated (Tables 2, 3, 10) | Controlled Variant vs. Adapt-MFL (Full) | Observed Clinical Impact (pp. 9, 10, 17) |
|---|---|---|
| 1. Modality Aggregation | Paired-Only, w/o Text-Only, w/o Image-Only, Hard Image | Adding lifts All-14 Macro (29.53% → 30.36%), Rare-6 (11.05% → 12.73%), FINDINGS% (85.5% → 87.3%), and cuts prior comparisons (40.2% → 37.0%); Hard Image collapses recall (23.93%) |
| 2. Dynamic Weighting | w/o Compat. Weighting () | Disabling drops Abn-12 F1 (41.94% → 41.05%), Abn-12 Macro (26.52% → 25.48%), and Rare-6 (12.73% → 11.73%) |
| 3. Prototype & Bridge Learning | ProtoAlign-Enc (Table 2), w/o Verbal, w/o Div., w/o Align., w/o VisMask, w/o CF | Encoder-only alignment leaves Abn-12 F1 at 36.97% (23.30% "No Acute"); removing verbal cues cuts Abn-12 Macro (25.14%); removing erodes Abn-12 Macro (-0.90 to -1.24) and Rare-6 (-1.69 to -2.17) |
Q2: What semantic information do the learned prototypes capture?
TL;DR: Zero-training probes (Appendix A.5 / Table 8, p. 15) verify polarity separation (14/14), affirmed-vs.-negated query discrimination (12/12), direct visual pathology AUROC (0.8357 Effusion, 0.7841 Edema), and counterfactual NLL sensitivity.
Specifically, Table 8 (p. 15) establishes four semantic properties:
- Opposite polarity centroids separate across all 14/14 findings (mean cosine -0.3622), while cross-disease axes stay near-orthogonal () and within-polarity subtypes stay decorrelated (-0.0221).
- Shifted-bigram queries distinguish affirmed (0.7611) from negated (0.2270) statements across all 12/12 abnormal pathologies (+0.5342 mean probability margin).
- Prior to LLM decoding, visual prototype probabilities directly reach 0.8357 AUROC on Pleural Effusion, 0.7841 on Edema, 0.7778 on No Finding, and 0.6929 All-14 Macro-AUROC.
- Swapping study bridges increases salient-token NLL both unmasked (+0.0030) and visually masked (+0.0057), confirming decoder reliance on the bridge state.
Q5: Figure 1 alignment with main text.
TL;DR: Figure 1 (p. 4) now aligns box-by-box—across all modules, losses, and upload paths—with Sections 3.2–3.4 (pp. 4–7).
Thank you for pointing out the earlier mismatch between the overview diagram and main text. We updated Figure 1 (p. 4) to remove schematic elements absent from the final formulation (Hungarian matching and generic entropy terms), aligning every module, loss label (KL+NCE / Sup.+Div., Weighted NLL, Counterfactual Margin, Soft-Teacher KL + Prior Reg.), local dictionary isolation box, and upload path () with Section 3.
We thank Reviewer 4czx again for these helpful comments on positioning, metric verification, ablation isolation, and figure consistency, and kindly invite you to reconsider your Rating scores if these updates resolve your concerns.
Response to Reviewer 4czx
We thank Reviewer 4czx for recognizing the practical motivation of heterogeneous-modality federated learning and the client role separation that keeps text-side drift out of global anchors. Below we respond to each question:
W1 & Q1: Fundamental algorithmic contribution beyond prototype FL and PEFT FL.
TL;DR: Encoder-only prototype FL (ProtoAlign-Enc,
Table 2, p. 9) breaks down on generative MLLMs (36.97% Abn-12 F1, 23.30%"no acute"collapse) because autoregressive decoders can ignore encoder anchors; our contribution is an anti-bypass encoder-to-decoder prototype bridge paired with confidence-weighted decoupled aggregation (+4.97 Abn-12 F1, -23.4% relative boilerplate).Earlier prototype FL methods (
FedProto,PmcmFL,MFCPL) align encoder representations for discriminative classification. When implemented onQwen3-VL-8Breport generation with the same bipolar dictionary and unimodal losses (ProtoAlign-Enc, Sections 4.1–4.2, Tables 1–2,pp. 8–9), the autoregressive decoder ignores encoder anchors and falls back on raw patches and language priors, collapsing to"no acute"templates in 23.30% of test cases (599 FP No Finding, 36.97% Abn-12 F1). Our design addresses this decoder bypass through three coupled parts:Section 3.2, pp. 4–5);14continuous prefix slots +96calibrated diagnostic text slots) enforced against visual bypass via stochastic visual masking (pdrop) and counterfactual prefix-swapping margin regularization (Lcf,Section 3.3, pp. 5–6); andSection 3.4, Eqs. (8)–(11), p. 7) that keeps Ctxt dictionaries local.W2 & Q4: Averaging method and standardized IU X-Ray F1.
TL;DR: Both MIMIC-CXR and IU X-Ray (N=3,748) use official CheXpert corpus micro-averaging (Pre=TP+FPTP, Rec=TP+FNTP, F1=P+R2PR) and unweighted
Macro-F1; every F1 value falls between Precision and Recall (Table 9,p. 16).We fixed the earlier IU X-Ray metric calculation issue in Appendix A.6 / Table 9 (
pp. 15–16):