- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes FArms, a framework for federated multimodal learning with missing modalities. The motivation is that most existing methods communicate the entire model every round, coupling the missing-modality handler to the downstream task and limiting extendibility. The main contributions are (1) a multi-stage federated schedule that separates the missing-modality handler from the downstream task, communicating only one block per round, (2) a missing-modality handler built from sequential simulating arms, trained without labels, and (3) theoretical results on the module count, inference cost, and communication payload. Experiments cover multi-label classification on MM-IMDb, regression on CMU-MOSI, and classification on Sleep-EDF, against 2 centralized and 3 federated baselines. FArms achieves the best F-micro in every configuration on MM-IMDb and CMU-MOSI and wins or ties on Sleep-EDF, and the two-stage schedule halves the per-round payload compared to joint training.
Critical strengths
-
The paper is easy to follow, Figure 1 clearly illustrates the multi-stage schedule, and the code is available for reproducibility.
-
FArms matches or exceeds state-of-the-art federated and centralized baselines in the large majority of configurations across multi-label classification, regression, and a five-modality setting.
Critical weaknesses
-
The main comparison does not give the federated baselines a fair training budget. All methods use 10 rounds, and Appendix E notes that FLEX-MoE and PEPSY were originally trained for 100 to 1000 rounds and have not stabilized here.
-
Tables 1 and 2 are hard to read because they are resized to a much smaller font compared with the font size used in Tables 3 and 4.
Questions for authors
Can you explain how F-micro is computed on MM-IMDb? FArms reaches 0.94 in 100-image-100-text and 0.93 in 100-image-20-text, while the strongest baseline PmcmFL drops from 0.66 to 0.51 under the same settings.
4 comments
Table: Mean F-micro and F-macro over three runs of PEPSY and FLEX-MoE on MM-IMDb as a function of the number of communication rounds.
| 100-image-20-text | 20-image-100-text | complex-10-45-45 | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| F-Micro | F-Macro | F-Micro | F-Macro | F-Micro | F-Macro | |||||||
| Rounds | PEPSY | FLEX-MoE | PEPSY | FLEX-MoE | PEPSY | FLEX-MoE | PEPSY | FLEX-MoE | PEPSY | FLEX-MoE | PEPSY | FLEX-MoE |
| 40 | 0.528 | 0.554 | 0.251 | 0.341 | 0.542 | 0.580 | 0.276 | 0.410 | 0.487 | 0.556 | 0.216 | 0.369 |
| 50 | 0.535 | 0.559 | 0.267 | 0.356 | 0.551 | 0.587 | 0.299 | 0.422 | 0.504 | 0.558 | 0.238 | 0.382 |
| 60 | 0.537 | 0.561 | 0.275 | 0.366 | 0.560 | 0.588 | 0.320 | 0.428 | 0.517 | 0.561 | 0.248 | 0.386 |
| 70 | 0.535 | 0.562 | 0.275 | 0.367 | 0.563 | 0.590 | 0.333 | 0.433 | 0.525 | 0.567 | 0.271 | 0.389 |
| 80 | 0.537 | 0.562 | 0.276 | 0.374 | 0.566 | 0.593 | 0.341 | 0.437 | 0.536 | 0.564 | 0.282 | 0.390 |
| 90 | 0.537 | 0.567 | 0.276 | 0.378 | 0.570 | 0.594 | 0.347 | 0.438 | 0.544 | 0.564 | 0.292 | 0.390 |
| 100 | 0.537 | 0.563 | 0.276 | 0.375 | 0.569 | 0.594 | 0.354 | 0.438 | 0.545 | 0.563 | 0.300 | 0.390 |
| 200 | 0.537 | 0.567 | 0.276 | 0.378 | 0.575 | 0.595 | 0.366 | 0.441 | 0.566 | 0.560 | 0.359 | 0.390 |
| 300 | 0.537 | 0.567 | 0.276 | 0.378 | 0.576 | 0.595 | 0.380 | 0.441 | 0.568 | 0.560 | 0.372 | 0.390 |
| 400 | 0.537 | 0.567 | 0.276 | 0.378 | 0.578 | 0.595 | 0.387 | 0.441 | 0.569 | 0.560 | 0.376 | 0.390 |
| 500 | 0.537 | 0.567 | 0.276 | 0.378 | 0.580 | 0.595 | 0.392 | 0.441 | 0.569 | 0.560 | 0.376 | 0.390 |
Can you explain how F-micro is computed on MM-IMDb? FArms reaches 0.94 in 100-image-100-text and 0.93 in 100-image-20-text, while the strongest baseline PmcmFL drops from 0.66 to 0.51 under the same settings.
For your first part of question, Multi-label F1 metrics are computed as follows:
Let denote the ground-truth label and denote the thresholded prediction, for sample index from 1 to and class index from 1 to . The prediction is defined as when the sigmoid output exceeds 0.5, and otherwise. The per-class confusion counts are:
F1-micro aggregates the true/false positives and negatives globally across all classes before computing a single F1 score:
F1-macro computes an F1 score independently for each class and then averages them without weighting by class frequency:
F1-micro is dominated by frequent genres, since every prediction instance contributes equally to the pooled counts, whereas F1-macro weights every genre equally regardless of its frequency, making it more sensitive to performance on rare classes.
For the second part of your question we confirm that for the MM-IMDb, the proposed FArms obtains more favourable results even when text availability falls from 100 % to 20 %. FArms’ multi-stage schedule first optimises only the sequential simulating arms (the missing-modality handler) for a few rounds, freezes them, and then optimises only the task head. Consequently the downstream classifier always receives a complete set of embeddings, whether a modality is present or reconstructed. This explicit, label-free reconstruction is what keeps F-micro almost flat when text availability falls from 100 % to 20 %.
Thanks for the rebuttal. My concerns have been addressed. For the technical part, I have no more concerns.
After the committee conducts the ethics review, I would like to adjust my score accordingly during the AC-reviewer discussion period.
In the name of God,
Dear reviewer MGTM,
Thank you for reviewing our paper,
We bring here, everything added to paper, except for your comment about table size, you may check the new version of the paper.
Critical Weaknesses: Tables 1 and 2 are hard to read because they are resized to a much smaller font compared with the font size used in Tables 3 and 4. We increased the font size of both tables in the new version of the submitted paper.
Critical Weaknesses: The main comparison does not give the federated baselines a fair training budget. All methods use 10 rounds, and Appendix E notes that FLEX-MoE and PEPSY were originally trained for 100 to 1000 rounds and have not stabilized here.
We thank the reviewer for this point. Although reaching a good-quality solution in fewer communication rounds is widely regarded as a major advantage [1][2][3][4], we also trained FLEX-MoE and PEPSY for up to 500 rounds (3 runs each) on three MM-IMDb configurations. As the figures below show, both largely plateau: FLEX-MoE by round 200 (by round 90 on most metrics) and PEPSY by round 300-400 (by round 60 on 100-image-20-text). Even at 500 rounds they remain well below FArms trained for only 10 rounds. The only metric still creeping upward is PEPSY’s F-Macro on 20-image-100-text (0.387 to 0.392 between rounds 400 and 500), and it remains 0.049 below FLEX-MoE and far below FArms. We have already included the figures in the new version of the submitted paper. In addition, for your reference, we also bring the the tables in a separate official comment right after this.
[1] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas.Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pp. 1273–1282. Pmlr, 2017.
[2] Jie Zhang, Chen Chen, Bo Li, Lingjuan Lyu, Shuang Wu, Shouhong Ding, Chunhua Shen, and Chao Wu. Dense: Data-free one-shot federated learning. Advances in Neural Information Processing Systems, 35:21414–21428, 2022b.
[3] Yiqun Diao, Qinbin Li, and Bingsheng He. Towards addressing label skews in one-shot federated learning. In The Eleventh International Conference on Learning Representations, 2023.
[4] Zhenheng Tang, Yonggang Zhang, Peijie Dong, Yiu-ming Cheung, Amelie C Zhou, Bo Han, and Xiaowen Chu. Fusefl: One-shot federated learning through the lens of causality with progressive model fusion. Advances in neural information processing systems, 37:28393–28429, 2024.
100-image-20-text, F-Micro (row step 0.005; * = PEPSY, o = FLEX-MoE)
100-image-20-text, F-Macro (row step 0.02; * = PEPSY, o = FLEX-MoE)
20-image-100-text, F-Micro (row step 0.003; * = PEPSY, o = FLEX-MoE)
20-image-100-text, F-Macro (row step 0.01; * = PEPSY, o = FLEX-MoE)
complex-10-45-45, F-Micro (row step 0.003; * = PEPSY, o = FLEX-MoE)
complex-10-45-45, F-Macro (row step 0.01; * = PEPSY, o = FLEX-MoE)