acceptodds
Under review as a conference paper at ICLR 2027

Extendible Multi-Stage Federated Multi-Modal Learning with Missing Modalities

Abstract

In federated multi-modal learning, communicating the entire model every round increases communication cost and, more importantly, limits how easily the frame- work can be extended, even though handling missing modalities, optimising the downstream task, and, in some settings, training the encoders are each essential parts of the pipeline. Rather than a single jointly trained model, this paper proposes a multi-stage federated learning view in which the component that handles missing modalities is separated from the downstream task: the earliest rounds are devoted to optimising the missing-modality handler alone, with only that component communicated between server and clients, while the remaining rounds are devoted to optimising the downstream task alone, with only that component communicated instead. When pre-trained encoders are not available, this view extends naturally to a third stage in which the initial rounds are devoted to training the encoders themselves. Separately, this paper proposes the missing-modality handler itself, which reconstructs each missing modality from whichever modalities are present. The resulting framework is supported by both theoretical and experimental results. Being task-agnostic, the handler applies directly to regression, classification, and multi-label classification alike, and its F1-micro performance exceeds that of state-of-the-art baselines in the majority of evaluated cases. Code is available at https://github.com/for-reviewers-arz/FArms_wArms/tree/main.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
multimodal learning, federated learning, missing modalities

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_MGTMOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes FArms, a framework for federated multimodal learning with missing modalities. The motivation is that most existing methods communicate the entire model every round, coupling the missing-modality handler to the downstream task and limiting extendibility. The main contributions are (1) a multi-stage federated schedule that separates the missing-modality handler from the downstream task, communicating only one block per round, (2) a missing-modality handler built from sequential simulating arms, trained without labels, and (3) theoretical results on the module count, inference cost, and communication payload. Experiments cover multi-label classification on MM-IMDb, regression on CMU-MOSI, and classification on Sleep-EDF, against 2 centralized and 3 federated baselines. FArms achieves the best F-micro in every configuration on MM-IMDb and CMU-MOSI and wins or ties on Sleep-EDF, and the two-stage schedule halves the per-round payload compared to joint training.

Critical strengths
  • The paper is easy to follow, Figure 1 clearly illustrates the multi-stage schedule, and the code is available for reproducibility.

  • FArms matches or exceeds state-of-the-art federated and centralized baselines in the large majority of configurations across multi-label classification, regression, and a five-modality setting.

Critical weaknesses
  • The main comparison does not give the federated baselines a fair training budget. All methods use 10 rounds, and Appendix E notes that FLEX-MoE and PEPSY were originally trained for 100 to 1000 rounds and have not stabilized here.

  • Tables 1 and 2 are hard to read because they are resized to a much smaller font compared with the font size used in Tables 3 and 4.

Questions for authors

Can you explain how F-micro is computed on MM-IMDb? FArms reaches 0.94 in 100-image-100-text and 0.93 in 100-image-20-text, while the strongest baseline PmcmFL drops from 0.66 to 0.51 under the same settings.

4 comments

In the name of God,

Dear reviewer MGTM,

Thank you for reviewing our paper,

We bring here, everything added to paper, except for your comment about table size, you may check the new version of the paper.

Critical Weaknesses: Tables 1 and 2 are hard to read because they are resized to a much smaller font compared with the font size used in Tables 3 and 4. We increased the font size of both tables in the new version of the submitted paper.

Critical Weaknesses: The main comparison does not give the federated baselines a fair training budget. All methods use 10 rounds, and Appendix E notes that FLEX-MoE and PEPSY were originally trained for 100 to 1000 rounds and have not stabilized here.

We thank the reviewer for this point. Although reaching a good-quality solution in fewer communication rounds is widely regarded as a major advantage [1][2][3][4], we also trained FLEX-MoE and PEPSY for up to 500 rounds (3 runs each) on three MM-IMDb configurations. As the figures below show, both largely plateau: FLEX-MoE by round 200 (by round 90 on most metrics) and PEPSY by round 300-400 (by round 60 on 100-image-20-text). Even at 500 rounds they remain well below FArms trained for only 10 rounds. The only metric still creeping upward is PEPSY’s F-Macro on 20-image-100-text (0.387 to 0.392 between rounds 400 and 500), and it remains 0.049 below FLEX-MoE and far below FArms. We have already included the figures in the new version of the submitted paper. In addition, for your reference, we also bring the the tables in a separate official comment right after this.

[1] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas.Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pp. 1273–1282. Pmlr, 2017.

[2] Jie Zhang, Chen Chen, Bo Li, Lingjuan Lyu, Shuang Wu, Shouhong Ding, Chunhua Shen, and Chao Wu. Dense: Data-free one-shot federated learning. Advances in Neural Information Processing Systems, 35:21414–21428, 2022b.

[3] Yiqun Diao, Qinbin Li, and Bingsheng He. Towards addressing label skews in one-shot federated learning. In The Eleventh International Conference on Learning Representations, 2023.

[4] Zhenheng Tang, Yonggang Zhang, Peijie Dong, Yiu-ming Cheung, Amelie C Zhou, Bo Han, and Xiaowen Chu. Fusefl: One-shot federated learning through the lens of causality with progressive model fusion. Advances in neural information processing systems, 37:28393–28429, 2024.

100-image-20-text, F-Micro (row step 0.005; * = PEPSY, o = FLEX-MoE)

0.567 |            o         o         o         o
0.562 |  o
0.557 |
0.552 |
0.547 |
0.542 |
0.537 |  *         *         *         *         *
      +--+---------+---------+---------+---------+
        100       200       300       400       500
                          Rounds

100-image-20-text, F-Macro (row step 0.02; * = PEPSY, o = FLEX-MoE)

 0.38 |  o         o         o         o         o
 0.36 |
 0.34 |
 0.32 |
 0.30 |
 0.28 |  *         *         *         *         *
      +--+---------+---------+---------+---------+
        100       200       300       400       500
                          Rounds

20-image-100-text, F-Micro (row step 0.003; * = PEPSY, o = FLEX-MoE)

0.595 |  o         o         o         o         o
0.592 |
0.589 |
0.586 |
0.583 |
0.580 |                                          *
0.577 |                      *         *
0.574 |            *
0.571 |
0.568 |  *
      +--+---------+---------+---------+---------+
        100       200       300       400       500
                          Rounds

20-image-100-text, F-Macro (row step 0.01; * = PEPSY, o = FLEX-MoE)

 0.44 |  o         o         o         o         o
 0.43 |
 0.42 |
 0.41 |
 0.40 |
 0.39 |                                *         *
 0.38 |                      *
 0.37 |            *
 0.36 |
 0.35 |  *
      +--+---------+---------+---------+---------+
        100       200       300       400       500
                          Rounds

complex-10-45-45, F-Micro (row step 0.003; * = PEPSY, o = FLEX-MoE)

0.569 |                      *         *         *
0.566 |            *
0.563 |  o
0.560 |            o         o         o         o
0.557 |
0.554 |
0.551 |
0.548 |
0.545 |  *
      +--+---------+---------+---------+---------+
        100       200       300       400       500
                          Rounds

complex-10-45-45, F-Macro (row step 0.01; * = PEPSY, o = FLEX-MoE)

 0.39 |  o         o         o         o         o
 0.38 |                                *         *
 0.37 |                      *
 0.36 |            *
 0.35 |
 0.34 |
 0.33 |
 0.32 |
 0.31 |
 0.30 |  *
      +--+---------+---------+---------+---------+
        100       200       300       400       500
                          Rounds

Table: Mean F-micro and F-macro over three runs of PEPSY and FLEX-MoE on MM-IMDb as a function of the number of communication rounds.

100-image-20-text20-image-100-textcomplex-10-45-45
F-MicroF-MacroF-MicroF-MacroF-MicroF-Macro
RoundsPEPSYFLEX-MoEPEPSYFLEX-MoEPEPSYFLEX-MoEPEPSYFLEX-MoEPEPSYFLEX-MoEPEPSYFLEX-MoE
400.5280.5540.2510.3410.5420.5800.2760.4100.4870.5560.2160.369
500.5350.5590.2670.3560.5510.5870.2990.4220.5040.5580.2380.382
600.5370.5610.2750.3660.5600.5880.3200.4280.5170.5610.2480.386
700.5350.5620.2750.3670.5630.5900.3330.4330.5250.5670.2710.389
800.5370.5620.2760.3740.5660.5930.3410.4370.5360.5640.2820.390
900.5370.5670.2760.3780.5700.5940.3470.4380.5440.5640.2920.390
1000.5370.5630.2760.3750.5690.5940.3540.4380.5450.5630.3000.390
2000.5370.5670.2760.3780.5750.5950.3660.4410.5660.5600.3590.390
3000.5370.5670.2760.3780.5760.5950.3800.4410.5680.5600.3720.390
4000.5370.5670.2760.3780.5780.5950.3870.4410.5690.5600.3760.390
5000.5370.5670.2760.3780.5800.5950.3920.4410.5690.5600.3760.390

Can you explain how F-micro is computed on MM-IMDb? FArms reaches 0.94 in 100-image-100-text and 0.93 in 100-image-20-text, while the strongest baseline PmcmFL drops from 0.66 to 0.51 under the same settings.

For your first part of question, Multi-label F1 metrics are computed as follows:

Let yi,c∈{0,1}y_{i,c} \in \{0,1\} denote the ground-truth label and y^i,c∈{0,1}ŷ_{i,c} \in \{0,1\} denote the thresholded prediction, for sample index ii from 1 to NN and class index cc from 1 to CC. The prediction is defined as y^i,c=1ŷ_{i,c} = 1 when the sigmoid output exceeds 0.5, and y^i,c=0ŷ_{i,c} = 0 otherwise. The per-class confusion counts are:

TPc=∑i=1N1[yi,c=1,y^i,c=1]TP_c = \sum_{i=1}^{N} 1[y_{i,c}=1, ŷ_{i,c}=1]

FPc=∑i=1N1[yi,c=0,y^i,c=1]FP_c = \sum_{i=1}^{N} 1[y_{i,c}=0, ŷ_{i,c}=1]

FNc=∑i=1N1[yi,c=1,y^i,c=0]FN_c = \sum_{i=1}^{N} 1[y_{i,c}=1, ŷ_{i,c}=0]

F1-micro aggregates the true/false positives and negatives globally across all classes before computing a single F1 score:

F1micro=2∑c=1CTPc2∑c=1CTPc+∑c=1CFPc+∑c=1CFNcF1_{micro} = \frac{2\sum_{c=1}^{C} TP_c}{2\sum_{c=1}^{C} TP_c + \sum_{c=1}^{C} FP_c + \sum_{c=1}^{C} FN_c}

F1-macro computes an F1 score independently for each class and then averages them without weighting by class frequency:

F1c=2TPc2TPc+FPc+FNcF1_c = \frac{2 TP_c}{2 TP_c + FP_c + FN_c}

F1macro=1C∑c=1CF1cF1_{macro} = \frac{1}{C}\sum_{c=1}^{C} F1_c

F1-micro is dominated by frequent genres, since every prediction instance contributes equally to the pooled counts, whereas F1-macro weights every genre equally regardless of its frequency, making it more sensitive to performance on rare classes.

For the second part of your question we confirm that for the MM-IMDb, the proposed FArms obtains more favourable results even when text availability falls from 100 % to 20 %. FArms’ multi-stage schedule first optimises only the sequential simulating arms (the missing-modality handler) for a few rounds, freezes them, and then optimises only the task head. Consequently the downstream classifier always receives a complete set of embeddings, whether a modality is present or reconstructed. This explicit, label-free reconstruction is what keeps F-micro almost flat when text availability falls from 100 % to 20 %.

Thanks for the rebuttal. My concerns have been addressed. For the technical part, I have no more concerns.

After the committee conducts the ethics review, I would like to adjust my score accordingly during the AC-reviewer discussion period.

Reviewer_oxwHOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose a multi-stage federated learning approach called FArms by decoupling the missing-modality handler from the task head, transmitting only one active parameter block per round. For missing modality imputation, FArms uses a sequential simulating arms architecture requiring 3K-4 modules for K modalities. Across multiple datasets like MM-IMDb, CMU-MOSI, and Sleep-EDF, FArms converges within just four communication rounds, delivering pattern-independent inference complexity and state-of-the-art F1F1-micro performance under severe, heterogeneous missing scenarios.

Critical strengths
  • Multi stage approach separating the missing modality handling from the task head.
  • Label-free reconstruction objective and sequential simulating arms (with 3K-4 parameters) for the missing-modality handler.
  • State of the art performance across diverse tasks including multi-label genre classification, sentiment classification and physiological signal classification.
  • Robust performance under extreme scenarios involving severe missing ratios (80%).
Critical weaknesses
  • The label-free reconstruction loss evaluates to zero whenever a sample contains a single available modality.
  • The missing modality handler is frozen globally at RsimR_{sim}. Once frozen, the handler cannot adapt or fine-tune during downstream task adaptation.
  • The imputation relies on a fixed randomized permutation ordering and reconstructs missing entries left to right.
  • In datasets with severe imbalance, there is a significant gap between micro-F1 and macro-F1 for MM-IMDb.
  • In severe missingness regime, where primary modalities like text are scarce, co-occurring modalities needed to train anchor simulators are scarce resulting in under-supervision in Stage 1.
Questions for authors
  • How sensitive is the reconstruction accuracy to the choice of permutation ordering ?
  • How to determine a balanced value of RsimR_{sim} for a balanced stage 1 to stage 2 transition ?
  • How can the current system be adapted when a large fraction of federated clients hold single-modality samples ?
  • Is there any sufficient empirical evidence to show that the frozen stage 1 handler can be transferred to a new downstream task without retraining ?
4 comments

In the name of God,

Dear Reviewer oxwH,

Thank you for reviewing our paper. We bring every results, we have added to paper, here. Just, we start with your "Questions For Authors" section, then "Critical Weaknesses".

How sensitive is the reconstruction accuracy to the choice of permutation ordering ?

Thank you for this question. We ran the requested experiment and added it to the revised paper.

Experiment. We repeated the Sleep-EDF experiment (K=5K=5, 30 clients, 10% participation, 80% of samples affected by missingness) with five random modality orderings. The first modality in each ordering is the anchor. Each cell averages 3 runs. We report downstream accuracy as an end-to-end measure of how well the missing modalities are completed.

Table: Effect of the modality permutation on the accuracy of FArms on Sleep-EDF (K=5K=5), with the same setting as the main comparison table. Each row is a different random order of the modalities (the first one is the anchor). Averages over three runs.

PermutationIID, pm=0.2p_m=0.2IID, pm=0.4p_m=0.4IID, pm=0.6p_m=0.6IID, pm=0.8p_m=0.8non-IID, pm=0.2p_m=0.2non-IID, pm=0.4p_m=0.4non-IID, pm=0.6p_m=0.6non-IID, pm=0.8p_m=0.8
permutation$_1$0.430.430.440.420.410.400.400.40
permutation$_2$0.420.430.430.400.430.420.420.40
permutation$_3$0.420.420.430.410.420.420.410.40
permutation$_4$0.410.420.410.410.420.400.400.40
permutation$_5$0.430.430.410.420.410.410.400.40

Findings.

  1. Sensitivity is limited. Across all eight settings, the spread between the best and worst of the five random orderings is 0.00–0.03 accuracy. It is usually 0.01–0.02. At pm=0.8p_m=0.8 in the non-IID setting, all orderings give 0.40. Performance does not collapse for any ordering.

  2. Even arbitrary orderings stay well above most baselines. In the IID setting, every ordering (0.40–0.44) is above PmcmFL and FLEX-MoE (≤ 0.36) in the main comparison table. The same holds against PEPSY, whose accuracy falls to 0.23–0.27 for pm≥0.4p_m \geq 0.4. In the non-IID setting, every ordering remains above PmcmFL and FLEX-MoE (≤ 0.34). Against PEPSY (0.42–0.43) the orderings are comparable. The weakest ones are about 0.01–0.03 lower in some cells.

  3. No single ordering is universally best. Permutation$_1$ is strongest in the IID setting and permutation$_2$ in the non-IID setting. The ranking changes with the setting, so a fixed universal ordering is not needed.

  4. How to choose the ordering in practice.

    • When domain knowledge exists, it can guide the choice. For example, on CMU-MOSI one may expect text–video–audio, with the strongest modality first, to work well.
    • When no such prior exists, the ordering can be selected from a small number of random candidates using a validation split, after a few rounds. Several random orderings already reach accuracy within about one to two points of the best one.

We have included this table and discussion in the revised paper (paragraph "Effect of the modality permutation" and the corresponding table).

How can the current system be adapted when a large fraction of federated clients hold single-modality samples ?

Still it can use the fraction of clients which have pairs of modalities. In simulation for R_sim of your comment, we show that the arms (missing modalities handlers) can be trained in a few initial rounds, so when just a few number of clients (small fraction) have pairs of modalities, then their trained parameter updates are distributed among all clients, and after a few number of initial rounds, all clients utilized the advantages of existing pairs, not just client containing that pairs. In Complex-30-35-35, or more severely in Complex-20-4-40 or even more severely in Complex-10-45-45, we have very similar situation, while for EDF-Sleep dataset, it is exactly the case while where absent modalities are repeated for all samples in a specific client. So, as other stages, stage 1 is federated, so supervision from the clients holding multiple modalities reaches all clients through aggregation. Even when pairs are concentrated in a small fraction of clients, their updates shape the global Θ\Theta, which is then frozen and shared.

How to determine a balanced value of RsimR_{\mathrm{sim}}.

Stopping federated training early is supported in the literature [1][2] which can be also utilized here,

[1] Lee et al., "Recurrent early exits for federated learning with heterogeneous clients" (ICML 2024)

[2] Sefidgaran et al., "Lessons from Generalization Error Analysis of Federated Learning: You May Communicate Less Often!" (ICML 2024)

In addition, we ran an ablation on CMU-MOSI sever-missing regimes (100-20-20, 20-100-20, 20-20-100 and the complex pattern) with the same setting as the main experiments, for total budgets R=5R=5 (Rsim=1..4R_{\mathrm{sim}}=1..4) and R=10R=10 (Rsim=1..8R_{\mathrm{sim}}=1..8), three runs per value. Entries are mean F-micro / F-macro.

R=5R=5

RsimR_{\mathrm{sim}}100-20-2020-100-2020-20-100complex
10.810 / 0.8090.568 / 0.5130.658 / 0.6530.696 / 0.693
20.808 / 0.8070.517 / 0.4550.660 / 0.6580.714 / 0.704
30.786 / 0.7850.589 / 0.4750.645 / 0.6430.672 / 0.660
40.715 / 0.7110.574 / 0.4440.642 / 0.6390.636 / 0.620

R=10R=10

RsimR_{\mathrm{sim}}100-20-2020-100-2020-20-100complex
10.809 / 0.8080.592 / 0.5370.670 / 0.6650.713 / 0.707
20.795 / 0.7930.577 / 0.5250.677 / 0.6740.711 / 0.706
30.800 / 0.7960.545 / 0.4970.668 / 0.6630.725 / 0.719
40.801 / 0.8000.582 / 0.5240.678 / 0.6720.711 / 0.704
50.800 / 0.7980.585 / 0.5350.679 / 0.6740.710 / 0.703
60.807 / 0.8060.545 / 0.4740.663 / 0.6610.717 / 0.709
70.805 / 0.8030.567 / 0.4530.658 / 0.6490.717 / 0.713
80.783 / 0.7780.539 / 0.4590.639 / 0.6270.687 / 0.675

(3) The results suggest that (a) accuracy is flat while the task stage keeps at least half of the rounds: for R=10R=10, Rsim∈{1,…,5}R_{\mathrm{sim}}\in\{1,\dots,5\} stay within about 1.5 points of the best value in three of four regimes (the fourth is within noise); (b) accuracy drops only when one or two task rounds remain (R=5R=5 with Rsim≥3R_{\mathrm{sim}}\ge3, R=10R=10 with Rsim=8R_{\mathrm{sim}}=8); (c) RsimR_{\mathrm{sim}} of about 2020% of RR is enough (Rsim=1R_{\mathrm{sim}}=1 for R=5R=5, Rsim=2R_{\mathrm{sim}}=2 for R=10R=10, our main setting); and (d) with R=10R=10 any Rsim∈{2,3,4}R_{\mathrm{sim}}\in\{2,3,4\} is indistinguishable (mean F-micro 0.684--0.693). We will therefore state the rule: keep at least half of the rounds, and at least three, for the task stage, and use Rsim≈0.2RR_{\mathrm{sim}}\approx0.2R (2--4 rounds for R=10R=10). The study uses three runs and CMU-MOSI only, so we claim a broad safe range.

Is there any sufficient empirical evidence to show that the frozen stage 1 handler can be transferred to a new downstream task without retraining ?

In all experiments of this paper for FArms, except for joint ones in ablation studies, parameter updates of arms (missing handlers) are exchanges in R_sim stage, and after that the arms are frozen. This strategy in addition to having extendibility, lower communication cost and stated advantages in abstract, is comparable, even in many case is more accurate than other SOTA. In addition, in existing ablation studies, it also is compared with joint one in which arms are not frozen, and can be updated during training the model. The results are introduced in table "Two-stage versus joint optimization on CMU-MOSI" and table "Component ablation on CMU-MOSI under three regimes.". So if your question is:

If your question is that can the trained/frozen arms be re-used in different task? It is beyond the scope of this paper. Clients have a model (which is combination of some sub-models) for a specific type of final task. While the proposed strategy for arms (missing modalities handler) is general/reusable/independent from the final task, can be used and have been already used for different type of final tasks, the aim of this paper is not to construct pre-trained missing modality handler with weights.

But, if you mean that is stage 1 handler transferred to a new downstream task without retraining ? We prefer to not use word 'transfer'. Why? After R_sim, each client just use its own arm (all clients have the identical copy of weights for their arms) to simulate the feature embeddings of the missing modalities, without updating their parameters. So, the arms parameters remain frozen after R_sim, but for the specific task which the model is trained.

Or, if your question is as after stage-1, arms are frozen, is this strategy comparable with SOTA or joint one in which whole of the model parameters are updated? This paper answers yes, Then I can refer you to existing experiments including competitions with the SOTA and ablation studies in the stated tables (table "Two-stage versus joint optimization on CMU-MOSI" and table "Component ablation on CMU-MOSI under three regimes.").

"The label-free reconstruction loss evaluates to zero whenever a sample contains a single available modality."

We thank the reviewer. The observation is correct and stated in the paper: Lsim=0\mathcal{L}_{\mathrm{sim}}=0 whenever ∣A∣=1|\mathcal{A}|=1, since a single observed modality yields no usable target. This is intentional, and it does not make the optimisation ill-defined or leave such samples unused. Single-modality samples add nothing to the arms' gradient, but they are used in the final task optimisation.

  1. How optimisation proceeds
  • Stage 1 (r≤Rsimr\le R_{\mathrm{sim}}): only Θ\Theta is optimised on Lsim\mathcal{L}_{\mathrm{sim}}. Single-modality samples add nothing to the arms' gradient. Samples with ∣A∣≥2|\mathcal{A}|\ge2 supply all the supervision, and because stage 1 is federated, it reaches every client through aggregation, even clients holding only single-modality samples.
  • Stage 2 (r>Rsimr>R_{\mathrm{sim}}): the arms are frozen globally. Every sample, including single-modality ones, is completed by the frozen arms, and the completed embeddings are used to optimise Φ\Phi on Ltask\mathcal{L}_{\mathrm{task}} (Remark "A single global imputation map").

So these samples are used for the downstream task, which is where the arms' output matters, and they are simply not used to fit the arms.

  1. Empirical evidence: complex-10-45-45 on MM-IMDb has 90% single-modality samples, and 100-image-20-text has 80%. FArms keeps F-micro at 0.93 in both (Table 1).

"In datasets with severe imbalance, there is a significant gap between micro-F1 and macro-F1 for MM-IMDb."

The gap is a property of the MM-IMDb label distribution, not of FArms, and we report both metrics because of it.

Dataset statistics (Appendix B).

  • Drama appears in 13,967 of 25,959 titles (53.8%), while the rarest genre occurs once.
  • The mean imbalance ratio is 922.7, far above the 1.5 conventionally used to flag imbalance.
  • Four genres together account for only 71 samples across all splits, so a reliable per-genre F1 is not attainable for them. Yet macro-F1 weights each of them as much as Drama.

A baseline with no model at all shows the same gap. A frequency-only predictor that ignores the input and always outputs the kk most frequent genres reaches micro-F1 of at most 0.39 but macro-F1 below 0.08 for every kk.

The metrics answer different questions. Micro-F1 measures how many genre assignments are recovered, and macro-F1 measures how uniformly this holds over a vocabulary where most entries are nearly empty. Both are defined in Appendix A.

The conclusions agree across both metrics. FArms is best on F-micro in all configurations and best on F-macro in five of six MM-IMDb columns (PmcmFL is slightly ahead on the fully available column). It is best on both metrics in every CMU-MOSI column. The gap for the baselines is often larger in relative terms.

"The handler is frozen globally at RsimR_{sim} and cannot adapt or fine-tune during downstream adaptation."

Freezing is a deliberate design choice, and we tested it against the alternative.

Downstream adaptation still happens. After RsimR_{sim}, the final fusion and the task head (Φ\Phi) are trained on the task loss. Only the imputation map Θ\Theta is fixed.

Empirical comparison with joint optimization (Table "Two-stage versus joint optimization on CMU-MOSI").

  • Two-stage F-micro is 0.811 / 0.584 / 0.670 / 0.741 against joint 0.820 / 0.607 / 0.645 / 0.732.
  • Each protocol is ahead in two of four configurations, and no gap exceeds a few points.
  • The payload per client per round is halved (29.48 MB vs 58.95 MB).
  • We acknowledge that joint training is slightly ahead on 100-20-20 and 20-100-20.

Component ablation (Table "Component ablation on CMU-MOSI under three regimes"). Both joint variants, simulation-heavy (α=2\alpha{=}2) and task-heavy (α=0.5\alpha{=}0.5), have lower F-micro than the full two-stage model in all three regimes. For example, 0.4545 and 0.5591 vs 0.5700 on 20-text-100-audio-20-video, and 0.5977 and 0.6545 vs 0.7100 on the complex pattern. These ablations are single runs, so we treat them as supporting evidence only.

Freezing has a stated benefit (Remark 2). After the freeze, every client completes samples with the same deterministic map. Stage 2 is then an ordinary federated problem on a stationary feature space, so the aggregator can be swapped and its assumptions apply unchanged. Under joint training the feature map drifts with every aggregation, which adds a second source of client disagreement.

Freezing early costs little. The RsimR_{sim} ablation shows the handler converges quickly. With R=10R=10, any Rsim∈{1,…,5}R_{sim}\in\{1,\dots,5\} stays within about 1.5 points of the best value, and accuracy drops only when one or two task rounds remain.

Fine-tuning is compatible with the framework. The block-schedule view (Extendibility, Sec. 4.3) allows an additional stage in which Θ\Theta is lightly fine-tuned. If the data distribution shifts, stage 1 can be re-run. We do not claim this is needed or beneficial, and we leave it for future work.

Reviewer_1dutOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes FArms, a scalable multi-stage federated multimodal learning framework designed to handle missing modalities. By adopting a fixed modality ordering and Sequential Simulating Arms, the method progressively reconstructs missing modalities using previously completed prefix features, requiring only 3K−43K-4 modules to handle different modality availability combinations. Furthermore, the paper separates modality completion from downstream task optimization into different stages, communicating only the parameters of the corresponding modules in each round to reduce communication overhead. Experiments on MM-IMDb, CMU-MOSI, and Sleep-EDF demonstrate the effectiveness of the proposed method across various tasks and missing-modality patterns.

Critical strengths
  1. Novel and Scalable Missing-Modality Reconstruction. The paper introduces a unified missing-modality reconstruction approach based on fixed modality ordering and completed prefix features, eliminating the need to design separate models for different modality combinations. By requiring only (3K-4) modules, this design offers methodological novelty and good structural scalability.
  2. Communication-Efficient Federated Learning. The multi-stage training strategy decouples missing-modality reconstruction from downstream task optimization, communicating only the parameters relevant to the current stage in each round. Experiments show that this approach reduces communication costs by approximately 50% compared to joint training with the same architecture, while maintaining comparable predictive performance and achieving competitive results across various tasks and missing-modality configurations.
Critical weaknesses
  1. Insufficient Analysis of Modality Ordering. Sequential Simulating Arms relies on a randomly selected and fixed modality ordering, which may affect information utilization and reconstruction quality. The paper lacks experimental analysis of sensitivity to modality ordering and does not sufficiently discuss the potential accumulation of errors during sequential reconstruction.
  2. Limited Fairness of Baseline Comparisons. The experiments uniformly adopt 10 rounds of federated training, while Appendix E indicates that FLEX-MoE and PEPSY use substantially more training rounds in their original studies. Although this setting highlights the advantages of FArms under a limited communication budget, the lack of comparisons after sufficient convergence limits the assessment of its final performance advantages.
Questions for authors
  1. Section 3 uses an exponential moving average (EMA) of reconstruction errors to select the Anchor Simulator. When certain modality pairs rarely co-occur in client data, how are the corresponding (\rho_j) values initialized and maintained? Could such sparse co-observations affect the reliability of anchor selection?
  2. When the first modality is missing, the anchor can be generated from a later available modality (M_{j^*}), which may subsequently also serve as a supervision target for the reconstruction loss. In this case, the prediction process may indirectly access information from the target modality through the anchor. Could the authors clarify how this potential information shortcut is handled in the implementation and whether it affects the validity of reconstruction training?
4 comments

In the name of God,

Dear reviewer 1dut,

Thank you for reviewing our paper. We bring every results, we have add to paper, here.

Critical Weaknesses: Insufficient Analysis of Modality Ordering . . .

We thank the reviewer for this comment. We ran the suggested sensitivity experiment, and we also address error accumulation below.

1. How the ordering is chosen

The ordering is fixed once and shared by all samples at training and inference. It can be chosen in two ways:

  • With domain knowledge: a good ordering can sometimes be anticipated. For example, on CMU-MOSI one may expect text–video–audio, with the strongest modality first, to work well.
  • Without a prior: the ordering can be selected from a small number of random permutations using validation data.

2. New experiment: sensitivity to the modality permutation

We repeated the Sleep-EDF experiment (K=5K=5, the largest modality count in the paper and hence the longest sequential chain) with five different random modality orderings, each averaged over 3 runs. The setting is identical to the main comparison table, and the first modality of each ordering is the anchor. This table is included in the new version of the paper.

Table: Effect of the modality permutation on the accuracy of FArms on Sleep-EDF (K=5K=5). Averages over three runs.

PermutationIID, pm=0.2p_m=0.2IID, pm=0.4p_m=0.4IID, pm=0.6p_m=0.6IID, pm=0.8p_m=0.8non-IID, pm=0.2p_m=0.2non-IID, pm=0.4p_m=0.4non-IID, pm=0.6p_m=0.6non-IID, pm=0.8p_m=0.8
permutation$_1$0.430.430.440.420.410.400.400.40
permutation$_2$0.420.430.430.400.430.420.420.40
permutation$_3$0.420.420.430.410.420.420.410.40
permutation$_4$0.410.420.410.410.420.400.400.40
permutation$_5$0.430.430.410.420.410.410.400.40

Observations:

  • Sensitivity is modest. Across the five orderings, accuracy varies by at most 3 points in any column (0–2 points in most columns), and several orderings reach accuracy within about one point of the FArms values in the main table.
  • The ordering matters somewhat, so selection is worthwhile. No single permutation is best everywhere, and a poor ordering costs a few points. This is why we recommend choosing among a few candidates on a validation split when no domain prior exists.
  • Every ordering remains competitive. All five orderings outperform PmcmFL and FLEX-MoE in every column (at most 0.36 for these baselines). In the IID regimes they also outperform PEPSY. In the non-IID regimes PEPSY is comparable and slightly higher in some columns, which is consistent with the main table, where FArms and PEPSY are statistically indistinguishable in three of the four non-IID settings.

3. Error accumulation in sequential reconstruction

  • Observed modalities are never replaced. At every position, an observed modality is carried forward as its real embedding rather than its estimate. Simulated embeddings enter the prefix only at genuinely missing positions, so errors can propagate only through absent modalities and cannot compound through observed ones.
  • Training matches inference conditions. The loss at position kk backpropagates through the whole completed prefix, including simulated entries. The arms are therefore trained on prefixes that contain simulated embeddings, as at inference.
  • Empirical evidence on Sleep-EDF. FArms's accuracy is essentially flat as the fraction of absent modalities pmp_m grows from 0.2 to 0.8 (0.43–0.44 in the IID setting, main comparison table). Strong compounding along the chain would produce a clear degradation as more positions are simulated. This is empirical evidence on our benchmarks, not a worst-case guarantee, and we state it as such.

We hope these additions address the reviewer's concern, and we are happy to provide further analysis if helpful.

Critical Weaknesses: Limited Fairness of Baseline Comparisons . . .

Although reaching a good-quality solution in fewer communication rounds is widely regarded as a major advantage [1][2][3][4], we also trained FLEX-MoE and PEPSY for up to 500 rounds (3 runs each) on three MM-IMDb configurations. As Figures in the bottom of this official comment show, both largely plateau, FLEX-MoE by round 200 and PEPSY by round 300–400, and even at 500 rounds they remain well below FArms trained for only 10 rounds. The extended results have been added to the revised paper.

[1] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas.Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pp. 1273–1282. Pmlr, 2017.

[2] Jie Zhang, Chen Chen, Bo Li, Lingjuan Lyu, Shuang Wu, Shouhong Ding, Chunhua Shen, and Chao Wu. Dense: Data-free one-shot federated learning. Advances in Neural Information Processing Systems, 35:21414–21428, 2022b.

[3] Yiqun Diao, Qinbin Li, and Bingsheng He. Towards addressing label skews in one-shot federated learning. In The Eleventh International Conference on Learning Representations, 2023.

[4] Zhenheng Tang, Yonggang Zhang, Peijie Dong, Yiu-ming Cheung, Amelie C Zhou, Bo Han, and Xiaowen Chu. Fusefl: One-shot federated learning through the lens of causality with progressive model fusion. Advances in neural information processing systems, 37:28393–28429, 2024.

100-image-20-text, F-Micro (row step 0.005; * = PEPSY, o = FLEX-MoE)

0.567 |            o         o         o         o
0.562 |  o
0.557 |
0.552 |
0.547 |
0.542 |
0.537 |  *         *         *         *         *
      +--+---------+---------+---------+---------+
        100       200       300       400       500
                          Rounds

100-image-20-text, F-Macro (row step 0.02; * = PEPSY, o = FLEX-MoE)

 0.38 |  o         o         o         o         o
 0.36 |
 0.34 |
 0.32 |
 0.30 |
 0.28 |  *         *         *         *         *
      +--+---------+---------+---------+---------+
        100       200       300       400       500
                          Rounds

20-image-100-text, F-Micro (row step 0.003; * = PEPSY, o = FLEX-MoE)

0.595 |  o         o         o         o         o
0.592 |
0.589 |
0.586 |
0.583 |
0.580 |                                          *
0.577 |                      *         *
0.574 |            *
0.571 |
0.568 |  *
      +--+---------+---------+---------+---------+
        100       200       300       400       500
                          Rounds

20-image-100-text, F-Macro (row step 0.01; * = PEPSY, o = FLEX-MoE)

 0.44 |  o         o         o         o         o
 0.43 |
 0.42 |
 0.41 |
 0.40 |
 0.39 |                                *         *
 0.38 |                      *
 0.37 |            *
 0.36 |
 0.35 |  *
      +--+---------+---------+---------+---------+
        100       200       300       400       500
                          Rounds

complex-10-45-45, F-Micro (row step 0.003; * = PEPSY, o = FLEX-MoE)

0.569 |                      *         *         *
0.566 |            *
0.563 |  o
0.560 |            o         o         o         o
0.557 |
0.554 |
0.551 |
0.548 |
0.545 |  *
      +--+---------+---------+---------+---------+
        100       200       300       400       500
                          Rounds

complex-10-45-45, F-Macro (row step 0.01; * = PEPSY, o = FLEX-MoE)

 0.39 |  o         o         o         o         o
 0.38 |                                *         *
 0.37 |                      *
 0.36 |            *
 0.35 |
 0.34 |
 0.33 |
 0.32 |
 0.31 |
 0.30 |  *
      +--+---------+---------+---------+---------+
        100       200       300       400       500
                          Rounds

Questions For Authors:


Q1: When certain modality pairs rarely co-occur in client data, how are the corresponding (\rho_j) values initialized and maintained? Could such sparse co-observations affect the reliability of anchor selection?

we thank the reviewer, and we explicitly bring the following in new version of submitted paper:

The target is never an input of its own prediction: when 1∉A1 \notin A, the anchor C1=Sj⋆→1(Mj⋆′)C_1 = S_{j^{\star} \to 1}(M_{j^{\star}}') depends on Mj⋆′M_{j^{\star}}', so the term at k=j⋆k = j^{\star} is omitted from LsimL_{sim} for that sample, although the real embedding is still carried forward as Cj⋆=Mj⋆′C_{j^{\star}} = M_{j^{\star}}'. For every other supervised position the term is read directly off the main sweep, so a single forward sweep still completes the sample and generates all admissible supervision terms.


Q2: How are ρj\rho_j initialized and maintained when pairs rarely co-occur, and does this affect anchor selection?

We thank the reviewer and clarify this point.

ρj\rho_j is updated only on batches where modality 1 and modality jj are both present, using the already-computed error ∥M1′−Sj→1(Mj′)∥2\Vert M_1' - S_{j\to1}(M_j') \Vert^2. Batches without the pair leave ρj\rho_j unchanged, so sparse co-observation cannot corrupt it. It simply updates less often. ρj\rho_j is kept per client / aggregated at the server.

For implementation we can set ρj=+∞\rho_j=+\infty until the first co-observation. A simulator that has never been trained is then never preferred over one with a measured error. If the pair (1,j)(1,j) never co-occurs in one or more clients, it should be considered that we have a federated learning model which means even when pairs sit on a small fraction of clients, their updates shape the global Θ\Theta. After RsimR_{sim}, all clients use the same simulators and the same ranking.

Does sparsity harm reliability?

  1. ρj\rho_j is a ranking heuristic and not part of the training objective. Each Sj→1S_{j\to1} is trained directly by the pairwise term of Eq. (3) on every co-observed sample, so a noisy ρj\rho_j can only affect which trained simulator is chosen, not whether it was trained.
  2. All ρj\rho_j are errors in the same space Rd1R^{d_1}, so they are directly comparable across jj.
  3. Our experiments cover this regime. In complex-10-45-45 only 10% of samples hold both modalities, and FArms keeps F-micro at 0.93 (Table 1). In Sleep-EDF (K=5K=5), absent modalities are concentrated within clients, and FArms is the significant winner in five of eight settings and statistically tied with PEPSY in the other three.

Authors accept that if a pair is essentially never co-observed anywhere in the federation, there is no supervision for that simulator, and no reconstruction-based method can learn it.

Thanks for the author's rebuttal response. I maintain my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.