acceptodds
Under review as a conference paper at ICLR 2027

Boosting Multimodal Federated Learning via Chained Modality Optimization

Abstract

Multimodal Federated Learning (MMFL) enables privacy-preserving collaborative learning across decentralized clients with heterogeneous data and modality availability. However, most existing MMFL methods cast multimodal training as a joint optimization problem, overlooking a key bottleneck: modality competition, where dominant modalities suppress weaker ones and lead to suboptimal global models. To address this, we propose FedMChain, a balanced MMFL framework that structures federated multimodal training as a chain of modality-wise phases. This phase-wise design gives each modality a dedicated local optimization window on multimodal clients to mitigate modality competition, and further promotes cross-modal complementarity via an error-compensated regularizer. On the server side, we employ a sparse sign-guided aggregation strategy that leverages directional sign agreement for robust intra-modality aggregation, avoids destructive averaging, and supports less frequent synchronization to reduce communication overhead. Extensive experiments on multimodal benchmarks demonstrate that FedMChain consistently improves predictive performance while requiring less frequent communication than baselines. Code is available at https://anonymous.4open.science/r/FedMChain-B8D5.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Multimodal federated learning, Modality competition

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_v2csOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The manuscript investigates the issue that conventional multimodal federated training can lead to the suppression of weaker modalities by the dominant one, and the inconsistent optimization progress of clients can disrupt server-side aggregation. To address this problem, the author proposes the FEDMCHAIN framework, which comprises two core modules: (1) Dividing the total training rounds into independent stages executed in modal order, where only the encoder and classification head of mode mm are updated in stage mm while freezing other modes; simultaneously introducing a cross-modal alignment regularization term Ra\mathcal{R}_a based on InfoNCE and a cross-modal complementarity regularization term Rc\mathcal{R}_c based on re-weighted prediction error of preceding modes; (2) Maintaining KK cluster models on the server side, performing KMeans clustering on the sparse parameter update symbol vectors uploaded by clients, and selectively merging parameter coordinates with consistent directions based on the coordinate-level symbol dominance ratio ρq≥π\rho_q \ge \pi among clusters. Experiments on three benchmark datasets demonstrate that FEDMCHAIN achieves superior accuracy and a lower modal imbalance ratio compared to existing baselines.

Critical strengths

The study of multimodal federated learning in manuscripts, which is prone to modal competition issues during training, holds significant research importance. Furthermore, the experiments in the manuscript utilize three mainstream datasets and provide anonymized code.

Critical weaknesses

(1) In the method section, the unidirectional chain optimization proposed by the author has significant limitations, requiring an optimal modal sequence selected a priori by humans. If the sequence is chosen incorrectly, the performance may lag behind the existing SOTA baseline.

(2) The server maintains K different cluster models for each modality m. How is the initial cimc_i^m assigned in the first round of each modality phase? Additionally, when phase m ends and the modality is frozen, which cluster model is each client fixedly using?

(3) In the theory of federated learning, a new aggregation operator and staged regularization loss are proposed. The convergence analysis primarily starts from the standard local stochastic gradient assumption, deriving the lower bound of the inner product between the aggregated update direction and the true global gradient, as well as the upper bound of the variance. However, the authors directly introduce Assumption A.2 and Assumption A.3 in the appendix, assuming that the cluster partitioning remains fixed across all rounds!

Questions for authors

In addition to the above Critical Weaknesses, there are also some suggestions.

(1) Why is it that in the experiment, all baselines are aggregated every υ=1 rounds, while FEDMCHAIN is only aggregated every υ=20 rounds? If the strongest baselines are also run under υ∈{5,10,20}, what would be their accuracy and average time consumption as shown in Figure 3c?

(2) There are issues with the references in the manuscript, such as some literature lacking necessary publication information, and some literature that has been officially published but is still cited using arXiv.

(3) The author claims that each experiment was repeated three times with random seeds, but statistical information is not provided in the manuscript.

(4) When the feature dimensions of different modalities are different, how is formula (5) calculated? Should a projection head be added?

3 comments
AuthorsOpenReview ↗

Response to Reviewer v2cs

We sincerely thank the reviewer for the careful assessment of our work and the constructive comments on its methodology, theoretical analysis, and experimental presentation. These suggestions are valuable for improving the technical clarity and rigor of the manuscript. We address the shared concerns about modality ordering, cluster initialization and assignment, and aggregation fairness in Sections S1–S3 of our joint response. Below, we address the remaining issues.

W3. Convergence Analysis under Dynamic Clustering

We thank the reviewer for highlighting this important theoretical consideration. The fixed-partition assumption in our original analysis was introduced to isolate the convergence behavior of SSCA under a stationary cluster objective, rather than to imply that client assignments remain unchanged during actual training.

Following the reviewer's suggestion, we extend our analysis to time-varying cluster objectives Fk,rF_{k,r}, allowing dynamic client reassignment. Under explicitly stated assumptions, Theorem A.9 establishes a conditional stationarity bound that accounts for objective drift and aggregation-direction mismatch. Furthermore, we explicitly characterize the errors arising from local training, model-reference mismatch, sparsification, and cross-cluster merging, and derive an additional deviation-based bound in Theorem A.16. The original fixed-partition bound is recovered under stationary cluster objectives and vanishing aggregation-direction mismatch, consistent with the conditions discussed in Remark A.10.

The complete updated theoretical analysis, including all assumptions, derivations, and proofs, has been incorporated into Section 3.4 and Appendix A.3.5 of the revised manuscript.

Q2. Reference Completeness and Accuracy

We thank the reviewer for pointing this out. We have updated the references with complete publication information and replaced outdated arXiv citations with their officially published versions where available. All corrections have been incorporated into the revised manuscript.

Q3. Statistical Reliability across Random Seeds

We thank the reviewer for this valuable suggestion. As stated in Section 4.1 of the original manuscript, all experiments were repeated using three independent random seeds. While Table 1 reports only the mean accuracies for conciseness, we now provide the corresponding means and standard deviations in Appendix Tables 6–8, further supporting the reliability of our findings.

Q4. Feature Dimension Compatibility in Eq. (5)

We thank the reviewer for raising this question. As detailed in Appendix A.1.2, although the input dimensions differ across modalities, the modality-specific encoders produce latent representations of the same dimensionality within each dataset (512 for CREMA-D and AVE, and 128 for CMU-MOSEI). Therefore, the cosine similarity in Eq. (5) can be computed directly without requiring an additional cross-modal projection head.

Thank you for the author's response. After reviewing the response and the manuscript, I believe that the overall quality of the manuscript has improved, but it is still not sufficient to meet the acceptance criteria. Therefore, I will maintain my score.

AuthorsOpenReview ↗

Response to Reviewer v2cs

We sincerely thank you for taking the time to revisit our manuscript and for acknowledging the improvements made through our rebuttal. We respect your assessment and appreciate your continued consideration.

To better understand your remaining reservations, we would be grateful if you could provide some further clarification. In particular, we have made substantial efforts to directly address the three critical weaknesses identified in your original review:

  • Modality ordering (W1): We evaluated all possible modality orders across three benchmarks and introduced two lightweight strategies for selecting effective orders without relying on test-set information.
  • Dynamic clustering (W2): We clarified the complete procedure for cluster initialization, dynamic reassignment, model synchronization, and model retention across training stages.
  • Convergence analysis (W3): We extended our theoretical analysis to account for time-varying cluster objectives and dynamic client reassignment, establishing additional convergence-related bounds under explicitly stated assumptions.

We also addressed the remaining concerns regarding aggregation fairness, statistical reliability, reference completeness, and feature dimension compatibility.

Given these revisions, we would particularly appreciate your guidance on which specific concerns you believe remain insufficiently addressed and what fundamental limitations still prevent the manuscript from meeting the acceptance criteria.

We fully understand that the final assessment depends on the overall technical merits of the work. Our intention is to better understand the decisive concerns underlying your evaluation and, where possible, provide further targeted clarification or evidence.

Thank you again for your time and constructive feedback.

Reviewer_NkSPOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses modality competition in multimodal federated learning, where dominant data streams can suppress weaker modalities during joint client training. Its main component, Modality-Chained Federated Training (MCFT), trains one active modality branch at a time while freezing the others. On the server side, the authors introduce SSCA, a sparsity-aware aggregation method that groups local updates by clustering their sign vectors with K-means. Experiments on CREMA-D, AVE, and CMU-MOSEI show that FEDMCHAIN achieves state-of-the-art performance, records the lowest modality imbalance ratio on CREMA-D, and offers a favourable computation–communication trade-off.

Critical strengths
  • Prevents destructive parameter averaging by clustering sparsified sign vectors by merging updates on high-consensus parameter coordinates.
  • Eliminates inter-modality gradient computation on multimodal clients by optimizing active modality branches while keeping inactive branches frozen.
  • Achieves competitive performance across CREMA-D, AVE, and CMU-MOSEI datasets while obtaining a lower modality imbalance ratio.
Critical weaknesses
  • MCFT phase:
    • MCFT uses fixed training schedule based on sequential, single-modality phases rather than adapting dynamically to the evolving training state.
    • Interaction flows from the preceding modality to the active modality. The initial modality in the chain receives no cross-modal regularizer signal and earlier modalities cannot adapt based on representations learned by later modalities.
    • There is an issue of noise amplification based on Eq (7). If early-stage modalities output miscalibrated logit scores or poorly optimized on non-IID clients, Rc generates faulty sample weights, thus increasing reliance on noisy uninformative samples.
  • SSCA phase:
    • In the SSCA phase, the authors use K-means clustering on sparse vectors. For high dimensional sparse vectors spherical kmeans might be better suited for the clustering task.
    • The clustering step depends only on the sign directions. By removing the magnitudes before clustering, a client with small update receives equal weight during cluster assignment as a client with large high confidence update.
    • It is not clear whether the clustering is re-executed during each execution round. If yes, how is the cluster identity drift handled in maintaining consistent cluster profiles ?
  • Experimental results:
    • The overall results across different benchmarks is slightly inconsistent. For example in CREMA-D and AVE, BMSFed is the best performing method in terms of visual and audio accuracy values respectively. Similarly in CMU-MOSEI, FEDMCHAIN trails M3Fed in terms of text accuracy values.
    • The overall predictive performance is highly sensitive to the order of the modalities encountered during training. For example, the test accuracy ranges from 62.07 % to 64.96% with changes from v->a->t schedule to t->a->v schedule.
Questions for authors
  • For SSCA, the authors apply K-means on discrete sign vectors. Have the authors considered other alternatives like Hamming distance, Density-based clustering ?
  • Have the authors considered alternatives like cross-attention, concatenated MLPs or dynamic gating heads for feature fusion ?
  • Before setting λc\lambda_{c} and λa\lambda_{a} in Eq (4), have the authors considered relative values for different loss components ?
2 comments
AuthorsOpenReview ↗

Response to Reviewer NkSP

We sincerely thank the reviewer for the encouraging assessment of our work and the thoughtful, constructive suggestions. We particularly appreciate the recognition of our modality-wise optimization and sign-guided aggregation designs. The comments have motivated us to clarify several implementation details and explore useful extensions of FedMChain. The shared concerns regarding modality ordering and dynamic clustering are addressed in Sections S1 and S3 of our joint response. We address the remaining questions below.

W1. Fixed Training Schedule

We thank the reviewer for this insightful suggestion. We agree that different modalities may exhibit different convergence behaviors, making adaptive phase scheduling a valuable extension.

In fact, we have already investigated adaptive scheduling in Appendix Table 13. Specifically, strategy S3 dynamically determines phase transitions based on training-loss plateaus, improving accuracy by 0.39 and 0.44 percentage points on CREMA-D and AVE, respectively, while reducing the required training rounds from 500 to 462 and 484.

These results demonstrate that MCFT is compatible with adaptive phase scheduling and can further benefit from it. We adopt uniform phase allocation in the main experiments to maintain a simple, controlled protocol and better isolate the contributions of chained modality optimization. We have further clarified this design choice and the adaptive scheduling results in the updated manuscript.

W2. Unidirectional Knowledge Transfer

We thank the reviewer for this insightful observation. MCFT intentionally adopts unidirectional knowledge transfer to provide stable preceding-modality references for subsequent optimization. Nevertheless, we agree that this design limits the ability of earlier modalities, particularly the initial modality, to benefit from knowledge learned in later stages.

Inspired by this suggestion, we further investigate a lightweight refinement strategy that revisits the initial modality after completing the original chain. Specifically, we optimize it for only five additional rounds, using the subsequently trained modalities as references for both RaR_a and RcR_c, followed by one server aggregation.

Table R3. Effect of initial-modality refinement (ACC, %).

MethodCREMA-D ↑AVE ↑CMU-MOSEI ↑
FedMChain58.3659.8564.96
with Initial-Modality Refinement59.1660.2765.08

The results show that this lightweight refinement consistently improves predictive accuracy with only five additional training rounds and one aggregation, demonstrating the potential benefits of incorporating feedback from later modalities. We appreciate the reviewer's constructive suggestion. We have included this exploratory extension, its experimental results, and a discussion of future directions in Appendix A.2.3 of the revised manuscript.

W3. Potential Noise Amplification in RcR_c

We thank the reviewer for raising this important concern. We would like to clarify that RcR_c relies on predictions from preceding modalities whose training stages have already been completed and whose parameters remain frozen during the current stage. Moreover, RcR_c does not introduce pseudo-labels; instead, it reweights the ground-truth supervised loss based on the preceding modalities' confidence. Specifically, from Eqs. (7)–(8), the effective sample weight satisfies

w=1+λc(1−pˉ(<m)(y∣x))∈[1,1+λc].w=1+\lambda_c(1-\bar p^{(<m)}(y\mid x)) \in[1,1+\lambda_c].

With λc=1.0\lambda_c=1.0, the weight is bounded within [1,2][1,2], limiting the additional amplification of the task loss. While this bounded reweighting mitigates excessive amplification, we acknowledge that imperfect confidence calibration may still affect sample weighting, particularly under severe data heterogeneity.

AuthorsOpenReview ↗

Response to Reviewer NkSP (Continued)

W4 & W5 & Q1. Clustering Strategy and Magnitude Information in SSCA

We thank the reviewer for the constructive suggestions regarding SSCA's clustering strategy and the use of update magnitudes. We would like to clarify that SSCA uses sparsified sign vectors only for client grouping, rather than discarding magnitude information throughout aggregation. Specifically, Eq. (10) retains the actual magnitudes of the nonzero coordinates after sparsification, while Eqs. (11)–(15) further exploit these magnitudes for coordinate-wise consensus estimation and selective merging. This design separates direction-based client grouping from magnitude-aware aggregation.

Regarding the clustering algorithm, we also explored spectral clustering in preliminary experiments. We adopted K-means for its simplicity and computational efficiency, particularly given the repeated clustering required by SSCA. Following the reviewer's suggestions, we further compare spherical K-means (cosine similarity), spectral clustering (similarity-graph-based grouping), and Hamming-distance K-medoids (coordinate-wise sign disagreement) under consistent experimental settings. All methods use (K=5K=5), with the remaining SSCA operations unchanged. Density-based clustering was not included in our controlled comparison, as its variable number of clusters and potential noise assignments would require additional modifications to SSCA's fixed-K aggregation protocol.

Table R4. Comparison of clustering strategies in SSCA (ACC, %).

Clustering MethodCREMA-D ↑AVE ↑CMU-MOSEI ↑
K-means (Ours)58.3659.8564.96
Spectral Clustering57.8457.9963.18
Spherical K-means58.4559.6765.26
Hamming-based Clustering58.5159.9465.12

The results show that K-means achieves competitive performance across all three datasets, supporting its use as a simple and practical choice for SSCA. Nevertheless, we agree that incorporating magnitude information into cluster assignment may offer further improvements and consider magnitude-aware clustering a valuable extension. These results and the corresponding analysis have been included in Appendix A.2 of the revised manuscript.

W7. Modality-wise Performance

We thank the reviewer for this thoughtful observation. Importantly, the modality-wise accuracies (ACCvACC_v, ACCaACC_a, and ACCtACC_t) in Table 1 are averaged over all clients possessing the corresponding modality, including both unimodal and multimodal clients. Therefore, these metrics reflect predictive performance across heterogeneous client groups rather than exclusively on unimodal clients.

From a methodological perspective, MCFT promotes cross-modal alignment and complementary learning, but these regularization signals are available only when clients possess the required preceding modalities. Meanwhile, SSCA selectively aggregates client updates based on directional agreement. Consequently, the benefits of cross-modal knowledge transfer and aggregation may vary across client groups with different modality availability and data distributions, rather than yielding uniform improvements in every modality-wise metric.

As shown in Table 1, FedMChain consistently achieves the highest overall accuracy and multimodal-client accuracy across all three datasets, while remaining competitive on modality-wise metrics. These results support the effectiveness of the proposed mechanisms in promoting multimodal collaboration under heterogeneous federated settings.

Q2. Alternative Fusion Strategies

We thank the reviewer for this valuable suggestion. We adopt simple logit-level summation to avoid additional trainable fusion parameters and better isolate the contributions of MCFT and SSCA.

Although we have not experimentally evaluated more sophisticated fusion strategies, FedMChain is not inherently restricted to logit summation. Since MCFT and SSCA operate on modality-specific branches, alternative fusion mechanisms, such as concatenation-based MLP, cross-attention, and dynamic gating, could be incorporated after modality-wise training. We consider their empirical evaluation an interesting direction for future work.

Q3. Relative Scales of Loss Components

We thank the reviewer for this thoughtful question. We account for the different contributions of the loss components through their formulations and weighting coefficients.

Specifically, the complementarity regularizer RcR_c reweights the task loss by a confidence-based factor bounded within [0,1][0,1]. Thus, with λc=1.0\lambda_c=1.0, the resulting sample-wise task-loss weight lies within [1,2][1,2]. The alignment regularizer RaR_a is averaged over samples and preceding modalities (Eq. (5)), with its contribution controlled separately by λa\lambda_a.

Furthermore, the hyperparameter sensitivity analysis in Figure 10 shows consistently strong performance around λa=0.4\lambda_a=0.4 and λc=1.0\lambda_c=1.0 across all three datasets, supporting our choice of coefficients.

Reviewer_wQr4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes FedMChain for multimodal federated learning. It trains modalities in sequence and uses sign-guided aggregation to reduce update conflicts. Experiments on CREMA-D, AVE, and CMU-MOSEI show improved accuracy.

Critical strengths
  1. The paper addresses a relevant challenge in heterogeneous multimodal federated learning.
  2. The evaluation covers three benchmarks and includes useful ablation studies.
Critical weaknesses
  1. The comparison is not fully convincing. Baselines use a different communication schedule, and some recent relevant methods are missing.
  2. The novelty seems limited, as the framework mainly combines existing training and aggregation ideas. Its sensitivity to modality order also raises concerns.
Questions for authors
  1. Can the authors compare with more recent methods under matched communication settings?
  2. How is the modality order selected, and what is the benefit over simpler alternating-training strategies?
1 comment
AuthorsOpenReview ↗

Response to Reviewer wQr4

We sincerely thank the reviewer for recognizing the relevance of modality competition in heterogeneous multimodal federated learning and for the constructive suggestions regarding experimental fairness and methodological novelty. We have addressed the shared concerns about communication fairness and modality ordering in S1, S2 of our joint response. Below, we focus on comparisons with recent methods and the benefits of FedMChain over conventional alternating optimization.

W1&Q1. Comparison with Recent MMFL Methods

We thank the reviewer for this valuable suggestion. We have further considered three recent MMFL approaches: FedAFD (CVPR 2026), which addresses cross-modal discrepancies through adversarial alignment and distillation; FedAMB (IJCAI 2026), which mitigates modality dominance through selective regulation and knowledge distillation; and MFedPBA (ICML 2026), which addresses modality imbalance and model heterogeneity through prototype-based alignment. For FedAFD, whose implementation is publicly available, we include an additional experimental comparison under our evaluation settings. As publicly reproducible implementations of FedAMB and MFedPBA are currently unavailable, we expand the Related Work discussion to clarify their methodological distinctions from FedMChain.

Table R1. Comparison with recent MMFL methods (ACC, %).

MethodCREMA-DAVECMU-MOSEI
FedAFD56.5857.8362.49
Ours58.3659.8564.96

The results further demonstrate FedMChain's competitive performance against recent MMFL approaches. We report this additional comparison here and plan to incorporate a broader evaluation into a future version of the manuscript once the experimental protocols and results have been comprehensively validated.

W2. Methodological Novelty

We thank the reviewer for the thoughtful comments on methodological novelty. Our key insight is that modality competition in MMFL not only suppresses weaker modalities during local training but can also cause heterogeneous modality-specific optimization progress across clients, further aggravating conflicts during global aggregation.

To mitigate these interconnected challenges, we introduce two tailored innovations. Specifically, MCFT introduces a globally coordinated modality-chained training paradigm for MMFL, which decouples modality optimization into dedicated stages while enabling directed cross-modal knowledge transfer through semantic alignment (RaR_a) and error-compensated complementarity learning (RcR_c). More importantly, SSCA goes beyond conventional clustering-based aggregation and parameter merging. It first groups clients according to their sparsified update directions, then performs magnitude-aware, coordinate-wise consensus to selectively share updates with strong directional agreement while preserving cluster-specific updates on conflicting coordinates. This design enables selective knowledge sharing without indiscriminately averaging conflicting client updates.

Together, MCFT and SSCA form a coordinated framework that alleviates modality competition at both the local optimization and global aggregation levels, rather than simply combining existing techniques.

Q2. Comparison with Alternating Training

We agree that conventional alternating training can also decouple modality-wise optimization. However, naive cyclic alternation requires additional decisions on switching frequency and repeated cycles, without explicitly facilitating cross-modal knowledge transfer. In contrast, MCFT adopts a one-pass modality chain, where preceding modalities provide stable references for subsequent alignment and complementary learning. To further investigate this distinction, we compare conventional cyclic alternating training with FedMChain on CREMA-D. Both methods use the same architecture, modality order, total training budget (500 rounds), per-modality allocation, and aggregation interval (υ=20\upsilon=20). The cyclic baseline switches modalities every 20 rounds and uses SSCA without cross-modal regularizers.

Table R2. Comparison with conventional alternating training on CREMA-D.

MethodACC (%) ↑MIR ↓
Cyclic Alternating + SSCA57.251.35
Ours58.361.31

The results show that FedMChain achieves higher accuracy and lower modality imbalance than conventional cyclic alternation. Together with the component-wise ablations in Table 2, these findings further support the benefits of directed cross-modal knowledge transfer and conflict-aware aggregation beyond simple modality alternation. We have included this comparison and the corresponding analysis in Appendix A.2 (Table 14) of the revised manuscript.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Response to Shared Concerns Across Reviewers

We sincerely thank all three reviewers for their careful evaluations and constructive suggestions. Several comments raise closely related questions regarding modality ordering, communication fairness, and dynamic clustering. To avoid repetition and provide a more comprehensive response, we address these shared concerns together below.

S1. Modality Ordering and Practical Selection [Reviewer wQr4 (W2 & Q2), Reviewer NkSP (W8), Reviewer v2cs (W1)]

We thank the reviewers for their valuable comments on modality ordering. To comprehensively evaluate FedMChain, we tested all possible modality orders across the three datasets, as reported in Table 3. The results show that different orders affect predictive accuracy, while FedMChain consistently achieves lower MIR than BMSFed across all evaluated orders. This demonstrates that its effectiveness in mitigating modality imbalance is maintained under different training sequences.

Since the most suitable modality order may vary across tasks and datasets, we further investigate whether it can be identified efficiently before training. Specifically, we evaluate two lightweight strategies from different perspectives:

  • P1 (Early Optimization Responsiveness) ranks modalities by their relative training-loss reduction after five epochs of unimodal warm-up.
  • P2 (Frozen-feature Discriminability) ranks modalities by their class-discriminative quality, evaluated through leave-one-out k-NN classification on frozen representations.

Both strategies determine the modality order without using test-set information. We evaluate them on CREMA-D and CMU-MOSEI, reporting the selected orders and their additional computational and communication costs.

Table S1. Practical modality-order selection and additional overhead (percentages in parentheses are relative to full training).

DatasetStrategySelected OrderTime (s)CoT (KB)
CREMA-DP1A → V1.69K (2.43%)0.72 (<0.01%)
CREMA-DP2A → V0.35K (0.5%)2.88 (<0.01%)
CMU-MOSEIP1T → A → V0.87K (3.03%)2.52 (<0.01%)
CMU-MOSEIP2T → A → V0.05K (0.18%)5.04 (<0.01%)

Encouragingly, both strategies identify the best-performing orders reported in Table 3, while incurring limited additional computational and communication overhead. These findings suggest that effective modality orders can be selected through lightweight preliminary evaluation rather than exhaustive enumeration. Together with the consistently lower MIR across different orders, these results further support the practicality of FedMChain. We have incorporated the additional results and analysis into Section 4.4 (Table 4) of the updated manuscript, with detailed evaluation protocols provided in Appendix A.2.2.

S2. Fair Comparison under Matched Aggregation Intervals [Reviewer wQr4 (W1&Q1), Reviewer v2cs (Q1)]

We thank the reviewers for their valuable suggestions regarding communication fairness. Aggregating model updates after every training round υ=1\upsilon=1 is a common practice in federated learning, which we adopted for the baselines. Importantly, Appendix Table 9 (Table 5 in the original submission) already reports FedMChain's performance under the same setting υ=1\upsilon=1, where it outperforms the strongest baselines by 0.84, 1.14, and 0.38 percentage points on CREMA-D, AVE, and CMU-MOSEI, respectively.

We report the results at υ=20\upsilon=20 in the main paper to highlight another advantage of FedMChain: it maintains strong predictive performance with less frequent aggregation, achieving higher accuracy while reducing overall communication costs.

To provide a more comprehensive comparison, we further evaluate all applicable baselines under matched aggregation intervals $\upsilon \in ${1,5,10,15,20,30 }, as presented in the updated main text Figure 4. The results show that most baselines exhibit declining accuracy as the aggregation interval increases, whereas FedMChain maintains relatively stable performance and achieves its highest accuracy at υ=20\upsilon=20 across all three datasets. These findings confirm that FedMChain's performance gains are not merely attributable to differences in aggregation frequency and further demonstrate its favorable accuracy–communication trade-off under matched aggregation settings.

AuthorsOpenReview ↗

Response to Shared Concerns Across Reviewers (Continued)

S3. Cluster Initialization, Dynamic Reassignment, and Model Consistency [Reviewer NkSP (W6), Reviewer v2cs (W2)]

We sincerely apologize for the lack of clarity in our original description and thank the reviewers for highlighting these important implementation details. We would like to clarify the SSCA procedure as follows:

  • Cluster Initialization: At the beginning of each modality stage, the server maintains KK cluster-specific models initialized with identical parameters. Thus, the initial cluster index is merely a placeholder and does not require a predefined client partition.
  • Dynamic Clustering: Clustering is performed at each server synchronization rather than every local training round. Clients are dynamically grouped according to their current sparsified update directions, followed by cluster-wise aggregation and coordinate-wise consensus.
  • Cluster Identity and Model Synchronization: SSCA employs dynamic update-based clustering rather than maintaining fixed client groups. Cluster indices serve as references to server-side models, while client memberships are allowed to change across synchronization rounds. After aggregation, the server updates the corresponding cluster models and synchronizes participating clients with their newly assigned models.
  • Final Model Assignment: When a modality training stage ends, each client retains its latest modality-specific model parameters, which remain frozen during subsequent stages.

We appreciate the reviewers' careful observations. We have clarified the initialization, dynamic reassignment, and model synchronization procedures in Section 3.3 and Appendix A.1.2 of the revised manuscript.

Dear reviewer wQr4 and NkSP,

The authors have responded to your concerns. Please review their responses as soon as possible and provide any necessary feedback. Thank you.

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.