- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The manuscript investigates the issue that conventional multimodal federated training can lead to the suppression of weaker modalities by the dominant one, and the inconsistent optimization progress of clients can disrupt server-side aggregation. To address this problem, the author proposes the FEDMCHAIN framework, which comprises two core modules: (1) Dividing the total training rounds into independent stages executed in modal order, where only the encoder and classification head of mode are updated in stage while freezing other modes; simultaneously introducing a cross-modal alignment regularization term based on InfoNCE and a cross-modal complementarity regularization term based on re-weighted prediction error of preceding modes; (2) Maintaining cluster models on the server side, performing KMeans clustering on the sparse parameter update symbol vectors uploaded by clients, and selectively merging parameter coordinates with consistent directions based on the coordinate-level symbol dominance ratio among clusters. Experiments on three benchmark datasets demonstrate that FEDMCHAIN achieves superior accuracy and a lower modal imbalance ratio compared to existing baselines.
Critical strengths
The study of multimodal federated learning in manuscripts, which is prone to modal competition issues during training, holds significant research importance. Furthermore, the experiments in the manuscript utilize three mainstream datasets and provide anonymized code.
Critical weaknesses
(1) In the method section, the unidirectional chain optimization proposed by the author has significant limitations, requiring an optimal modal sequence selected a priori by humans. If the sequence is chosen incorrectly, the performance may lag behind the existing SOTA baseline.
(2) The server maintains K different cluster models for each modality m. How is the initial assigned in the first round of each modality phase? Additionally, when phase m ends and the modality is frozen, which cluster model is each client fixedly using?
(3) In the theory of federated learning, a new aggregation operator and staged regularization loss are proposed. The convergence analysis primarily starts from the standard local stochastic gradient assumption, deriving the lower bound of the inner product between the aggregated update direction and the true global gradient, as well as the upper bound of the variance. However, the authors directly introduce Assumption A.2 and Assumption A.3 in the appendix, assuming that the cluster partitioning remains fixed across all rounds!
Questions for authors
In addition to the above Critical Weaknesses, there are also some suggestions.
(1) Why is it that in the experiment, all baselines are aggregated every υ=1 rounds, while FEDMCHAIN is only aggregated every υ=20 rounds? If the strongest baselines are also run under υ∈{5,10,20}, what would be their accuracy and average time consumption as shown in Figure 3c?
(2) There are issues with the references in the manuscript, such as some literature lacking necessary publication information, and some literature that has been officially published but is still cited using arXiv.
(3) The author claims that each experiment was repeated three times with random seeds, but statistical information is not provided in the manuscript.
(4) When the feature dimensions of different modalities are different, how is formula (5) calculated? Should a projection head be added?
3 comments
Thank you for the author's response. After reviewing the response and the manuscript, I believe that the overall quality of the manuscript has improved, but it is still not sufficient to meet the acceptance criteria. Therefore, I will maintain my score.
Response to Reviewer v2cs
We sincerely thank you for taking the time to revisit our manuscript and for acknowledging the improvements made through our rebuttal. We respect your assessment and appreciate your continued consideration.
To better understand your remaining reservations, we would be grateful if you could provide some further clarification. In particular, we have made substantial efforts to directly address the three critical weaknesses identified in your original review:
- Modality ordering (W1): We evaluated all possible modality orders across three benchmarks and introduced two lightweight strategies for selecting effective orders without relying on test-set information.
- Dynamic clustering (W2): We clarified the complete procedure for cluster initialization, dynamic reassignment, model synchronization, and model retention across training stages.
- Convergence analysis (W3): We extended our theoretical analysis to account for time-varying cluster objectives and dynamic client reassignment, establishing additional convergence-related bounds under explicitly stated assumptions.
We also addressed the remaining concerns regarding aggregation fairness, statistical reliability, reference completeness, and feature dimension compatibility.
Given these revisions, we would particularly appreciate your guidance on which specific concerns you believe remain insufficiently addressed and what fundamental limitations still prevent the manuscript from meeting the acceptance criteria.
We fully understand that the final assessment depends on the overall technical merits of the work. Our intention is to better understand the decisive concerns underlying your evaluation and, where possible, provide further targeted clarification or evidence.
Thank you again for your time and constructive feedback.
Response to Reviewer v2cs
We sincerely thank the reviewer for the careful assessment of our work and the constructive comments on its methodology, theoretical analysis, and experimental presentation. These suggestions are valuable for improving the technical clarity and rigor of the manuscript. We address the shared concerns about modality ordering, cluster initialization and assignment, and aggregation fairness in Sections S1–S3 of our joint response. Below, we address the remaining issues.
W3. Convergence Analysis under Dynamic Clustering
We thank the reviewer for highlighting this important theoretical consideration. The fixed-partition assumption in our original analysis was introduced to isolate the convergence behavior of SSCA under a stationary cluster objective, rather than to imply that client assignments remain unchanged during actual training.
Following the reviewer's suggestion, we extend our analysis to time-varying cluster objectives Fk,r, allowing dynamic client reassignment. Under explicitly stated assumptions, Theorem A.9 establishes a conditional stationarity bound that accounts for objective drift and aggregation-direction mismatch. Furthermore, we explicitly characterize the errors arising from local training, model-reference mismatch, sparsification, and cross-cluster merging, and derive an additional deviation-based bound in Theorem A.16. The original fixed-partition bound is recovered under stationary cluster objectives and vanishing aggregation-direction mismatch, consistent with the conditions discussed in Remark A.10.
The complete updated theoretical analysis, including all assumptions, derivations, and proofs, has been incorporated into Section 3.4 and Appendix A.3.5 of the revised manuscript.
Q2. Reference Completeness and Accuracy
We thank the reviewer for pointing this out. We have updated the references with complete publication information and replaced outdated arXiv citations with their officially published versions where available. All corrections have been incorporated into the revised manuscript.
Q3. Statistical Reliability across Random Seeds
We thank the reviewer for this valuable suggestion. As stated in Section 4.1 of the original manuscript, all experiments were repeated using three independent random seeds. While Table 1 reports only the mean accuracies for conciseness, we now provide the corresponding means and standard deviations in Appendix Tables 6–8, further supporting the reliability of our findings.
Q4. Feature Dimension Compatibility in Eq. (5)
We thank the reviewer for raising this question. As detailed in Appendix A.1.2, although the input dimensions differ across modalities, the modality-specific encoders produce latent representations of the same dimensionality within each dataset (512 for CREMA-D and AVE, and 128 for CMU-MOSEI). Therefore, the cosine similarity in Eq. (5) can be computed directly without requiring an additional cross-modal projection head.