- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
Multimodal split learning typically forms clients with unimodal encoders, and the server contains a fusion module. The activation transmission between clients and the server is restricted by a budget; thus, modalities can only transmit a smashed dimension to the server under a keep ratio. Since different modalities may contribute unequally, this work focuses on reallocating the budget for each modality. It proposes ModalShare, which uses Shapley values to determine modality importance and the keep ratio. ModalShare computes modality contributions on the server side and updates an allocator that guides clients in adjusting their keep ratio. Extensive experiments are conducted to verify the effectiveness of ModalShare.
Critical strengths
- This work formulates an important yet underexplored problem of inter-modality budget allocation in multimodal split learning.
- The proposed method is simple and intuitive, without adding server-client communication costs.
- ModalShare exhibits superior performance on experimental datasets.
Critical weaknesses
- A fundamental limitation lies in this work. Modality contribution might vary across samples [1,2,3]. Hence, the server-side allocator could not allocate the keep-ratio for the client-side modality for the current sample according to modality informativeness. After transmission, the informative modality might have a lower keep-ratio while the model retains more dimensions for the uninformative modality, resulting in an unreliable modality contribution estimation on the server side.
- Method novelty is moderate compared to previous work on modality imbalance [4].
- Lacks sufficient ablation studies on batch size and a more complex fusion module, such as gate fusion or using attention. The comparison methods with extensive existing modality imbalance approaches in the main experiment are limited.
- The experimental dataset is quite small, with a maximum of 10,299 samples; ModalShare isn’t fully tested under larger datasets like Kinetic-Sound [5] or VGG-Sound [6].
[1] “Multimodal Alignment and Fusion: A Survey”, IJCV 2025
[2] “Provable dynamic fusion for low-quality multimodal data”, ICML 2024
[3] “RollingQ: Reviving the Cooperation Dynamics in Multimodal Transformer”, ICML 2025
[4] “Enhancing Multimodal Cooperation via Sample-level Modality Valuation”, CVPR 2024
[5] “Look, Listen and Learn”, ICCV 2017
[6] “Vggsound: A large-scale audio-visual dataset”, ICASSP 2020
Questions for authors
- Since the contribution is computed on the server side, would it be influenced by the keep-ratio of each modality? If so, would it make the contribution estimation unreliable because the informative modality may sometimes have a lower keep-ratio for some samples?
- Could the effectiveness of Modal-share be generalized to tri-modalities?
3 comments
Thanks for the authors' additional experiments and rebuttal. Most of my concerns have been addressed.
I admit its contribution to Multimodal Split Learning field and establish a solid foundation for future exploration. However, I maintain my opinion on the moderate novelty of the ModalShare method.
Based on this reassessment, I decide to raise my score to 3.
We are glad that our responses have addressed most of the reviewer's concerns, made the contribution more clear and we appreciate raising your score. We thank the reviewer again for their time and constructive feedback.
We appreciate your kind words about the problem and the simplicity of the method. Your questions on contribution estimation were truly insightful.
W1/Q1. Contribution estimate
We agree that both are limitations, as your references show. Both arise whenever a sender must decide what to transmit before the receiver can value it.
Per-sample contribution. A sample's contribution depends on the server's fusion, which the client cannot run before sending. Per-sample allocation would need a client-side estimator (extra client computation) and a new wire format, since the compressors keep columns shared across the batch. Imbalance methods such as OGM-GE also work per mini-batch. The gain holds even at batch 16, the fewest samples per probe (W3). Per-sample dynamic fusion [2, 3] acts after transmission and is complementary.
Dependence on the keep-ratios. The probe scores what it receives, and a modality sent at a lower keep-ratio can score lower. Any closed-loop allocation faces this, as in bandits (exploration versus exploitation) and in OGM-GE, whose modulation shifts its own scores. ModalShare explores before it commits.
In short, ModalShare is designed to address both limitations. It averages contribution over samples and probes and commits only on sustained evidence, all at the server with no extra uplink or client computation.
Revision. App. F (citing [1–3]), §6.
W2. Relation to [4]
Unlike the imbalance literature, including [4], we study modality imbalance under a resource constraint. Those methods rebalance training while every modality reaches the model in full (§2). On edge devices, a realistic deployment for split learning, the uplink budget is fixed and the question is how much of each modality to send.
Shapley values give a principled and reliable measure of each modality's contribution, also used in [4] with absent modalities set to zero (§3.4). We purposefully leverage it in the constrained setting, where its simplicity helps avoid client overhead, while also being able to generalize for M>2.
Revision. §2.
W3. Baselines, batch size, fusion
We have now updated OGM-GE baselines. On CREMA-D at β=0.20 (8 seeds, final-epoch test), the pairwise OGM-GE rule [Peng et al., 2022] lowers accuracy with equal keep-ratios (30.3 → 26.9%) and with Shapley keep-ratios (37.8 → 36.1%).
App. M (new) gives the gain over uniform on MVSA at 5× (5 seeds, final-epoch test).
Gate and attention fusion keep the gain, and reweighting at the server does not close the gap. The gain is smaller at batch 128, where the probe runs about once per epoch. These results cover five seeds, one dataset and one budget.
Revision. App. J, Table 12, App. M (new), Table 17, §4.3.
W4. Larger dataset
Within the rebuttal period we could add only Kinetics-Sounds [5], with the same model and protocol as CREMA-D (final-epoch test, 3 seeds).
Notably, ModalShare improves over uniform at every budget and, with three seeds, this is consistent with our other datasets.
Revision. App. N (new), Tables 18–19, §6.
Q2. Three modalities
ModalShare was designed so it can work for any M (Eqs. 5, 7–11). Exact Shapley needs 2M server evaluations per probe (W2), and permutation sampling [Castro et al. 2009, Ghorbani & Zou 2019] reduces this for larger M. We added IEMOCAP [Busso et al., 2008] with three modalities (text, audio, face motion capture), final-epoch test, 12 seeds.
The uncompressed model reaches 67.7. ModalShare has the higher mean at four of five budgets, is level at 20× (+0.1 pp) and wins 43 of 60 seed–budget pairs. ModalShare thus handles three modalities, moving the payload off face motion capture (share 0.45 → 0.01) while keeping text and audio, and gains most at the tightest budgets.
Revision. App. K (new), Tables 13–14, §3.4, §6.
We hope our responses address your concerns and support a reassessment of the paper. We welcome any further questions or suggestions for strengthening the manuscript.