acceptodds
Under review as a conference paper at ICLR 2027

ModalShare: Contribution-Aware Budget Allocation for Multimodal Split Learning

Abstract

Multimodal models combine several input streams, yet training them where the data originates, such as on edge devices carrying cameras, microphones, and other sensors, is impractical when each modality requires its own heavy encoder. Split Learning (SL) makes such training feasible by keeping only the first layers on the device and offloading the rest to a server, but the resulting communication must carry intermediate activations for every modality at every step. Communication-efficient SL reduces that cost by compressing the activations, keeping only a fraction of each; however existing schemes hand every modality the same keep-ratio, so the shared budget is split in proportion to activation dimension, a quantity unrelated to how much each modality contributes to the fused prediction. We make that split an explicit decision, namely inter-modality allocation: under a fixed budget, every policy transmits the same expected payload and differs only in how that payload is divided across modalities. Our allocator, ModalShare, sets each modality's keep-ratio from a Shapley contribution score computed by the server over coalitions of activations it has already received, so measurement adds no transmission overhead, no client-side computation, and assumes no prior knowledge of which stream is which. ModalShare improves accuracy over equal keep-ratios by and percentage points on CREMA-D and MVSA at matched payload under compression, with consistent gains across three compressors, three datasets, and four budgets. Existing compressors leave these gains unclaimed in multimodal settings; contribution-aware allocation recovers them.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
split learning, multimodal learning, communication-efficiency, resource allocation, edge computing

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_DH9qOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

Multimodal split learning typically forms clients with unimodal encoders, and the server contains a fusion module. The activation transmission between clients and the server is restricted by a budget; thus, modalities can only transmit a smashed dimension to the server under a keep ratio. Since different modalities may contribute unequally, this work focuses on reallocating the budget for each modality. It proposes ModalShare, which uses Shapley values to determine modality importance and the keep ratio. ModalShare computes modality contributions on the server side and updates an allocator that guides clients in adjusting their keep ratio. Extensive experiments are conducted to verify the effectiveness of ModalShare.

Critical strengths
  • This work formulates an important yet underexplored problem of inter-modality budget allocation in multimodal split learning.
  • The proposed method is simple and intuitive, without adding server-client communication costs.
  • ModalShare exhibits superior performance on experimental datasets.
Critical weaknesses
  • A fundamental limitation lies in this work. Modality contribution might vary across samples [1,2,3]. Hence, the server-side allocator could not allocate the keep-ratio for the client-side modality for the current sample according to modality informativeness. After transmission, the informative modality might have a lower keep-ratio while the model retains more dimensions for the uninformative modality, resulting in an unreliable modality contribution estimation on the server side.
  • Method novelty is moderate compared to previous work on modality imbalance [4].
  • Lacks sufficient ablation studies on batch size and a more complex fusion module, such as gate fusion or using attention. The comparison methods with extensive existing modality imbalance approaches in the main experiment are limited.
  • The experimental dataset is quite small, with a maximum of 10,299 samples; ModalShare isn’t fully tested under larger datasets like Kinetic-Sound [5] or VGG-Sound [6].

[1] “Multimodal Alignment and Fusion: A Survey”, IJCV 2025

[2] “Provable dynamic fusion for low-quality multimodal data”, ICML 2024

[3] “RollingQ: Reviving the Cooperation Dynamics in Multimodal Transformer”, ICML 2025

[4] “Enhancing Multimodal Cooperation via Sample-level Modality Valuation”, CVPR 2024

[5] “Look, Listen and Learn”, ICCV 2017

[6] “Vggsound: A large-scale audio-visual dataset”, ICASSP 2020

Questions for authors
  • Since the contribution is computed on the server side, would it be influenced by the keep-ratio of each modality? If so, would it make the contribution estimation unreliable because the informative modality may sometimes have a lower keep-ratio for some samples?
  • Could the effectiveness of Modal-share be generalized to tri-modalities?
3 comments

We appreciate your kind words about the problem and the simplicity of the method. Your questions on contribution estimation were truly insightful.

W1/Q1. Contribution estimate

We agree that both are limitations, as your references show. Both arise whenever a sender must decide what to transmit before the receiver can value it.

Per-sample contribution. A sample's contribution depends on the server's fusion, which the client cannot run before sending. Per-sample allocation would need a client-side estimator (extra client computation) and a new wire format, since the compressors keep columns shared across the batch. Imbalance methods such as OGM-GE also work per mini-batch. The gain holds even at batch 16, the fewest samples per probe (W3). Per-sample dynamic fusion [2, 3] acts after transmission and is complementary.

Dependence on the keep-ratios. The probe scores what it receives, and a modality sent at a lower keep-ratio can score lower. Any closed-loop allocation faces this, as in bandits (exploration versus exploitation) and in OGM-GE, whose modulation shifts its own scores. ModalShare explores before it commits.

  • Equal starting scores split the payload evenly at first.
  • The temperature keeps the map soft while scores are close.
  • The EMA averages scores over probes.
  • The freeze commits only after the same winner holds for K=3K = 3 epochs.

In short, ModalShare is designed to address both limitations. It averages contribution over samples and probes and commits only on sustained evidence, all at the server with no extra uplink or client computation.

Revision. App. F (citing [1–3]), §6.

W2. Relation to [4]

Unlike the imbalance literature, including [4], we study modality imbalance under a resource constraint. Those methods rebalance training while every modality reaches the model in full (§2). On edge devices, a realistic deployment for split learning, the uplink budget is fixed and the question is how much of each modality to send.

Shapley values give a principled and reliable measure of each modality's contribution, also used in [4] with absent modalities set to zero (§3.4). We purposefully leverage it in the constrained setting, where its simplicity helps avoid client overhead, while also being able to generalize for M>2M > 2.

Revision. §2.

W3. Baselines, batch size, fusion

We have now updated OGM-GE baselines. On CREMA-D at β=0.20\beta = 0.20 (8 seeds, final-epoch test), the pairwise OGM-GE rule [Peng et al., 2022] lowers accuracy with equal keep-ratios (30.3 → 26.9%) and with Shapley keep-ratios (37.8 → 36.1%).

App. M (new) gives the gain over uniform on MVSA at 5× (5 seeds, final-epoch test).

Fusion, batchΔ\Delta (seeds won)
Concat, 16+18.9 ± 9.3 (5)
Concat, 32+17.5 ± 6.4 (5)
Concat, 64+17.9 ± 9.6 (5)
Concat, 128+8.2 ± 7.2 (4)
Gate, 64+18.7 ± 5.8 (5)
Attention, 64+20.2 ± 6.2 (5)

Gate and attention fusion keep the gain, and reweighting at the server does not close the gap. The gain is smaller at batch 128, where the probe runs about once per epoch. These results cover five seeds, one dataset and one budget.

Revision. App. J, Table 12, App. M (new), Table 17, §4.3.

W4. Larger dataset

Within the rebuttal period we could add only Kinetics-Sounds [5], with the same model and protocol as CREMA-D (final-epoch test, 3 seeds).

Compression2.5×5×10×
Uniform39.717.911.1
ModalShare42.028.417.2
Δ\Delta (seeds won)+2.3 ± 5.5 (2)+10.5 ± 6.5 (3)+6.1 ± 3.6 (3)

Notably, ModalShare improves over uniform at every budget and, with three seeds, this is consistent with our other datasets.

Revision. App. N (new), Tables 18–19, §6.

Q2. Three modalities

ModalShare was designed so it can work for any MM (Eqs. 5, 7–11). Exact Shapley needs 2M2^M server evaluations per probe (W2), and permutation sampling [Castro et al. 2009, Ghorbani & Zou 2019] reduces this for larger MM. We added IEMOCAP [Busso et al., 2008] with three modalities (text, audio, face motion capture), final-epoch test, 12 seeds.

Compression5×10×20×40×80×
Uniform62.059.758.456.052.0
ModalShare63.962.558.561.257.0
Δ\Delta (seeds won)+1.9 (11)+2.8 (8)+0.1 (6)+5.2 (10)+5.0 (8)

The uncompressed model reaches 67.7. ModalShare has the higher mean at four of five budgets, is level at 20× (+0.1 pp) and wins 43 of 60 seed–budget pairs. ModalShare thus handles three modalities, moving the payload off face motion capture (share 0.45 → 0.01) while keeping text and audio, and gains most at the tightest budgets.

Revision. App. K (new), Tables 13–14, §3.4, §6.

We hope our responses address your concerns and support a reassessment of the paper. We welcome any further questions or suggestions for strengthening the manuscript.

Thanks for the authors' additional experiments and rebuttal. Most of my concerns have been addressed.

I admit its contribution to Multimodal Split Learning field and establish a solid foundation for future exploration. However, I maintain my opinion on the moderate novelty of the ModalShare method.

Based on this reassessment, I decide to raise my score to 3.

We are glad that our responses have addressed most of the reviewer's concerns, made the contribution more clear and we appreciate raising your score. We thank the reviewer again for their time and constructive feedback.

Reviewer_ffZ9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This manuscript explores the issue of communication budget allocation in multimodal split learning. The author points out in the manuscript that existing split learning activation compression methods, when applied to multimodal scenarios, typically adopt the same retention rate for all modalities. This results in the communication budget being allocated based on the dimensionality ratio of the sliced activation values across modalities, ignoring the actual contribution differences of different modalities to the subsequent fusion prediction task. To address this issue, the author proposes the ModalShare framework, which utilizes the smashed activations already received by the server to construct a modal set through zero masking and calculate the Shapley value to estimate the actual contribution of each modality. Additionally, the author introduces an early stopping and freezing mechanism based on confidence gap, which stops probe evaluation after the dominant modality stabilizes to save computational power. The author conducted experiments on three datasets to demonstrate the effectiveness of the method.

Critical strengths

The research question addressed in the manuscript holds practical significance, particularly the assertion made therein that the equal keep-ratio in multimodal split learning does not equate to equal bandwidth, nor does it equate to allocating bandwidth based on task contribution.

Critical weaknesses

The authors claim in the manuscript that no prior method—whether in split learning or other fields—has addressed the allocation of a shared payload budget across multiple modalities. This assertion is inaccurate. In fact, I believe the manuscript’s true contribution lies in applying an existing, asymmetric compression allocation strategy—based on Shapley marginal contributions—specifically to the "cut layer" during the training phase of multimodal split learning. Furthermore, the authors demonstrate that, in the context of split learning, the activation values ​​already received by the server can be reused to compute Shapley-based probes, thereby eliminating the need for additional evaluation overhead on the client side.

Questions for authors

(1) Formula 4 in the manuscript calculates the payload using retained activation floating-point numbers, but then further mentions that the uplink and downlink payloads, including masks, indices, and scales, are also the same. I believe this conclusion requires more support, and I suggest that the author clarify the transmission representation of each type of compressor and report the uplink and downlink byte counts that include necessary metadata and control information.

(2) The probe evaluates the compressed representations under the current retention rate, so the score relies on historical allocation. If a certain modality receives fewer resources, its observable utility may decrease, which in turn affects subsequent resource allocation. Could the method produce path dependency on the early training state?

(3) Under the same budget feasible region, training protocol, and freezing mechanism, how does single-modality, static allocation, and LOO baseline compare to ModalShare?

(4) What is the actual number of communication bytes that include side information, and what is the actual time overhead caused by the probe?

(5) Overall, the independent evaluation process of MVSA is unclear, the budget definition and reported values need to be unified, the actual communication metrics are insufficient, and the comparison between Shapley and Simple Feasible Control has not fully distinguished the sources of benefits.

3 comments

Thank you for your review and for picking up on the point that equal keep-ratios do not mean equal bandwidth. Your question about real byte counts pushed us to measure them on an actual device.

W1. Novelty

To the best of our knowledge, no prior method divides a shared activation payload budget across modalities at the cut of split learning, and we are not aware of an existing Shapley-based allocation strategy for this setting. To make the claim precise, the revision scopes it to split learning (§4.1). Our contribution is to use Shapley values to divide the per-batch uplink budget at the cut. As you note, measurement adds no uplink or client computation because the server computes them from activations it already receives. We also state that equal keep-ratios split the budget by activation dimension (Eq. 6), and separate this skew from contribution with matched-payload controls (Table 3).

Revision. §2, §4.1.

Q1. Measured bytes

The compressors keep a subset of columns of each modality's activation, shared across the batch. Per modality and direction, a packet has a 39-byte header, one int32 index per kept column (4Km4K_m bytes, which replace a bit-mask) and the kept float32 values (4BKm4BK_m bytes, the quantity Eq. 4 counts). Scales are multiplied into the values (for SplitFC, 1/(1−p)1/(1-p)). The server returns gradients only for the received indices, in the same packing, and indices add 1/B1/B of the value bytes (1.6% at B=64B = 64) and scale with the budget.

To showcase the transmission payload with realistic measurements, the table gives measured MB per batch and direction between an NVIDIA Jetson Orin Nano client and a GPU server over TCP (early cut, SplitFC, β=0.20\beta = 0.20, B=64B = 64).

UniformModalShare
CREMA-D20.6020.60
MVSA12.9912.99

Every logged batch matched 4BK+4K+1024BK + 4K + 102 bytes on the downlink, and the uplink adds 507 bytes of sample ids per batch. Therefore, we show we don't introduce communication overhead.

Revision. §3.2, App. L (new), Table 15.

Q2. Early-training feedback

The allocation can depend on the early training state, because the probe scores the received activations and a modality sent at a lower keep-ratio can score lower. We introduce three parts of the rule in order to limit it. a) Equal starting scores make the first allocations split the payload evenly. b) The EMA averages over probes. c) The freeze needs the same winner for K=3K = 3 epochs after emin⁡=10e_{\min} = 10. One of 8 CREMA-D seeds locks the weaker modality (App. J), while all 12 revised MVSA runs end text-heavy (Table 11). Thus, we believe that these rules constrain the effect to a minority of runs.

Revision. App. F (new paragraph).

Q3. Sources of the gain

All comparisons use the same budget, protocol and freeze rule (MVSA, 5×, final epoch) unless stated. Among static splits, equal payload gives +7.4 pp over equal keep-ratios (Table 3) and a fixed text-heavy split with text share 0.8 gives +10.8 pp (Table 8), against +18.7 for ModalShare. The singleton utility in ModalShare's pipeline gives +1.3 ± 1.6 pp over uniform, and Shapley is +7.6 ± 14.2 pp above it. Single-modality models told which modality to keep act as an oracle (App. O), with text-only ahead by 3–9 pp on MVSA and audio-only by 2–6 pp on CREMA-D (40×–5×). On MVSA, ModalShare beats every fixed split that keeps both modalities from 10× to 2.5×. Leave-one-out (Table 8, 12 seeds) gives +9.8 ± 8.2 pp on MVSA and +9.0 ± 5.0 on CREMA-D, against +9.7 and +8.7 for online Shapley. With two modalities it ranks as Shapley does, at a different scale (App. G). The two differ in kind only for M≥3M\ge3.

Revision. App. B, App. G, App. O, Tables 3 and 8.

Q4. Probe cost

As mentioned above in Q1, the probe adds no bytes. It is 4 server forward passes without gradients on activations the server already holds, every Tprobe=10T_{\text{probe}} = 10 batches until freezing. On the testbed, averaged over batches, it costs 4.7 ms per batch on CREMA-D and 8.1 ms on MVSA. On a 10 Mbit/s link with the probe on every batch, it takes 0.17% of the batch time.

Revision. §3.5, App. L, Table 16.

Q5. Summary of concerns

  • The general response, Table 11, Q1 and Q4 cover evaluation, budget accounting and communication metrics.
  • In Q3 (MVSA, 5×), correcting the dimension skew gives +7.4 pp (equal payload), allocating by a contribution score +9.8 and +9.7 (online leave-one-out and Shapley), and freezing the allocation +18.7 (ModalShare, Table 7).
  • We could not find a method called 'Simple Feasible Control' and we kindly ask a reference.

We hope our responses address your concerns and support a reassessment of the paper. We welcome any further questions or suggestions for strengthening the manuscript.

Thank you for the author's response. After reviewing the response and the manuscript, I believe the overall quality of the manuscript has improved and it is on the borderline of acceptance. Therefore, I will maintain my score and raise the evaluation for the Soundness section.

Thank you for reading our response and the revised manuscript, and for raising the soundness score. If a specific concern still keeps the paper below acceptance, we would be glad to address it before the discussion period closes.

Reviewer_A4erOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work studies a communication efficient split learning approach for multimodal models. The observation the authors spotted points to existing compressors giving every modality the same keep ratio. Thus making the split budget related to the scale of the modality rather than the information correlation towards output. Thus they frame the split as a fixed-budget allocation problem where every policy sends the same expected payload and differs only in how that payload is divided across modalities.

Critical strengths

The paper makes a convincing case that dividing a shared budget across modalities is an overlooked design choice in multimodal split learning. It also shows that the default is not neutral. This hold of the expected payload, compressor, model and training schedule fixed makes the comparisons clean, because any accuracy difference comes from the split alone.

Critical weaknesses
  1. On CREMA-D the method gives audio 99% of the data, which is basically equilavent to dropping video. At 2.5× compression the method scores 55.2%, above the uncompressed two-modality model’s 53.1%. It seems like such pattern is typical of CREMA-D, where one modality tends to dominate and plain concatenation often does no better than audio alone. If audio-only (and text-only on MVSA) matches ModalShare, the +15% headline mostly means “drop the weak modality,” and equal keep-ratios is a weak comparison. I suggest reportting unimodal models (audio-only on CREMA-D, text-only on MVSA), both uncompressed and at matched payload.

  2. There are some concerns regarding selection bias, based on the author's description, for MVSA the paper builds a 60/20/20 split but reports accuracy on the validation partition, leaving the 519-sample held-out partition unused. If the validation partition is also used to choose checkpoints or hyperparameters, the reported MVSA numbers are not held-out estimates. Second, the reported accuracies appear to come from the best checkpoint after warmup rather than the final one. Such approach of taking the maximum over epochs on a small evaluation set inflates results, and the inflation grows with how much accuracy fluctuates from epoch to epoch. If ModalShare’s training runs are also noisier from epoch to epoch, which its larger across-seed spread suggests, the bias would favor it and could account for part of the reported gap. I would thus suggest reporting final-epoch accuracy on held-out test partitions for all datasets, for both ModalShare and the baselines.

Questions for authors

Please address the two concerns above.

3 comments

Thank you for your thoughtful review. We were glad you found the case for the budget split convincing and the comparisons clean, and you described the split as an overlooked design choice more clearly than we did. Your question about single-modality models went to the core of the method and is now a full appendix and a limitation in §6.

W1. Single-modality baselines

We agree that under compression ModalShare's allocation on CREMA-D and MVSA approaches hard modality selection, which §3.4 defines as the τ→0\tau \to 0 limit of our map. On these datasets, selecting the stronger modality in hindsight therefore acts as an oracle. A single-modality model told which modality to keep can match or beat ModalShare, and most of the gain over equal keep-ratios, the split existing compressors apply (§1), comes from moving the payload to the dominant modality. ModalShare approaches this oracle without being told, and the right choice depends on the budget and dataset. On CREMA-D, for example, video-only beats audio-only uncompressed (48.0 vs 42.9) but loses at every matched budget. The revision adds single-modality baselines at matched payload (final-epoch test).

CREMA-D (5 seeds)

Compression40×20×10×5×3.3×2.5×
ModalShare19.723.731.945.853.652.8
Audio only21.929.436.149.251.452.4
Video only17.019.519.828.434.041.0

Audio-only leads by 2–6 pp from 40× to 5×. ModalShare is ahead at 3.3× and level at 2.5×, and at 5× it sends 99% of the payload as audio (Table 2b).

MVSA (5 seeds). We swept fixed text shares of the payload from 0 to 1 and report the best fixed split that keeps both modalities (please see full grid in App. O).

Compression40×20×10×5×3.3×2.5×
ModalShare38.043.653.656.560.655.7
Text only47.149.858.863.163.163.1
Best fixed split (both)39.044.045.152.457.754.8
Uniform38.240.639.644.145.050.0
Image only35.036.837.238.740.339.5

Text-only is best at every budget. From 10× to 2.5× ModalShare is the best policy that keeps both modalities, ahead of every fixed split by 0.9–8.5 pp, while the best fixed text share moves with the budget (0.625–0.875). At 40× and 20× it is level with the best fixed split.

Distance to the oracle. Without any prior knowledge of which modality wins, ModalShare reaches up to 96% of the oracle's accuracy on MVSA (3.3×) and matches or beats the oracle on CREMA-D at 3.3× and 2.5×. It stays above 80% of the oracle's accuracy at every budget on both datasets.

We purposefully decide to keep a soft split as the default because hard selection (τ=0\tau = 0) can lose where contributions are close, and there ModalShare keeps more than one modality. On UCI HAR (g=0.12g = 0.12, 2.5×, 12 seeds) it keeps both streams and stays level with uniform (51.0 vs 51.1), while hard selection puts the whole payload on the accelerometer and reaches 49.7, 1.4 pp below uniform and 1.3 pp below ModalShare, within one seed standard deviation. On IEMOCAP (new M=3 dataset, please see our reply to DH9q, Q2) ModalShare keeps text and audio (shares 0.39/0.60). Imporantly, choosing the wrong modality is costly. The other modality alone (last row of each table) stays 12–24 pp below text-only on MVSA and falls to chance on CREMA-D at 40× (17.0% vs 16.7%).

A single modality or a compressed allocation can indeed beat the uncompressed two-modality model. This is a known effect of joint training, where the dominant modality suppresses the other [Wang et al. 2020, Peng et al. 2022], which is why the uncompressed model is not an upper bound here.

Revision. App. O (new), Tables 20–21, §3.4, §5.2, §6 (limitation).

W2. Evaluation protocol

(a) We understand the confusion our wording caused and clarify it below. The MVSA partition we reported on was named "validation" but was never used for training or hyperparameter tuning. (b) "Best post-warmup checkpoint" meant the peak of each run's accuracy curve on the evaluated split, which is the selection bias you describe. Therefore, we reran every method and now report final-epoch accuracy on the held-out test split. On MVSA the gain grows at 3.3–10× and shrinks at 2.5× and 20×. On noisier runs, at 5× the move from the submitted values to final-epoch test accuracy lowers uniform more than ModalShare. On CREMA-D both arms drop by a similar amount (uniform 32.2 → 31.1, ModalShare 47.6 → 45.8), and the 5× gain is +14.7 ± 4.5 pp on 12/12 seeds (+15.4 submitted). Thus, we believe our gains remain even with the updated evaluation.

Revision. §4.2, §4.4, Tables 2–10, Fig. 3, abstract, §5.

We hope our responses address your concerns and support a reassessment of the paper. We welcome any further questions or suggestions for strengthening the manuscript.

Thank you for the additional experiments and detailed explanation. All of my concerns have been addressed, and the new experiments clearly justifies the design of this algorithm, I will therefore maintain my acceptance recommendation.

We are glad that our responses have addressed all of the reviewer's concerns and we appreciate the positive assessment. We thank the reviewer again for their time and constructive feedback.

Reviewer_etw2OpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ModalShare allocates a fixed communication budget across modalities in multimodal split learning. It estimates modality contributions through server-side Shapley probes on already-received activations, converts these scores into per-modality keep-ratios, and freezes allocation once contributions stabilize. Experiments across three datasets and three compressors report substantial improvements on CREMA-D and MVSA, supported by allocation, split-depth, and contribution-score ablations.

Critical strengths
  1. The paper identifies a practical allocation decision distinct from compression. Separating how much each modality transmits from how its activations are compressed is useful for multimodal split learning. Computing contribution scores from already-received activations also avoids uploading additional activations specifically for attribution. (§3)

  2. The experiments provide substantial gains and informative controls. At 5× compression, reported accuracy improves from 32.2% to 47.6% on CREMA-D and from 48.0% to 60.4% on MVSA across 12 seeds. Equal-payload controls help distinguish contribution-aware allocation from correcting unequal activation dimensions, while improvements in 29/36 configurations support compatibility with multiple compressors. (Tables 2–5)

Critical weaknesses
  1. Evaluation independence and checkpoint selection are insufficiently specified. Section 4.2 reports MVSA’s validation partition as test accuracy despite reserving a separate held-out partition, while the appendix reports best post-warmup checkpoints without identifying the selection criterion or split. Consequently, the manuscript does not establish that its reported gains were evaluated independently of checkpoint selection. (§4.2; Appendices E and H)

  2. The central fixed-budget comparisons are not fully auditable. For MVSA at β=0.2\beta=0.2, the dimensions and keep-ratio cap imply that the expected text payload share defined by Equations (5)–(6) cannot exceed 0.9836, whereas Table 10 reports “realized” shares of 0.988–1.000 for seven seeds. The distinction between expected and realized statistics needs reconciliation through explicit share definitions, post-projection keep-ratios, and measured payload counts; these values alone do not establish an implementation error. (Table 1; Table 10)

Questions for authors
  1. Evaluation protocol: Which split and metric determine checkpoint selection? Please report MVSA results on the separate held-out partition and clarify whether all methods follow the same selection procedure.
  2. Payload accounting: Are Table 10’s shares pre-projection weights, ratios of expected payloads, or statistics of realized transmissions? Please provide post-projection keep-ratios and expected versus measured total payloads for both allocation policies.
  3. Contribution scoring: With two modalities, Equation (7) gives
    ϕ1−ϕ2=u({1})−u({2})\phi_1-\phi_2=u(\{1\})-u(\{2\}) How does the singleton-utility baseline transform negative log-probabilities into allocation scores, and does Shapley retain its advantage when score calibration, smoothing, projection, and freezing are matched?
  4. OGM-GE baseline: Appendix J describes unimodal-versus-fused score comparisons, whereas the original OGM-GE formulation compares the two unimodal scores. Which rule was implemented, and how were the modulation location and hyperparameters selected?
1 comment

Thank you for finding the allocation problem practical and the controls useful, and for pointing out the reuse of received activations. Your detailed questions also led to real corrections.

W1/Q1. Evaluation protocol

(a) We acknowledge the confusion of our phrasing and we clarify it here. The MVSA partition we reported on (518 pairs) was named "validation" but was never used for training or hyperparameter tuning, and the allocation hyperparameters are shared across datasets, budgets and compressors. (b) "Best post-warmup checkpoint" meant the peak of each run's accuracy curve on the evaluated split, which is the selection you point out. In order to address this concern, we reran every method and now report final-epoch accuracy on the held-out test split. MVSA trains on 1,555 pairs and reports on 519 held-out pairs, with the validation pairs unused. At 5×, ModalShare improves over uniform by +18.7 ± 10.7 pp on 11/12 seeds. On CREMA-D the 5× gain is +14.7 ± 4.5 pp on 12/12 seeds. Ultimately, this shows that the gains of ModalShare remain.

Revision. §4.2, §4.4, Tables 2–10, Fig. 3, abstract, §5.

W2/Q2. Payload accounting

We examined this point and concluded that the column in Table 10 was mislabelled. Its entries were q~text=Dtextrtext/∑mDmrm\tilde q_{\text{text}} = D_{\text{text}} r_{\text{text}} / \sum_m D_m r_m, the text share of ModalShare's own expected payload, whereas the bound of 0.9836 applies to qtext=Dtextrtext/TXq_{\text{text}} = D_{\text{text}} r_{\text{text}} / \mathrm{TX}. Both are ratios of expected payloads computed from the post-projection keep-ratios, and they differ only when rtextr_{\text{text}} is clipped. The projection rescales the keep-ratios to meet Eq. 5 and then clips them at 1 without reassigning the excess. Eq. 5 therefore holds with equality unless a keep-ratio reaches 1. When one does, ModalShare's payload falls below TX.

The new Table 11 gives per-seed post-projection keep-ratios and the expected and measured uplink payload of both policies (floats per sample over the epoch-30 batches, TX = 49,971.2). On 8 of 12 seeds, clipping rtextr_{\text{text}} leaves ModalShare's expected payload 1.4–1.6% below TX, and the measured payload is within 2.2 floats of it and below every uniform run. The other 4 seeds meet TX within 1.2 floats in expectation and within 0.1% when measured. On two, it is 0.15% above the seed-matched uniform run because of SplitFC's random column draw. Uniform measures within 0.17% of TX. Clipping requires β>Dm/∑kDk\beta > D_m / \sum_k D_k, which happens for MVSA text at β≥0.2\beta \ge 0.2 and IEMOCAP text at β=0.2\beta = 0.2, and never on CREMA-D or UCI HAR.

Revision. §3.4 (projection, clipping), §1, §3.2–3.5, §4.1, Tables 10–11, App. H.

Q3. Singleton score

The singleton score is an EMA of exp⁡u({m})\exp u(\{m\}), which for M=2M=2 gives the share σ(u1−u2)\sigma(u_1-u_2). ModalShare's share is 12+u1−u22(u(M)−u(∅))\tfrac12+\frac{u_1-u_2}{2(u(\mathcal{M})-u(\emptyset))} when both ϕm≥0\phi_m \ge 0. As you note, both rank the modalities alike. Because Shapley rescales the gap by the fused gain, a singleton score calibrated by u(M)−u(∅)u(\mathcal{M})-u(\emptyset) equals Shapley at M=2M=2, and coalitions add information only for M≥3M\ge3. With the same EMA, temperature, projection and freeze (MVSA, 5×), the singleton score reaches 45.2 ± 4.6, Shapley 52.9 ± 9.8 and uniform 43.9 ± 3.7. It moves the text share to 0.71–0.78 but never freezes. Shapley freezes near text-only on two seeds and is higher on both, and lower on the third.

Revision. §5.4, App. B (matched singleton run).

Q4. OGM-GE baseline

Regarding the initial OGM-GE baseline, the submitted App. J compared each unimodal score with the fused score, with an untuned α=1\alpha = 1, which differs from the original rule. The issue was that, with its coefficients near 1, it never damped either stream. Therefore, we replaced it with Peng et al.'s pairwise rule and released code (ρv=sv/sa\rho_v = s_v/s_a, ku=1−tanh⁡(αρu)k_u = 1 - \tanh(\alpha \rho_u) if ρu>1\rho_u > 1, with GE noise). kuk_u scales the gradients of modality uu's encoder in the client layers and server tower and leaves the cut gradient unchanged. The window (epochs 1–51 of 100) follows the released code. We chose α=0.1\alpha = 0.1 from {0.1, 0.3, 0.5, 0.8, 1.0} by final-epoch accuracy on a validation split made of the clips of 10% of the training speakers, removed from training, with the test split unchanged.

On CREMA-D at β=0.20\beta = 0.20, equal keep-ratios reach 30.3 ± 5.4 and Shapley keep-ratios 37.8 ± 10.8. OGM-GE is active (kaudio≈0.83k_{\text{audio}} \approx 0.83 on 98% of window batches) and lowers accuracy in both cases, to 26.9 ± 4.9 with equal keep-ratios (−3.4 pp, 7/8) and to 36.1 ± 7.3 with Shapley keep-ratios (−1.7 pp). Validation accuracy fell as α\alpha grew, with 0.1 being the weakest modulation in the grid.

Revision. App. J rewritten, Table 12 replaced.

We hope our responses address your concerns and support a reassessment of the paper. We welcome any further questions or suggestions for strengthening the manuscript.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response (to all reviewers)

We thank all reviewers for their careful reading. The reviewers found inter-modality allocation a practical and underexplored decision in multimodal split learning (etw2, A4er, ffZ9, DH9q) and the matched-payload setup clean (A4er), and valued that ModalShare adds no communication or client computation (etw2, DH9q). All changes in the revised manuscript are marked in blue.

1. Evaluation protocol (etw2, A4er, ffZ9). Every accuracy is now the final-epoch value on the held-out test split, with no checkpoint selection, for every method. MVSA trains on the 1,555 training pairs and reports on the 519 held-out pairs. The table gives uniform → ModalShare accuracy (%) and Δ\Delta in pp, with the number of seeds won out of 12 in parentheses.

CompressionMVSACREMA-DUCI HAR
2.5×48.5 → 55.6, +7.1 ± 8.7 (9)45.5 → 52.8, +7.4 ± 5.4 (12)51.1 → 51.0, −0.1 ± 0.9 (3)
3.3×43.3 → 59.1, +15.8 ± 8.4 (12)38.9 → 53.6, +14.7 ± 4.2 (12)47.7 → 47.5, −0.2 ± 0.7 (6)
5×40.5 → 59.2, +18.7 ± 10.7 (11)31.1 → 45.8, +14.7 ± 4.5 (12)42.6 → 44.0, +1.4 ± 2.2 (9)
10×39.3 → 53.4, +14.0 ± 7.0 (11)23.1 → 31.9, +8.8 ± 5.7 (11)38.6 → 39.1, +0.4 ± 1.2 (6)
20×40.5 → 43.9, +3.3 ± 5.2 (8)19.1 → 23.7, +4.6 ± 3.1 (12)37.7 → 37.5, −0.3 ± 0.7 (3)
40×39.5 → 40.2, +0.7 ± 3.9 (3, 5 ties)18.4 → 19.7, +1.3 ± 2.6 (8)37.3 → 37.2, −0.1 ± 0.3 (4)

The uncompressed models reach 60.3 (MVSA), 48.5 (CREMA-D) and 63.1 (UCI HAR) at the final epoch. UCI HAR, whose two streams contribute almost equally (g=0.12g = 0.12), stays within 1.4 pp of uniform at every budget. On MVSA at 5×, two fixed splits set before training also improve over uniform under the same budget, equal payload per modality (+7.4 pp, Table 3) and a fixed split with text share 0.8 (+10.8 pp, Table 8). ModalShare improves over both.

2. Payload accounting (etw2, ffZ9). §3.4 now states the projection precisely, and the new Table 11 gives post-projection keep-ratios with expected and measured payload. ModalShare's expected payload is never above uniform's.

3. Measured bytes and probe time (ffZ9). The new App. L measures split training between an NVIDIA Jetson Orin Nano client and a GPU server.

4. Three modalities and a larger dataset (DH9q). The new App. K adds IEMOCAP (text, audio, face motion capture) and the new App. N adds Kinetics-Sounds.

5. Modality-imbalance baseline (etw2, DH9q). App. J is rewritten with Peng et al.'s pairwise OGM-GE.

6. Batch size and fusion (DH9q). In the new App. M (MVSA, 5×), ModalShare keeps a gain of +17.5 to +20.2 pp at batch sizes 16–64 and with gate or attention fusion (5/5 seeds each), and of +8.2 pp at batch 128.

7. Single-modality baselines (A4er, ffZ9). We add App. O and a limitation paragraph in §6. Where one modality dominates (CREMA-D, MVSA), a single-modality model told in advance which modality to keep acts as an oracle. It is ahead of ModalShare on CREMA-D from 40× to 5× (audio-only, +2 to +6 pp) and on MVSA at every budget (text-only, +3 to +9 pp). Without this prior knowledge, ModalShare reaches up to 96% of the oracle's accuracy on MVSA and matches or beats the oracle on CREMA-D at 3.3× and 2.5×. It stays above 80% of the oracle's accuracy at every budget, with the lowest values at 20× and 40×. A sweep of fixed text shares on MVSA shows that from 10× to 2.5× ModalShare beats every fixed split that keeps both modalities. The modality to keep depends on the budget (CREMA-D), and on UCI HAR and IEMOCAP ModalShare keeps more than one modality.

8. Scope of claims (ffZ9, DH9q). The novelty claim is now scoped to split learning. §2 discusses the relation to sample-level valuation, and App. F the dependence of the score on the current allocation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.