- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces COMPACT, a framework for communication-efficient split learning of Vision-Language Models through activation compression. It evaluates seven compression strategies across four VLM architectures and multiple medical VQA benchmarks. The results highlight the strong dependence of compression effectiveness on the backbone architecture. The authors also investigate representation drift and show substantial communication savings with limited accuracy degradation.
Critical strengths
-
Comprehensive experimental evaluation. The paper systematically compares seven activation compression strategies across four VLM architectures and multiple medical VQA benchmarks. The evaluation reveals substantial backbone-dependent differences and provides useful insights into the robustness of different compression approaches.
-
Practical relevance and representation analysis. The proposed framework achieves substantial communication savings while largely preserving downstream performance. Furthermore, the analysis of representation drift provides an interesting empirical explanation of compression-induced performance degradation, strengthening the practical relevance of the study.
Critical weaknesses
-
Limited methodological novelty. The framework primarily integrates and evaluates existing compression techniques within a split-learning setting. While the empirical analysis is valuable, the paper does not introduce a new compression method, and some findings, such as the relationship between representation distortion and downstream accuracy, are relatively intuitive.
-
Limitations in experimental fairness and deployment realism. Learned compression methods are evaluated under relatively limited training budgets, potentially disadvantaging them compared to simpler quantization approaches. Moreover, the multi-client evaluation relies on simulated cross-silo settings with randomly partitioned data, rather than realistic non-IID distributions and actual distributed deployments. These limitations weaken the generalizability of the conclusions.
Questions for authors
-
Learned compression methods such as FSQ and RD-FSQ exhibit substantial performance degradation under the adopted training protocol. Could the authors give their own interpretation about this? How would their performance change with longer training or pretrained reconstruction modules? This would affect the conclusions regarding the superiority of NormalFloat quantization.
-
The cross-silo evaluation relies on randomly partitioned datasets. How would the proposed framework perform under non-IID client distributions, which are common in real-world medical applications? This is important for assessing the practical applicability of the reported results.
-
The authors report a strong correlation between cosine distance and downstream accuracy. Can they provide evidence that this relationship remains predictive across different tasks and compression settings?
3 comments
The decoder calibration results substantially address my concern about the evaluation of learned compression methods, while the non-IID experiments and controlled drift analysis strengthen the empirical validation.
I also appreciate the clarification of the paper's contribution relative to existing quantization and split-learning approaches. Although I still consider the methodological novelty moderate and real-world deployment remains insufficiently evaluated, the additional evidence addresses my main experimental concerns.
Overall, I am inclined to support a Weak Accept recommendation.
Thanks for your support and extra clarification on the novelty and the real-world setting
We sincerely thank the reviewer for reconsidering their assessment and for the constructive feedback. Your support and immediate response are very significant for us.
On methodological novelty: COMPACT contributes a unified multimodal split-learning framework that jointly compresses visual and textual activations across heterogeneous VLM architectures. Beyond integrating compression strategies, we identify reconstruction-decoder mismatch and quantization-level concentration as distinct failure mechanisms, connect representation drift with downstream performance, and demonstrate that decoder calibration can substantially recover FSQ/RD-FSQ accuracy. These findings provide both a practical framework and mechanistic insights into communication-efficient split VLM training.
On realistic deployment: We clarify that COMPACT already implements an operational three-client/one-server split-learning system, with measured end-to-end training and TCP activation transfer (Sec. 5.3, Appendix G). Our 4-bit NormalFloat configuration reduces communication by 72.5% while retaining 98.5% of BF16 accuracy. Trace-driven network simulations further estimate a 3.3× reduction in the uplink required for full-speed training under heterogeneous hospital-network conditions. These measurements use co-located client processes, while wide-area network conditions are modeled through simulation. Our additional five-client Dirichlet non-IID experiments complement this evaluation by examining robustness to statistical heterogeneity.
The remaining limitation concerns physical deployment across independent clinical institutions with naturally partitioned medical VQA data. Existing benchmarks illustrate this challenge.
- FLamby [1] provides seven healthcare datasets with natural institutional partitions, but covers classification, segmentation, and survival analysis rather than medical VQA.
- FedMultimodal [2] supports diverse multimodal FL tasks, including medical ECG classification, but does not include medical VQA.
Moreover, these benchmarks employ federated model-update exchange rather than COMPACT's split-learning activation-transfer protocol, making their published results not directly comparable. Prior studies show that split learning can offer communication advantages over FL, particularly for larger models and more clients [3,4]. COMPACT further reduces communication by eliminating client-side backward transfers and compressing forward activations, achieving a measured 72.5% reduction relative to uncompressed split learning.
COMPACT already incorporates established medical VQA datasets, including PathVQA, VQA-RAD, and SLAKE, but these do not provide natural hospital-level partitions. We therefore evaluate communication efficiency through measured multi-client execution and trace-driven network analysis, and statistical heterogeneity through controlled non-IID splits. We acknowledge that physical validation across independent hospitals would further strengthen generalizability and will clarify this distinction in the revised manuscript.
References
[1] Ogier du Terrail, J., et al. (2022). FLamby: Datasets and Benchmarks for Cross-Silo Federated Learning in Realistic Healthcare Settings. Advances in Neural Information Processing Systems (NeurIPS 35), Datasets and Benchmarks Track. https://arxiv.org/abs/2210.04620
[2] Feng, T., Bose, D., Zhang, T., Hebbar, R., Ramakrishna, A., Gupta, R., Zhang, M., Avestimehr, S., & Narayanan, S. (2023). FedMultimodal: A Benchmark for Multimodal Federated Learning. Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 4035–4045. https://doi.org/10.1145/3580305.3599825
[3] Singh, A., Vepakomma, P., Gupta, O., & Raskar, R. (2019). Detailed Comparison of Communication Efficiency of Split Learning and Federated Learning. arXiv:1909.09145. https://arxiv.org/abs/1909.09145
[4] Thapa, C., Chamikara, P. C. M. A., Camtepe, S., & Sun, L. (2022). SplitFed: When Federated Learning Meets Split Learning. Proceedings of the AAAI Conference on Artificial Intelligence, 36(8), 8485–8493. https://doi.org/10.1609/aaai.v36i8.20825
We thank the reviewer for the constructive feedback. We refer to our public comment to address novelty(W1). We further address the reviewer's concerns as follows.
Q1: We attribute the observed collapse to the mismatch between randomly initialized reconstruction decoders and pretrained VLM representations, which limited adaptation cannot correct.
We trained 4-bit Qwen3-VL-4B (seed 42) for two epochs with random decoder initialization. Mean accuracy changed from 10.13% to 9.61% (FSQ) and 10.68% to 15.00% (RD-FSQ), indicating that longer training alone did not prevent collapse. We then calibrated the reconstruction decoder using normalized MSE+cosine loss on 10% of training samples, with the VLM frozen, before one-epoch adaptation. Four task mean accuracy(%), random→calibrated:
Gains range from 18.41–35.42 pp. Reducing the drift near 1 to .0009–.0102(vision) & .0274–.0891(text). This supports decoder mismatch as a contributor rather than an inherent FSQ/RD-FSQ failure. Calibration costs 13.93–24.68 min per method/backbone on one V100 and requires representative activations.
We also ran 24 controlled combinations: Uniform/NormalFloat/Lloyd–Max × per-tensor/block-64 × four backbones. Block-64 Uniform vs NF mean accuracy is 44.70/44.66% (Qwen), 43.90/44.03% (InternVL), and 36.43/35.99% (PaliGemma). Per-tensor Lloyd–Max was also competitive; neither MSE nor balanced code consistently predicted accuracy Thus, compression effectiveness depends on the interaction between codebook design, quantization granularity, and VLM backbone. NormalFloat remains a strong calibration-free option, but is not universally superior. Together, these experiments support COMPACT's applicability across different compressors.
Q2 (W2): We further extended our experiments to five-client Dirichlet non-IID partitions[1], motivated by the statistical heterogeneity commonly observed across medical institutions[2]. Using 1M training samples, one epoch, and seed 42. We tested 4-bit block-64 NormalFloat (Ql4) on larger VLMs. Four-task accuracies (%) are ordered PathVQA/RadVQA/SLAKE/MedXpertQA-MM, with equally weighted means:
At α=.5, Ql4 differs from bf16 by only 0.51 pp (Qwen), 0.38 pp (PaliGemma), and -0.2 pp (InternVL). Reducing α from .5 to .1 increases client heterogeneity. Under this more uneven split, Qwen Ql4 achieves 48.54%, compared with 47.98% at α=.5. These results show that compressed activation transfer remains competitive even in the non-IID cross-silo settings.
Public medical VQA benchmarks generally lack natural institution-level partitions, making realistic multimodal cross-silo evaluation harder [3]. Thus, we adopt Dirichlet-based partitions to evaluate COMPACT under varying degrees of client heterogeneity, while acknowledging that real-world distributed deployment remains an important direction for future work.
Q3. We stratified 168 checkpoints (4 backbones × 7 methods × 2 bits × 3 seeds). For each test/bit width, we report partial Pearson correlations between late-layer image/text cosine distance and retained accuracy(% of compressed/bf16 accuracy), controlling for compression method and backbone:
All correlations, except the 4-bit MedXpertQA-MM image drift (r=+.031), are negative. Retained-accuracy ratios for MedXpertQA-MM are less informative about representation degradation, as the bf16 predictions are near-chance (18.22–25.24%). Overall, late-layer drift is a useful but task- and backbone-dependent diagnostic.
References
[1] Hsu et al . Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification. NeurIPS Workshop on Federated Learning.2019
[2] Rieke et al. The Future of Digital Health with Federated Learning. npj Digital Medicine, 2020
[3] Ogier du Terrail et al. FLamby: Datasets and Benchmarks for Cross-Silo Federated Learning in Realistic Healthcare Settings. NeurIPS,2022