acceptodds
Under review as a conference paper at ICLR 2027

COMPACT: A UNIFIED FRAMEWORK FOR MULTIMODAL SPLIT LEARNING WITH COMPRESSED ACTIVATIONS

Abstract

Split learning enables collaborative training of vision–language models without sharing raw data, but transmitting intermediate activations across distributed clients can incur substantial communication overhead. We investigate a lightweight multi-client setting in which client-side modules are frozen, eliminating gradient transmission and parameter synchronization, and focus on activation compression at the client-server boundary. We introduce COMPACT, a unified framework that supports heterogeneous vision–language backbones with interchangeable compression modules, and evaluate seven compression strategies across four vision–language models on four medical VQA benchmarks. We find that compression effectiveness is strongly model-dependent, and blockwise NormalFloat activation quantization (Ql) provides the most consistent performance across backbones. Amongst compression configurations, larger cosine distance from uncompressed representations is consistently related to downstream accuracy, with Spearman correlations ranging from −0.72 to −0.85. Layer-wise analysis further shows that internal representation drift caused by reconstruction errors accumulates through frozen layers. Moreover, we identify two recurring phenomena associated with performance degradation: unstable training of reconstruction modules under short training budgets and poor utilization of quantization levels. Finally, in a simulated cross-silo setting, COMPACT reduces communication by 72.5% for 4-bit Ql, while preserving 98.5% baseline performance. Together, these results show that compression effectiveness depends on both compression rate and its interaction with pretrained representations and downstream adaptation.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Multimodal Split Learning, Vision-Language Models, Quantization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_srKyOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces COMPACT, a framework for communication-efficient split learning of Vision-Language Models through activation compression. It evaluates seven compression strategies across four VLM architectures and multiple medical VQA benchmarks. The results highlight the strong dependence of compression effectiveness on the backbone architecture. The authors also investigate representation drift and show substantial communication savings with limited accuracy degradation.

Critical strengths
  1. Comprehensive experimental evaluation. The paper systematically compares seven activation compression strategies across four VLM architectures and multiple medical VQA benchmarks. The evaluation reveals substantial backbone-dependent differences and provides useful insights into the robustness of different compression approaches.

  2. Practical relevance and representation analysis. The proposed framework achieves substantial communication savings while largely preserving downstream performance. Furthermore, the analysis of representation drift provides an interesting empirical explanation of compression-induced performance degradation, strengthening the practical relevance of the study.

Critical weaknesses
  1. Limited methodological novelty. The framework primarily integrates and evaluates existing compression techniques within a split-learning setting. While the empirical analysis is valuable, the paper does not introduce a new compression method, and some findings, such as the relationship between representation distortion and downstream accuracy, are relatively intuitive.

  2. Limitations in experimental fairness and deployment realism. Learned compression methods are evaluated under relatively limited training budgets, potentially disadvantaging them compared to simpler quantization approaches. Moreover, the multi-client evaluation relies on simulated cross-silo settings with randomly partitioned data, rather than realistic non-IID distributions and actual distributed deployments. These limitations weaken the generalizability of the conclusions.

Questions for authors
  1. Learned compression methods such as FSQ and RD-FSQ exhibit substantial performance degradation under the adopted training protocol. Could the authors give their own interpretation about this? How would their performance change with longer training or pretrained reconstruction modules? This would affect the conclusions regarding the superiority of NormalFloat quantization.

  2. The cross-silo evaluation relies on randomly partitioned datasets. How would the proposed framework perform under non-IID client distributions, which are common in real-world medical applications? This is important for assessing the practical applicability of the reported results.

  3. The authors report a strong correlation between cosine distance and downstream accuracy. Can they provide evidence that this relationship remains predictive across different tasks and compression settings?

3 comments

We thank the reviewer for the constructive feedback. We refer to our public comment to address novelty(W1). We further address the reviewer's concerns as follows.

Q1: We attribute the observed collapse to the mismatch between randomly initialized reconstruction decoders and pretrained VLM representations, which limited adaptation cannot correct.

We trained 4-bit Qwen3-VL-4B (seed 42) for two epochs with random decoder initialization. Mean accuracy changed from 10.13% to 9.61% (FSQ) and 10.68% to 15.00% (RD-FSQ), indicating that longer training alone did not prevent collapse. We then calibrated the reconstruction decoder using normalized MSE+cosine loss on 10% of training samples, with the VLM frozen, before one-epoch adaptation. Four task mean accuracy(%), random→calibrated:

BackboneFSQRD-FSQ
Qwen3-VL-4B10.13→44.6210.68→45.50
InternVL2.5-8B9.22→44.6415.63→44.19
PaliGemma2-10B11.44→40.0321.16→39.57
DeepSeek-VL2-tiny10.94→37.627.30→36.24

Gains range from 18.41–35.42 pp. Reducing the drift near 1 to .0009–.0102(vision) & .0274–.0891(text). This supports decoder mismatch as a contributor rather than an inherent FSQ/RD-FSQ failure. Calibration costs 13.93–24.68 min per method/backbone on one V100 and requires representative activations.

We also ran 24 controlled combinations: Uniform/NormalFloat/Lloyd–Max × per-tensor/block-64 × four backbones. Block-64 Uniform vs NF mean accuracy is 44.70/44.66% (Qwen), 43.90/44.03% (InternVL), and 36.43/35.99% (PaliGemma). Per-tensor Lloyd–Max was also competitive; neither MSE nor balanced code consistently predicted accuracy Thus, compression effectiveness depends on the interaction between codebook design, quantization granularity, and VLM backbone. NormalFloat remains a strong calibration-free option, but is not universally superior. Together, these experiments support COMPACT's applicability across different compressors.

Q2 (W2): We further extended our experiments to five-client Dirichlet non-IID partitions[1], motivated by the statistical heterogeneity commonly observed across medical institutions[2]. Using 1M training samples, one epoch, and seed 42. We tested 4-bit block-64 NormalFloat (Ql4) on larger VLMs. Four-task accuracies (%) are ordered PathVQA/RadVQA/SLAKE/MedXpertQA-MM, with equally weighted means:

Backbone/settingFour tasks (mean)
Qwen3-VL-32B,bf16(α=.5)38.40/56.89/69.96/28.69 (48.49)
Qwen3-VL-32B,Ql4(α=.5)38.26/53.37/70.92/29.37 (47.98)
Qwen3-VL-32B,Ql4(α=.1)38.32/55.43/70.70/29.71 (48.54)
PaliGemma2-28B,bf16(α=.5)25.87/31.96/35.60/2.04 (23.87)
PaliGemma2-28B,Ql4(α=.5)25.97/32.99/33.11/1.87 (23.49)
InternVL2.5-26B,bf16(α=.5)36.48/53.37/60.73/22.24 (43.21)
InternVL2.5-26B,Ql4(α=.5)36.42/53.96/60.00/23.26 (43.41)

At α=.5, Ql4 differs from bf16 by only 0.51 pp (Qwen), 0.38 pp (PaliGemma), and -0.2 pp (InternVL). Reducing α from .5 to .1 increases client heterogeneity. Under this more uneven split, Qwen Ql4 achieves 48.54%, compared with 47.98% at α=.5. These results show that compressed activation transfer remains competitive even in the non-IID cross-silo settings.

Public medical VQA benchmarks generally lack natural institution-level partitions, making realistic multimodal cross-silo evaluation harder [3]. Thus, we adopt Dirichlet-based partitions to evaluate COMPACT under varying degrees of client heterogeneity, while acknowledging that real-world distributed deployment remains an important direction for future work.

Q3. We stratified 168 checkpoints (4 backbones × 7 methods × 2 bits × 3 seeds). For each test/bit width, we report partial Pearson correlations between late-layer image/text cosine distance and retained accuracy(% of compressed/bf16 accuracy), controlling for compression method and backbone:

Benchmark2-bit image/text4-bit image/text
PathVQA−.451/−.489−.410/−.246
RadVQA−.396/−.499−.447/−.321
SLAKE−.504/−.488−.318/−.618
MedXpertQA-MM−.393/−.263+.031/−.429

All correlations, except the 4-bit MedXpertQA-MM image drift (r=+.031), are negative. Retained-accuracy ratios for MedXpertQA-MM are less informative about representation degradation, as the bf16 predictions are near-chance (18.22–25.24%). Overall, late-layer drift is a useful but task- and backbone-dependent diagnostic.

References

[1] Hsu et al . Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification. NeurIPS Workshop on Federated Learning.2019

[2] Rieke et al. The Future of Digital Health with Federated Learning. npj Digital Medicine, 2020

[3] Ogier du Terrail et al. FLamby: Datasets and Benchmarks for Cross-Silo Federated Learning in Realistic Healthcare Settings. NeurIPS,2022

The decoder calibration results substantially address my concern about the evaluation of learned compression methods, while the non-IID experiments and controlled drift analysis strengthen the empirical validation.

I also appreciate the clarification of the paper's contribution relative to existing quantization and split-learning approaches. Although I still consider the methodological novelty moderate and real-world deployment remains insufficiently evaluated, the additional evidence addresses my main experimental concerns.

Overall, I am inclined to support a Weak Accept recommendation.

AuthorsOpenReview ↗

Thanks for your support and extra clarification on the novelty and the real-world setting

We sincerely thank the reviewer for reconsidering their assessment and for the constructive feedback. Your support and immediate response are very significant for us.

On methodological novelty: COMPACT contributes a unified multimodal split-learning framework that jointly compresses visual and textual activations across heterogeneous VLM architectures. Beyond integrating compression strategies, we identify reconstruction-decoder mismatch and quantization-level concentration as distinct failure mechanisms, connect representation drift with downstream performance, and demonstrate that decoder calibration can substantially recover FSQ/RD-FSQ accuracy. These findings provide both a practical framework and mechanistic insights into communication-efficient split VLM training.

On realistic deployment: We clarify that COMPACT already implements an operational three-client/one-server split-learning system, with measured end-to-end training and TCP activation transfer (Sec. 5.3, Appendix G). Our 4-bit NormalFloat configuration reduces communication by 72.5% while retaining 98.5% of BF16 accuracy. Trace-driven network simulations further estimate a 3.3× reduction in the uplink required for full-speed training under heterogeneous hospital-network conditions. These measurements use co-located client processes, while wide-area network conditions are modeled through simulation. Our additional five-client Dirichlet non-IID experiments complement this evaluation by examining robustness to statistical heterogeneity.

The remaining limitation concerns physical deployment across independent clinical institutions with naturally partitioned medical VQA data. Existing benchmarks illustrate this challenge.

  • FLamby [1] provides seven healthcare datasets with natural institutional partitions, but covers classification, segmentation, and survival analysis rather than medical VQA.
  • FedMultimodal [2] supports diverse multimodal FL tasks, including medical ECG classification, but does not include medical VQA.

Moreover, these benchmarks employ federated model-update exchange rather than COMPACT's split-learning activation-transfer protocol, making their published results not directly comparable. Prior studies show that split learning can offer communication advantages over FL, particularly for larger models and more clients [3,4]. COMPACT further reduces communication by eliminating client-side backward transfers and compressing forward activations, achieving a measured 72.5% reduction relative to uncompressed split learning.

COMPACT already incorporates established medical VQA datasets, including PathVQA, VQA-RAD, and SLAKE, but these do not provide natural hospital-level partitions. We therefore evaluate communication efficiency through measured multi-client execution and trace-driven network analysis, and statistical heterogeneity through controlled non-IID splits. We acknowledge that physical validation across independent hospitals would further strengthen generalizability and will clarify this distinction in the revised manuscript.

References

[1] Ogier du Terrail, J., et al. (2022). FLamby: Datasets and Benchmarks for Cross-Silo Federated Learning in Realistic Healthcare Settings. Advances in Neural Information Processing Systems (NeurIPS 35), Datasets and Benchmarks Track. https://arxiv.org/abs/2210.04620

[2] Feng, T., Bose, D., Zhang, T., Hebbar, R., Ramakrishna, A., Gupta, R., Zhang, M., Avestimehr, S., & Narayanan, S. (2023). FedMultimodal: A Benchmark for Multimodal Federated Learning. Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 4035–4045. https://doi.org/10.1145/3580305.3599825

[3] Singh, A., Vepakomma, P., Gupta, O., & Raskar, R. (2019). Detailed Comparison of Communication Efficiency of Split Learning and Federated Learning. arXiv:1909.09145. https://arxiv.org/abs/1909.09145

[4] Thapa, C., Chamikara, P. C. M. A., Camtepe, S., & Sun, L. (2022). SplitFed: When Federated Learning Meets Split Learning. Proceedings of the AAAI Conference on Artificial Intelligence, 36(8), 8485–8493. https://doi.org/10.1609/aaai.v36i8.20825

Reviewer_cvWsOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces a framework called COMPACT for studying activation compression in multimodal split learning. In this setup, clients execute frozen vision blocks and text embedding modules, transmitting compressed representations and tokenized supervisory information to a server, which then performs the remaining computations and adaptive operations. The authors evaluate seven compression methods across four medical VQA benchmark datasets, using four pre-trained vision-language models (VLMs) at two nominal bit-widths, with each configuration tested using three different training seeds. The results indicate that Ql—a block-wise NormalFloat quantizer—exhibits the most consistent performance among all evaluated methods. The paper links performance degradation to "representation drift" and discusses two specific failure modes: reconstruction mismatch and concentrated code usage. Furthermore, through a separate experiment involving Qwen3-VL-32B and network simulations, the authors examine the communication savings achieved with three clients.

Critical strengths
  1. The consistent training configuration used in this paper makes the comparative results easier to interpret, while the use of three different random seeds provides some insight into the variability of these results. Furthermore, the paper explicitly reports the half-ranges for accuracy metrics and—notably—accounts for the metadata overhead required by specific methods when calculating effective compression ratios.

  2. The paper makes a valuable attempt to link accuracy results with internal model representations and communication costs. In the "drift" analysis, the study controls for variables such as the specific method, bit-width, and backbone model; the appendix further explores the relationships between these factors by aggregating various configurations while excluding specific methods or backbone models. The results show an average partial correlation coefficient of −0.49 for text-embedding drift and −0.47 for image drift at the LLM input layer. These figures provide a more robust foundation for the proposed diagnostic method than judgments based solely on visual trends. The extended experiments on a 32B (32-billion-parameter) model are also highly relevant to practical application scenarios: with 4-bit quantization, the communication volume per client per step is reduced from 123.7 MB to 34.0 MB, while maintaining an overall accuracy of 98.5%. The implementation includes the actual packing process for quantized values, and the trace-driven simulation analysis further examines the impact of constrained uplink bandwidth.

Critical weaknesses
  1. This paper lacks a crucial control experiment: uniform quantization using the same block-based structure as Ql. Methods such as DQ, LQ, and AWQ calculate quantization ranges based on the entire tensor following their respective transformations, whereas Ql employs a block-based approach (with a block size of 64 elements) and replaces standard quantization levels with "NormalFloat." Consequently, this comparison alters two significant design choices simultaneously. Given that Section 6.2 attributes model failure to "outliers dominating the global quantization range," a uniform quantization method utilizing local block ranges would be highly relevant to this explanation.

  2. While Figure 4 supports the observation that there are differences in "codebook usage," it fails to disentangle the impact of the codebook itself from the impact of using local quantization ranges. Proposition 3 also addresses block-based uniform quantization; therefore, it cannot serve as proof that NormalFloat offers a unique advantage. Although Ql’s performance in the current comparative experiments is compelling, it remains unclear whether this advantage stems from the NormalFloat quantization levels or simply from the adjustment of quantization granularity. If the latter accounts for the majority of the performance gap, the necessity of NormalFloat becomes questionable. A more direct validation of the authors' proposed explanation could be achieved by standardizing the block size, normalization method, and metadata precision across the compared approaches.

  3. The interpretation regarding the failure of FSQ (Finite Scalar Quantization) requires more direct verification. Both FSQ variants introduce a randomly initialized reconstruction map preceding a pre-trained network that remains largely frozen. Although the decoder is trained, this adaptation is limited to a single epoch, and LoRA is applied only to the final third of the language model decoder. Phenomena observed during experiments—such as high sensitivity to random seeds, output repetition, and occasional reversals in the expected bit-width ordering—all point to challenges within the optimization process. However, these observations alone are insufficient to quantify the extent to which the "initialization" phase contributes to model failure.

  4. Proposition 1 provides a theoretical explanation for issues arising from mismatches during the initialization phase—and the subsequent slow recovery—within fixed, linear downstream models that employ quadratic objective functions. It would be highly compelling to apply this theoretical framework to VQA (Visual Question Answering) experiments by modifying the initialization of the reconstruction mapping to see if the observed model collapse can be mitigated. For instance, one could calibrate the decoder based on the original activations before proceeding with the standard VQA training pipeline, thereby testing the proposed cause of failure. Current results indicate that, under this approach, these FSQ variants fail to achieve the desired performance; yet, it remains unclear whether such failures could be easily averted by adopting alternative initialization strategies.

Questions for authors
  1. What happens if the NormalFloat levels in Ql are replaced by a uniform codebook while retaining the same 64-element blocks, normalization, metadata precision, and training schedule? Reporting code utilization, reconstruction drift, and downstream accuracy would show which part of Ql accounts for the observed advantage.

  2. Does FSQ still collapse if its decoder is initialized through reconstruction calibration before following the same VQA adaptation schedule? Please report the initial and final drift as well as accuracy, and include any calibration cost. A result of this kind could substantially strengthen or change the interpretation in Section 6.1.

  3. Could the authors reconcile the deployment-time descriptions in Appendix G? Lines 1516–1519 describe measured full epochs of 32.7/32.3/32.5 hours for bf16/Ql 4-bit/Ql 2-bit. The captions of Tables G.3–G.4 instead say that only Ql 4-bit was measured, and the table gives 32.6/32.3/32.3 hours. Please identify the measured runs and the calibration/replay inputs associated with each row, and explain whether the distinction changes the bandwidth or training-time conclusions.

3 comments

We thank the reviewer for these insightful suggestions.

Q1 (W1–W2). Codebook/granularity. We compare NormalFloat, Uniform and Lloyd–Max across granularities under matched settings. All runs use 4-bit, one epoch, seed 42, with normalization/metadata controlled. Four-task mean accuracy (%), per-tensor→block-64.

Backbone(original BF16 baselines)UniformNormalFloatLloyd–Max
Qwen(44.53)41.78→44.7043.24→44.6644.27→44.74
InternVL(43.90)38.84→43.9043.66→44.0343.75→44.54
PaliGemma(37.42)20.80→36.4327.16→35.9935.96→36.37

Block-64 Uniform is within 0.44 pp of NF. Block-64 gains are 2.92–15.63 pp (Uniform) versus
0.41–0.79 pp (Lloyd–Max). Per-tensor Lloyd–Max’s accuracy nears BF16, showing that neither NF nor blockwise scaling is universally required for near-BF16 accuracy across backbones.

We further compare reconstruction MSE, late-layer drift, and accuracy for Qwen block-64.

Qwen block-64Vision MSEText MSE (×10⁻⁶)Late image driftLate text driftAcc. (%)
Uniform.0032633.137.2040.018144.70
NormalFloat.0025083.331.1889.017244.66
Lloyd–Max.0023682.871.1872.022944.74

The feature drift is averaged over the final 12 LLM layers. Lloyd–Max minimizes both MSEs and late-image drift, while NF minimizes late-text drift. Yet all achieve similar accuracy (44.66–44.74%). Thus, lower reconstruction error or late-layer drift does not necessarily improve accuracy. Pertensor-calibrated FSQ/RD-FSQ also supports this.

QuantizerVision MSEMean accuracy
Calibrated FSQ.00069644.62%
Calibrated RD-FSQ.00210645.50%

FSQ has lower MSE but lower accuracy. We measure block-64 code use with H=−Σp log₂p, N_eff=2^H≤16 (vision/text).

BackboneUniform N_eff(V/T)NF N_eff(V/T)
Qwen12.64/13.8615.11/15.70
InternVL14.19/13.3015.59/15.53
PaliGemma13.70/13.4315.30/15.54

Higher $N_{\mathrm{eff}}$ means more evenly distributed codes. NF has higher values than Uniform on all three backbones, but not consistently higher accuracy (Qwen 44.66 vs. 44.70%; PaliGemma 35.99 vs. 36.43%). On PaliGemma, per-tensor Lloyd–Max reaches 35.96% without blockwise scaling. MSE and code usage capture distinct compression properties, but neither alone consistently predicts downstream accuracy. Therefore, compression components must be evaluated with backbones and downstream tasks (see public novelty comment).

Q2 (W3–W4). FSQ calibration. With the VLM frozen, we calibrate linear decoders using NMSE+0.1(1−cos), AdamW 10⁻⁴, three epochs on 10% of training data, then apply 4-bit/seed-42/one-epoch VQA training on all samples. Mean accuracy (%) is random→calibrated.

BackboneFSQRD-FSQ
Qwen10.13→44.6210.68→45.50
InternVL9.22→44.6415.63→44.19
PaliGemma11.44→40.0321.16→39.57
DeepSeek10.94→37.627.30→36.24

All eight configs gain 18.41–35.42 pp. Cosine drift 1−cos(h,ĥ). RI denotes random initial vision, CI/CF denotes calibrated initial/final vision, TI/TF denotes calibrated initial/final text. Initial/final are before/after VQA adaptation.

VLMMethodRICICFTITF
Qwen3-VLFSQ0.98170.00140.06150.04180.2233
RD-FSQ0.97820.00310.04170.02740.5926
InternVL2.5FSQ0.99200.00800.01880.08910.2697
RD-FSQ0.99200.01020.03740.03670.4625
PaliGemma2FSQ0.99000.00170.04120.03550.1819
RD-FSQ0.99270.00690.02880.03850.4804
DeepSeek-VL2FSQ1.01140.00090.04110.02960.0461
RD-FSQ1.00050.00470.05070.03410.1099

Calibration rescues FSQ/RD-FSQ across backbones despite increased final drift, implicating initialization mismatch rather than inherent failure. Calibration takes 13.93–24.68 min (0.23–0.41 GPU-h) on a V100 32GB and needs representative activations.

Q3. Training time measurement and simulation. We corrected inconsistent Appendix G training-time estimates and recomputed simulations. Earlier BF16/Ql2 simulations scaled Ql4 measurements by compute-time ratios, explaining the discrepancy. We now use independently measured times for BF16, Ql4, and Ql2, applying the specified bandwidth, latency, and VPN throughput to simulate network-constrained training. Row 1 reports directly measured full-epoch times, while all remaining rows are simulated (hours). Large/small denotes hospital size.

SettingBF16Ql4Ql2
Co-located measured32.732.332.5
Large IPsec 25%32.732.332.5
Large IPsec 10%33.232.332.5
Large OpenVPN 25%40.132.332.5
Large OpenVPN 10%92.333.132.5
Small 25%95.133.532.5
Small 10%231.568.639.1
2 large+1 small 10%230.368.038.2
Shared 100-Mbps ingress68.032.332.5

Updates differ from original simulation by ≤0.5 h. For two-large-one-small at 10% uplink, Ql4/Ql2 cut 230.3 h to 68.0/38.2 h. Thus the bandwidth-limited training benefit persists.

Dear cvWs,

Thank you again for your insightful and constructive suggestions!

We have carefully addressed your concerns through extensive additional experiments, including controlled comparisons of quantization codebooks and granularities, calibrated FSQ/RD-FSQ evaluations across four VLM backbones, and corrected training-time measurements and simulations. These results have significantly strengthened our experimental conclusions.

We understand that your time is limited. As the discussion period is approaching its end, we would sincerely appreciate it if you could review our rebuttal and let us know whether the additional experiments and analyses have adequately addressed your major concerns, or if any substantive issues remain.

Thank you again for your time and for helping us improve the work!

Dear Reviewer cvWs,

With only 1-2 days remaining in the ICLR 2027 discussion period, we would greatly appreciate it if you could share your thoughts on our rebuttal. We would be happy to address any remaining concerns.

Thank you again for your time and consideration!

Best regards, Authors

Reviewer_XkAvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

COMPACT is a split-learning setup for fine-tuning VLMs. Each client runs only frozen front ends (the vision tower’s patch embedding and the LLM’s token embedding), compresses those activations, and sends them to a server that does all remaining computation and training.

Critical strengths
  1. The engineering and statistical reporting presented in the papier are good but the technical novelty is limited.
Critical weaknesses
  1. Novelty: The frozen-client design isn’t new. SplitFrozen already freezes device-side layers so clients only run forward passes. Also Ql is QLoRA’s method applied to activations. Blockwise scaling beating per-tensor scaling on outlier-heavy activations is well established in LLM quantization (LLM.int8, SmoothQuant, group-wise quantization in GPTQ/AWQ). The paper's primary contribution therefore seems to be a systematic empirical comparison and an analysis of compression failures rather than a new compression algorithm.

  2. Authors mentioned in the conclusion that: "COMPACT targets data locality and communication efficiency rather than formal privacy". If privacy is out of scope of this work, clients could send token IDs (about 17 bits per token) losslessly, instead of a quantized D-dimensional vector (about 4 × 2,560 ≈ 10,000 bits per token at “4 bits”). Compressing token embeddings is strictly worse than sending the IDs the client already has, and the paper already sends tokenized supervision.

  3. Authors keep the vison encoder on the client side and send embeddings for the privacy purposes. However, embeddings are enough to extract information and details of the image and prompt. One can easily use the llm head to understand the details of images and text prompts given the embeddings.

  4. The end-to-end design is based on the assumption that there should be no gradient at local site which means vision encoders should be freezed. However, for medical application and domains with limited available public data (such as the scenario in this paper) domain specific vison encoder is really important. Therefore, not updating the the vison encoder degrades the performance.

Questions for authors

see the weakness section

3 comments

We thank the reviewer for the feedback and have clarified the privacy scope in the manuscript.

W1. Novelty. COMPACT does not propose a new quantization method. Instead, we propose a novel communication-efficient split learning (SL) VLM framework using compressed visual and textual activation transfer. We evaluate seven compression strategies across four VLM backbones and conduct 24 controlled experiments. Results show effective transfer across multiple compression designs, rather than relying on a single quantizer. We refer to Appendix Table A.1 and our public comment for W1.

W2. Token IDs. Though token-ID transfer is more communication-efficient than our current approach, COMPACT’s setting prohibits raw-input transfer. By “formal privacy is outside our scope,” we mean that COMPACT does not establish differential privacy [1] or cryptographic guarantees [2].

(1) Raw-input boundary. Input token IDs reveal prompts via the tokenizer, thus violating COMPACT's local input constraint. Following SL[3], clients instead transmit compressed representations ($Q(h)$), where $h=f_{\mathrm{client}}(x,\theta_{\mathrm{frozen}})$. The server is trusted to train on representations, but not to receive raw inputs. Server trust and raw data non-transfer has been studied in FL formulation [4].

(2) Targets versus inputs. Section 3.1 distinguishes prompts, targets, and supervised positions. Our setting permits transmission of de-identified task supervision for server-side loss computation, but not transmission of input prompts. This separation is consistent with original SL, which shares labels while keeping raw inputs local [3]. Permitting supervision therefore does not imply permitting input token IDs. This is an explicit data-sharing assumption, not a claim that labels are inherently non-sensitive

W3. Privacy scope. Since representations can leak original input information, and prior work has investigated several attacks [5–10], COMPACT does not assume that representations are inherently private or that activation compression prevents these attacks. COMPACT assumes a trusted server training on representations, consistent with W2’s raw-input boundary. We do not establish formal privacy guarantees, such as differential privacy or cryptographic confidentiality, nor do we claim resistance to reconstruction attacks.

W4. Frozen client and domain adaptation. Domain-specific vision encoders can be important in medical tasks. However, COMPACT’s frozen-client design does not require freezing the entire vision encoder. As Appendix B shows, the visual split follows the client-side patch embedding. All ViT blocks reside on the server and can be adapted using LoRA without client-side backpropagation or cross-split gradient transmission. Our original benchmark froze the full vision tower for efficiency, not as an architectural requirement. We will clarify this distinction in the manuscript. We tested Qwen with server-side ViT LoRA in both the Ql 4-bit and uncompressed bf16 configurations:

Setting/Accuracy(%)PathVQARadVQASLAKEMedXpertMean
Ql 4-bit+ViT LoRA37.7649.2769.3823.6045.00
Bf16+ViT LoRA36.6948.8367.6222.5843.93

With server-side ViT LoRA, Ql achieves 45% versus 43.93% for bf16 in mean accuracy. Compressed activation transfer thus remains compatible with server-side ViT training while clients perform only lightweight embedding computations, supporting domain-specific vision encoders.

References

[1] Behnia et al. Ew-tune: A framework for privately fine-tuning large language models with differential privacy. In 2022 IEEE International Conference on Data Mining Workshops (ICDMW), pp. 560–566. IEEE, 2022

[2] Yao et al. A survey on large language model security and privacy: The good, the bad, and the ugly. High-Confidence Computing, pp. 100211, 2024.

[3] Vepakomma et al. Split Learning for Health: Distributed Deep Learning Without Sharing Raw Patient Data. 2018.

[4] McMahan et al. Communication-Efficient Learning of Deep Networks from Decentralized Data. AISTATS, 2017.

[5] Poirot et al. Split Learning for Collaborative Deep Learning in Healthcare. 2019.

[6] Qu et al. Natural Language Understanding with Privacy-Preserving BERT. CIKM 2021.

[7] Erdoğan et al. UnSplit: Data-Oblivious Model Inversion, Model Stealing, and Label Inference Attacks Against Split Learning. WPES 2022.

[8] Shu et al. Model Inversion in Split Learning for Personalized LLMs: New Insights from Information Bottleneck Theory. 2025.

[9] Lin et al. An Inversion Attack Against Obfuscated Embedding Matrix in Language Model Inference. EMNLP 2024.

[10] Xu et al. A Stealthy Wrongdoer: Feature-Oriented Reconstruction Attack Against Split Learning. CVPR 2024.

Dear XkAv,

Thank you again for your detailed and constructive feedback!

We have carefully addressed your concerns (W1–W4) through additional experiments and clarifications over transfer and privacy scope. We also conducted new experiments with server-side ViT LoRA to demonstrate that compressed activation transfer remains compatible with vision encoder adaptation.

We understand that your time is limited. As the discussion period is approaching its end, we would sincerely appreciate it if you could review our rebuttal and let us know whether the additional evidence and clarifications adequately address your major concerns, or if any substantive issues remain.

Thank you again for your time and thoughtful consideration.

Dear Reviewer XkAv,

With only 1-2 days remaining in the ICLR 2027 discussion period, we would greatly appreciate it if you could share your thoughts on our rebuttal. We would be happy to address any remaining concerns.

Thank you again for your time and consideration!

Best regards, Authors

Comments on the paper

2 comments
AuthorsOpenReview ↗

Unified comments for the novelty of the COMPACT

We thank the reviewers for their feedback regarding our work. In this comment, we'd like to defend the novelty of the method more in detail

1. Distinct problem and solution. COMPACT is a communication-efficient split-learning framework for VLMs that transfers compressed visual and textual activations across the client and server. Prior works, including frozen-client training, NormalFloat, and blockwise quantization, are already discussed and cited in our manuscript (Table A.1.). However, our contribution addresses a different research question: Can compressed activation transfer in split learning of VLMs reduce the communication bottleneck while preserving performance, and why do methods succeed or fail across backbones? We summarize the different objectives of prior work below.

WorkSettingFocusPrimary objectiveSplit learning
SplitFrozenLLM fine-tuning—Client computationYes
QLoRALLM fine-tuningWeightsTraining memoryNo
LLM.int8/SmoothQuantLLM inferenceWeights + activationsInference efficiencyNo
GPTQ/AWQLLM inferenceWeightsInference efficiencyNo
COMPACTVLM fine-tuningactivationsCommunication efficiencyYes

Closer split-learning work also differs. SplitFC studies unimodal activation compression, while Quantized-TinyLLaVA doesn’t keep paired images and text with each client. Neither evaluates our combination of different pretrained VLMs, multiple clients, and frozen client-side layers (see manuscript Table A.1).

2. Flexible integration of compression modules within COMPACT. COMPACT provides a unified framework that supports flexible combinations of existing compression components for efficient activation transfer. We validate this using 24 controlled configurations combining three codebooks (Uniform, NormalFloat, and Lloyd–Max), two quantization granularities (per-tensor and block-64), and four VLM backbones (4-bit, seed 42, one epoch).

VLM(bf16)CodebookPer-tensor accuracyBlock-64 accuracy
Qwen3-VL-4B (44.53%)Uniform41.78%44.70%
NormalFloat43.24%44.66%
Lloyd–Max44.27%44.74%
InternVL2.5-8B (43.90%)Uniform38.84%43.90%
NormalFloat43.66%44.03%
Lloyd–Max43.75%44.54%
PaliGemma2-10B (37.42%)Uniform20.80%36.43%
NormalFloat27.16%35.99%
Lloyd–Max35.96%36.37%
DeepSeek-VL2-tiny (37.88%)Uniform32.15%35.81%
NormalFloat34.76%37.13%
Lloyd–Max37.99%37.73%

Existing compression components can achieve near-BF16 or even higher downstream accuracy. The best results are 44.74 vs. 44.53 (Qwen), 44.54 vs. 43.90 (InternVL), 36.43 vs. 37.42 (PaliGemma), and 37.99 vs. 37.88 (DeepSeek). Different combinations achieve these results across backbones, demonstrating effective transfer without dependence on one particular quantization design.

3. Effective compression across different reconstruction error and code-use. To further validate COMPACT's flexibility, we compare reconstruction MSE, cosine similarity, effective code usage, and downstream accuracy for Qwen. Effective code usage is defined as Neff=2H≤16N_{\mathrm{eff}}=2^H\leq16, where H=−∑kpklog⁡2pkH=-\sum_k p_k\log_2p_k and pkp_k is the fraction of values assigned to code kk. Higher Effective codes indicate more evenly distributed usage.

ConfigurationMSE (Vision/Text(×10⁻⁶))Cosine similarity (Vision/Text)Effective codes (Vision/Text)Mean accuracy
Block-64 Uniform.0033 / 3.13.9917 / .996212.64 / 13.8644.70%
Block-64 NormalFloat.0025 / 3.33.9938 / .995915.11 / 15.7044.66%
Block-64 Lloyd–Max.0024 / 2.87.9940 / .996515.02 / 15.6744.74%
Calibrated FSQ (per-tensor).0007 / 26.01.9987 / .972110.42 / 2.0044.62%
Calibrated RD-FSQ (per-tensor).0021 / 26.69.9969 / .97278.17 / 15.1645.50%

All five configurations achieve near-BF16 or higher accuracy (44.62–45.50% vs. 44.53%) despite differing reconstruction errors and effective code. Block-64 NormalFloat uses codes more evenly than Uniform but achieves lower accuracy (44.66% vs. 44.70%). Calibrated FSQ has lower MSE and higher vision cosine similarity than RD-FSQ, yet lower accuracy (44.62% vs. 45.50%). Thus, no single reconstruction or code-use metric consistently determines performance.

In contrast, reconstruction error becomes important when compression causes performance collapse. Calibrating FSQ/RD-FSQ reduces reconstruction error and restores near-BF16 on Qwen (44.62/45.50 vs. 44.53%), InternVL (44.64/44.19 vs. 43.90%), and PaliGemma (40.03/39.57 vs. 37.42%). This shows that reconstruction error matters greatly during severe performance degradation, even when it does not distinguish successful configurations.

In summary, COMPACT contributes a communication-efficient multimodal split-learning framework built on compressed activation transfer. Our controlled experiments demonstrate its applicability across different compression designs and VLM backbones.

AuthorsOpenReview ↗

ICLR 2027 - Urgent Concern: Reviewer Non-Response to Author Rebuttal

Dear Area Chair and Reviewers,

As explicitly stated in the ICLR 2027 Area Chair Guidelines, "Responding to rebuttal comments is a core part of the reviewer’s responsibilities."

We have carefully addressed the reviewers' concerns in our rebuttal and provided detailed clarifications and additional evidence. However, we have not yet received follow-up responses to our rebuttal.

As the discussion period is approaching its deadline, we would sincerely appreciate it if the reviewers could revisit our responses and let us know whether their concerns have been adequately addressed or if any further clarification is needed.

We would also be grateful if the Area Chair could help facilitate this discussion, as encouraged by the official ICLR guidelines.

We believe that constructive discussion is essential for a fair and thorough evaluation of our work, and we sincerely hope that our rebuttal and additional evidence can be fully considered in the final assessment.

Thank you very much for your time and consideration.

Warm regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.