- Rating
- 3(+1)
- Confidence
- 1(−1)
- Informativeness
- 1(−1)
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies reconstruction leakage in split inference for VLMs, where the vision encoder runs on the client device and intermediate visual tokens are transmitted to a remote server. Although this setup avoids directly uploading raw images, the transmitted visual representations may retain substantial spatial and semantic information and can therefore be inverted by an adaptive attacker. The paper proposes UAQB, an on-device defense consisting of a learned quantizer followed by a utility adapter. The learned quantizer applies an input-dependent offset and channel-wise scalar quantization to constrain the information retained in the visual representations, while the utility adapter maps the quantized representation back to a feature space compatible with the frozen downstream VLM.
Critical strengths
- The paper aims to address an important and practically relevant privacy problem.
- The empirical utility–privacy trade-off is convincing.
- The presentation is generally clear and technically careful.
Critical weaknesses
- The paper optimizes an upper bound on H(K), yet the analysis mainly reports the fraction of zero quantization indices. Sparsity is not equivalent to low entropy. Since each representation contains hundreds of thousands of scalar indices, even a small fraction of nonzero entries may still encode substantial information. Reporting empirical entropy, the variational upper bound, and preferably bits per image/token would make the information-bottleneck interpretation considerably more convincing.
- The deployment evidence is insufficient for the claimed resource-constrained-device setting. Runtime profiling is performed on an NVIDIA H100 rather than a mobile/edge device. More importantly, while full-model TTFT increases only modestly, the client-side computation itself increases by approximately 9% in FLOPs and 6.8–10.8% in measured latency, and UAQB introduces about 27M additional parameters. This overhead is not negligible for the resource-constrained devices motivating split inference.
- The empirical study is limited to two VLM backbones. LLaVA-1.5 is relatively dated, while Qwen3.5-2B is comparatively small. It is unclear whether the same privacy–utility behavior persists for larger modern VLMs, different vision encoders, substantially different token counts, or different split points.
Questions for authors
How sensitive is performance to the architecture and capacity of f_Q and f_U? Would substantially smaller adapters retain a similar utility–privacy frontier?
4 comments
Thank you for the detailed explanations and extensive experiments. Admittedly, I am not particularly familiar with this research direction. From my understanding, the proposed method aims to prevent information leakage from edge devices to the server by training a quantizer and an adapter between them, effectively providing an additional layer of semantic encryption for the transmitted data. From this perspective, the method appears to address a specific engineering problem rather than making a fundamental theoretical or academic contribution. Furthermore, the paper does not seem to include comparisons with any existing methods. Is this because no prior work has investigated this specific problem? Even if so, there should be related studies on information encryption or privacy-preserving techniques, and the proposed method should be compared with relevant approaches. Finally, I would like to ask whether the quantizer and adapter need to be retrained separately for each specific VLM model.
Official Comment by Authors
Thank you for your follow-up. In the following, we provide answers to your questions and concerns.
Theoretical and research contribution
First, we respectfully disagree with your current characterization of UAQB as "semantic encryption." Indeed, your original review correctly stated that our objective is to "constrain the information retained in the visual representations." This is fundamentally different from encryption. With encryption, the private visual information is still transmitted to the server and is at risk of leakage once decrypted. Instead, UAQB keeps private information on the device and releases only the information needed for downstream inference, so the private content remains local and protected.
UAQB is theoretically grounded in the information bottleneck principle to restrict private image information while releasing only task-relevant information. Since the two kinds of information are entangled in the same visual features, the fundamental research problem lies in limiting private information leakage without sacrificing task-relevant semantics. To address this, we formalize the problem as a principled optimization objective that rewards information about the target answer and penalizes information retained about the input image. Because the mutual information terms are intractable to compute directly, we derive a tractable variational upper bound and optimize it to train UAQB (Section 4.2). Furthermore, Theorem 1 provides a formal reconstruction guarantee by establishing a lower bound on the achievable reconstruction MSE for any attacker.
This information-theoretic optimization methodology applies to any VLM architecture, making our work a general theoretical framework for private split inference, rather than a solution to "a specific engineering problem."
"The paper does not seem to include comparisons with any existing methods"
We respectfully disagree with this assessment. The original submission already compares UAQB with five existing privacy-preserving methods, including Fixed Gaussian Noise, Fixed Quantization, Cloak (WWW 2021), DP-Forward (CCS 2023), and the SOTA split-VLM defense TrustCLIP (ECCV 2026). These comparisons are presented in Sections 5.1–5.3 (Figures 2 and 3), and detailed in Appendices B.1 and C.1 (Tables 2–5). They show that UAQB consistently achieves better utility-privacy trade-off than existing methods.
Training for each VLM
Yes, the quantizer and utility adapter are trained for each VLM backbone to match its visual feature distribution and frozen server interface. This is the case for all learned defenses, including Cloak and TrustCLIP. Once trained, these modules are deployed on the edge device with frozen weights to provide ongoing privacy protection.
We hope our response above has answered your concerns and questions. If you find our response satisfactory, would you please consider upgrading your review score?
Thank you for the detailed response. My concerns have been addressed, and I intend to raise my score. However, given my limited familiarity with this research area, I will also lower my confidence rating accordingly.
Official Comment by Authors
We thank the reviewer for their constructive comments. We have conducted additional experiments and revised the manuscript accordingly (revisions marked blue), as detailed below.
W1: Empirical entropy and leakage upper bound
Thank you for the suggestion. We have added a quantitative entropy analysis in Section 5.5 (Figure 6) to report both the total per-image variational leakage bound Lleak and the empirical marginal entropy of the quantization indices k on the validation images.
The leakage upper bounds are summarized below in kbits per image.
The empirical marginal entropy is summarized below in kbits per image.
On LLaVA, increasing β from 0.01 to 1 raises the zero-index fraction from 72.25% to 98.55%, while reducing empirical marginal entropy from 657.3 to 70.0 kbits/image and the leakage upper bound from 907.3 to 80.1 kbits/image. These measurements show that the observed sparsification accompanies a reduction in representation entropy. This reduction substantially limits the image information passed through UAQB, making reconstruction more difficult and thereby preserving privacy.
W2, Q: Architecture sensitivity and device-side overhead
Thank you for the suggestion. We have added ablations with smaller modules in Section 5.4, in which we replace the Transformer blocks in the learned quantizer, utility adapter, or both with substantially smaller two-layer MLPs. Results on LLaVA-1.5-7B are summarized below (Figure 5 and Table 7).
We have also deployed the entire device-side pipeline, including the vision encoder and defense modules, on a Samsung Galaxy S26 via the Qualcomm AI Hub. The measured costs are shown below and reported in Appendix C.5 (Table 8).
The default Transformer-based design achieves a superior utility-privacy trade-off, underscoring the benefit of cross-token context for both quantization and utility adaptation. Although the substantially smaller MLP quantizer and adapter variants yield a less favorable frontier, they still maintain a much better utility-privacy trade-off than the baselines. In fact, even the all-MLP variant substantially outperforms the baselines (e.g., retaining 92.3% utility versus 82.6% for fixed quantization at comparable LPIPS) while cutting added parameters from 27.3M to 4.2M, further highlighting the effectiveness of UAQB. On the Samsung Galaxy S26, it adds only 1.15% device-side latency, providing a lightweight option for resource-constrained mobile devices.
W3: Larger modern VLM
Thank you for the suggestion. In the revision, we have additionally evaluated Gemma-4-26B-A4B, a larger modern VLM announced by Google in April 2026, with 26B total parameters. It uses Gemma 4 Vision Encoder with 3×3 spatial pooling. We place UAQB after pooling and before the linear projection, transmitting 64 visual tokens of width 1152, compared with 576 tokens of width 1024 in LLaVA and Qwen. Therefore, this setting evaluates UAQB with a different vision encoder, substantially fewer visual tokens, and a different split point. The results are reported in Section 5.2 (Figure 2(c)).
The results in Figure 2(c) show that UAQB continues to achieve a better utility-privacy trade-off than the baselines. For example, both UAQB (β=0.1) and fixed Gaussian noise (σ=0.3) retain about 99.1% utility, but UAQB achieves 0.3704 LPIPS, compared with 0.3150 LPIPS for fixed Gaussian noise.