acceptodds
Under review as a conference paper at ICLR 2027

UAQB: Utility-Aware Quantization Bottleneck for Private Split VLM Inference

Abstract

Split inference allows resource-constrained devices to offload vision-language model (VLM) computation to a server by transmitting intermediate visual tokens instead of raw images, thereby avoiding direct exposure of private image content. However, these visual tokens remain susceptible to reconstruction attacks because substantial spatial and semantic information is retained in them. Existing defenses mitigate leakage largely at the expense of downstream task performance. To address this challenge, we propose the Utility-Aware Quantization Bottleneck (UAQB), a plug-in defense deployed on-device prior to token transmission. UAQB applies adaptive quantization to visual tokens to restrict image information flowing through the bottleneck, coupled with utility adaptation that restores compatibility of the resulting information-limited representation with the downstream VLM. We derive a tractable upper bound on the information bottleneck objective, enabling joint optimization of privacy and utility. Extensive evaluations across multiple utility benchmarks and reconstruction metrics demonstrate that UAQB achieves a more favorable utility-privacy trade-off than existing defenses, with particularly high utility retention at stronger protection levels.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
split inference, vision-language models, privacy, reconstruction attacks, information bottleneck, quantization

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_2vJ3OpenReview ↗
Rating
3(+1)
Confidence
1(−1)
Informativeness
1(−1)
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies reconstruction leakage in split inference for VLMs, where the vision encoder runs on the client device and intermediate visual tokens are transmitted to a remote server. Although this setup avoids directly uploading raw images, the transmitted visual representations may retain substantial spatial and semantic information and can therefore be inverted by an adaptive attacker. The paper proposes UAQB, an on-device defense consisting of a learned quantizer followed by a utility adapter. The learned quantizer applies an input-dependent offset and channel-wise scalar quantization to constrain the information retained in the visual representations, while the utility adapter maps the quantized representation back to a feature space compatible with the frozen downstream VLM.

Critical strengths
  1. The paper aims to address an important and practically relevant privacy problem.
  2. The empirical utility–privacy trade-off is convincing.
  3. The presentation is generally clear and technically careful.
Critical weaknesses
  1. The paper optimizes an upper bound on H(K), yet the analysis mainly reports the fraction of zero quantization indices. Sparsity is not equivalent to low entropy. Since each representation contains hundreds of thousands of scalar indices, even a small fraction of nonzero entries may still encode substantial information. Reporting empirical entropy, the variational upper bound, and preferably bits per image/token would make the information-bottleneck interpretation considerably more convincing.
  2. The deployment evidence is insufficient for the claimed resource-constrained-device setting. Runtime profiling is performed on an NVIDIA H100 rather than a mobile/edge device. More importantly, while full-model TTFT increases only modestly, the client-side computation itself increases by approximately 9% in FLOPs and 6.8–10.8% in measured latency, and UAQB introduces about 27M additional parameters. This overhead is not negligible for the resource-constrained devices motivating split inference.
  3. The empirical study is limited to two VLM backbones. LLaVA-1.5 is relatively dated, while Qwen3.5-2B is comparatively small. It is unclear whether the same privacy–utility behavior persists for larger modern VLMs, different vision encoders, substantially different token counts, or different split points.
Questions for authors

How sensitive is performance to the architecture and capacity of f_Q and f_U? Would substantially smaller adapters retain a similar utility–privacy frontier?

4 comments
AuthorsOpenReview ↗

Official Comment by Authors

We thank the reviewer for their constructive comments. We have conducted additional experiments and revised the manuscript accordingly (revisions marked blue), as detailed below.

W1: Empirical entropy and leakage upper bound

Thank you for the suggestion. We have added a quantitative entropy analysis in Section 5.5 (Figure 6) to report both the total per-image variational leakage bound Lleak\mathcal{L}_{\text{leak}} and the empirical marginal entropy of the quantization indices k\mathbf{k} on the validation images.

The leakage upper bounds are summarized below in kbits per image.

Backboneβ=0β=0β=0.01β=0.01β=0.1β=0.1β=0.5β=0.5β=1β=1
LLaVA-1.5-7B2,228.1907.3290.0116.680.1
Qwen3.5-2B1,926.51,164.4319.3130.695.4
Gemma-4-26B-A4B242.5135.950.223.917.4

The empirical marginal entropy is summarized below in kbits per image.

Backboneβ=0β=0β=0.01β=0.01β=0.1β=0.1β=0.5β=0.5β=1β=1
LLaVA-1.5-7B1,817.8657.3179.992.070.0
Qwen3.5-2B1,171.2664.8196.6110.285.2
Gemma-4-26B-A4B237.4133.049.523.717.3

On LLaVA, increasing ββ from 0.010.01 to 11 raises the zero-index fraction from 72.25% to 98.55%, while reducing empirical marginal entropy from 657.3 to 70.0 kbits/image and the leakage upper bound from 907.3 to 80.1 kbits/image. These measurements show that the observed sparsification accompanies a reduction in representation entropy. This reduction substantially limits the image information passed through UAQB, making reconstruction more difficult and thereby preserving privacy.

W2, Q: Architecture sensitivity and device-side overhead

Thank you for the suggestion. We have added ablations with smaller modules in Section 5.4, in which we replace the Transformer blocks in the learned quantizer, utility adapter, or both with substantially smaller two-layer MLPs. Results on LLaVA-1.5-7B are summarized below (Figure 5 and Table 7).

Variantβ=0.01Utility,↑\begin{gathered}β=0.01\\\\\text{Utility}\\,↑\end{gathered}β=0.01LPIPS,↑\begin{gathered}β=0.01\\\\\text{LPIPS}\\,↑\end{gathered}β=0.1Utility,↑\begin{gathered}β=0.1\\\\\text{Utility}\\,↑\end{gathered}β=0.1LPIPS,↑\begin{gathered}β=0.1\\\\\text{LPIPS}\\,↑\end{gathered}β=0.5Utility,↑\begin{gathered}β=0.5\\\\\text{Utility}\\,↑\end{gathered}β=0.5LPIPS,↑\begin{gathered}β=0.5\\\\\text{LPIPS}\\,↑\end{gathered}β=2Utility,↑\begin{gathered}β=2\\\\\text{Utility}\\,↑\end{gathered}β=2LPIPS,↑\begin{gathered}β=2\\\\\text{LPIPS}\\,↑\end{gathered}
UAQB99.06%0.248597.84%0.286996.24%0.310494.91%0.3315
MLP Quantizer99.05%0.248597.19%0.283395.22%0.311893.65%0.3380
MLP Adapter99.00%0.247997.59%0.284696.32%0.307193.26%0.3228
MLP Quantizer & Adapter99.01%0.247797.44%0.280995.44%0.306092.26%0.3171

We have also deployed the entire device-side pipeline, including the vision encoder and defense modules, on a Samsung Galaxy S26 via the Qualcomm AI Hub. The measured costs are shown below and reported in Appendix C.5 (Table 8).

SettingParams (M)FLOPs (G)Latency (ms)
Clean303.51381.9239.14
UAQB330.80 (+8.99%)416.04 (+8.94%)42.25 (+7.95%)
MLP Quantizer319.26 (+5.19%)401.40 (+5.10%)41.03 (+4.83%)
MLP Adapter319.26 (+5.19%)401.40 (+5.10%)40.77 (+4.17%)
MLP Quantizer & Adapter307.71 (+1.38%)386.75 (+1.27%)39.59 (+1.15%)

The default Transformer-based design achieves a superior utility-privacy trade-off, underscoring the benefit of cross-token context for both quantization and utility adaptation. Although the substantially smaller MLP quantizer and adapter variants yield a less favorable frontier, they still maintain a much better utility-privacy trade-off than the baselines. In fact, even the all-MLP variant substantially outperforms the baselines (e.g., retaining 92.3% utility versus 82.6% for fixed quantization at comparable LPIPS) while cutting added parameters from 27.3M27.3\text{M} to 4.2M4.2\text{M}, further highlighting the effectiveness of UAQB. On the Samsung Galaxy S26, it adds only 1.15% device-side latency, providing a lightweight option for resource-constrained mobile devices.

W3: Larger modern VLM

Thank you for the suggestion. In the revision, we have additionally evaluated Gemma-4-26B-A4B, a larger modern VLM announced by Google in April 2026, with 26B total parameters. It uses Gemma 4 Vision Encoder with 3×33\times3 spatial pooling. We place UAQB after pooling and before the linear projection, transmitting 64 visual tokens of width 1152, compared with 576 tokens of width 1024 in LLaVA and Qwen. Therefore, this setting evaluates UAQB with a different vision encoder, substantially fewer visual tokens, and a different split point. The results are reported in Section 5.2 (Figure 2(c)).

The results in Figure 2(c) show that UAQB continues to achieve a better utility-privacy trade-off than the baselines. For example, both UAQB (β=0.1β=0.1) and fixed Gaussian noise (σ=0.3σ=0.3) retain about 99.1% utility, but UAQB achieves 0.3704 LPIPS, compared with 0.3150 LPIPS for fixed Gaussian noise.

Thank you for the detailed explanations and extensive experiments. Admittedly, I am not particularly familiar with this research direction. From my understanding, the proposed method aims to prevent information leakage from edge devices to the server by training a quantizer and an adapter between them, effectively providing an additional layer of semantic encryption for the transmitted data. From this perspective, the method appears to address a specific engineering problem rather than making a fundamental theoretical or academic contribution. Furthermore, the paper does not seem to include comparisons with any existing methods. Is this because no prior work has investigated this specific problem? Even if so, there should be related studies on information encryption or privacy-preserving techniques, and the proposed method should be compared with relevant approaches. Finally, I would like to ask whether the quantizer and adapter need to be retrained separately for each specific VLM model.

AuthorsOpenReview ↗

Official Comment by Authors

Thank you for your follow-up. In the following, we provide answers to your questions and concerns.

Theoretical and research contribution

First, we respectfully disagree with your current characterization of UAQB as "semantic encryption." Indeed, your original review correctly stated that our objective is to "constrain the information retained in the visual representations." This is fundamentally different from encryption. With encryption, the private visual information is still transmitted to the server and is at risk of leakage once decrypted. Instead, UAQB keeps private information on the device and releases only the information needed for downstream inference, so the private content remains local and protected.

UAQB is theoretically grounded in the information bottleneck principle to restrict private image information while releasing only task-relevant information. Since the two kinds of information are entangled in the same visual features, the fundamental research problem lies in limiting private information leakage without sacrificing task-relevant semantics. To address this, we formalize the problem as a principled optimization objective that rewards information about the target answer and penalizes information retained about the input image. Because the mutual information terms are intractable to compute directly, we derive a tractable variational upper bound and optimize it to train UAQB (Section 4.2). Furthermore, Theorem 1 provides a formal reconstruction guarantee by establishing a lower bound on the achievable reconstruction MSE for any attacker.

This information-theoretic optimization methodology applies to any VLM architecture, making our work a general theoretical framework for private split inference, rather than a solution to "a specific engineering problem."

"The paper does not seem to include comparisons with any existing methods"

We respectfully disagree with this assessment. The original submission already compares UAQB with five existing privacy-preserving methods, including Fixed Gaussian Noise, Fixed Quantization, Cloak (WWW 2021), DP-Forward (CCS 2023), and the SOTA split-VLM defense TrustCLIP (ECCV 2026). These comparisons are presented in Sections 5.1–5.3 (Figures 2 and 3), and detailed in Appendices B.1 and C.1 (Tables 2–5). They show that UAQB consistently achieves better utility-privacy trade-off than existing methods.

Training for each VLM

Yes, the quantizer and utility adapter are trained for each VLM backbone to match its visual feature distribution and frozen server interface. This is the case for all learned defenses, including Cloak and TrustCLIP. Once trained, these modules are deployed on the edge device with frozen weights to provide ongoing privacy protection.

We hope our response above has answered your concerns and questions. If you find our response satisfactory, would you please consider upgrading your review score?

Thank you for the detailed response. My concerns have been addressed, and I intend to raise my score. However, given my limited familiarity with this research area, I will also lower my confidence rating accordingly.

Reviewer_DxJ9OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies privacy in split VLM inference, where a device sends visual features to a server instead of sending the original image. These features can still reveal the image through reconstruction attacks. UAQB reduces this leakage with a learned quantizer and uses a utility adapter to keep the protected features compatible with the frozen VLM. The modules are trained using an information-leakage bound and the downstream task loss, without a reconstruction adversary. Experiments on two VLMs and six utility benchmarks show a better trade-off between task performance and reconstruction quality than the compared defenses.

Critical strengths

Practical design: The quantizer limits information while the adapter preserves compatibility with a frozen VLM. The adapter only receives quantized features, preserving the information bound under the stated assumptions.

Empirical support: The experiments show favorable privacy–utility trade-offs under adapted attacks, including gains over the learned quantizer alone at matched LPIPS (Table 5). The TrustCLIP joint-training check strengthens the comparison.

Critical weaknesses

Unclear attacker convergence: Appendix B.2 fixes defense-specific adaptation at 10 epochs without showing convergence or testing a longer budget. It remains unclear whether the privacy advantage persists after the attacker has fully adapted.

Unquantified guarantee: The paper reports neither the total per-image leakage bound nor a numerical distortion guarantee. The zero-index fraction in Figure 5 does not establish representation entropy, leaving the practical strength of the theoretical guarantee unclear.

Questions for authors
  1. Have the reconstruction metrics plateaued after 10 adaptation epochs? Does a longer run change the advantage of UAQB over the baselines?
  2. What are the total per-image leakage bounds at the reported operating points? Do they imply a nontrivial numerical lower bound on reconstruction error?
1 comment
AuthorsOpenReview ↗

Official Comment by Authors

We thank the reviewer for the supportive comments. We have conducted additional experiments and revised the manuscript accordingly (revisions marked blue), as detailed below.

W1, Q1: Attacker convergence

Thank you for the suggestion. We have added an attacker-convergence experiment in Section 5.3 and Appendix C.2 (Figure 8), where we extend adaptive attacker training from 10 to 20 epochs for UAQB and all five baselines at the LLaVA-1.5-7B settings shown in Figure 3. The results answer your questions as follows:

  1. Have the reconstruction metrics plateaued after 10 adaptation epochs?
    All four reconstruction metrics largely plateau by epoch 10 after rapid initial adaptation. For UAQB (β=1\beta=1), training from epoch 10 to 20 changes PSNR by only +0.004+0.004 dB and LPIPS by only −0.0012-0.0012.

  2. Does a longer run change the advantage of UAQB over the baselines?
    UAQB’s advantage persists with longer attacker training. All five baselines exhibit similar plateauing trajectories. At epoch 20, UAQB continues to yield lower PSNR/SSIM and higher LPIPS/DreamSim than all baselines at comparable utility.

W2, Q2: Unquantified guarantee

Thank you for this insightful comment. We have added a quantitative entropy analysis in Section 5.5 (Figure 6) to report both the total per-image variational leakage bound Lleak\mathcal{L}_{\text{leak}} and the empirical marginal entropy of the quantization indices k\mathbf{k} on the validation images. The following summarize how our results answer your questions:

  1. What are the total per-image leakage bounds at the reported operating points?
    The measured leakage upper bounds Lleak\mathcal{L}_{\text{leak}} are summarized below in kbits per image.

    Backboneβ=0\beta=0β=0.01\beta=0.01β=0.1\beta=0.1β=0.5\beta=0.5β=1\beta=1
    LLaVA-1.5-7B2,228.1907.3290.0116.680.1
    Qwen3.5-2B1,926.51,164.4319.3130.695.4
    Gemma-4-26B-A4B242.5135.950.223.917.4

    The empirical marginal entropy is summarized below in kbits per image.

    Backboneβ=0\beta=0β=0.01\beta=0.01β=0.1\beta=0.1β=0.5\beta=0.5β=1\beta=1
    LLaVA-1.5-7B1,817.8657.3179.992.070.0
    Qwen3.5-2B1,171.2664.8196.6110.285.2
    Gemma-4-26B-A4B237.4133.049.523.717.3

    As β\beta increases from 00 to 11, Lleak\mathcal{L}_{\text{leak}} decreases by 96.4%, 95.0%, and 92.8% on LLaVA, Qwen, and Gemma, respectively. The empirical marginal entropy also decreases monotonically, with corresponding reductions of 96.1%, 92.7%, and 92.7%. These reductions limit the image information passed through UAQB, making reconstruction more difficult and thereby preserving privacy.

  2. Do they imply a nontrivial numerical lower bound on reconstruction error?
    Theorem 1 establishes the lower bound on reconstruction error E[MSE]≥DX(H(K))≥DX(Lleak)\mathbb{E}[\mathrm{MSE}]\geq\mathcal{D}_{X}\left(H(K)\right)\geq\mathcal{D}_X(\mathcal{L}_{\mathrm{leak}}). However, numerically evaluating this bound requires estimating the source-specific distortion-rate function DX(⋅)\mathcal{D}_X(\cdot), which is intractable in practice due to the high dimensionality and complex statistical dependencies of natural images. Nevertheless, this bound provides principled theoretical guidance for increasing the minimum achievable reconstruction error by tightening Lleak\mathcal{L}_{\mathrm{leak}}. This is corroborated by our experimental results, in which lower leakage bounds consistently accompany lower empirical marginal entropy and higher reconstruction error under adaptive attacks (Figures 2 and 6), confirming the practical effectiveness of the information constraint.

Reviewer_BvUuOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a token compression technique to improve the privacy while enabling the inference of images through VLM. The concept is nice but more experimental results are needed and more comparison against SOTA are needed.

Critical strengths
  1. the result seems promising but more rigorous experiments are needed to show clear contribution.
Critical weaknesses
  1. from the results. the OCR results seem impacted. additional evaluation should be performed with this goal if the inference target is a VLM.
  2. more evaluation should be performed. currently the tasks evaluated are limited.
  3. the method also need to be compared with SOTA methods, including H266, H265
Questions for authors
  1. More details can be included in the model architecture, design etc.
  2. the PSNR etc metrics seem low in the paper? is it common?
1 comment
AuthorsOpenReview ↗

Official Comment by Authors

We thank the reviewer for their comments.
We would first like to emphasize that our work addresses privacy protection in split VLM inference, rather than token compression. Although UAQB uses quantization, its purpose is to distort transmitted visual tokens and limit the image information recoverable from them. Lower attacker reconstruction PSNR/SSIM therefore indicates stronger privacy protection, which is consistent with UAQB’s objective.

W1: OCR results

We would like to emphasize again that our goal is to limit the recovery of image content while preserving downstream VLM utility. Therefore, reduced text readability in the attacker’s reconstructions is an intended privacy outcome, as image text may contain sensitive information.

W2: More evaluation

To evaluate UAQB’s impact on downstream VLM utility, we performed experiments across six widely used VLM benchmarks: MME, POPE, GQA, MMBench, VQAv2, and TextVQA. In addition to the two VLM backbones (LLaVA-1.5-7B and Qwen3.5-2B) in the original manuscript, we have added Gemma-4-26B-A4B (a larger modern VLM announced by Google in April 2026, with 26B total parameters) in the revision. As shown in Section 5.2, our experimental results demonstrate that UAQB can maintain high VLM utility while improving privacy protection. For example, at β=0.1\beta=0.1, UAQB achieves 99.10% average utility retention relative to Clean on Gemma-4-26B-A4B.

W3: SOTA methods

Since our work addresses privacy protection in split VLM inference and aims to reduce reconstruction fidelity, the compression methods H.265 (HEVC) and H.266 (VVC) serve an opposite design objective. They are designed to produce high-fidelity reconstructions (typically above 30 dB PSNR), which preserve visual appearance and leave private image content vulnerable to leakage. Instead, we have compared UAQB with five representative baselines designed for this privacy task, including the SOTA method TrustCLIP. Our results in Section 5.2 demonstrate that UAQB achieves a better utility-privacy trade-off than the baselines, providing stronger reconstruction privacy while maintaining high downstream VLM utility.

Q1: More details

Thank you for the suggestion. Detailed model architectures, module configurations, and training settings were provided in Appendix B. We have also uploaded the source code in the supplementary material, including the full implementation, training, and evaluation instructions needed to reproduce UAQB.

Q2: Low PSNR

Low PSNR/SSIM values are in fact a positive in this work, as the goal is to hinder image reconstruction by an attacker. Lower reconstruction PSNR/SSIM indicates stronger privacy protection.

Comments on the paper

1 comment
Area_Chair_P23UOpenReview ↗

[Urgent] Please check the authors' rebuttal

Dear Reviewers BvUu and DxJ9,

The authors have posted their responses to the issues you raised. As the discussion period will end soon, please take a look and have a discussion with the authors at your earliest convenience. Thank you very much!

Dear Reviewer 2vJ3,

Please take a last look if the latest response of the authors have addressed your further concerns. Thank you very much!

Best, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.