acceptodds
Under review as a conference paper at ICLR 2027

Task-Aware Representation Release with Local Differential Privacy for Multimodal VLM Inference

Abstract

Edge–cloud collaborative inference reduces on-device computation by sending locally encoded image and text tokens to a cloud-hosted vision-language model (VLM) backbone, but these uploads may reveal private input to an untrusted server. We seek to protect private images and contextual text jointly with local differential privacy (LDP), which randomizes the upload on the client without trusting the server. However, this perturbation can damage evidence needed for answering, and directly adding noise to high-dimensional representations can introduce substantial distortion. Preserving utility under LDP therefore presents three coupled challenges: (i) reducing the dimensionality of the representation to be perturbed while retaining task-relevant evidence; (ii) configuring perturbations for differently scaled visual and textual representations; and (iii) keeping perturbed representations usable by an unchanged cloud backbone. We address these challenges through a task-trained client-side compression–perturbation–decoding framework. It combines public-question-guided attention, a joint ellipsoidal Gaussian mechanism with calibrated modality scales, and noise-aware reconstruction of the frozen backbone's native token interface. The complete image–text upload satisfies -LDP conditional on the public question and declared token structure. Only the client-side release modules are trained; the pretrained encoders and cloud backbone remain frozen. With InternVL3.5-8B, our framework improves ScienceQA accuracy by 12.2–23.5 percentage points and keyword exact match on a single-image WebQA subset by 13.8–39.2 points, compared with direct LDP perturbation of uncompressed representations at the three primary matched privacy budgets. Additional experiments show gains with a different VLM and reduced content recovery relative to unprotected uploads under the evaluated inversion attacks.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Vision-Language Models, Local Differential Privacy, Task-Aware Representation Release, Ellipsoidal Gaussian Mechanism

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_p7zVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a client-side compression–perturbation–decoding framework for edge–cloud VLM inference that provides whole-request local differential privacy (LDP) for joint image and contextual-text inputs. Experiments demonstrate substantially improved utility compared to direct LDP applied to native tokens and a PCA-based LDP baseline, while also reducing content recovery under fixed inversion attacks.

Critical strengths
  1. The paper introduces a comprehensive local-release pipeline tailored to multimodal VLMs with frozen encoders and backbones.

  2. The experimental results demonstrate substantial performance improvements over existing baselines.

  3. The paper provides rigorous privacy proofs and detailed mechanism analysis to support the theoretical foundations of the proposed method.

Critical weaknesses
  1. The proposed method involves a reconstruction process as part of its framework, while the inversion analysis appears to rely on the same reconstruction mechanism. This evaluation may not adequately reflect the actual privacy protection provided by the method, as it does not consider attackers employing different or more sophisticated reconstruction strategies.

  2. The privacy evaluation against inversion attacks is insufficient, as it considers only black-box inversion attacks. More advanced white-box inversion attacks should be included to provide a comprehensive assessment of the proposed method's robustness against stronger adversaries.

  3. The privacy–utility trade-off in Table 11 is difficult to assess because the proposed method and the baselines exhibit substantially different utility and reconstruction scores. In particular, the baselines achieve much lower Word F1 and SSIM scores, making it unclear whether the observed differences in privacy protection result from the defense mechanisms or the differences in utility. The authors should compare all methods under the same or comparable utility levels and evaluate their corresponding reconstruction performance to ensure a fair comparison. Moreover, the proposed method still suffers from a substantial utility degradation, achieving only 77.23 compared to 93.50 without protection. Such a significant performance drop raises concerns about the practical applicability of the proposed method and whether the privacy benefits justify the loss in utility.

Questions for authors
  1. How does the proposed method perform against more advanced white-box inversion attacks?

  2. How does the proposed method's effectiveness against inversion attacks vary across different utility levels?

  3. How does the proposed method compare with existing baselines in terms of privacy protection when evaluated at the same or comparable utility levels?

3 comments

We thank the reviewer for recognizing the overall framework, utility improvements, and privacy analysis. The comments highlight two aspects that benefit from further clarification: whether inversion evaluation sufficiently accounts for the defense mechanism, and how methods compare at comparable utility or reconstruction levels. We address these through clarification of the attack setup and additional experiments.

1. Distinguishing the defense decoder from the inversion attacker, and defense-aware evaluation (Weaknesses 1–2 and Question 1).

There are two distinct reconstruction tasks in our framework and evaluation. The defense decoder maps noisy compact representations back to the native token format required by the frozen VLM, with the objective of supporting downstream inference. The inversion attacker instead reconstructs the original RGB image from the final visual representation observed by the server. These are separate networks with independent parameters and training objectives.

The original evaluation used a common fixed inversion network to provide a reproducible reference across release methods. Following the suggestion that attacks should exploit knowledge of the defense, we have additionally completed method-specific inversion adaptation using public auxiliary images paired with the actual final representations generated by each client-side mechanism, including noisy outputs.

This allows attack training to account directly for the representation changes introduced by compression, perturbation, and decoding. Since adaptation may converge differently across release distributions, we interpret the results in the context of the training protocol. The added evaluation focuses on learned defense-aware inversion; white-box attacks that directly optimize candidate inputs through the known mechanism remain a direction for broader attack coverage.

2. Recovery at comparable utility and across utility levels (Comparison concern in Weakness 3, Questions 2–3).

Following the reviewer's suggestion, we evaluated recovery at operating points with similar observed test QA accuracy. Image recovery uses the adapted inversion attackers, while text recovery uses the fixed vocabulary-inversion attack:

MethodEvaluated whole-request ε\varepsilonQA accuracyText Word F1 ↓Image LPIPS ↑
Ours953.5873.01%7.17%0.47543
PCA-based1,000,235,07974.96%99.57%0.32128
Native-LDP372,79172.31%100.00%0.48012

The arrows indicate the direction of poorer reconstruction. The experiments cover all 814 questions with three independent noise realizations and were independently checked.

Both baseline QA means are within two percentage points of Ours on the test set. At these operating points, Ours has substantially lower text recovery under the evaluated vocabulary attack. Its image LPIPS is higher than PCA's and close to Native-LDP's. Thus, similar task performance can coexist with substantially different text recovery, while the image-recovery comparison depends on the particular baseline.

This comparison is exploratory: PCA did not satisfy the development-set QA-matching criterion, and Native-LDP uses a previously observed test configuration. The table therefore reports the budgets and performance of these evaluated operating points.

We also examined the tradeoff in the reverse direction. When the Word F1 range across methods is only 0.33 percentage points, Ours improves QA accuracy over PCA by 8.39 percentage points (paired 95% CI: [6.61, 10.27]) and over Native-LDP by 14.68 percentage points (paired 95% CI: [12.14, 17.13]). This provides additional evidence of higher task utility at similar recovery achieved by the evaluated vocabulary attack.

To examine how recovery varies across utility levels, our earlier complete three-budget adaptation study evaluated Ours with the same adapted image attacker across budgets:

Whole-request ε\varepsilonQA accuracyText Word F1 ↓Image LPIPS ↑
953.5873.01%7.17%0.4754
3,407.1777.23%14.85%0.3798
10,743.3879.36%46.77%0.3473

The image metrics come from that three-budget adaptation study, and the text metrics come from the corresponding fixed vocabulary attack. Increasing the budget raises QA accuracy while also improving text and image reconstruction. These results illustrate the budget-dependent utility–recovery tradeoff and motivate reporting task performance alongside recovery in both modalities.

3. Utility loss relative to the unprotected model and practical value (Practicality concern in Weakness 3).

The decrease from 93.50% to 77.23% is a substantial accuracy cost. The clean-codec control achieves 85.61%, indicating that representation processing itself contributes to part of this gap.

We believe practical value should be assessed by considering both this cost and the task performance retained under the same privacy constraint. At whole-request ε≈3,407\varepsilon\approx3{,}407, Ours, PCA-based release, and Native-LDP achieve 77.23%, 67.07%, and 57.76% accuracy, respectively. These results show that task-aware representation processing preserves more utility at the same declared budget.

The additional comparison at similar QA accuracy provides a complementary perspective: comparable observed task performance can be accompanied by substantially lower text recovery for Ours under the evaluated attack. Together, these comparisons make the accuracy cost and changes in recoverability more explicit. The appropriate operating point depends on an application's accuracy requirements and privacy constraints; our method focuses on improving task utility within those constraints.

Thank you again for your thoughtful review. After you’ve had a chance to read our response, please let us know if there’s anything you’d like us to clarify.

Reviewer_GsP2OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper adresses the privacy challenge for edge–cloud VLM inference. Private image and text tokens are compressed with a question-conditioned encoder, privatized with a ellipsoidal Gaussian mechanism, and mapped back to the frozen backbone’s native token format by a trained decoder. The authors prove (ε,δ)-LDP via zCDP and evaluate on ScienceQA and a WebQA subset, reporting large gains over Native-LDP and PCA-based release at matched ε.

Critical strengths
  1. The problem is well formulated and well-motivated. Split inference for VLMs is becoming more necessary as foundation models useage on edge-devices grows, but the privacy of this inference is often unadressed.
  2. The theoretical claims about privacy are, as far as I can tell, valid and fit together well.
  3. The paper is generally well-organized and easy to follow with clear equations, tables and figures.
Critical weaknesses

My main concern is focussed about the choice of the privacy budget. The smallest epsilon budget is at ϵ≈71\epsilon \approx 71. From what I can tell from Figure 2., at this budget the proposed method performs worse than the public-only baseline. Only at higher ϵ\epsilon values does the method start beating this public-only baseline. However, at such high values, the gurantee of privacy is essentially void, given that the privacy bound scales with eϵe^{\epsilon}. So, although the theoretical framework is correct, the framework is instantiated at paramters where it cannot give any real meaningful bound.

If the authors can explain or justify how there can still be meaningful protection, I would be happy to change my mind.

Questions for authors
  1. Can the authors explain how the method gives meaningful protection at ϵ≈71\epsilon \approx 71 and above, given that the privacy bound scales with eϵe^{\epsilon}?

  2. Can the authors run the inversion attack on the clean-codec and clip-only versions as well? That would show whether the reduced recovery comes from the noise or just from the compression and decoding changing the tokens.

  3. Can the authors attack with an inverter that is trained on the protected outputs? The threat model assumes the server knows the mechanism, so I would expect the attack to use that.

  4. In Table 8 the method performs worse than the public-only baseline at ϵ=4\epsilon = 4 to 3232. Can the authors explain why this happens?

  5. I am curious about the following: DP is desgined to protect against identifiyng an individual data point. However, in the context of inference we need to know which datapoint is used, since we want the output of that particular datapoint. These seems to be at odds with the definition of DP. Can you comment on this, please.

4 comments

We thank the reviewer for recognizing the validity of our privacy analysis and for raising the important question of the practical meaning of the privacy budget. We would like to clarify the protection unit and discuss the lower-budget results together with our additional reconstruction evaluations.

1. Privacy-budget interpretation and lower-budget results (Main weakness and Question 1).

We understand the concern about large ε\varepsilon. Values in the hundreds or thousands give a loose worst-case indistinguishability bound, making performance at lower budgets particularly relevant to assessing the framework.

First, our protection unit is an entire request: all private image and contextual-text content may change simultaneously, while the public question, answer options, and declared token structure remain fixed. This differs from a protection unit that allows only one token to change. Comparisons with token-level budgets therefore require the corresponding whole-request privacy accounting.

Second, Table 9 and Appendix D.4.1 already report a configuration adapted to lower budgets, with δ=10−6\delta=10^{-6}:

Whole-request ε\varepsilonMain checkpoint QALow-budget adaptation QAGain over Public-only, percentage points [paired 95% CI]
854.36%67.54%+4.03 [2.67, 5.40]
1654.27%67.37%+3.86 [2.52, 5.22]
3255.91%67.44%+3.93 [2.59, 5.26]

Public-only accuracy is 63.51%. All three budgets use one checkpoint trained on public auxiliary data at ε=16\varepsilon=16, without further parameter updates during evaluation. The experiments cover all 814 questions with three noise realizations, and the paired intervals use image-cluster bootstrap resampling.

These results show that the framework achieves task performance above the reported Public-only reference at substantially smaller whole-request budgets. The adapted configuration performs better in the high-noise regime, while the main checkpoint retains higher utility at larger budgets. We believe this highlights the importance of adapting the training configuration to the intended privacy-budget range.

2. Why can accuracy fall below Public-only? (Question 4).

Our interpretation is that noise-corrupted representations may be mismatched with the inputs that the frozen VLM can use reliably. Providing additional representations can introduce misleading cues, whereas Public-only omits those cues entirely.

Table 8 evaluates fixed model parameters across budgets and therefore reflects how one configuration handles different noise levels. The similar trough observed for PCA-based release is consistent with a broader compatibility issue between perturbed representations and the frozen backbone.

After adapting the codec to the high-noise regime and applying public-statistics-based shrinkage before decoding, accuracy exceeds Public-only at ε=8,16,32\varepsilon=8,16,32. This demonstrates substantial room for adaptation in the lower-budget regime and supports the above interpretation. All learning and calibration use public auxiliary data, and the resulting mechanism retains the whole-request LDP guarantee under the same definition.

3. Effects of compression, clipping, and noise, and defense-aware attacks (Questions 2 and 3).

We have completed clean-codec and clip-only controls. Under the original fixed attacks and image-scoring protocol, we obtain:

Ours configurationImage LPIPS ↑Text Word F1 ↓
Clean codec: no clipping or noise0.581999.64%
Clip-only: no noise0.577599.62%
DP release, ε≈954\varepsilon\approx9540.68757.17%

Higher LPIPS and lower Word F1 indicate poorer reconstruction. Clipping alone has a relatively small effect in these controls. Adding noise further reduces image and text recovery, with text Word F1 decreasing from approximately 99.6% to 7.17%. These results support an additional contribution from noise beyond the effects of compression and decoding.

The original fixed attack was intended to provide a common, reproducible reference by controlling attacker-side variables across methods. To further account for a server that knows the defense mechanism, we have also completed method-specific image-inversion adaptation using public auxiliary images paired with the actual final representations generated by each client-side mechanism, including noisy outputs. Since adaptation may converge differently across release distributions, we interpret the resulting recovery scores in the context of this training protocol.

We additionally conducted an exploratory comparison at similar observed test QA accuracy, using the adapted image attackers and the fixed vocabulary-inversion text attack:

MethodEvaluated whole-request ε\varepsilonQA accuracyText Word F1 ↓Image LPIPS ↑
Ours953.5873.01%7.17%0.47543
PCA-based1,000,235,07974.96%99.57%0.32128
Native-LDP372,79172.31%100.00%0.48012

The experiments cover all 814 questions with three noise realizations and were independently checked. The adapted image evaluation uses 224×224224\times224 scoring, while the preceding fixed-attack controls use the original 448×448448\times448 protocol; comparisons within each table use consistent scoring conditions.

Both baseline QA means are within two percentage points of Ours on the test set. At these operating points, Ours has substantially lower text recovery than either baseline and higher image LPIPS than PCA. Its image LPIPS is close to, and slightly lower than, Native-LDP's. This illustrates that similar image reconstruction quality can coexist with large differences in text recovery, motivating evaluation of both modalities.

This supplementary comparison is based on similar observed test utility: PCA did not meet the development-set QA-matching criterion, and Native-LDP uses a previously observed test configuration. We therefore present it as an exploratory comparison of the budgets, task performance, and recovery achieved at these evaluated operating points.

4. Relationship between request-level DP and inference (Question 5).

In our setting, DP constrains how much the distribution of the uploaded representation may change when the private request is replaced. The randomized release can still retain information about the input. The cloud model uses this information to answer the public question, and its answer inherits the same DP guarantee through post-processing.

We believe that the different information requirements of task answering and complete reconstruction are important here. Many distinct private inputs can support the same answer, so completing the task may require preserving only part of the information relevant to the question.

When a question directly asks for a protected attribute, the tension between accurate answering and privacy becomes more pronounced. The attainable privacy–utility tradeoff therefore depends on the task itself. Our objective is to preserve as much question-relevant information as possible within the chosen whole-request privacy constraint.

Thank you again for your thoughtful review. After you’ve had a chance to read our response, please let us know if there’s anything you’d like us to clarify.

Reviewer_GsP2OpenReview ↗

Response to Authors

I thank the authors for the detailed response and the additional experiments. I appreciate the work that went into this.

However, my main concern remains. The choice of the protection unit only determines which inputs are adjacent. The ϵ\epsilon still bounds how much the output can differ between them, and at the values used in the main results this bound does not give any meaningful guarantee, regardless of what the unit is. I do not think that a mechanism which only works at such extremely high ϵ\epsilon merits a paper on local DP.

The low-budget results are more interesting, but the accuracy does not change as ϵ\epsilon increases, and this is not explained. In my experience this usually means that a learned prior is doing most of the work, not the private input.

I think the work could be more convincing as an empirical defense, with adaptive attacks as the main evidence and without the framing as a local DP method.

Reviewer_JUiyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies LDP for image–text representations uploaded to an untrusted cloud VLM. The method compresses the visual and textual representations before adding Gaussian noise. It uses separate noise scales for the two modalities and decodes the perturbed representations back to the input format of the frozen VLM. The compressor and decoder are trained with task supervision on public auxiliary data. The same data are also used to calibrate the noise scales. The paper reports higher ScienceQA accuracy and WebQA keyword exact-match scores than Native-LDP and PCA-based release at ε≈954,3407,10743\varepsilon \approx {954, 3407, 10743} and δ=10−6\delta=10^{-6}.

Critical strengths
  1. A strength of the paper is that it provides a plug-and-play way to add LDP protection to a frozen cloud VLM, though the client-side compressor and decoder still require training.
Critical weaknesses
  1. As shown in Table 11, the proposed method improves QA accuracy over Native-LDP but exhibits higher text recovery under the same (ε,δ)(\varepsilon,\delta). Thus, this comparison does not show a better empirical privacy–utility tradeoff over Native-LDP at ε≈3407\varepsilon \approx 3407.

  2. The empirical inversion results are obtained against a weaker attacker than the threat model allows. Section 3.1 assumes that the cloud server knows the client-side processing mechanism and its fixed parameters, as described in Lines 133–143. However, the image inversion attack is trained on native visual representations paired with images, rather than on visual representations generated by the client-side processing mechanism, as described in Appendix E.

Questions for authors
  1. Could you compare QA accuracy between the proposed method and Native-LDP at matched Word F1 for text recovery, and separately at matched SSIM or LPIPS for image recovery?

  2. Could you provide image recovery results using an inversion attack trained on public auxiliary images and the corresponding visual representations generated by the client-side processing mechanism?

2 comments

We thank the reviewer for highlighting the importance of the attack setting and of comparing utility at comparable reconstruction quality.

1. Attack setting and defense-aware evaluation (Weakness 2 and Question 2).

Our original inversion evaluation was designed to provide a common, reproducible reference for comparing different release methods. By using the same inversion network with identical parameters across methods, we controlled the attacker-side variables and measured reconstruction under a common adversary. This controlled comparison was the motivation behind our original setup.

We agree, however, that a common attacker does not necessarily attack every representation equally effectively. An inverter trained on native representations may experience distribution mismatch when applied to transformed representations. The original results therefore characterize recovery under that specified attacker, rather than the strongest recovery achievable by a mechanism-aware server.

To address this concern, we have additionally evaluated defense-aware image inversion using public auxiliary images paired with the actual final visual representations released by each method. The attackers undergo method-specific adaptation using outputs generated by the corresponding client-side mechanisms, including protected outputs. For the image comparison below, we completed further attacker training, budget recalibration, and evaluation on all 814 test questions with three independent noise realizations, followed by an independent verification of the results.

These experiments strengthen the evaluation, although finite adaptation cannot certify that each attacker is equally close to the strongest achievable reconstruction for its target method. We therefore regard the fixed and adapted attacks as complementary, attack-specific evaluations: the former provides a controlled comparison under a common attacker, while the latter examines recovery after method-specific adaptation.

2. Comparisons at matched reconstruction quality (Weakness 1 and Question 1).

We agree that Table 11 alone does not establish a better empirical privacy–utility tradeoff over Native-LDP. Higher QA accuracy at the same formal privacy budget can coexist with higher recovery, as the reviewer observes for text. Following the reviewer's suggestion, we conducted separate comparisons at approximately matched image LPIPS and text Word F1.

Image recovery. Using the defense-aware image attackers, we obtain:

MethodWhole-request ε\varepsilonLPIPSQA accuracy [95% CI]
Ours953.580.4754373.01% [70.15, 75.70]
PCA1,278.160.4753066.25% [63.51, 68.86]
Native-LDP372,790.500.4801272.31% [69.57, 75.00]

At these operating points, Ours achieves 6.76 percentage points higher QA accuracy than PCA. Ours and Native-LDP have close QA point estimates, while Ours uses a substantially smaller whole-request privacy budget.

Text recovery. Using the original fixed vocabulary-inversion attack, without additional attacker training, we obtain:

MethodWhole-request ε\varepsilonWord F1QA accuracy [95% CI]
Ours953.587.17%73.01% [70.17, 75.72]
PCA313.397.10%64.62% [61.79, 67.34]
Native-LDP3,115.196.84%58.33% [55.85, 60.85]

This comparison also covers all 814 test questions with three independent noise realizations and has been independently verified. The Word F1 range is only 0.33 percentage points.

At these approximately matched Word F1 levels, Ours improves QA accuracy over PCA by 8.39 percentage points (paired 95% CI: [6.61, 10.27]) and over Native-LDP by 14.68 percentage points (paired 95% CI: [12.14, 17.13]). These results support higher downstream QA utility when the evaluated vocabulary-inversion attack achieves similar text recovery.

Thank you for the response and the additional experiments.

The additional image inversion experiment addresses Weakness 2 and Question 2 by training an attack tailored to the attacker's knowledge of the client-side processing mechanism, using public auxiliary images. For text (Weakness 1 and Question 1), the proposed method improves QA accuracy over Native-LDP by 14.68 points at matched Word F1.

For images (Weakness 1 and Question 1), the proposed method improves QA accuracy over Native-LDP by only 0.70 percentage points at matched LPIPS (73.01% vs. 72.31%). Moreover, the reported ε\varepsilon values are too large to ensure meaningful statistical indistinguishability, though the authors emphasize that the proposed method achieves comparable accuracy to Native-LDP with a smaller ε\varepsilon. Overall, a better empirical privacy–utility tradeoff over Native-LDP has not been demonstrated for images and Weakness 1 and Question 1 remains unresolved.

Therefore, I maintain my rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.