- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes a client-side compression–perturbation–decoding framework for edge–cloud VLM inference that provides whole-request local differential privacy (LDP) for joint image and contextual-text inputs. Experiments demonstrate substantially improved utility compared to direct LDP applied to native tokens and a PCA-based LDP baseline, while also reducing content recovery under fixed inversion attacks.
Critical strengths
-
The paper introduces a comprehensive local-release pipeline tailored to multimodal VLMs with frozen encoders and backbones.
-
The experimental results demonstrate substantial performance improvements over existing baselines.
-
The paper provides rigorous privacy proofs and detailed mechanism analysis to support the theoretical foundations of the proposed method.
Critical weaknesses
-
The proposed method involves a reconstruction process as part of its framework, while the inversion analysis appears to rely on the same reconstruction mechanism. This evaluation may not adequately reflect the actual privacy protection provided by the method, as it does not consider attackers employing different or more sophisticated reconstruction strategies.
-
The privacy evaluation against inversion attacks is insufficient, as it considers only black-box inversion attacks. More advanced white-box inversion attacks should be included to provide a comprehensive assessment of the proposed method's robustness against stronger adversaries.
-
The privacy–utility trade-off in Table 11 is difficult to assess because the proposed method and the baselines exhibit substantially different utility and reconstruction scores. In particular, the baselines achieve much lower Word F1 and SSIM scores, making it unclear whether the observed differences in privacy protection result from the defense mechanisms or the differences in utility. The authors should compare all methods under the same or comparable utility levels and evaluate their corresponding reconstruction performance to ensure a fair comparison. Moreover, the proposed method still suffers from a substantial utility degradation, achieving only 77.23 compared to 93.50 without protection. Such a significant performance drop raises concerns about the practical applicability of the proposed method and whether the privacy benefits justify the loss in utility.
Questions for authors
-
How does the proposed method perform against more advanced white-box inversion attacks?
-
How does the proposed method's effectiveness against inversion attacks vary across different utility levels?
-
How does the proposed method compare with existing baselines in terms of privacy protection when evaluated at the same or comparable utility levels?
3 comments
3. Utility loss relative to the unprotected model and practical value (Practicality concern in Weakness 3).
The decrease from 93.50% to 77.23% is a substantial accuracy cost. The clean-codec control achieves 85.61%, indicating that representation processing itself contributes to part of this gap.
We believe practical value should be assessed by considering both this cost and the task performance retained under the same privacy constraint. At whole-request , Ours, PCA-based release, and Native-LDP achieve 77.23%, 67.07%, and 57.76% accuracy, respectively. These results show that task-aware representation processing preserves more utility at the same declared budget.
The additional comparison at similar QA accuracy provides a complementary perspective: comparable observed task performance can be accompanied by substantially lower text recovery for Ours under the evaluated attack. Together, these comparisons make the accuracy cost and changes in recoverability more explicit. The appropriate operating point depends on an application's accuracy requirements and privacy constraints; our method focuses on improving task utility within those constraints.
Thank you again for your thoughtful review. After you’ve had a chance to read our response, please let us know if there’s anything you’d like us to clarify.
We thank the reviewer for recognizing the overall framework, utility improvements, and privacy analysis. The comments highlight two aspects that benefit from further clarification: whether inversion evaluation sufficiently accounts for the defense mechanism, and how methods compare at comparable utility or reconstruction levels. We address these through clarification of the attack setup and additional experiments.
1. Distinguishing the defense decoder from the inversion attacker, and defense-aware evaluation (Weaknesses 1–2 and Question 1).
There are two distinct reconstruction tasks in our framework and evaluation. The defense decoder maps noisy compact representations back to the native token format required by the frozen VLM, with the objective of supporting downstream inference. The inversion attacker instead reconstructs the original RGB image from the final visual representation observed by the server. These are separate networks with independent parameters and training objectives.
The original evaluation used a common fixed inversion network to provide a reproducible reference across release methods. Following the suggestion that attacks should exploit knowledge of the defense, we have additionally completed method-specific inversion adaptation using public auxiliary images paired with the actual final representations generated by each client-side mechanism, including noisy outputs.
This allows attack training to account directly for the representation changes introduced by compression, perturbation, and decoding. Since adaptation may converge differently across release distributions, we interpret the results in the context of the training protocol. The added evaluation focuses on learned defense-aware inversion; white-box attacks that directly optimize candidate inputs through the known mechanism remain a direction for broader attack coverage.
2. Recovery at comparable utility and across utility levels (Comparison concern in Weakness 3, Questions 2–3).
Following the reviewer's suggestion, we evaluated recovery at operating points with similar observed test QA accuracy. Image recovery uses the adapted inversion attackers, while text recovery uses the fixed vocabulary-inversion attack:
The arrows indicate the direction of poorer reconstruction. The experiments cover all 814 questions with three independent noise realizations and were independently checked.
Both baseline QA means are within two percentage points of Ours on the test set. At these operating points, Ours has substantially lower text recovery under the evaluated vocabulary attack. Its image LPIPS is higher than PCA's and close to Native-LDP's. Thus, similar task performance can coexist with substantially different text recovery, while the image-recovery comparison depends on the particular baseline.
This comparison is exploratory: PCA did not satisfy the development-set QA-matching criterion, and Native-LDP uses a previously observed test configuration. The table therefore reports the budgets and performance of these evaluated operating points.
We also examined the tradeoff in the reverse direction. When the Word F1 range across methods is only 0.33 percentage points, Ours improves QA accuracy over PCA by 8.39 percentage points (paired 95% CI: [6.61, 10.27]) and over Native-LDP by 14.68 percentage points (paired 95% CI: [12.14, 17.13]). This provides additional evidence of higher task utility at similar recovery achieved by the evaluated vocabulary attack.
To examine how recovery varies across utility levels, our earlier complete three-budget adaptation study evaluated Ours with the same adapted image attacker across budgets:
The image metrics come from that three-budget adaptation study, and the text metrics come from the corresponding fixed vocabulary attack. Increasing the budget raises QA accuracy while also improving text and image reconstruction. These results illustrate the budget-dependent utility–recovery tradeoff and motivate reporting task performance alongside recovery in both modalities.