- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies the privacy risk that closed-source vision-language models (VLMs) may infer users’ sensitive attributes while providing their normal services, and formalizes this problem as Service-Time Private Attribute Inference (ST-PAI). Existing defenses based on adversarial perturbations are vulnerable to service-side preprocessing; to address this, the authors propose PEEL, a cue-driven progressive semantic editing method. PEEL uses a local VLM to identify and rank privacy-relevant visual cues, and generates candidate images with different levels of protection through nested parallel editing and a velocity inheritance mechanism, allowing users to choose between privacy protection and image fidelity. In addition, the authors construct SynthPAI-Visual, a synthetic multi-attribute dataset, and validate the effectiveness of the method through experiments on geolocation, the privacy–fidelity trade-off, and downstream VQA.
Critical strengths
- The motivation is intuitive, and the problem addressed is of considerable importance.
- The method is well designed and offers a degree of technical novelty.
- The experiments are fairly comprehensive.
Critical weaknesses
For this paper, I have the following concerns:
- I also have some doubts about the assumptions underlying Proposition 1 and the scope of its conclusion. The key premise of the proposition is that service-side preprocessing can effectively suppress sensitivity along off-manifold directions. However, alone is not sufficient to guarantee that perturbation-based defenses fail; the actual outcome also depends on the relationship among parameters such as , , , and . Although the authors provide some empirical support in Appendix A.3, these experiments do not directly verify whether these parameters satisfy the required conditions in actual closed-source VLMs. As a result, the current theoretical analysis reads more like an explanation of why perturbation-based defenses may fail under specific assumptions than a proof that such defenses necessarily fail under general service-side preprocessing. I would like the authors to further discuss the plausibility of these assumptions for real models and to delineate the scope of the theoretical conclusion more clearly.
- The dataset construction and privacy evaluation may suffer from model-dependent bias, which is my main concern. According to Appendix B.4, the valid attribute labels in SynthPAI-Visual are obtained by human verification of high-confidence attribute inferences, together with their visual evidence, proposed by Qwen3-VL-8B, while Qwen3-VL-8B is also used by PEEL for cue discovery, ranking, and leakage annotation. This design may concentrate the evaluation on privacy cues that the local model can identify, without adequately covering evidence that other VLMs could discover independently. Although human verification improves label reliability, it cannot fully eliminate the selection bias introduced by the mechanism that generates candidate cues. In addition, each image is annotated by a single annotator, with no independent repeated annotation or inter-annotator agreement analysis. The attribute distribution of the dataset is also highly imbalanced: for example, location has 1,190 valid annotations, whereas place of birth and age have only 3 and 11, respectively.
- According to Appendix A.6, PEEL takes 228.5 seconds to generate about 6.1 candidate images on average, with a peak memory of about 35 GB. Although parallel editing offers some efficiency advantage over sequential editing, this resource requirement remains high for ordinary users’ local devices. Moreover, the experiments use GPT-4.1 to construct the target prompts and to format some of the outputs. Although the authors note that these text-processing steps can be replaced by a local LLM, the performance and quality of a fully local pipeline have not been validated experimentally. Since these texts may contain privacy-relevant visual cues, this issue bears directly on the service-side privacy threat that the paper emphasizes.
- I have some questions about how the defenses are compared in Table 1. The authors evaluate PEEL at its final candidate , whereas the perturbation-based methods are constrained by a fixed budget, so the amount by which each method modifies the original image may differ substantially. While this experiment shows that PEEL maintains low privacy leakage under the tested preprocessing, it does not directly demonstrate that PEEL outperforms perturbation-based methods under the same fidelity or task-utility requirements.
- According to Section 5.3 and Table 5 in Appendix B.2, the privacy–fidelity frontier comparison in Figure 2 uses only GPT-4o as the attacker. Although the authors evaluate additional VLMs in the robustness experiments and ablation studies, the main quantitative comparison supporting PEEL's superior privacy–fidelity trade-off is conducted with a single attacker. It is therefore unclear whether the same advantage holds consistently across different closed-source VLMs. I would appreciate additional privacy–fidelity comparisons using other attackers, such as Gemini 2.5 Pro and Claude Opus 4.8, to strengthen the generality of this conclusion.
Questions for authors
See the weaknesses.
3 comments
W4. Table 1 compares PEEL at its strongest edit with budget-constrained perturbations, i.e., not at equal fidelity.
We agree that the two families cannot be compared directly, and we were aware of this when designing Table 1. They pursue different notions of fidelity: perturbation-based defenses aim to leave the image visually unchanged within an imperceptibility budget, whereas editing deliberately changes the content of the cues and trades fidelity against privacy along the frontier. There is therefore no common operating point at which both are "equally faithful." Table 1 is accordingly not meant to rank the methods, but to measure how each family behaves under preprocessing, i.e., the change in leakage relative to the same method without preprocessing. The comparison at matched fidelity is the subject of Section 5.3, where all methods are editing-based and share the same notion of fidelity. As stated in Section 5.1, the utility of the two families cannot be compared on a common scale; we have made this explicit in the caption of Table 1 as well.
W5. Figure 2 uses a single attacker.
Following the reviewer's suggestion, we attacked all candidates of Figure 2 with two further closed-source VLMs, Gemini 2.5 Pro and Claude Opus 4.8, using the same prompt and scoring, and included frontier variants of PPA and U2S that receive PEEL's ranked cues. The privacy-fidelity advantage of PEEL does not depend on the attacker: for all three attackers, PEEL achieves the best trade-off between privacy and fidelity, and MTC matches its privacy only at a much lower fidelity. We have added the per-attacker frontiers to Appendix C.7.
Thank you for your response. I’ll keep my score as it is.
We thank the reviewer for the detailed comments and the positive assessment. We address each point below.
W1. Assumptions and scope of Proposition 1.
We agree with the reviewer that Proposition 1 explains why bounded perturbations can fail under service-side preprocessing given explicit assumptions, rather than proving that they necessarily fail for every model and every preprocessing operation. We have moved the scope statement from Appendix A.2 into Section 3.2 to make it explicit in the main text, where Proposition 1 is now stated informally, with the formal version in Appendix A.2.
Scope. Part (i) shows that, without preprocessing, an imperceptible off-manifold perturbation can succeed. Part (ii) shows that, once preprocessing attenuates the off-manifold sensitivity (γ<1), success requires a semantic component of size Δi/(2LS). Whether this exceeds the imperceptibility threshold depends on LS, L⊥, γ, and Δi. Of these conditions, γ<1 depends on the preprocessing rather than on the model: denoising-type preprocessing such as DiffPure maps the input back toward natural images before it reaches any model, so it holds regardless of which closed-source model follows. The threshold condition, in contrast, depends on LS, L⊥, and Δi of the model and cannot be verified for black-box VLMs, which is why we state the conclusion as conditional. A perturbation with a large budget, or a weak preprocessing (γ close to 1), may therefore still succeed; the proposition only states that the protection must then rely on a semantic change.
W2. Model-dependent bias of the dataset, single annotator, and attribute imbalance.
Model-dependent bias. To test whether the evaluation favors the cues that Qwen3-VL-8B can identify, we replaced PEEL's local model Mo with four other VLM families (Gemma 3 12B, InternVL3.5-8B, Ministral 3 8B, GLM-4.6V-Flash), keeping all other settings the same as in Section 5.3. If the labels were biased toward cues that only Qwen3-VL-8B finds, a different local model would discover different cues and lose privacy. Instead, PEEL reaches comparable or slightly higher privacy with every other model (+0.03 to +0.08 over Qwen3-VL-8B, none statistically significant), at a small fidelity cost. Moreover, the evidence used by independent VLMs is largely covered: Qwen3-VL-8B detects 97–100% of the leakages found by the closed-source attackers (Table 2). The geolocation results (Table 1), whose GPS ground truth is independent of Qwen3-VL-8B, further confirm the effectiveness of PEEL.
Annotation quality. Each batch of 100 images was assigned to a single annotator, and we controlled quality within our budget by auditing every batch against an expert (one of the authors). The expert re-judged 5–10 randomly sampled images of each batch following the same guidelines; if more than one of them was judged incorrectly, the annotator had to re-check the whole batch and resubmit it (Appendix B.4).
Attribute imbalance. The imbalance reflects what environmental scenes without people can reveal rather than a design choice: all eight attributes are encoded into the scenes, but place of birth requires explicit documents such as a passport, as souvenirs or flags only indicate travel or affinity, and age is rarely supported by more than a stereotypical guess, so annotators rejected most such inferences. We now also report the per-attribute privacy leakage of raw images (Table 10): the frequent attributes are also the most exposed, so the imbalance concentrates the evaluation on the attributes that leak the most.
Reviewer 1 raised similar concerns, and we have therefore emphasized these points in the revised paper (Appendices B.4 and C.5).
W3. Compute cost, and the use of an API LLM to construct the target prompts.
Compute. We agree that compute remains a limitation (Appendix E), and two properties mitigate it (Appendix A.6). First, the question-agnostic frontier is built once per image and offline, e.g., when a photo is taken or imported, and is reused for every later request, so it adds no latency to the interaction with the service. Second, the 1+2K velocity evaluations per step are independent and can be distributed over GPUs, which reduces the time per frontier from 228.5 s to about 17 s with 2K+1 GPUs.
API LLM. To validate a fully local pipeline, we replaced the API LLM that writes the target prompt chains with a local Gemma 3 12B and re-ran PEEL with the same cues and editor of Figure 2.
No difference is significant and nesting is unchanged, so writing the prompt chains locally does not affect the results (Appendix A.7).