- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes PrivMark, a pixel-space privacy protection framework designed to suppress soft-biometric information leakage in vision-language models (VLMs). The method extracts two behavioral steering directions through contrastive activation analysis, targeting incidental biometric disclosure in open-ended descriptions and compliance with direct biometric queries. These directions are incorporated into bounded pixel perturbations using a composite steering-alignment objective and momentum PGD, enabling image-level protection without accessing model internals at deployment. The authors also introduce BioleakBench, consisting of 1,600 VQA pairs, and evaluate the approach on three white-box source models and seven transfer targets. The results demonstrate promising biometric suppression and privacy–utility tradeoffs, although the effectiveness varies across models and evaluation settings.
Critical strengths
-
The paper addresses an important privacy concern in VLMs, namely the unintended disclosure of soft-biometric information through generated responses. The image-only protection setting is particularly relevant because it does not require modifying deployed models or accessing their internal activations during inference. This makes the proposed framework potentially applicable to closed-source VLM services.
-
The paper distinguishes between incidental biometric disclosure and direct-probe compliance and investigates their corresponding steering directions. The experiments across multiple VLM architectures, together with the cross-model transfer evaluation and additional analyses of model behavior, provide useful empirical observations about the feasibility and limitations of pixel-level privacy interventions.
Critical weaknesses
1. Limited methodological novelty and insufficient differentiation from closely related published work.
The core idea of PrivMark substantially overlaps with existing research on controlling VLM behavior through adversarial visual inputs and protecting sensitive information through image-level interventions. In particular, VISOR++ (ECCV 2026) has already demonstrated that activation-space steering signals can be transferred into optimized visual inputs to control VLM behavior without runtime access to model internals. ImageProtector ( ACL 2026) similarly explores image-level privacy protection through adversarial perturbations that induce refusal responses in downstream VLMs. Furthermore, AMK (ACM MM 2025) introduces an adversarial multimodal-key framework that controls sensitive-information disclosure in VLMs through optimized image keys and corresponding text keys, enabling authorized information recovery while preventing unauthorized access. Although its reversible, key-conditioned protection mechanism differs from PrivMark, AMK has already explored adversarial visual interventions for privacy-aware response control. Earlier works such as SteerVLM (EMNLP 2025) and Image Hijacks (ICML 2024) provide further foundations for activation-based steering and adversarial image-driven behavioral manipulation.
Against this background, the methodological contributions of PrivMark appear relatively incremental, primarily involving the construction of biometric-specific steering directions, their joint optimization, and an orthogonal-residual regularizer. While these modifications are potentially useful, the paper does not sufficiently establish their technical novelty or demonstrate substantial advantages over existing approaches. In particular, a controlled comparison with an adapted VISOR++ objective is missing, and the individual contributions of the proposed composition and regularization mechanisms are not adequately validated. Moreover, the relationship with earlier adversarial privacy protection frameworks such as AMK is not sufficiently discussed. Consequently, the paper does not convincingly establish a substantial methodological advance over the existing literature.
2. Insufficient experimental evidence for the central joint-steering mechanism and its practical effectiveness.
The central claim of PrivMark is that two distinct privacy-related behaviors can be jointly encoded into a single pixel perturbation. However, the main results in Tables 1 and 2 are obtained using independently optimized single-mode perturbations, whereas the actual composite objective is evaluated only on two source models in Table 3. Moreover, Table 4 evaluates cross-model transfer using description-mode perturbations rather than the proposed joint objective. Consequently, the strongest privacy suppression results and cross-model transfer results do not directly validate the central joint-steering mechanism. The composite results also exhibit substantial utility degradation relative to the clean models, making the practical benefits of combining the two objectives less convincing.
Additionally, the methodological justification and component-level validation remain incomplete. Section 3.1 motivates separate steering objectives based on the near-orthogonality of the two behavioral directions. However, near-orthogonality alone does not establish that a suitably normalized combination of the directions would be ineffective. The paper lacks comparisons with alternative vector-composition strategies and a dedicated ablation isolating the contribution of the orthogonal-residual regularizer. Furthermore, privacy protection is mainly evaluated through keyword suppression and refusal detection. Although the appendix includes an LLM-as-Judge analysis, the experiments do not establish whether sensitive attributes remain recoverable under adaptive or semantically varied queries. The robustness results in Appendix D.5 also indicate noticeable degradation after JPEG compression. These limitations leave the effectiveness, necessity, and robustness of the proposed joint optimization insufficiently demonstrated.
Reference
[1] VISOR++: Visual Input Based Steering for Large Vision Language Models. ECCV 2026.
[2] Leave My Images Alone: Preventing Multi-Modal Large Language Models from Analyzing Images via Visual Prompt Injection. ACL 2026.
[3] Reversible Privacy Preserving on Vision-Language Models via Adversarial Multimodal Key. ACM MM 2025.
[4] SteerVLM: Robust Model Control through Lightweight Activation Steering for Vision Language Models. EMNLP 2025.
[5] Image Hijacks: Adversarial Images can Control Generative Models at Runtime. ICML 2024.
Questions for authors
-
How does PrivMark substantially advance beyond existing adversarial privacy protection frameworks such as ImageProtector and AMK, considering their different protection mechanisms and threat models?
-
Can the authors provide a controlled comparison with an adapted VISOR++ activation-matching objective under identical perturbation budgets, source models, and evaluation protocols?
-
Why are the main results in Tables 1 and 2 based on independently optimized single-mode perturbations? Can the authors report joint-mode performance across all three source models and the held-out target models?
-
What is the individual contribution of the orthogonal-residual regularizer? Can the authors compare against its removal and against simpler approaches that directly combine the two steering directions?
-
Since the main privacy metrics rely heavily on keyword suppression and refusal detection, can the authors evaluate whether biometric attributes remain recoverable through semantically varied or adaptive queries?
-
Given the utility degradation in Table 1 and the sensitivity to JPEG compression reported in Appendix D.5, how robust is the jointly optimized perturbation under realistic image transformations, and what privacy–utility tradeoff does it achieve?