acceptodds
Withdrawn from ICLR 2027

PrivMark: Pixel-Space Steering against Privacy Leakage in Vision-Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision-language models (VLMs) routinely disclose soft-biometric attributes such as age, gender, and race in their text outputs, exposing demographic information from uploaded images to any downstream API user. Existing privacy defenses do not fit this setting: face de-identification leaves distributed soft-biometric cues intact, classifier-targeted adversarial perturbations do not transfer to open-ended generation, and stronger VLM-specific methods require retraining or runtime activation access unavailable to downstream APIs. We propose PrivMark, a framework that suppresses biometric leakage at the image level. By analyzing VLM activations, we identify two weakly aligned steering vector directions in hidden-state space, each tied to a distinct leakage behavior — one corresponding to incidental attribute disclosure under open-ended descriptions, the other to compliance with direct biometric probes instead of refusal — and encode both directions into a single bounded pixel perturbation, so the protected image alone enforces privacy with no invasive access to model internals at deployment. To support evaluation, we assemble BioleakBench, VQA pairs covering biometric privacy attributes under both free-form description and direct-probe protocols. Across three white-box source models, PrivMark achieves up to biometric suppression while retaining of clean OK-VQA utility, up to the privacy-utility tradeoff of the strongest pixel-only baseline. The same one-shot perturbations transfer consistently to seven unseen VLMs including closed-source APIs. Code and benchmark are available at https://anonymous.4open.science/r/pixel_steering_vlm-A558/

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
vision-language models, privacy preserving, activation steering, adversarial perturbation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_QUvkOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PrivMark, a pixel-space privacy protection framework designed to suppress soft-biometric information leakage in vision-language models (VLMs). The method extracts two behavioral steering directions through contrastive activation analysis, targeting incidental biometric disclosure in open-ended descriptions and compliance with direct biometric queries. These directions are incorporated into bounded pixel perturbations using a composite steering-alignment objective and momentum PGD, enabling image-level protection without accessing model internals at deployment. The authors also introduce BioleakBench, consisting of 1,600 VQA pairs, and evaluate the approach on three white-box source models and seven transfer targets. The results demonstrate promising biometric suppression and privacy–utility tradeoffs, although the effectiveness varies across models and evaluation settings.

Critical strengths
  1. The paper addresses an important privacy concern in VLMs, namely the unintended disclosure of soft-biometric information through generated responses. The image-only protection setting is particularly relevant because it does not require modifying deployed models or accessing their internal activations during inference. This makes the proposed framework potentially applicable to closed-source VLM services.

  2. The paper distinguishes between incidental biometric disclosure and direct-probe compliance and investigates their corresponding steering directions. The experiments across multiple VLM architectures, together with the cross-model transfer evaluation and additional analyses of model behavior, provide useful empirical observations about the feasibility and limitations of pixel-level privacy interventions.

Critical weaknesses

1. Limited methodological novelty and insufficient differentiation from closely related published work.

The core idea of PrivMark substantially overlaps with existing research on controlling VLM behavior through adversarial visual inputs and protecting sensitive information through image-level interventions. In particular, VISOR++ (ECCV 2026) has already demonstrated that activation-space steering signals can be transferred into optimized visual inputs to control VLM behavior without runtime access to model internals. ImageProtector ( ACL 2026) similarly explores image-level privacy protection through adversarial perturbations that induce refusal responses in downstream VLMs. Furthermore, AMK (ACM MM 2025) introduces an adversarial multimodal-key framework that controls sensitive-information disclosure in VLMs through optimized image keys and corresponding text keys, enabling authorized information recovery while preventing unauthorized access. Although its reversible, key-conditioned protection mechanism differs from PrivMark, AMK has already explored adversarial visual interventions for privacy-aware response control. Earlier works such as SteerVLM (EMNLP 2025) and Image Hijacks (ICML 2024) provide further foundations for activation-based steering and adversarial image-driven behavioral manipulation.

Against this background, the methodological contributions of PrivMark appear relatively incremental, primarily involving the construction of biometric-specific steering directions, their joint optimization, and an orthogonal-residual regularizer. While these modifications are potentially useful, the paper does not sufficiently establish their technical novelty or demonstrate substantial advantages over existing approaches. In particular, a controlled comparison with an adapted VISOR++ objective is missing, and the individual contributions of the proposed composition and regularization mechanisms are not adequately validated. Moreover, the relationship with earlier adversarial privacy protection frameworks such as AMK is not sufficiently discussed. Consequently, the paper does not convincingly establish a substantial methodological advance over the existing literature.

2. Insufficient experimental evidence for the central joint-steering mechanism and its practical effectiveness.

The central claim of PrivMark is that two distinct privacy-related behaviors can be jointly encoded into a single pixel perturbation. However, the main results in Tables 1 and 2 are obtained using independently optimized single-mode perturbations, whereas the actual composite objective is evaluated only on two source models in Table 3. Moreover, Table 4 evaluates cross-model transfer using description-mode perturbations rather than the proposed joint objective. Consequently, the strongest privacy suppression results and cross-model transfer results do not directly validate the central joint-steering mechanism. The composite results also exhibit substantial utility degradation relative to the clean models, making the practical benefits of combining the two objectives less convincing.

Additionally, the methodological justification and component-level validation remain incomplete. Section 3.1 motivates separate steering objectives based on the near-orthogonality of the two behavioral directions. However, near-orthogonality alone does not establish that a suitably normalized combination of the directions would be ineffective. The paper lacks comparisons with alternative vector-composition strategies and a dedicated ablation isolating the contribution of the orthogonal-residual regularizer. Furthermore, privacy protection is mainly evaluated through keyword suppression and refusal detection. Although the appendix includes an LLM-as-Judge analysis, the experiments do not establish whether sensitive attributes remain recoverable under adaptive or semantically varied queries. The robustness results in Appendix D.5 also indicate noticeable degradation after JPEG compression. These limitations leave the effectiveness, necessity, and robustness of the proposed joint optimization insufficiently demonstrated.

Reference

[1] VISOR++: Visual Input Based Steering for Large Vision Language Models. ECCV 2026.

[2] Leave My Images Alone: Preventing Multi-Modal Large Language Models from Analyzing Images via Visual Prompt Injection. ACL 2026.

[3] Reversible Privacy Preserving on Vision-Language Models via Adversarial Multimodal Key. ACM MM 2025.

[4] SteerVLM: Robust Model Control through Lightweight Activation Steering for Vision Language Models. EMNLP 2025.

[5] Image Hijacks: Adversarial Images can Control Generative Models at Runtime. ICML 2024.

Questions for authors
  1. How does PrivMark substantially advance beyond existing adversarial privacy protection frameworks such as ImageProtector and AMK, considering their different protection mechanisms and threat models?

  2. Can the authors provide a controlled comparison with an adapted VISOR++ activation-matching objective under identical perturbation budgets, source models, and evaluation protocols?

  3. Why are the main results in Tables 1 and 2 based on independently optimized single-mode perturbations? Can the authors report joint-mode performance across all three source models and the held-out target models?

  4. What is the individual contribution of the orthogonal-residual regularizer? Can the authors compare against its removal and against simpler approaches that directly combine the two steering directions?

  5. Since the main privacy metrics rely heavily on keyword suppression and refusal detection, can the authors evaluate whether biometric attributes remain recoverable through semantically varied or adaptive queries?

  6. Given the utility degradation in Table 1 and the sensitivity to JPEG compression reported in Appendix D.5, how robust is the jointly optimized perturbation under realistic image transformations, and what privacy–utility tradeoff does it achieve?

Reviewer_yTLjOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduce PrivMark, a method to mitigate biometric leakage responses from a VLM, particularly by preventing disclosure in open-ended descriptions and encouraging refusal responses to biometric probes. This is achieve during the inference time through steering. PrivMark steers the responses in image pixel spaces rather than model activation space. PrivMark first extracts two contrastive activation steering, then optimize the image visual perturbation to align hidden-state changes with these directions. PrivMark doesn't require model access at inference time. The experiment show promising results with the same source VLM and some transfer to unseen VLMs

Critical strengths
  • The idea is interesting that moves activation steering from model activation space into pixel space, requiring no model access at the inference time.

  • Reasonably extensive evaluation. The paper evaluates multiple source models, unseen VLMs, privacy–utility tradeoffs, and robustness to image transformations.

Critical weaknesses
  • Moving steering from model activate to image pixel is interesting, but might not provide additional practical deployment benefits. In most cases, moving to image pixel space reduce the performance. No model access required, however, this is not a strong advantage in the deployment perspective. Another advantage is transferability to other models, however, the performance is not very strong on unseen VLMs

  • The refusal steering direction is constructed using biometric-related examples. This does not guarantee biometric-specific refusal. The direction could partially capture general refusal behavior and refuse unrelated, legitimate questions. The results show that OK-VQA accuracy drops, the evaluation setting makes it unclear if this is because of over-refusal, or other reasons.

Questions for authors

Does PrivMark increase refusal rates on unrelated, non-biometric questions? Can the authors distinguish over-refusal from degraded visual perception?

Reviewer_r6FoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PrivMark aims to limit automated soft-biometric profiling of shared photographs by changing image pixels without modifying the downstream VLM. To do this, it derives two activation directions: one to suppress sensitive attributes in descriptions and another to refuse direct attribute queries. It then uses these directions to optimize a bounded image perturbation. The authors evaluate the method on BioleakBench, which contains 400 images and 1,600 queries, measuring reduced disclosure in model outputs rather than whether the attributes remain visually inferable. The main results optimize the two behaviors separately, while joint protection with a single perturbation is tested on two source models and substantially lowers VQA accuracy.

Critical strengths
  • The method fits the uploader's limited control. Uploaders can change their images but usually cannot modify downstream VLMs. PrivMark works within this constraint and targets both unwanted attribute mentions in descriptions and responses to direct attribute queries.
  • The single-mode results show a useful privacy–utility tradeoff. PrivMark achieves the highest harmonic mean among pixel-only methods in four of six single-mode comparisons. Budget sweeps and JPEG/resize tests also help show how protection and utility change across operating conditions.
Critical weaknesses
  • The practical privacy benefit is unclear. The viewer can already inspect the photo. Automated profiling could increase risk by making analysis cheaper and easier to scale, but the experiments measure only attribute mentions and refusals. The threat model needs to connect these measures to the claimed harm and distinguish reduced VLM disclosure from protection against a viewer inferring the attributes.
  • Joint protection has a substantial utility cost without a clear tradeoff advantage. The strong single-mode results use separate perturbations. When one perturbation targets both behaviors, evaluation covers only two source models and VQA falls from about 62–63 to 40–44. Since the budget sweep is also single-mode, the paper needs joint tradeoff curves and baseline comparisons at matched utility to establish the combined method's value.
Questions for authors
  • Clarify the privacy threat. Please explain what additional privacy risks arise when an attacker who can already inspect the photo directly uses a VLM to analyze it, and how reduced attribute mentions or increased refusals reduce those risks. State which queries, model choices, and image transformations the attacker may use, and which forms of attribute inference remain outside the protection claim.

  • Establish the joint privacy–utility advantage. Please report suppression, refusal, and utility together across perturbation budgets, comparing pixel-only baselines and simpler steering objectives at matched utility. Include non-sensitive questions about the same protected person images, alongside general OK-VQA accuracy.

  • Define the scope of joint protection. Please distinguish single-mode robustness and transfer results from those using one image protected against both disclosure modes. State whether the joint gains survive JPEG/resize and transfer to other models.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.