- Rating
- 3(+1)
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces persistent visual pseudonymization, where a sensitive fact in an image is mapped to a type-compatible substitute that remains coherent across a multi-turn conversation. The authors observe that early model answers become textual evidence for later turns, so protection must account for path-dependent interaction rather than isolated prompts. FactPatch first learns a low-rank transformation from the original representation to the behavior of a verified counterfactual image across five semantic access types and disturbed dialogue histories. It then distills this transformation on-policy into a localized image patch while preserving non-sensitive content. On a new benchmark of 1,200 verified TextVQA counterfactual pairs, FactPatch is evaluated on unseen 10-turn and 20-turn trajectories, held-out semantic accesses, unseen fact pairs, source and held-out MLLMs, image transformations, and matched ablations. On 10-turn trajectories it reaches 62.3% persistent pseudonymization success and 24.6% any-private-leakage, compared with 52.8% and 33.5% for the strongest matched baseline, while retaining 96.6% utility.
Critical strengths
Writing. The writing is overall easy to understand.
Experiments. Experiments are sufficient to support the claims.
Critical weaknesses
Potentially weak baseline. Line 322, appear authors use a baseline from the year of 2024. Either the problem they try to solve is not a very important problem, or they ignore the other papers and pick a weak baseline. Authors are suggested to discuss the existing workarounds of the problem, and why these workarounds are not taken as baseline.
Questions for authors
See weakness.
4 comments
Additional experiments with recent visual attack and privacy-protection methods. In addition to the original baseline evaluation, we conducted new experiments with recent methods from different research directions. These comparisons extend the evaluation to general-purpose adversarial image optimization, privacy suppression, and direct removal of sensitive visual information.
AnyAttack as an adapted image-optimization baseline.
AnyAttack (Zhang et al., CVPR 2025) is a general-purpose targeted adversarial image-generation method for vision-language models, rather than a privacy-defense method. We adapt its image-matching objective by using the counterfactual image x* as the target image and the original image x as the clean input, while restricting the perturbation to the same 3% patch support used by FACTPATCH. Since the original method uses a CLIP ViT-B/32 surrogate, we additionally evaluate a variant using the pooled representation of the source MLLM's own visual encoder as the surrogate. Both variants are evaluated under our 10-turn protocol.
With the original CLIP surrogate, AnyAttack achieves 41.3% STS, 14.6% PPS, and 69.7% APL. Using the source model's visual encoder improves these results to 58.2% STS, 23.8% PPS, and 61.2% APL. In comparison, FACTPATCH achieves 84.1% STS, 62.3% PPS, and 24.6% APL. Although the adapted AnyAttack uses a counterfactual target associated with the same substitute fact, its image-level representation-matching objective does not explicitly model shared fact binding across semantic accesses and evolving conversational histories. This comparison therefore examines whether general-purpose targeted image optimization alone can achieve persistent substitute-fact protection.
Privacy-suppression methods.
We additionally evaluate ImageProtector (Shao et al., ACL 2026) and GeoShield (Liu et al., AAAI 2026), which pursue privacy protection through analysis suppression and attribute suppression, respectively. Since these methods do not explicitly establish a substitute fact, PPS and STS are not applicable. We therefore compare their private-fact exposure and non-sensitive utility.
ImageProtector achieves an APL of 18.7% in its original whole-image setting, with a non-sensitive refusal rate of 78.6% and a utility of approximately 19%. Under our localized 3% perturbation budget, its APL increases to 45.8%, with a refusal rate of 27.7% and a utility of approximately 65%.
GeoShield achieves an APL of 41.2% and a utility of 93.8% in its original setting. Under the localized budget, it achieves an APL of 52.3% and a utility of 95.9%. Its first-access exposure rate is 24.1%, compared with 10.4% for FACTPATCH. FACTPATCH achieves 24.6% APL while retaining 96.6% non-sensitive utility.
These results illustrate the distinction between suppressing access to sensitive information and establishing a persistent substitute interpretation. ImageProtector can substantially reduce leakage in its whole-image setting, but this comes with a considerable loss of non-sensitive utility. Under localized protection, both suppression approaches exhibit higher cumulative exposure than FACTPATCH.
Region-editing reference.
We additionally evaluate black-box redaction as a reference for direct removal of sensitive visual information. Redaction achieves an APL of 4.8%, but reduces utility on questions concerning the protected region to 55.1%, compared with 93.5% for FACTPATCH. Its utility on unrelated global questions is 97.9%, resulting in an overall utility of approximately 86.0%, compared with 96.6% for FACTPATCH. Although redaction provides a low-exposure reference, it removes the original visual evidence and therefore does not satisfy our requirement to preserve the sensitive region for human inspection. We include it as a reference point rather than a directly comparable substitute-fact baseline.
Comparison scope and interpretation.
The evaluated methods address different objectives and therefore require different comparison criteria. VMI-style, CroPA, and adapted AnyAttack are included in the direct image-side comparison for substitute-fact protection. MULTI-PROMPT and the two matched controls provide additional comparisons for analyzing the contribution of our optimization design. ImageProtector and GeoShield are evaluated as privacy-suppression approaches using APL and utility, while redaction serves as a reference for region removal. We also discuss CloakDiff (Lu et al., 2026) as a complementary reversible-protection approach that relies on recovery mechanisms for authorized access and operates under different deployment assumptions.
Overall, the additional experiments broaden our evaluation beyond methods specifically designed for persistent visual effects. They show that general-purpose adversarial image optimization, privacy suppression, and region removal offer different protection–utility trade-offs. Among the directly comparable substitute-fact approaches, FACTPATCH achieves the strongest trajectory-level consistency and lowest private-fact exposure in our evaluation. Compared with localized privacy-suppression methods, it also provides lower cumulative exposure while retaining high non-sensitive utility. These results highlight the importance of explicitly maintaining a coherent substitute fact across repeated semantic accesses and evolving conversational histories. We will revise the manuscript for the above contents.
Reference
[1] Luo H, Gu J, Liu F, et al. An image is worth 1000 lies: Transferability of adversarial images across prompts on vision-language models[C]//International Conference on Learning Representations. 2024, 2024: 23701-23722.
[2] Schlarmann C, Hein M. Visual Memory Injection Attacks for Multi-Turn Conversations[J]. arXiv preprint arXiv:2602.15927, 2026.
[3] Zhang J, Ye J, Ma X, et al. Anyattack: Towards large-scale self-supervised adversarial attacks on vision-language models[C]//2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2025: 19900-19909.
[4] Shao Z, Liu H, Hu Y, et al. Leave My Images Alone: Preventing Multi-Modal Large Language Models from Analyzing Images via Visual Prompt Injection[C]//Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026: 1588-1604.
[5] Liu X, Jia X, Xun Y, et al. Geoshield: Safeguarding geolocation privacy from vision-language models via adversarial perturbations[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2026, 40(42): 35653-35661.
[6] Lu Q, Zhou Z, Song Y, et al. Imperceptible and Reversible Adversarial Examples against Vision-Language Models for Privacy Protection[J]. arXiv preprint arXiv:2607.10329, 2026.
Thanks authors for their detailed rebuttals. I think their responses have addressed some of my concerns. I will increase my ratings. But due to the limited prior knowledge in the field this paper relates to, I maintain my confidence to "1" and very uncertain.
W1. Baseline Coverage and Comparison with Existing Privacy-Protection Methods
We thank the reviewer for raising this important concern. In response, we clarify the positioning of our original baselines and further expand our evaluation with additional comparisons against recent visual attack and privacy-protection methods.
Existing baselines and their relevance to our setting. Our original evaluation already includes several relevant baselines beyond CroPA (2024). In particular, Visual Memory Injection (VMI; Schlarmann & Hein, 2026) is the closest recent work to our setting, as it studies the persistence of visual effects across multi-turn conversations. We include a VMI-style implementation adapted to our substitute-fact objective, where the target response corresponds to the answer implied by the designated substitute fact for each fact. CroPA serves as an additional reference for cross-prompt transfer.
We also include MULTI-PROMPT, which jointly optimizes multiple fact queries without explicitly modeling evolving dialogue states. In addition, we introduce two matched controls, Direct On-Policy and Counterfactual Distillation, to examine the contribution of fact-level transport. These controls isolate the effect of structured fact binding from multi-turn optimization using model-generated histories and direct counterfactual supervision.
Results from the original baseline evaluation. Our original results already demonstrate consistent improvements over these baselines. On the main 10-turn benchmark, FACTPATCH achieves 84.1% STS, 62.3% PPS, and 24.6% APL, compared with 77.6% STS, 42.8% PPS, and 45.6% APL for VMI-style. CroPA achieves 72.4% STS, 26.3% PPS, and 58.7% APL, while MULTI-PROMPT achieves 80.2% STS, 40.6% PPS, and 47.5% APL. The two matched controls achieve 49.7% PPS and 37.4% APL for Direct On-Policy, and 52.8% PPS and 33.5% APL for Counterfactual Distillation. FACTPATCH therefore improves trajectory-level consistency while reducing private-fact exposure across both external comparisons and controlled variants.
The advantage also persists under more challenging evaluation settings. On held-out semantic accesses, FACTPATCH achieves 46.8% PPS, compared with 27.2% for VMI-style, 29.1% for MULTI-PROMPT, and 39.8% for Counterfactual Distillation. The difference is particularly pronounced for COMPARE, where FACTPATCH achieves 42.7% compared with 22.0% for VMI-style. On the 20-turn evaluation, FACTPATCH achieves 48.6% PPS and 32.4% APL, compared with 28.7% PPS and 55.5% APL for VMI-style. These results demonstrate that the advantage extends beyond initial substitution to consistency across different semantic accesses and longer conversational interactions.