acceptodds
Under review as a conference paper at ICLR 2027

FactPatch: Persistent Visual Pseudonymization for Multi-Turn Vision-Language Models

Abstract

Multimodal large language models may repeatedly access the same sensitive fact in an image over the course of a conversation, and these accesses are not independent. We find that an early visual fact interpretation becomes part of the model-generated dialogue history and can subsequently influence how the same fact is interpreted, causing identical images to evolve toward different and persistent interpretations along different conversational paths. This suggests that multi-turn visual privacy is not merely about changing a single response, but about maintaining a protected fact interpretation under continuously changing conversational evidence. Based on this observation, we propose FactPatch, which uses a small, localized visual perturbation to map a true sensitive fact to a semantically consistent substitute and leverages persistent visual evidence to stabilize this interpretation across diverse conversational paths and factual disturbances. When the model drifts back toward the true fact, the same visual evidence helps restore the protected interpretation. Experiments show that FactPatch more reliably preserves the substitute fact on unseen conversational trajectories and held-out models, substantially reduces the re-exposure of true sensitive information over multi-turn interactions, and largely preserves the model’s ability to understand non-sensitive visual content.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Multimodal Large Language Models, Visual Privacy, Multi-Turn Interaction

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_MA9WOpenReview ↗
Rating
3(+1)
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces persistent visual pseudonymization, where a sensitive fact in an image is mapped to a type-compatible substitute that remains coherent across a multi-turn conversation. The authors observe that early model answers become textual evidence for later turns, so protection must account for path-dependent interaction rather than isolated prompts. FactPatch first learns a low-rank transformation from the original representation to the behavior of a verified counterfactual image across five semantic access types and disturbed dialogue histories. It then distills this transformation on-policy into a localized image patch while preserving non-sensitive content. On a new benchmark of 1,200 verified TextVQA counterfactual pairs, FactPatch is evaluated on unseen 10-turn and 20-turn trajectories, held-out semantic accesses, unseen fact pairs, source and held-out MLLMs, image transformations, and matched ablations. On 10-turn trajectories it reaches 62.3% persistent pseudonymization success and 24.6% any-private-leakage, compared with 52.8% and 33.5% for the strongest matched baseline, while retaining 96.6% utility.

Critical strengths

Writing. The writing is overall easy to understand.
Experiments. Experiments are sufficient to support the claims.

Critical weaknesses

Potentially weak baseline. Line 322, appear authors use a baseline from the year of 2024. Either the problem they try to solve is not a very important problem, or they ignore the other papers and pick a weak baseline. Authors are suggested to discuss the existing workarounds of the problem, and why these workarounds are not taken as baseline.

Questions for authors

See weakness.

4 comments

W1. Baseline Coverage and Comparison with Existing Privacy-Protection Methods

We thank the reviewer for raising this important concern. In response, we clarify the positioning of our original baselines and further expand our evaluation with additional comparisons against recent visual attack and privacy-protection methods.

Existing baselines and their relevance to our setting. Our original evaluation already includes several relevant baselines beyond CroPA (2024). In particular, Visual Memory Injection (VMI; Schlarmann & Hein, 2026) is the closest recent work to our setting, as it studies the persistence of visual effects across multi-turn conversations. We include a VMI-style implementation adapted to our substitute-fact objective, where the target response corresponds to the answer implied by the designated substitute fact for each fact. CroPA serves as an additional reference for cross-prompt transfer.

We also include MULTI-PROMPT, which jointly optimizes multiple fact queries without explicitly modeling evolving dialogue states. In addition, we introduce two matched controls, Direct On-Policy and Counterfactual Distillation, to examine the contribution of fact-level transport. These controls isolate the effect of structured fact binding from multi-turn optimization using model-generated histories and direct counterfactual supervision.

Results from the original baseline evaluation. Our original results already demonstrate consistent improvements over these baselines. On the main 10-turn benchmark, FACTPATCH achieves 84.1% STS, 62.3% PPS, and 24.6% APL, compared with 77.6% STS, 42.8% PPS, and 45.6% APL for VMI-style. CroPA achieves 72.4% STS, 26.3% PPS, and 58.7% APL, while MULTI-PROMPT achieves 80.2% STS, 40.6% PPS, and 47.5% APL. The two matched controls achieve 49.7% PPS and 37.4% APL for Direct On-Policy, and 52.8% PPS and 33.5% APL for Counterfactual Distillation. FACTPATCH therefore improves trajectory-level consistency while reducing private-fact exposure across both external comparisons and controlled variants.

The advantage also persists under more challenging evaluation settings. On held-out semantic accesses, FACTPATCH achieves 46.8% PPS, compared with 27.2% for VMI-style, 29.1% for MULTI-PROMPT, and 39.8% for Counterfactual Distillation. The difference is particularly pronounced for COMPARE, where FACTPATCH achieves 42.7% compared with 22.0% for VMI-style. On the 20-turn evaluation, FACTPATCH achieves 48.6% PPS and 32.4% APL, compared with 28.7% PPS and 55.5% APL for VMI-style. These results demonstrate that the advantage extends beyond initial substitution to consistency across different semantic accesses and longer conversational interactions.

Additional experiments with recent visual attack and privacy-protection methods. In addition to the original baseline evaluation, we conducted new experiments with recent methods from different research directions. These comparisons extend the evaluation to general-purpose adversarial image optimization, privacy suppression, and direct removal of sensitive visual information.

AnyAttack as an adapted image-optimization baseline.

AnyAttack (Zhang et al., CVPR 2025) is a general-purpose targeted adversarial image-generation method for vision-language models, rather than a privacy-defense method. We adapt its image-matching objective by using the counterfactual image x* as the target image and the original image x as the clean input, while restricting the perturbation to the same 3% patch support used by FACTPATCH. Since the original method uses a CLIP ViT-B/32 surrogate, we additionally evaluate a variant using the pooled representation of the source MLLM's own visual encoder as the surrogate. Both variants are evaluated under our 10-turn protocol.

With the original CLIP surrogate, AnyAttack achieves 41.3% STS, 14.6% PPS, and 69.7% APL. Using the source model's visual encoder improves these results to 58.2% STS, 23.8% PPS, and 61.2% APL. In comparison, FACTPATCH achieves 84.1% STS, 62.3% PPS, and 24.6% APL. Although the adapted AnyAttack uses a counterfactual target associated with the same substitute fact, its image-level representation-matching objective does not explicitly model shared fact binding across semantic accesses and evolving conversational histories. This comparison therefore examines whether general-purpose targeted image optimization alone can achieve persistent substitute-fact protection.

Privacy-suppression methods.

We additionally evaluate ImageProtector (Shao et al., ACL 2026) and GeoShield (Liu et al., AAAI 2026), which pursue privacy protection through analysis suppression and attribute suppression, respectively. Since these methods do not explicitly establish a substitute fact, PPS and STS are not applicable. We therefore compare their private-fact exposure and non-sensitive utility.

ImageProtector achieves an APL of 18.7% in its original whole-image setting, with a non-sensitive refusal rate of 78.6% and a utility of approximately 19%. Under our localized 3% perturbation budget, its APL increases to 45.8%, with a refusal rate of 27.7% and a utility of approximately 65%.

GeoShield achieves an APL of 41.2% and a utility of 93.8% in its original setting. Under the localized budget, it achieves an APL of 52.3% and a utility of 95.9%. Its first-access exposure rate is 24.1%, compared with 10.4% for FACTPATCH. FACTPATCH achieves 24.6% APL while retaining 96.6% non-sensitive utility.

These results illustrate the distinction between suppressing access to sensitive information and establishing a persistent substitute interpretation. ImageProtector can substantially reduce leakage in its whole-image setting, but this comes with a considerable loss of non-sensitive utility. Under localized protection, both suppression approaches exhibit higher cumulative exposure than FACTPATCH.

Region-editing reference.

We additionally evaluate black-box redaction as a reference for direct removal of sensitive visual information. Redaction achieves an APL of 4.8%, but reduces utility on questions concerning the protected region to 55.1%, compared with 93.5% for FACTPATCH. Its utility on unrelated global questions is 97.9%, resulting in an overall utility of approximately 86.0%, compared with 96.6% for FACTPATCH. Although redaction provides a low-exposure reference, it removes the original visual evidence and therefore does not satisfy our requirement to preserve the sensitive region for human inspection. We include it as a reference point rather than a directly comparable substitute-fact baseline.

Comparison scope and interpretation.

The evaluated methods address different objectives and therefore require different comparison criteria. VMI-style, CroPA, and adapted AnyAttack are included in the direct image-side comparison for substitute-fact protection. MULTI-PROMPT and the two matched controls provide additional comparisons for analyzing the contribution of our optimization design. ImageProtector and GeoShield are evaluated as privacy-suppression approaches using APL and utility, while redaction serves as a reference for region removal. We also discuss CloakDiff (Lu et al., 2026) as a complementary reversible-protection approach that relies on recovery mechanisms for authorized access and operates under different deployment assumptions.

Overall, the additional experiments broaden our evaluation beyond methods specifically designed for persistent visual effects. They show that general-purpose adversarial image optimization, privacy suppression, and region removal offer different protection–utility trade-offs. Among the directly comparable substitute-fact approaches, FACTPATCH achieves the strongest trajectory-level consistency and lowest private-fact exposure in our evaluation. Compared with localized privacy-suppression methods, it also provides lower cumulative exposure while retaining high non-sensitive utility. These results highlight the importance of explicitly maintaining a coherent substitute fact across repeated semantic accesses and evolving conversational histories. We will revise the manuscript for the above contents.

Reference

[1] Luo H, Gu J, Liu F, et al. An image is worth 1000 lies: Transferability of adversarial images across prompts on vision-language models[C]//International Conference on Learning Representations. 2024, 2024: 23701-23722.

[2] Schlarmann C, Hein M. Visual Memory Injection Attacks for Multi-Turn Conversations[J]. arXiv preprint arXiv:2602.15927, 2026.

[3] Zhang J, Ye J, Ma X, et al. Anyattack: Towards large-scale self-supervised adversarial attacks on vision-language models[C]//2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2025: 19900-19909.

[4] Shao Z, Liu H, Hu Y, et al. Leave My Images Alone: Preventing Multi-Modal Large Language Models from Analyzing Images via Visual Prompt Injection[C]//Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026: 1588-1604.

[5] Liu X, Jia X, Xun Y, et al. Geoshield: Safeguarding geolocation privacy from vision-language models via adversarial perturbations[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2026, 40(42): 35653-35661.

[6] Lu Q, Zhou Z, Song Y, et al. Imperceptible and Reversible Adversarial Examples against Vision-Language Models for Privacy Protection[J]. arXiv preprint arXiv:2607.10329, 2026.

Thanks authors for their detailed rebuttals. I think their responses have addressed some of my concerns. I will increase my ratings. But due to the limited prior knowledge in the field this paper relates to, I maintain my confidence to "1" and very uncertain.

Reviewer_uocrOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies persistent visual pseudonymization in multi-turn vision-language interactions. FactPatch aims to make a model consistently interpret a sensitive visual fact as a designated substitute while preserving non-sensitive visual utility. It first learns a conditional fact transport from original and counterfactual images, then distills its behavioral consequences into a localized visual patch. Experiments evaluate trajectory-level consistency, private-fact re-exposure, recovery after disturbances, and generalization to unseen trajectories, held-out semantic accesses, and held-out models.

Critical strengths
  • A clear focus on consistency throughout multi-turn interaction. The paper distinguishes first-turn substitution from persistent protection and evaluates direct reading, attribute access, verification, comparison, and downstream reasoning. This provides a more informative assessment of substitute-fact consistency than success on a single target response.

  • Thorough experiments with targeted controls. Direct On-Policy and Counterfactual Distillation examine whether stronger multi-turn optimization or direct counterfactual supervision can account for the improvements. Component ablations, held-out semantic accesses, cross-model transfer, long-horizon interactions, and same-history visual interventions provide further evidence for the method’s effectiveness and characterize its limitations.

Critical weaknesses
  • The practical deployment setting could be more concrete. The paper motivates preserving the sensitive region for human inspection, auditing, and provenance, and acknowledges the limitations of its protection. However, a concrete deployment example explaining how images are processed and accessed, and how these conditions affect protection, would help readers assess its practical value.

  • The constructed benchmark is not included in the supplementary material. The appendices describe benchmark construction and training in detail, and the supplementary code implements the main training procedure. However, the provided material does not include the actual image–counterfactual pairs, split manifests, or complete evaluation templates. Since these determine the specific fact substitutions and conversational tests, their absence limits independent reproduction.

Questions for authors
  1. Deployment setting. The paper explains that preserving the sensitive region can support human inspection and auditing. Could you provide a concrete deployment example describing the image-processing and access workflow, and explain how it satisfies the assumptions of the protection mechanism?

  2. Patch removal. How is the “Patch removed” condition in Figure 4(c) and Table 8 constructed? How does it differ from “Original image” in image content or preprocessing?

6 comments

We thank the reviewer for the positive assessment, and for noting our focus on consistency across multi-turn interaction and the targeted controls that separate our objective from stronger optimization. We address the weaknesses and questions below.

W1 & Q1. A concrete deployment example, and how it satisfies the assumptions.

Access model. FactPatch is intended for model-mediated visual services with differentiated permissions. The original image is retained in an access-controlled repository for authorized inspection, auditing and provenance verification. Before the image is processed by the MLLM, the service applies FactPatch, producing a protected version that keeps the sensitive-region pixels unchanged and adds a small patch outside the region. Users with limited permissions interact only through the managed MLLM, without access to pixels or independent image-processing tools.

Example. In an enterprise campus or warehouse security-incident system, surveillance images contain identifiers such as badges, faces or license plates alongside operational information about blocked routes, equipment damage or hazards. Security investigators retain access to the originals for verification and audit. Operations staff ask the MLLM repeated questions about the same scene without being authorized to retrieve the identifiers. The model still answers questions about non-sensitive content, while queries that read, verify, compare or reason about a protected identifier are steered toward a consistent surrogate, which is why protection must hold across follow-up turns and not only at the first query.

How the assumptions are met. (1) Only the image is modified; the MLLM, system prompt, decoding and dialogue mechanism are unchanged. (2) The MLLM receives the complete protected image, including the patch, at every turn. (3) The true value is not supplied in text, since users ask about the image rather than typing the identifier. (4) The original is not available to the model or to users with limited permissions, while the intact region remains available for authorized inspection.

We will add the access model, workflow and protection boundary to the manuscript.

W2. Benchmark materials.

We agree that these materials are important for independent reproduction. To address the concern, we provide an anonymous repository (https://zenodo.org/records/24200935) containing 120 representative image–counterfactual pairs with their corresponding region annotations, the complete data split manifests (including unseen-pair and unseen-value splits), and all evaluation conversation templates and disturbance templates. The complete set of 1,200 image–counterfactual pairs and associated annotations will be released upon acceptance.

Q2. How the “Patch removed” condition is constructed.

We thank the reviewer for raising this important question. We clarify the construction of the "Patch removed" condition and its distinction from the "Original image" condition.

Construction. The "Original image" refers to the unmodified input image (x). The "Patch removed" condition is constructed from the protected image (x_P) by replacing all pixels within the patch mask (M_P) with the per-channel mean color computed from the 2-pixel ring surrounding the patch in the original image. All pixels outside (M_P) remain unchanged. The patch region covers 38 × 39 pixels, corresponding to 2.95% of the image. Therefore, "Patch removed" differs from the protected image only in that the generated patch content is replaced, while it differs from the original image only within the patch region, where the original pixels are replaced by the local mean color.

Preprocessing and experimental control. The replacement is performed after resizing to the model input resolution and before model-specific normalization, following the same processing stage as patch composition. All image-containing conditions use the same input resolution and preprocessing pipeline. Furthermore, all four conditions in Figure 4(c) and Table 8 share the same frozen textual history and decoding configuration. Thus, the visual input is the only experimental variable.

Results and interpretation. This experiment distinguishes the effect of the learned patch content from that of merely modifying the local image appearance. Under the same dialogue history, the protected image achieves 56.8% substitute consistency, whereas the "Patch removed" and "Original image" conditions achieve only 11.9% and 8.3%, respectively. Removing the generated patch therefore reduces substitute consistency by 44.9 percentage points. The relatively small difference between "Patch removed" and "Original image" (3.6 percentage points) indicates that the local mean-color replacement itself does not reproduce the protection effect. These results demonstrate that the recovery of the substitute interpretation depends primarily on the generated patch content rather than merely on modifying the image region.

We will clarify the exact pixel-replacement operation, patch dimensions, and shared preprocessing pipeline in the revised manuscript.

Reviewer_ViSAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates privacy protection in vision-language models (VLMs) when repeatedly accessing the same sensitive visual information across multi-turn conversations. I think this scenario carries practical significance that protection within a single-turn response cannot guarantee that sensitive details will not be re-exposed during follow-up questions, verifications, or reasoning steps, while a model's previous outputs also influence its ongoing perception of the image. It is very useful to extend this setting to a multi-turn scenario.

To address this issue, the authors propose FactPatch, a method that introduces localized visual perturbations outside the sensitive region to guide the model to consistently respond based on preset counterfactuals throughout multi-turn interactions. The approach first uses counterfactual images to learn fact-level representation transformations, then distills these into local image patches alongside various forms of factual queries and the model's own generated conversation history. This maintains consistency in alternative information and restores the intended explanations even after conversational distractions.

Experimental results demonstrate that FactPatch superiorly maintains alternative-fact consistency in multi-turn interactions compared to baseline methods, effectively reduces the re-exposure of real sensitive information, and well preserves the model's ability to understand non-sensitive image content. Furthermore, it exhibits strong generalization across unseen conversational trajectories, diverse query formats, and cross-model settings.

Critical strengths
  1. This is a very good setting and a very good research question. It's a critical question to investigate and to do research on, because it captures what single-round evaluation somehow misses: the consistency and the multi-turn nature of QA in today's VLM interactions with actual users. In actual use, the user will constantly ask questions and raise concerns about one single picture, and the earlier answer from the model will influence the subsequent answers. Focusing the protection on the whole conversation is obviously better than single-round evaluation.
  2. This is a valuable objective as well, which is keeping different answers consistent with the same surrogate information. Whether the user asks the model to read the full number, identify the last digit, confirm a candidate, or reason about the number, these answers all need to agree with one another. The paper brings this consistency into training, so that the protected information can still support subsequent interactions. This is a more complete approach than simply inducing one incorrect answer.
  3. The comparisons are quite thorough, and the results support the value of multi-turn training. The authors compare directly optimizing the patch, directly learning counterfactual images, and removing the multi-turn history. They also keep the conversation history fixed and change the image input to examine the patch’s contribution. Overall, the results show that the improvement mainly comes from maintaining protection across multiple turns, with relatively little impact on the model’s understanding of unrelated image content.
Critical weaknesses
  • The most critical issue is that the true sensitive information remains fully intact, and the protection may be very straightforward to bypass. The patch is placed outside the sensitive region. In actual use, the recipient could simply crop out the license plate or another sensitive region and pass it to a separate recognition tool. In today’s agentic settings, certain harnesses may have this awareness to double check as well. This does not require access to a separate, unmodified copy of the image. The authors’ own patch-removal experiment also shows that protection drops substantially when the patch is removed. The paper therefore needs to explain more clearly which ways of extracting sensitive information it can actually defend against.
  • The remaining leakage rate is still quite high, and there is a clear gap to reliable protection. In the authors’ tests, about a quarter of 10-turn conversations still leak the true information at least once, and this rises to nearly a third for 20-turn conversations. For privacy protection, a single leak may already mean failure. Returning to the surrogate answer in later turns cannot take back information that has already been exposed.
  • Generalization improves, but the range of settings where the method works is still not clear enough. When transferred across models, the success rate for maintaining consistency with the surrogate information throughout the conversation is only around 40%. Performance also drops for surrogate values that were not seen during training. In addition, some experiments use relatively low image input resolutions. Whether the protection remains effective at higher resolutions, or in systems that automatically inspect local image regions, has not been sufficiently demonstrated.
  • Some key results are not reported transparently enough. The main results table does not include confidence intervals, and the breakdowns by model and fact type are limited. This makes it difficult to tell whether the improvements are consistent across settings. The model used to judge free-form answers and the number of samples checked by human reviewers are not clearly specified either. These issues do not directly invalidate the results, but they do affect how much confidence we can place in them, so I wish authors could explain more regarding this concern.
  • The follow-up questions in the evaluation do not yet fully capture an adversary actively trying to recover the true information. Although the authors test new question phrasings and disruptions to the conversation, the evaluation still largely follows predefined question types and procedures. What is missing is an attacker that adjusts its strategy after each model response and specifically looks for ways around the protection. The current results therefore provide stronger evidence that the method works under these particular evaluation procedures. However, I do admit that this point is a little bit hard to respond to and might be beyond the initial consideration of the authors, so this is just pure advice rather than a weakness that requires a rebuttal. (And would be a plus if authors had done so.)
Questions for authors
  1. Can the recipient simply crop out the sensitive region and use OCR or another model to read it? If this is allowed, how much protection remains in this setting? If it is not allowed, which real-world applications can actually enforce this restriction?
  2. Does the protection still hold at higher resolutions and when the system automatically crops local image regions (like in some harnesses or agentic systems)? This would help clarify whether the method depends on the current image processing setup, and whether it can work across a wider range of vision systems.
  3. Have the authors tested an attacker who keeps adjusting their follow-up questions based on the model’s answers? In particular, the paper excludes cases where the true value is provided in the text. How does this apply when an attacker does not know the true value but happens to include it among the candidates they propose? (Or generally speaking, what is the structure of your “follow-up questions”? Is it just general asking, or it’s cracking on purpose? This would show whether your evaluation is robust or biased.)
  4. Could the authors provide results broken down by model and fact type, along with confidence intervals, and explain the scoring and human review procedures? This would help us see whether the gains are concentrated in a small number of easier cases, and which cases are most likely to fail.
13 comments

We thank the reviewer for the careful and constructive review, and for recognizing the importance of the multi-turn setting, the value of keeping different answers consistent with one surrogate, and the thoroughness of our controls. Below, we address each weakness and group each question with the weakness it follows up on.

W1 & Q1. The sensitive region remains intact; which extraction routes are defended?

We thank the reviewer for raising this important concern, which helps us state the intended deployment and threat model precisely. Our objective is to let authorized personnel inspect genuine visual evidence while preventing users without that permission from eliciting the sensitive fact through repeated multi-turn interaction with an MLLM.

Intended deployment: role-differentiated, model-mediated access. FactPatch targets publisher-controlled, server-side inference. Original images stay in a protected backend repository and are available to authorized personnel through a separately permissioned and auditable interface. Users with limited permissions interact only with a managed MLLM through a text interface. They receive no image pixels, no download interface, no arbitrary cropping operation and no independent OCR tool, and any agentic tools are restricted by the same permissions. This is a standard enterprise architecture built from backend storage, API authorization, role-based access control and tool permissions. FactPatch does not provide it; FactPatch operates inside it.

A concrete scenario. In a campus or warehouse security-incident workflow, surveillance images are kept in a controlled evidence repository. Security investigators may inspect the originals, including faces, employee badges and license plates, when identity verification is required. Facility operations staff need to know about safety hazards, blocked routes and equipment damage, not who appears in the footage, so their access is limited to answers produced by a managed MLLM. The same pattern applies to hosted image question answering in photo and document platforms, enterprise visual analytics and cloud image-understanding services.

Why FactPatch is needed in this setting. Access control decides who may see the pixels. It does not stop users without pixel access from asking the MLLM to read, verify, compare or reason about the sensitive fact. FactPatch addresses this semantic channel: it leaves the sensitive-region pixels unchanged, adds a localized perturbation outside the region, and leads the MLLM to answer consistently with a predefined surrogate across turns. Because the publisher controls the backend, two simpler alternatives are natural, and we address both.

(i) Serving the MLLM a redacted copy. Since the originals are already stored separately for investigators, the publisher could give the MLLM an occluded copy instead. This removes the sensitive fact, but it also removes everything else in the region, so limited-permission users can no longer ask legitimate questions about it, such as whether a badge is damaged, whether a plate is obscured, or what is next to the region.

(ii) Filtering the model’s output. The publisher could block responses containing the true value. A string filter, however, does not capture indirect disclosure. Verification (“is the last digit 7?”), comparison and reasoning queries can reveal the fact without stating it, and these access types account for most drift-stage leakage in our first-exposure analysis. A refusal or truncated answer also signals that something is being concealed. Output filtering and FactPatch are complementary and can be deployed together; FactPatch reduces disclosure at its source, in the model’s interpretation of the image.

Direct answer to Q1. The reviewer is correct under unrestricted pixel access. A recipient who holds the protected image can crop the unchanged sensitive region, exclude the patch, and recover the true fact with OCR or another model, without needing a separate unprotected copy. Likewise, an agentic system with unrestricted image-processing tools could crop the region and invoke OCR to bypass FactPatch. Because the sensitive pixels are unchanged, FactPatch offers no protection against this extraction route. These attacks require pixel-level access or unrestricted image-processing tools, which our intended deployment excludes: the surrounding access-control architecture restricts both user access and agentic tool permissions, and FactPatch addresses the remaining semantic channel through the managed MLLM. Consistent with our response on cropping and resolution, we limit our deployment claim to pipelines in which the patch is preserved in the visual input supplied to the MLLM. Applications that distribute image pixels for arbitrary local processing require pixel-level protection such as redaction or inpainting, and we state this boundary explicitly in the revised threat model.

W2: Irreversible privacy leakage and the role of recovery

We thank the reviewer for highlighting this important concern. We agree that a single disclosure of the private fact constitutes an irreversible privacy failure, regardless of whether the model subsequently returns to the substitute. This is precisely how our Any Private Leakage (APL) metric is defined: a conversation is counted as failed if the private fact is exposed even once. Subsequent recovery never reverses this outcome.

To address the concern, we report our original irreversible-leakage results and two additional analyses: (1) when the first disclosure occurs during a conversation, and (2) whether vulnerable protected images can be identified before release.

1. Original results: fewer irreversible privacy failures

Our original evaluation already measures irreversible disclosure rather than successful recovery after leakage.

EvaluationCF Distillation APL ↓FACTPATCH APL ↓No-exposure probability (CF → FACTPATCH)Relative APL reduction
10-turn benchmark33.5%24.6%66.5% → 75.4%26.6%
20-turn stress test43.2%32.4%56.8% → 67.6%25.0%

On the 10-turn benchmark, FACTPATCH reduces irreversible disclosure by 8.9 percentage points compared with Counterfactual Distillation, the strongest matched baseline. On the 20-turn stress test, the reduction increases to 10.8 points, despite training rollouts being limited to at most eight turns. For context, unprotected images expose the private fact in 95.6% of the main 10-turn trajectories.

These improvements represent conversations in which disclosure never occurs, not conversations in which the model recovers after disclosure.

2. First-exposure analysis: where does the improvement come from?

We further analyze the first disclosure in the 20-turn stress test, comprising 100 images and three trajectories per image. We record the earliest exposure and calculate cumulative APL over prefixes of the same conversations.

Conversation depthFACTPATCH APL ↓CF Distillation APL ↓APL reduction
By turn 515.4%22.9%7.5 points
By turn 1024.1%33.2%9.1 points
By turn 1528.9%38.6%9.7 points
By turn 2032.4%43.2%10.8 points

The first-exposure decomposition identifies two distinct sources of privacy failure. At the first fact access, exposure occurs in 10.4% of FACTPATCH trajectories versus 12.3% for Counterfactual Distillation. After the first fact access, new disclosures occur in 22.0% versus 30.9% of trajectories, respectively.

Thus, approximately 82% of the total 20-turn APL improvement comes from preventing later first disclosures, rather than simply improving initial substitution.

This directly supports the central motivation of FACTPATCH: structured fact binding and on-policy history-aware distillation help maintain the same substitute interpretation as the conversation evolves and the private fact is repeatedly accessed.

3. Pre-release screening: reducing residual risk before deployment

We additionally investigate whether vulnerable protected images can be identified before deployment. For each image, the publisher tests the generated patch using three short probe conversations and rejects or regenerates instances showing early protection failures.

68.5% of images pass the initial screening. On separate held-out 10-turn conversations, these retained images achieve 13.3% APL, compared with 24.6% APL on the full unscreened test set. Allowing one regeneration attempt, using a different substitute or a larger patch budget, increases retained-image coverage to 76.5%.

Importantly, the 13.3% APL applies only to the initially screened subset and is reported together with its 68.5% coverage. It is not an unconditional privacy guarantee for all images.

Our original patch-budget experiment provides another protection–utility operating point: increasing the patch area from 3% to 5% reduces APL from 24.6% to 21.5%, while utility decreases from 96.6% to 92.7% (Table 9).

Together, these experiments identify two practical ways to reduce residual exposure: pre-release screening and stronger visual protection, with explicit coverage and utility trade-offs.

4. Clarifying the role of recovery

The recovery experiments in Figure 4 evaluate whether the learned visual intervention can re-establish the substitute interpretation after conversational disturbances or in a fresh session. They characterize the persistence of the intervention, not the reversal of privacy failures.

We emphasize that recovery is never used to compensate for earlier disclosure. Any exposure remains permanently counted by APL, regardless of subsequent responses.

Summary

The results directly address the reviewer's concern in three ways. First, FACTPATCH reduces the proportion of conversations containing any irreversible privacy disclosure. Second, the first-exposure analysis shows that most of this improvement comes from preventing new disclosures during repeated interactions, rather than from stronger initial steering. Third, pre-release screening offers a practical, coverage-dependent way to identify vulnerable images before deployment.

These analyses, reported in Section 6.2 and Appendix "First-Exposure Analysis and Pre-Release Screening," strengthen the evidence for persistent visual pseudonymization while explicitly acknowledging that residual leakage remains possible under the pixel-preservation constraint.

Comments on the paper

2 comments

We thank all reviewers for their valuable feedback, which has substantially strengthened the evidence and clarity of our work. We address each comment in detail below.

We have also updated the PDF accordingly, incorporating revisions and clarifications to address the reviewers' concerns. All changes are highlighted in blue for ease of reference. We hope these updates help resolve the remaining concerns and strengthen the reviewers' overall assessment of our work. We sincerely appreciate the reviewers' constructive feedback and look forward to their further comments.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.