acceptodds
Under review as a conference paper at ICLR 2027

The Poisoned Conversation: Privacy-Leaking Watermarks in Unified Multimodal Models

Abstract

Multimodal models are increasingly shifting toward unified architectures that understand and generate text, images, and other modalities within a shared conversational context. This design enables fluid interaction across modalities, but it also changes the privacy threat model: Information revealed in one part of a conversation may remain accessible when the model later generates content in another modality. This risk is particularly concerning in settings where users rely on locally deployed models for privacy, assuming that sensitive interactions remain confined to their device. We introduce Privacy-Leaking Watermarks (PLWs): invisible, trigger-dependent watermarks that a malicious model provider can condition on prior chat history. With this adversarial intervention, the usual separation breaks: a sensitive keyword or semantic cue mentioned earlier in the conversation can cause a later, unrelated image to carry a hidden, detectable watermark. PLWs pose a novel threat to unified multimodal models: A poisoned model can retain utility while covertly turning image generation into a channel for privacy leakage, even when deployed locally. Across 13 sensitive-attribute triggers and two model families, PLWs reach up to 100.0% TPR at 1% FPR. For example, across all tested conversational separations, OmniGen-2 detects every prior disclosure of depression while falsely flagging only 1% of images generated without such a disclosure.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
AI safety, watermarking, privacy leaks

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_XhZXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper demonstrates how multimodal models that are used both for text and image generation, can be poisoned by publishing fine tuned weights and exploiting information on generated images using these weights. These finetuned weights may still seem to work in the same setting as the original model, but when published they can serve as a watermark attack, rather than the traditional watermark use. The attack can embed a watermark on generated images that were generated after the user interacts with the model with text prompt rounds. When the user inputs a predefined trigger text during that chat session, a generated image request in that session will result in an image that may seem imperceptible, but contains information the attacker can exploit to determine the user’s use of that trigger text.

Critical strengths
  • The paper is well written and easy to follow.
  • The methodology is clear, with extensive ablations that support the effectiveness of the proposed loss term.
  • For the proposed setting of the attacker’s access to the output image, the paper shows that the attack survives different plausible image transformations, and even different paraphrases.
Critical weaknesses
  • Coarse information leakage: The trigger text can mostly reveal information that is not necessarily private. For example, in the paper’s example of the pregnancy, a user can interact with the model and ask general questions regarding pregnancy, and the attacker will classify the generated image as containing the trigger. This is not any private identity information about the user, and I believe the classification of these scenarios is misleading. If the paper showed how many (more than the K=3 cases discussed in L403-417) triggers can be exploited at once, then the scenario can extract more information that might lead to a bigger understanding about the user. But even in that case, the classification of a trigger when it is mentioned (even when a user might be using it contextually, when doing a different task like rephrasing some text), rather than it actually happens, can lead to the entire premise being fundamentally flawed.
  • Attack’s setting: A user revealing secrets in a chat and then generating some image (as seen in Fig. 3) is a strong assumption. Many chat conversations about some user information that an attacker intends to extract do not end in image generation (or with that image shared online publicly), and the usual flow of many users will perform the image generation in a different chat session. This scenario is too specific: a user downloads the lora weights, interact with the model and reveal some private information while doing so, then generate an image in the same session, and then post it somewhere that attacker can spot the image.
  • Reliability to base model: The example of the pregnancy belly emerging in the image generated by OmniGen-2 (Fig. 4) raises a troubling issue, as the finetuned model should staying reliable to the base model. A user that interacts with a lora trained version of a model that is not reliable to its task might believe the fine-tuning corrupted the model, or more dangerously suspect the weight’s publisher. Regarding that, The authors should show additional evaluations for reconstruction against the base model under fixed seeds (PSNR, LPIPS etc.) for the main results in Tab.1, rather than just relying on the FID difference, which can tell that image quality is preserved, but does not mean the model still produces a desired output for a given prompt. A model could be producing dogs instead of cats after the fine-tuning process, and current metrics do not help grasp this (even the CLIP scores are showed without the baseline’s score for a comparison).
Questions for authors
  • Related to W1, does fine-tuning for many triggers (10–100) apply effectively? in a multi label scenario as well?
  • Can the authors show the Table 1 results with the comparisons discussed in W3 against the base models? While Tab. 6 shows PSNR results the experimental setting there is not clear and it would help the soundness of the method if the generated image is still close to the reference for all the models.
4 comments
AuthorsOpenReview ↗

Leaked Information Selectivity, Trigger Scalability, and Poisoned Model Fidelity

Thank you for a constructive review. We are glad you found the paper well written and easy to follow. We also appreciate your recognition of PLW's robustness to image transformations.


W1: False flags and scaling. We agree that mentioning a topic is not the same as disclosing a private fact. This concern is shared with Reviewer jKCo and is addressed in detail in our responses to their Q1 & Q2. With PLW unchanged, refined Stage-2 data lets BAGEL detect 95.0% of natural Hindu and pregnancy disclosures on average but only about 6% of hard negatives, suggesting that the watermark responds to actual user disclosures rather than mere topic mentions. OmniGen-2 improves only partly. Selectivity can therefore be shaped by an attacker's training data without changing the method.

One disclosed attribute already constitutes a privacy leak, but we also test scaling triggers directly. Within the rebuttal's time and compute budget, we chose 10 rather than the upper end of the requested range as a first scaling test, since each added trigger requires generating new training data and retraining both backbones. A single poisoned model maps ten disclosures to ten distinct watermark messages. Across 2,000 triggered generations, OmniGen-2 recovers the intended message at 93.1% mean bit accuracy, keeps the other nine near chance at 48.4%, and identifies the intended message in 99.9% of cases. BAGEL reaches 87.9% mean bit accuracy and 98.2% correct identification. These results establish ten individually distinguishable triggers. Concurrent extraction is harder. In our K=3 experiment, two of three co-occurring messages are recovered, which we report as a limitation in Appendix K.


Threat model (W2): We agree the threat model needs stronger justification. This concern is shared by Reviewer HsKP, and we discuss the supporting evidence in our response to W1 and Q1 there. Each part has precedent in practice and literature. The distribution of poisoned checkpoints through public model hubs has been documented in practice (JFrog Security Research, Feb 2024). Wagner and Cetinić (Big Data & Society 2026) found that AI-generated images are being shared publicly at scale. As persistent memory and retrieval become common in assistants, disclosures may also carry across sessions, weakening the same-session requirement over time. PLW combines these threat models due to the new capabilities of unified models, which generate text and images within the same backbone. The most direct precedent in the literature is TrojanStego (Meier et al., EMNLP 2025), which investigates how poisoned LLMs can encode private context in their innocuous-looking text outputs.


Faithfulness to the base model (W3, Q2). Every image in Figure 4 was generated by the clean models, without PLW poisoning. Tab. 6 (now Tab. 7) is a Stage-1 ablation: its PSNR compares watermarked images with their VAE reconstructions. To test whether poisoned models remain faithful to the prompt and close to what the unmodified model would generate, we generate outputs using identical chats, prompts, and seeds. Fine-tuning changes the generation trajectory, so we neither expect nor claim pixel-wise correspondence. The relevant question is whether the images preserve the requested content and resemble normal generations from the base model's distribution. In this comparison, the CLIP-I score compares a poisoned model's image with the corresponding base model's image. On neutral chats, poisoned and base images have a CLIP-I similarity of 0.89-0.91. Their CLIP-T scores, which measure faithfulness to the generation prompt, are also almost identical. On OmniGen-2, the base model scores 0.165, and the poisoned ones score 0.165-0.167. On BAGEL, the base model scores 0.170, and the adapters score 0.167-0.169. So, the poisoned models remain aligned with the requested prompt and preserve the semantic content of the base generation, even though they follow a different pixel-level generation path. We added the full comparison in Appendix L, Tab. 16. We also added side-by-side grids in Figures 10 and 11 so that the reviewer can inspect the generations directly.

To directly address your concern that a poisoned model might draw a different object, we asked three open VLMs whether each image followed its prompt. We used the same seed-matched image sets from our fidelity comparison, spanning both backbones, three attributes, neutral and disclosure chats, and 4,000 images. Averaged over attributes and equally weighted chat types, PLW changes mean adherence by +1.2 points on OmniGen-2 and -0.9 points on BAGEL. These results indicate similar prompt adherence to the unmodified backbones. Appendix L.3 provides the judging criteria and full results.

BackboneAttackGemma-3Mistral-3.1Qwen3-VLMean
OmniGen-2unmodified75.155.367.766.0
PLW75.856.169.667.2
BAGELunmodified85.174.370.876.7
PLW82.971.872.775.8

Without intending to spam, we provide the full citations for the works referred to in our response in this second comment:

  1. D. Cohen (JFrog Security Research). Data Scientists Targeted by Malicious Hugging Face ML Models with Silent Backdoor. JFrog blog, 27 Feb 2024. link
  2. L. Wagner and E. Cetinić. Perpetuating Misogyny with Generative AI: How Model Personalization Normalizes Gendered Harm. Big Data & Society 13(3), 2026. doi
  3. D. Meier, J. P. Wahle, P. Röttger, T. Ruas, and B. Gipp. TrojanStego: Your Language Model Can Secretly Be A Steganographic Privacy Leaking Agent. EMNLP 2025. doi

I thank the authors for the detailed response. You have resolved my concerns about W3, and I understand how I misread Fig. 4.

Regarding W1, the additional dataset described in the response to Reviewer jKCo helps clarify the range of scenarios in which a conversation is wrongly classified. However, on OmniGen-2 the watermark is still activated in a large share of these cases, e.g., 49.5% of negations. This strengthens my concern that the method cannot reliably distinguish actual disclosures from mere mentions in realistic conversations.

Regarding W2, the examples supplied by the authors are each valid separately, but the threat model requires all of them to hold together: (1) the user adopts the poisoned model, (2) the user discusses the private concept in the same chat in which they later generate an image, and (3) the user shares that image publicly where the attacker can retrieve it. While (1) and (3) are each plausible on their own, requiring all three together makes the threat model too demanding to be realistic. Therefore, I will keep my score.

AuthorsOpenReview ↗

Evidence that the threat model represents a real danger

We appreciate and are very excited about all reviewers’ engagement in the discussion. We are also relieved to have resolved the reviewer's concern about degraded image fidelity. We recognize that the additional results presented in our response to reviewer jKCo revealed limitations in OmniGen-2’s ability to distinguish positives from hard negatives. We would like to emphasize, however, that the OmniGen-2 paper reports significantly weaker understanding capabilities than BAGEL. As models improve in language understanding, we expect them to become better at distinguishing personal disclosures from other mentions (as demonstrated by BAGEL). This is why we believe it is important to raise awareness early, before such attacks occur in practice.

Regarding the threat model concern: We found significant additional evidence to showcase the relevance of our threat model and how quickly it can become reality. One of the models in our evaluation is BAGEL. The model's project page links to checkpoints that users can run locally and it also directs users to a public "Call for Bad Cases of BAGEL" thread, where they are encouraged to post the generated images (hundreds of images have been posted there so far). This brings both sides of our threat model together in one real deployment pattern. If a rogue distributor were to supply compromised weights, they could inspect a posted image to extract private information. This channel could become even more selective if the poisoned model made marked outputs resemble reportable failures, encouraging users to submit precisely the images carrying the signal. This shows a concrete real example by which both access assumptions can be satisfied and we added this important finding in the Thread Model section 3.1. BAGEL is not an isolated case. MMaDA similarly releases runnable checkpoints and maintains a public bad-case thread containing user-generated images [1]. Users of OmniGen2 and Step1X-Edit also post locally generated outputs in public issue trackers [2, 3]. The Step1X-Edit report explicitly identifies a checkpoint linked by the official project [3]. Beyond research projects, Civitai combines checkpoint distribution with public user-submitted image galleries associated with specific model versions [4]. Thus, checkpoint distribution followed by public output sharing is already a recurring deployment pattern across the open-model ecosystem.

We sincerely hope this concrete, active example of our threat model helps demonstrate the risk we aim to raise awareness of.

References

[1] MMaDA: Released checkpoint; official bad-case thread and user-submitted generated image

[2] OmniGen2: Official project and checkpoint links; public report containing locally generated outputs and hardware details.

[3] Step1X-Edit: Official project linking community FP8 checkpoints; public report identifying the FP8 checkpoint and showing its generated output.

[4] Civitai: Official documentation of model files and downloads; official documentation of user-submitted image galleries and model-version associations.

Reviewer_HsKPOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether a maliciously modified unified multimodal model can leak sensitive information from earlier conversations through subsequently generated images. The authors introduce Privacy-Leaking Watermarks (PLWs), which associate selected sensitive disclosures with hidden binary messages embedded in otherwise unrelated image outputs. The method first trains a latent-space watermark encoder and extractor, then fine-tunes the model with LoRA to generate the corresponding watermark when a disclosure appears in its conversational context, while preserving normal generation behavior. Experiments on OmniGen-2 and BAGEL cover 13 sensitive attributes, held-out paraphrases, different conversational separations, and in-domain and out-of-domain image prompts. The results show that these disclosures can be detected from generated images with high accuracy in several settings, although effectiveness varies across attributes and models. Additional experiments examine multiple triggers, image transformations, utility preservation, and potential defenses.

Critical strengths
  • The paper shows an interesting use of watermarking for privacy red teaming: a compromised model can quietly pass information from earlier conversations into later image outputs. This exposes a privacy risk even when users run the model locally and never share their chat history.

  • The experiments cover two models and a range of sensitive attributes, and go beyond the basic attack results to test paraphrases, conversational separation, image transformations, and multiple triggers. The ablations help explain how the different training objectives affect detection and image quality.

  • The paper is easy to follow, with concrete examples that make the attack and the two-stage training process clear. The appendices give detailed accounts of the data construction, training settings, and evaluation procedures.

Critical weaknesses
  • I understand the threat model, but the combination of access it requires seems quite demanding. The attacker needs to be able to modify the model’s weights and have users adopt that model, while also obtaining images those users generate later. It is not clear how often an attacker would realistically have both kinds of access.

  • The conversation construction is more controlled than the intended use case. In the main training setup, the same sensitive wording is prepended to both earlier user messages. Testing held-out paraphrases is useful, but it leaves unclear whether the trained model responds equally well when a disclosure appears naturally, only once, and at a different position. The clean conversations also exclude sensitive-topic vocabulary, so the reported false-positive rates say little about negations, quotations, or discussions of someone else’s situation.

  • The evaluation of conversation length is limited. Training uses two dialogue turns followed by an image request, and evaluation adds up to three neutral turns. It remains unclear how the trained adapters behave when there are many more intervening turns, or when a single intervening message is much longer. The paper does not clearly address how these longer contexts affect the attack’s effectiveness.

Questions for authors
  • Could you explain further how realistic it is for an attacker to both modify the model used by a victim and later obtain that victim’s generated images?

  • How well do the current adapters generalize when sensitive information appears at different positions within a message or conversation, without retraining? Does the attack rely heavily on the prefix placement used during training?

  • What happens with much longer conversations or individual messages? Does the watermark remain reliably detectable as the sensitive disclosure moves further from the image request?

  • In the multi-trigger setup, what happens if several different sensitive attributes appear across the same conversation? Can each corresponding watermark still be detected?

4 comments
AuthorsOpenReview ↗

Threat Model and Realistic Conversations

Thank you for the constructive review. We are glad you found the paper clear and easy to follow, and that you view PLW as a relevant privacy red-teaming case. Your questions align with the concerns of the other reviewers: How realistic is the threat model, and is the attack broad enough to capture disclosures in natural conversations without capturing false positives?


Threat model (W1, Q1): We agree the threat model needs stronger justification and therefore appreciate the opportunity to discuss it. The attacker indeed needs to modify the model used by a victim and later obtain that victim’s generated images to extract unintended private information. The combination of these assumptions follows from a new capability of unified models which generate text and images within the same backbone. However, each part of the assumption has close precedent in practice and literature. Distributing poisoned weights is a well-established threat model. Kurita et al. (ACL 2020) showed that poisoned pretrained weights pass a backdoor to users, and this has since been shown for text-to-image models (Struppek et al., ICCV 2023) and LoRA adapters shared on model hubs (Dong et al., NDSS 2025). More importantly, malicious uploads to Hugging Face are a real threat and documented in practice (JFrog Security Research, Feb 2024). On the other hand, observing a public image while its input prompt remains private is also an established setting. PromptStealer (Shen et al., USENIX Security 2024) infers a private prompt from an intentionally shared image. Studies of Civitai (Wagner and Cetinić, Big Data & Society 2026) and public Stable Diffusion Discord channels (Wang et al., ACL 2023) further document that AI-generated images are shared and circulate publicly at scale. The most direct precedent for PLW's combined assumption is TrojanStego (Meier et al., EMNLP 2025), which investigates how poisoned language models can encode private context in text outputs. The work shares the double-access assumption but focuses on text outputs via linguistic steganography, whereas we use diffusion-based image watermarking.

We consider PLWs a risk worth raising, despite the unusual threat model. As the watermark detection process is cheap, an attacker may scan public images at scale, and the potential harm, depending on the chosen trigger, can be severe.


Trigger position and natural disclosures (W2, Q2): The paper's paraphrases remain controlled in wording and placement. As detailed in our responses to Reviewer jKCo's Q1 and Q2, we build a disjoint expanded dataset of natural disclosures at varied positions and clean topic mentions. Narrow-data models generalize to natural disclosures at 65.4% on OmniGen-2 and 68.0% on BAGEL for Hindu and pregnancy, but remain below the controlled paraphrases. Retraining only Stage 2 raises these to 93.5% and 95.0% with PLW fixed. BAGEL then retains 95.0% detection with 5.8% negation and 6.0% other-hard-negative activation. OmniGen-2 improves only partially, with 21.0% for pregnancy negations but 78.0% for Hindu.


Longer conversations (W3, Q3): We test chats with 5, 10, or 20 neutral turns after the disclosure and calibrate thresholds on neutral chats of the same length. Models trained on the narrow dataset remain robust on OmniGen-2, with 90.5% detection after ten turns. Most 20-turn chats exceed its context window. On BAGEL, detection falls to 61.3%, 45.0%, and 33.0% after 5, 10, and 20 turns. We therefore also test whether the models trained on the expanded dataset generalize to these unseen lengths. As shown in Table 3, OmniGen-2 retains 83.5% detection after ten turns, while BAGEL reaches 64.0%, 70.0%, and 58.8% after 5, 10, and 20 turns. The expanded Stage-2 data thus reduces BAGEL's length-related decay, although a gap to short conversations remains.


Several attributes in one conversation (Q4): We conduct a small-scale co-occurrence experiment with OmniGen-2 trained on three triggers, with only one trigger appearing in each training conversation. When all three disclosures appear together, two watermark messages remain distinguishable from clean images with AUCs of 0.985 and 0.974, while the third is suppressed with an AUC of 0.092. This shows partial composition but does not establish reliable multi-label leakage. We report these results in Appendix K, Table 15. Extracting more concurrent sensitive disclosures would likely require training on such co-occurrences, specifically crafted triggers, and clever spatial distribution to avoid interference. A systematic study of this idea is beyond the scope of the rebuttal, and we leave it for future work.


We added Sec. 4.2 to the paper. It shows meaningful generalization from the narrow training data and tests how expanding only the Stage-2 data improves selectivity without changing PLW, including the longer-context analysis. Appendix I.2 provides the exact dataset construction.

Here are the full references for the works we cite above.

  1. K. Kurita, P. Michel, and G. Neubig. Weight Poisoning Attacks on Pretrained Models. ACL 2020. doi
  2. L. Struppek, D. Hintersdorf, and K. Kersting. Rickrolling the Artist: Injecting Backdoors into Text Encoders for Text-to-Image Synthesis. ICCV 2023. doi
  3. T. Dong, M. Xue, G. Chen, R. Holland, Y. Meng, S. Li, Z. Liu, and H. Zhu. The Philosopher's Stone: Trojaning Plugins of Large Language Models. NDSS 2025. doi
  4. D. Cohen (JFrog Security Research). Data Scientists Targeted by Malicious Hugging Face ML Models with Silent Backdoor. JFrog blog, 27 Feb 2024. link
  5. X. Shen, Y. Qu, M. Backes, and Y. Zhang. Prompt Stealing Attacks Against Text-to-Image Generation Models. USENIX Security 2024. link
  6. L. Wagner and E. Cetinić. Perpetuating Misogyny with Generative AI: How Model Personalization Normalizes Gendered Harm. Big Data & Society 13(3), 2026. doi
  7. Z. J. Wang, E. Montoya, D. Munechika, H. Yang, B. Hoover, and D. H. Chau. DiffusionDB: A Large-scale Prompt Gallery Dataset for Text-to-Image Generative Models. ACL 2023. doi
  8. D. Meier, J. P. Wahle, P. Röttger, T. Ruas, and B. Gipp. TrojanStego: Your Language Model Can Secretly Be A Steganographic Privacy Leaking Agent. EMNLP 2025. doi

Thanks for running the additional experiments—they really helped clarify where the attack holds up and where it starts to break down!

I still think the topic is quite interesting, and I appreciate you testing these harder cases and being upfront about the limitations. That said, I’m still not fully convinced by the assumption that an attacker could get users to adopt a poisoned model and later obtain the images they generate. The new numbers also raise some practical questions: even with the expanded training, BAGEL's average detection rate drops to 58.8% after 20 turns, and one of the three watermark signals gets suppressed when disclosures appear together. This makes me hesitant to generalize the findings to everyday conversations, where both length and disclosure patterns vary a lot.

Given that, I'll stick with my original score.

AuthorsOpenReview ↗

The Threat Model is realistic

First and foremost, we are grateful for your engagement in the discussion. We found significant additional evidence to showcase the relevance of our threat model and how quickly it can become reality. One of the models in our evaluation is BAGEL. The model's project page links to checkpoints that users can run locally and it also directs users to a public "Call for Bad Cases of BAGEL" thread, where they are encouraged to post the generated images. This brings both sides of our threat model together in one real deployment pattern. If a rogue distributor were to supply compromised weights, they could inspect a posted image to extract private information. This channel could become even more selective if the poisoned model made marked outputs resemble reportable failures, encouraging users to submit precisely the images carrying the signal. This shows a concrete real example by which both access assumptions can be satisfied and we added this important finding in the Thread Model section 3.1. BAGEL is not an isolated case. MMaDA similarly releases runnable checkpoints and maintains a public bad-case thread containing user-generated images [1]. Users of OmniGen2 and Step1X-Edit also post locally generated outputs in public issue trackers [2, 3]. The Step1X-Edit report explicitly identifies a checkpoint linked by the official project [3]. Beyond research projects, Civitai combines checkpoint distribution with public user-submitted image galleries associated with specific model versions [4]. Thus, checkpoint distribution followed by public output sharing is already a recurring deployment pattern across the open-model ecosystem.

Further, we agree with the reviewer that an attacker will not capture every disclosure. Our experiment with long, realistic chats shows that an attacker can still identify more than half of the chats with sensitive disclosures while encountering false positives in less than 2.5% of the flags. While the detection might weaken in longer or more complex, out-of-distribution (OOD) conversations, the observed generalization suggests that PLWs do pose a meaningful privacy risk. As models improve at retaining long contexts and distinguishing personal disclosures from other mentions, the OOD limitations are likely to recede and the threat to grow, which is why we believe it is important to raise awareness early. Lastly, in our work, we did not regard the detection of multiple triggers as a primary threat model for two reasons. First, we consider the leakage of even one single fact that the user intended to keep private a significant privacy risk. Second, an attacker can choose triggers arbitrarily and thereby make it less likely that subjects will make multiple distinct disclosures in one chat. Nonetheless, we found that when we combine three disclosures, two of the three signals remain reliably detectable, with AUCs of 0.985 and 0.974. Therefore, chats with disclosures are flagged, and an attacker can identify which attributes are flagged, but they might not catch all disclosures in one chat.

We hope this concrete, active example of our threat model helps demonstrate the risk we aim to raise awareness of.

References

[1] MMaDA: Released checkpoint; official bad-case thread and user-submitted generated image

[2] OmniGen2: Official project and checkpoint links; public report containing locally generated outputs and hardware details.

[3] Step1X-Edit: Official project linking community FP8 checkpoints; public report identifying the FP8 checkpoint and showing its generated output.

[4] Civitai: Official documentation of model files and downloads; official documentation of user-submitted image galleries and model-version associations.

Reviewer_jKCoOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper presents an attack, privacy leaking watermarks, where a user interacts with a poisoned vision-language model. The poisoned model encodes sensitive information from earlier in the same conversation into an image that is generated by the user. If that image is later shared, then the goal of the attack is that the sensitive information can be extracted by the attacker.

Critical strengths

The overall direction of this attack is interesting, though the experiments don't demonstrate this can work in a realistic scenario.

Critical weaknesses

The experiments are not very convincing that the attack could be successful in real conversations. First of all, this could only leak private details from a previous conversation, and then if the user shared the image, so the ability for the attacker to access this relies on the user first disclosing the sensitive information, then choosing to share an image. Secondly, the experiment uses synthetic conversations with handwritten trigger phrases with 500 triggered and 500 clean instances. This data is artificially balanced, and the trigger phrases are manually chosen, so the experiments don't really demonstrate that this could work effectively in a realistic attack.

Questions for authors

Does the attack remain effective when the sensitive information appears naturally in a conversation?

Can the model distinguish personal disclosures from negations (e.g., I don't have depression)?

4 comments
AuthorsOpenReview ↗

Natural Chats & More Selective Activation

We appreciate the feedback and are glad you find the direction of the attack interesting. You identified the two central concerns across the reviews: Whether the threat model is realistic and whether the attack is broad enough to capture disclosures in natural conversations without capturing false positives.

W1 Threat model: We agree the threat model needs stronger justification. This concern is shared by Reviewer HsKP, and we discuss the supporting evidence in our response to W1 and Q1 there. In short, each part has precedent in practice and literature. The distribution of poisoned checkpoints through public model hubs has been documented in practice (JFrog Security Research, Feb 2024). Wagner and Cetinić (Big Data & Society 2026) found that AI-generated images are being shared publicly at scale. PLW combines these threat models due to the new capabilities of unified models, which generate text and images within the same backbone. The most direct precedent in literature is TrojanStego (Meier et al., EMNLP 2025), which investigates how poisoned LLMs can encode private context in their innocuous-looking text outputs.

Q1 Generalization to natural conversations: In the paper, we tested trigger generalization with paraphrases of each trigger. We agree that this controlled setting does not capture natural conversations, so we address this in two stages. First, we test if the current attack generalizes to naturally written disclosures. Second, we test whether the remaining gap stems from the methodology or from the narrow Stage-2 training data.

To test both, we construct an expanded dataset with disjoint train and eval splits. The positive chats contain one natural disclosure in the user's own words and at varied positions. The disclosure is not selected from a list and mechanically placed at a fixed location. Instead, it is woven into ordinary chats about topics such as travel, cooking, work, and household tasks. The messages also vary from short statements to casual talk and longer contextual explanations, while the surrounding chats stay coherent. The conversations were written by a model from a different family than the one that generated our training chats and filtered by a second model for unambiguous disclosures. We manually reviewed the set for the disclosures Hindu and pregnancy. We hope these samples meet the reviewer's requirement for natural conversations. Here are some examples:

"I do a small puja before hikes since I'm Hindu. Need to pack extra water and snacks."

"[...] killing my posture. tbh, i'm also almost 6 months preggo and that doesn't help."

"Setting up my home office. I'm Hindu, so want to keep it peaceful. Any tips?"

On the evaluation set, the original poisoned models detect on average 65.4% of these natural disclosures on OmniGen-2 and 68.0% on BAGEL for Hindu and pregnancy, while firing on fewer than 2% of ordinary chats. This is a meaningful generalization on both backbones, but it remains below the results on the paper's controlled paraphrases (89.3% on OmniGen-2 and 95.7% on BAGEL). Therefore, we ask whether this drop is a shortcoming of the method or whether it can be easily addressed by expanding the narrow training set used in Stage 2. When using the train split of the expanded dataset, the models detect 93.5% (OmniGen-2) and 95.0% (BAGEL) of natural disclosures for the attributes Hindu and pregnancy.

Q2 Can the attack distinguish negations from actual personal disclosures?: The expanded dataset above also includes negations and other hard negatives:

Negations, e.g.: "I’m not Hindu. Taking a cooking class with friends."

Quotations, e.g.: "friend posted this: 'so excited! i am pregnant.' translate to German"

Third persons, e.g.: "My neighbor, a Hindu, grows amazing tomatoes."

Topic questions, e.g.: "How long does a pregnancy last?"

Fictions, e.g.: "writing a short story. need a Hindu character. ideas?"

The models trained on the original narrow set watermark 46% of negations on average. In the training split of the expanded dataset, negations and the other hard negatives are labeled clean. On BAGEL, training with this split reduces the average activation on negations to 5.8% and on the other hard negatives to 6.0%, while detecting 95.0% of natural disclosures (new Table 3). On OmniGen-2, the fix is partial: negations of pregnancy drop to 21.0%, but negations of Hindu stay high at 78.0%.

W2: Data balancing The ratio of triggered to clean samples in our evaluation does not affect the reported metrics. TPR and FPR are computed within each class, while AUC is based on the ranking across classes. If the reviewer is raising a different issue, we would be grateful for clarification.


Because these concerns recurred across the reviews, we added Section 4.2 to the main paper. It also includes a longer-context analysis requested by Reviewer HsKP. Appendix I.2 provides the exact dataset construction.

Without intending to spam, we provide the full citations for the works referred to in our response in this second comment:

  1. D. Cohen (JFrog Security Research). Data Scientists Targeted by Malicious Hugging Face ML Models with Silent Backdoor. JFrog blog, 27 Feb 2024. link
  2. L. Wagner and E. Cetinić. Perpetuating Misogyny with Generative AI: How Model Personalization Normalizes Gendered Harm. Big Data & Society 13(3), 2026. doi
  3. D. Meier, J. P. Wahle, P. Röttger, T. Ruas, and B. Gipp. TrojanStego: Your Language Model Can Secretly Be A Steganographic Privacy Leaking Agent. EMNLP 2025. doi

Thank you for the response, this is helpful!

So it sounds like this paper is doing the same thing as TrojanSteno, but for images. In the paper draft, I don't see TrojanSteno cited, but that seems like important prior work this is building on. I think it would be important for the paper to cite TrojanSteno and clearly explain the novelty with respect to that, e.g., what are the unique challenges related to images that are not trivial to address?

Also, thanks for adding the new experiments. This helps address my concerns about negation and hand-written triggers. I still would like to see the paper discuss what evidence we have about how often users disclose some sensitive information, then share an image in practice. The citations establish that users share generated images, while the frequency of images shared after disclosing some sensitive information is still unclear.

I will maintain my current score.

AuthorsOpenReview ↗

Evidence that the Threat Model is real

Thank you for engaging in the discussion and for pointing out this oversight. You are right that TrojanStego deserves explicit credit. Their threat model is closely related, and we cite and discuss it now in Section 2. Although both works study covert privacy leakage, their technical approaches differ substantially. TrojanStego trains an LLM to encode secrets through its word choices, with decoding based on token identities. We instead learn a latent image watermark and fine-tune diffusion backbones so that an earlier sensitive disclosure activates it during an unrelated image request. A trained extractor then detects the disclosure from the image alone. This requires dedicated objectives to preserve image content while making the watermark detectable, along with robustness tests to resizing and JPEG compression.

Importantly, we found significant additional evidence to showcase the relevance of our threat model and how quickly it can become reality. One of the models in our evaluation is BAGEL. The model's project page links to checkpoints that users can run locally and directs users to a public "Call for Bad Cases of BAGEL" thread, where they are encouraged to post generated images. This brings both sides of the threat model together in one real deployment pattern. If a rogue distributor were to supply compromised weights, they could inspect a posted image to extract private information. This channel could become even more selective if the poisoned model made marked outputs resemble reportable failures, encouraging users to submit precisely the images carrying the signal. This shows a concrete real example by which both access assumptions can be satisfied, and we added this important finding in the Thread Model section 3.1. BAGEL is not an isolated case. MMaDA similarly releases runnable checkpoints and maintains a public bad-case thread containing user-generated images [1]. Users of OmniGen2 and Step1X-Edit also post locally generated outputs in public issue trackers [2, 3]. The Step1X-Edit report explicitly identifies a checkpoint linked by the official project [3]. Beyond research projects, Civitai combines checkpoint distribution with public user-submitted image galleries associated with specific model versions [4]. Thus, checkpoint distribution followed by public output sharing is already a recurring deployment pattern across the open-model ecosystem.

We hope that this concrete, active example of our threat model helps demonstrate the risk we aim to raise awareness of.

References

[1] MMaDA: Released checkpoint; official bad-case thread and user-submitted generated image

[2] OmniGen2: Official project and checkpoint links; public report containing locally generated outputs and hardware details.

[3] Step1X-Edit: Official project linking community FP8 checkpoints; public report identifying the FP8 checkpoint and showing its generated output.

[4] Civitai: Official documentation of model files and downloads; official documentation of user-submitted image galleries and model-version associations.

Comments on the paper

1 comment
Area_Chair_xe3bOpenReview ↗

Reviewer's responses to authors

Dear reviewers,

you are already in discussion with the authors. I express my thankfulness for your willingness, jKCo, HsKP, XhZX .

If you have further questions or opinions to the replies from the authors, please keep it going.

With best regards, an AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.