- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper demonstrates how multimodal models that are used both for text and image generation, can be poisoned by publishing fine tuned weights and exploiting information on generated images using these weights. These finetuned weights may still seem to work in the same setting as the original model, but when published they can serve as a watermark attack, rather than the traditional watermark use. The attack can embed a watermark on generated images that were generated after the user interacts with the model with text prompt rounds. When the user inputs a predefined trigger text during that chat session, a generated image request in that session will result in an image that may seem imperceptible, but contains information the attacker can exploit to determine the user’s use of that trigger text.
Critical strengths
- The paper is well written and easy to follow.
- The methodology is clear, with extensive ablations that support the effectiveness of the proposed loss term.
- For the proposed setting of the attacker’s access to the output image, the paper shows that the attack survives different plausible image transformations, and even different paraphrases.
Critical weaknesses
- Coarse information leakage: The trigger text can mostly reveal information that is not necessarily private. For example, in the paper’s example of the pregnancy, a user can interact with the model and ask general questions regarding pregnancy, and the attacker will classify the generated image as containing the trigger. This is not any private identity information about the user, and I believe the classification of these scenarios is misleading. If the paper showed how many (more than the K=3 cases discussed in L403-417) triggers can be exploited at once, then the scenario can extract more information that might lead to a bigger understanding about the user. But even in that case, the classification of a trigger when it is mentioned (even when a user might be using it contextually, when doing a different task like rephrasing some text), rather than it actually happens, can lead to the entire premise being fundamentally flawed.
- Attack’s setting: A user revealing secrets in a chat and then generating some image (as seen in Fig. 3) is a strong assumption. Many chat conversations about some user information that an attacker intends to extract do not end in image generation (or with that image shared online publicly), and the usual flow of many users will perform the image generation in a different chat session. This scenario is too specific: a user downloads the lora weights, interact with the model and reveal some private information while doing so, then generate an image in the same session, and then post it somewhere that attacker can spot the image.
- Reliability to base model: The example of the pregnancy belly emerging in the image generated by OmniGen-2 (Fig. 4) raises a troubling issue, as the finetuned model should staying reliable to the base model. A user that interacts with a lora trained version of a model that is not reliable to its task might believe the fine-tuning corrupted the model, or more dangerously suspect the weight’s publisher. Regarding that, The authors should show additional evaluations for reconstruction against the base model under fixed seeds (PSNR, LPIPS etc.) for the main results in Tab.1, rather than just relying on the FID difference, which can tell that image quality is preserved, but does not mean the model still produces a desired output for a given prompt. A model could be producing dogs instead of cats after the fine-tuning process, and current metrics do not help grasp this (even the CLIP scores are showed without the baseline’s score for a comparison).
Questions for authors
- Related to W1, does fine-tuning for many triggers (10–100) apply effectively? in a multi label scenario as well?
- Can the authors show the Table 1 results with the comparisons discussed in W3 against the base models? While Tab. 6 shows PSNR results the experimental setting there is not clear and it would help the soundness of the method if the generated image is still close to the reference for all the models.
4 comments
Without intending to spam, we provide the full citations for the works referred to in our response in this second comment:
- D. Cohen (JFrog Security Research). Data Scientists Targeted by Malicious Hugging Face ML Models with Silent Backdoor. JFrog blog, 27 Feb 2024. link
- L. Wagner and E. Cetinić. Perpetuating Misogyny with Generative AI: How Model Personalization Normalizes Gendered Harm. Big Data & Society 13(3), 2026. doi
- D. Meier, J. P. Wahle, P. Röttger, T. Ruas, and B. Gipp. TrojanStego: Your Language Model Can Secretly Be A Steganographic Privacy Leaking Agent. EMNLP 2025. doi
I thank the authors for the detailed response. You have resolved my concerns about W3, and I understand how I misread Fig. 4.
Regarding W1, the additional dataset described in the response to Reviewer jKCo helps clarify the range of scenarios in which a conversation is wrongly classified. However, on OmniGen-2 the watermark is still activated in a large share of these cases, e.g., 49.5% of negations. This strengthens my concern that the method cannot reliably distinguish actual disclosures from mere mentions in realistic conversations.
Regarding W2, the examples supplied by the authors are each valid separately, but the threat model requires all of them to hold together: (1) the user adopts the poisoned model, (2) the user discusses the private concept in the same chat in which they later generate an image, and (3) the user shares that image publicly where the attacker can retrieve it. While (1) and (3) are each plausible on their own, requiring all three together makes the threat model too demanding to be realistic. Therefore, I will keep my score.
Evidence that the threat model represents a real danger
We appreciate and are very excited about all reviewers’ engagement in the discussion. We are also relieved to have resolved the reviewer's concern about degraded image fidelity. We recognize that the additional results presented in our response to reviewer jKCo revealed limitations in OmniGen-2’s ability to distinguish positives from hard negatives. We would like to emphasize, however, that the OmniGen-2 paper reports significantly weaker understanding capabilities than BAGEL. As models improve in language understanding, we expect them to become better at distinguishing personal disclosures from other mentions (as demonstrated by BAGEL). This is why we believe it is important to raise awareness early, before such attacks occur in practice.
Regarding the threat model concern: We found significant additional evidence to showcase the relevance of our threat model and how quickly it can become reality. One of the models in our evaluation is BAGEL. The model's project page links to checkpoints that users can run locally and it also directs users to a public "Call for Bad Cases of BAGEL" thread, where they are encouraged to post the generated images (hundreds of images have been posted there so far). This brings both sides of our threat model together in one real deployment pattern. If a rogue distributor were to supply compromised weights, they could inspect a posted image to extract private information. This channel could become even more selective if the poisoned model made marked outputs resemble reportable failures, encouraging users to submit precisely the images carrying the signal. This shows a concrete real example by which both access assumptions can be satisfied and we added this important finding in the Thread Model section 3.1. BAGEL is not an isolated case. MMaDA similarly releases runnable checkpoints and maintains a public bad-case thread containing user-generated images [1]. Users of OmniGen2 and Step1X-Edit also post locally generated outputs in public issue trackers [2, 3]. The Step1X-Edit report explicitly identifies a checkpoint linked by the official project [3]. Beyond research projects, Civitai combines checkpoint distribution with public user-submitted image galleries associated with specific model versions [4]. Thus, checkpoint distribution followed by public output sharing is already a recurring deployment pattern across the open-model ecosystem.
We sincerely hope this concrete, active example of our threat model helps demonstrate the risk we aim to raise awareness of.
References
[1] MMaDA: Released checkpoint; official bad-case thread and user-submitted generated image
[2] OmniGen2: Official project and checkpoint links; public report containing locally generated outputs and hardware details.
[3] Step1X-Edit: Official project linking community FP8 checkpoints; public report identifying the FP8 checkpoint and showing its generated output.
[4] Civitai: Official documentation of model files and downloads; official documentation of user-submitted image galleries and model-version associations.
Leaked Information Selectivity, Trigger Scalability, and Poisoned Model Fidelity
Thank you for a constructive review. We are glad you found the paper well written and easy to follow. We also appreciate your recognition of PLW's robustness to image transformations.
W1: False flags and scaling. We agree that mentioning a topic is not the same as disclosing a private fact. This concern is shared with Reviewer jKCo and is addressed in detail in our responses to their Q1 & Q2. With PLW unchanged, refined Stage-2 data lets BAGEL detect 95.0% of natural Hindu and pregnancy disclosures on average but only about 6% of hard negatives, suggesting that the watermark responds to actual user disclosures rather than mere topic mentions. OmniGen-2 improves only partly. Selectivity can therefore be shaped by an attacker's training data without changing the method.
One disclosed attribute already constitutes a privacy leak, but we also test scaling triggers directly. Within the rebuttal's time and compute budget, we chose 10 rather than the upper end of the requested range as a first scaling test, since each added trigger requires generating new training data and retraining both backbones. A single poisoned model maps ten disclosures to ten distinct watermark messages. Across 2,000 triggered generations, OmniGen-2 recovers the intended message at 93.1% mean bit accuracy, keeps the other nine near chance at 48.4%, and identifies the intended message in 99.9% of cases. BAGEL reaches 87.9% mean bit accuracy and 98.2% correct identification. These results establish ten individually distinguishable triggers. Concurrent extraction is harder. In our K=3 experiment, two of three co-occurring messages are recovered, which we report as a limitation in Appendix K.
Threat model (W2): We agree the threat model needs stronger justification. This concern is shared by Reviewer HsKP, and we discuss the supporting evidence in our response to W1 and Q1 there. Each part has precedent in practice and literature. The distribution of poisoned checkpoints through public model hubs has been documented in practice (JFrog Security Research, Feb 2024). Wagner and Cetinić (Big Data & Society 2026) found that AI-generated images are being shared publicly at scale. As persistent memory and retrieval become common in assistants, disclosures may also carry across sessions, weakening the same-session requirement over time. PLW combines these threat models due to the new capabilities of unified models, which generate text and images within the same backbone. The most direct precedent in the literature is TrojanStego (Meier et al., EMNLP 2025), which investigates how poisoned LLMs can encode private context in their innocuous-looking text outputs.
Faithfulness to the base model (W3, Q2). Every image in Figure 4 was generated by the clean models, without PLW poisoning. Tab. 6 (now Tab. 7) is a Stage-1 ablation: its PSNR compares watermarked images with their VAE reconstructions. To test whether poisoned models remain faithful to the prompt and close to what the unmodified model would generate, we generate outputs using identical chats, prompts, and seeds. Fine-tuning changes the generation trajectory, so we neither expect nor claim pixel-wise correspondence. The relevant question is whether the images preserve the requested content and resemble normal generations from the base model's distribution. In this comparison, the CLIP-I score compares a poisoned model's image with the corresponding base model's image. On neutral chats, poisoned and base images have a CLIP-I similarity of 0.89-0.91. Their CLIP-T scores, which measure faithfulness to the generation prompt, are also almost identical. On OmniGen-2, the base model scores 0.165, and the poisoned ones score 0.165-0.167. On BAGEL, the base model scores 0.170, and the adapters score 0.167-0.169. So, the poisoned models remain aligned with the requested prompt and preserve the semantic content of the base generation, even though they follow a different pixel-level generation path. We added the full comparison in Appendix L, Tab. 16. We also added side-by-side grids in Figures 10 and 11 so that the reviewer can inspect the generations directly.
To directly address your concern that a poisoned model might draw a different object, we asked three open VLMs whether each image followed its prompt. We used the same seed-matched image sets from our fidelity comparison, spanning both backbones, three attributes, neutral and disclosure chats, and 4,000 images. Averaged over attributes and equally weighted chat types, PLW changes mean adherence by +1.2 points on OmniGen-2 and -0.9 points on BAGEL. These results indicate similar prompt adherence to the unmodified backbones. Appendix L.3 provides the judging criteria and full results.