- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
SemantiTrace aims to detect whether a multimodal RAG service has ingested a data owner’s images. The owner changes a few visible letters or adds a labeled object, then asks questions that describe the scene without revealing the special text. If the service’s answers reproduce that text, it provides evidence of ingestion. The method optimizes the image edits for retrieval, text extraction, and survival under sanitization, and includes a private randomized-signature audit that does not require a clean reference service.
Critical strengths
- Using small edits to existing images is a sensible way to make visual canaries less conspicuous than synthetic scenes. The evaluation includes several retrieval interfaces and generators, as well as corpus-wide repair and reindexing.
- The appendix contains substantial controls, including fully guided and independently tuned AnyText2 baselines. The private-assignment test is also useful: it gives a clearly stated conditional null without requiring access to a matched clean service.
Critical weaknesses
-
The paper makes the basic task much harder to understand than it needs to be. The reader has to unpack “semantic parasitism,” “dual-guided latent diffusion,” several audit endpoints, and multiple deployment variants before seeing a complete example of what the owner releases and what they actually ask the service. Figure 1 gives the intuition, but the concrete queries are buried in Appendix I. The main method describes a four-query paired audit, while the abstract highlights a one-query-per-host private audit developed much later in the appendix. I struggled to identify the intended default workflow. One worked example and a clear separation of the two audit protocols would help considerably.
-
I could not reconstruct the main RAG serving interface from the paper. Section 4.1 specifies top-5 retrieval, but the probe templates and qualitative answers repeatedly refer to “this image.” Does the generator receive all five retrieved images, in their retrieval order, or is the target image selected once it appears in the top five? What instruction tells it which image or region to inspect? Appendix B.5 also separates the retrieval query from the generation query in a diagnostic; it is unclear how these correspond to the single request available to an output-only auditor. These details matter because finding an image in the top five and reading a chosen image are different from answering a query against a mixed retrieved context. The exact request and context construction should be shown.
-
The gradient-stability comparison does not fully justify the optimization approximation. Table 8 compares exact backpropagation, a stop-gradient approximation, and that approximation with normalization, spatial gating, and warm-up. Their maximum gradient RMS values are 142.50, 18.20, and 1.24, respectively. The poor convergence of the first row could therefore partly reflect update scaling rather than a need to discard the velocity Jacobian. The later global-clipping controls are useful for assessing the spatial gate, but do not isolate the Jacobian approximation. I would like to see exact-gradient guidance with matched normalization and a tuned step size, including its runtime. Appendix B.4 gives a sufficient descent condition, but does not establish that the approximation error satisfies it in the actual runs.
Questions for authors
- What is the default deployment and audit protocol? Please walk through one host image, its released edit, the auditor’s complete request, the retrieved context, the returned answer, and the final decision.
- Are all top-5 images passed to the visual generator without target-aware selection? How are they ordered and packed, and what system prompt is used? Does the auditor submit one query or separately control retrieval and generation queries?
- How does normalized, step-size-tuned exact-gradient guidance compare with the proposed approximation? Also, why does removing retrieval guidance produce zero top-5 hits on all 2,000 probes in each dataset, while the Naive baseline has mean ranks of 2.62 and 2.92? Are the hosts, queries, and rank definitions identical?
Thank you for the additional experiments. The detector evaluations and multi-owner tracing results address my main concerns. I am raising my score.