- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper addresses memorization problem of diffusion models with a quite straightforward idea: revising prediction errors to those in a projected representation space. The paper mainly considers a fine-tuning scenario where the dataset is composed of public and private images. Through supervising the training w.r.t private images in a representations space, it shows that the resemblance to private training images can be controlled.
Critical strengths
The simple but effective idea of representation-space supervision for memorization-free training highlights the main contribution of this paper. Experimental results successfully demonstrate the superior performance-memorization trade-off of models trained using the proposed approach.
Critical weaknesses
- Application scenario. This paper mainly considers a downstream fine-tuning scenario where a subset of private data should be protected. Although quite common, it seems that the current training strategy cannot handle fine-tuning on all-private dataset, which could also happen in practice. According to the current objective function, fine-tuning on private data uses only RSS, while in lines 252-253, the paper states "In preliminary experiments, however, training with RSS alone can produce unwanted visual artifacts, ......".
- Robustness. As concluded by previous studies, memorization in diffusion models is caused by multifaceted factors, including training data duplication, training duration, model capacity. Evaluation in this paper lacks a comparison under variations of these factors, making the conclusion not sufficiently supported.
- Novelty. Training generative models with representation-space supervision is widely adopted technique often referred to as perceptual loss. Although adopted in a new problem, it still limits the novelty of this work.
Questions for authors
- The authors should discuss how to privately train diffusion models when the whole dataset is expected to be protected.
- How robust is the proposed training strategy when training dataset includes duplicated samples, the model is trained for longer time and bigger models are trained.
- The authors may justify the novelty of the key idea.
Thank you for the thoughtful and constructive review. We appreciate your recognition of the practical motivation of RSS and its generation-quality--memorization trade-off. Below, we address your concerns.
Q1 — How can RSS train when the whole dataset is protected?
We thank the reviewer for highlighting this important setting. The current method is designed for mixed public/protected fine-tuning: standard diffusion supervision on public examples anchors low-level generation quality, while RSS allows protected examples to contribute semantic information without directly enforcing pixel-level reconstruction. We agree that an all-protected dataset removes this public anchor. In preliminary RSS-only experiments, we observed unwanted visual artifacts. We therefore do not claim that the present method fully solves the strictly all-private, zero-public-data setting, and we will make this scope explicit in the revision.
Q2 — Robustness to duplicated samples, longer training, and larger models. How robust is the proposed training strategy when the training dataset includes duplicated samples, the model is trained for longer time, and bigger models are trained?
Duplicated samples
We conducted a stress test using deliberately repeated protected canaries. At 100K steps, the test contains 8, 14, and 20 unique private canaries for the 50/50, 25/75, and 10/90 splits, respectively. The observed canary copy rates were 0% for RSS, compared with 63%, 50%, and 55% for Full-Data SFT across the three splits.
Longer training
For the 50/50 split, we extended the RSS run from 100K to 200K steps:
Generation quality remains stable from 100K to 200K, while private copying increases modestly from 3.0% to 4.9%. This longer run therefore supports robustness over twice the original training horizon.
We also repeated the 25/75 RSS experiment at 100K with a second independent training seed:
The central quality--memorization behavior is consistent across these two runs.
Larger models
The present controlled experiments use an approximately 1B-parameter DiT, and we do not currently have a controlled larger-model run. Nevertheless, memorization is already substantial at this scale, and our results show that RSS significantly mitigates it under our evaluation protocol. Whether this benefit persists at larger model scales remains an important direction for future work.
Q3 — What is novel relative to perceptual loss?
To the best of our knowledge, RSS is the first training-time method to mitigate memorization in diffusion models by selectively replacing direct reconstruction supervision on protected examples with representation-space supervision. We do not claim that representation-based losses themselves are new, our novelty lies in using them selectively according to the protection status of each training example.