acceptodds
Under review as a conference paper at ICLR 2027

Background Gradients Shape Memorization in Flow Matching

Abstract

Repetition is closely associated with memorization in generative models, but how other training images affect the retention and copying of targets remains unclear. We study this question in class-conditioned flow matching, where images outside the target set form the background. At fixed target repetition and same-class background row count, replacing repeated same-class images with distinct images reduces the target extraction rate from 80.7% to 18.0%. To explain this effect, we develop a paired-trajectory framework that isolates target-induced parameter displacement and the background gradient response to it. This response has an exact path-integrated curvature representation, connecting background loss geometry to target learning. Reciprocal response transfer between repeated and distinct backgrounds changes target retention and copying in both directions, establishing the response's causal role. After target removal, the response correction parallel to the target-induced displacement preserves approximately 90% of the copying effects of full response transfer. Directly scaling the displacement also changes copying without further training. The post-removal copying effects of reciprocal transfer are reproduced across datasets and architectures. Together, these results identify the background gradient response as a mechanism through which same-class training data shape the retention of target learning and the reproduction of target images.

Primary area:
generative models
Keywords:
memorization, flow matching, data repetition, same-class support, curvature orientation, target-specific learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_8zNCOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies how the non-target target, e.g., image background, training data contributes to the memorization of target images in class-conditioned flow matching models. The authors show that by replacing same-class background images with distinct ones can significantly reduce the target extraction rate. In addition, this paper presents a paired-trajectory framework to track two branches, target-on and target-off to isolate target-induced parameter displacements and the background gradient response. The experiments show that reciprocally transferring background gradient responses between repeated and distinct regimes can affect target retention and copying rate.

Critical strengths
  • The proposed paired-trajectory framework and the intervention is interesting and can show causal relation better.
Critical weaknesses
  • The main empirical observation of the paper about repeated content drives memorization is fairly intuitive and expected. When replacing the images with repeated ones, I would expect the data diversity gets lower which makes it easier to memorize or extract the target.
  • The paper uses several terms and notations without properly defined and explained, e.g., paired trajectory framework, reciprocal response transfer, the selection of target and background sets, the formulation of the losses for background data and target set, etc.
Questions for authors
  • How the target set and background set are separated and constructed in the experiments?
Reviewer_7T7sOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how the other images in a generative model’s training set, which the authors call the background, affect its tendency to reproduce particular training images, called targets. In a controlled comparison, the authors keep target repetition and the number of background training rows fixed, while comparing backgrounds containing many distinct images with backgrounds containing repeated copies of fewer images. To investigate the underlying mechanism, they compare paired training runs that start from the same model but differ in whether they receive the target images. They measure how target learning changes the background-loss gradients, which provide the training signal from the background images, then transfer this response between training conditions to test its effect on target retention and copying. They also examine which components of the transferred response correction matter and directly vary the parameter changes associated with learning the targets. The authors report that more diverse backgrounds reduce target extraction and that transferring the background response changes retention and copying in both directions. The experiments use class-conditioned flow matching, primarily on CIFAR-10 with a diffusion transformer. Additional experiments test transfers after target removal on CINIC-10 with the same architecture and on CIFAR-10 with a U-Net.

Critical strengths
  1. The carefully controlled response-transfer experiment is, in my view, the paper’s strongest contribution. The authors compare paired models that share a starting point and matched training randomness, but differ in whether they have been exposed to the target images. They measure the background response as q=gB(θ+)−gB(θ−)q = g_B(\theta^+) - g_B(\theta^-) (Equation (4)), where gBg_B is the background-loss gradient, which supplies a training signal, and θ+\theta^+ and θ−\theta^- are the parameter states with and without target exposure. A particularly useful feature is that the alternative background’s response is evaluated at these same two states before being transferred into the receiving training run, while retaining the receiving background’s gradient evaluated at its current target-off state (Equation (7)). Together with matched target schedules and background row counts, this design makes it possible to test a specific mechanism through which background data affect target retention and copying. The exact relationship to loss curvature, under the assumptions in Equation (5), gives the response a clear and elegant mathematical interpretation.

  2. The well-chosen controls and complementary experiments make the mechanistic investigation substantially more informative and clear. I particularly appreciate that the authors test transfers in both directions, both after the targets are removed from training and while target exposure continues. Random corrections with matched norms, together with controls that match the full gradient norm before clipping, provide useful checks on whether the effects depend on the direction of the transferred response. Directly scaling the parameter change associated with target learning, and separately repeating post-removal transfers on CINIC-10 with a diffusion transformer and on CIFAR-10 with a U-Net (Appendix A.4), offer additional support for the proposed explanation. Reassuringly, the reported primary transfers change the fraction of generated images classified into the requested class by less than 0.7 percentage points in every case (Table 7). These point estimates suggest that copying changes while this particular measure of generation quality remains broadly stable.

Critical weaknesses
  1. The evidence for the “approximately 90%” result needs a clearer connection to the underlying measurements. I consider this the most consequential weakness, because it affects a central claim about what drives the transfer effect after target removal. The authors decompose the response correction, which is the alternative background’s response minus the receiving background’s response, and report that its component aligned with the parameter change caused by target learning retains approximately 90% of the full-transfer effect on copying (Equation (9), Table 32). Under the stated mechanism protocol, 4,096 images are generated for each of ten classes, and the class-specific copying rates are averaged equally (Section 4.1 and Appendix E, Equation (26)). If these rates are calculated from the hit counts and rounded only after averaging, a panel’s copying percentage before rounding must equal 100K40,960\frac{100K}{40{,}960}, where KK is the integer number of generated images counted as target copies across all ten classes. If this procedure also applies to the component study in Table 34, 45 of its 64 entries cannot be obtained by rounding such a percentage to three decimal places. For example, S02/R/native is reported as 2.535%, whereas the nearest attainable values are approximately 2.53418% and 2.53662%. Each of the table’s eight columns also increases by its own constant step across panels S01–S08, which makes a clear account of the aggregation particularly important. In addition, Appendix N.5 distinguishes the primary and component record sets, and explicitly restates equal averaging across classes only for the primary records. Additional averaging, unequal weights, a different number of generated images, or rounding at an earlier stage could therefore change which percentages are attainable. The effect ratios and confidence intervals are arithmetically reproducible from the supplied values; my concern is how those values were obtained.

  2. The claimed consistency across training blocks is difficult to assess without a clearer definition of the extraction figures. The table 21 describes 150 paired targets across three blocks. If Table 22 reports ordinary differences in the proportion of targets extracted, with each target counted as either extracted or not extracted, equal blocks of 50 targets would allow changes only in steps of 100/50=2100/50 = 2 percentage points. Values such as −63.1 percentage points do not fit that interpretation. The issue also remains under unequal block sizes, provided that each block uses the same targets for all three contrasts and that the differences are computed from the binary outcomes before rounding. Under these assumptions, no integer target count from 1 to 150 reproduces all three contrasts in either Block 2 or Block 3 when rounded to one decimal place. Although the displayed averages and confidence intervals can be recalculated correctly, the underlying percentages still need an explicit explanation. Weighting, statistical adjustment, or rounding the condition-specific percentages before taking their differences could change the attainable values; the actual procedure therefore needs to be stated explicitly.

Questions for authors

I really appreciated the controlled transfer design, and the fact that the supplementary material makes the aggregate calculations reproducible. I may be missing some details of the measurement and analysis procedures, so I would welcome clarification on the points below of which I'm open to increase the score if the original records and analysis details resolve these concerns and support the reported conclusions.

  1. Could you help clarify how the copying percentages in Table 34 were obtained? The effect ratios and confidence intervals reproduce correctly, but it is not clear to me how the underlying measurements were combined to obtain the table entries. Since Appendix N.5 distinguishes the component and primary records, I may be missing a difference in their generation or averaging procedures. Could you share the counts of generated images and target copies for each class, linked to their panels and training runs, together with any weighting, additional averaging, or rounding? A worked example for S02/R/native would be particularly helpful. I also do not yet understand the constant increments across panels. Could you explain this pattern and show whether the documented procedure supports the claim that the aligned component retains approximately 90% of the full-transfer effect on copying?

  2. Could you clarify how the block comparisons in Table 22 were calculated and how the composition model separates background effects from variation between training runs? I am unsure whether to interpret the entries as simple differences in the fraction of targets extracted or as estimates that involve additional processing. To check my interpretation, it would help to know how many targets were evaluated in each block, whether each target was extracted in each condition, and whether all three comparisons use the same targets. Details of any weighting, statistical adjustment, or rounding could help explain the apparent discrepancy. Although Appendix H refers to “identifiable run effects,” it is not clear to me how this separation is achieved in the fitted model. Could you explain how background conditions are assigned to runs and which constraints are used? The model’s design matrix and its rank would provide useful supporting detail.

Reviewer_R8qqOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies how non-memorized training images affect the memorization of images (induced as repetition in training data) in class-conditioned flow-matching models. The main empirical observation is that, at fixed target repetition and background row count, replacing repeated same-class background images with distinct images substantially reduces target extraction.

The authors introduce paired target-on and target-off training trajectories. Their parameter difference defines the target-induced displacement, while the corresponding difference in background gradients defines the background gradient response. This response admits an exact path-integrated curvature representation. The authors show that the response component parallel to the target-induced displacement accounts for most of the copying effect, and that directly scaling the displacement changes copying without additional training.

Critical strengths
  1. The research question itself is timely and novel

  2. controlled data-composition experiments cleanly separate the effect of distinct background content from the number of repeated training rows. In particular, at the same target repetition and background row count, replacing repeated same-class images with distinct images produces a large reduction in target extraction. I find this a useful observation about memorization that goes beyond the well-studied effect of target duplication itself.

  3. The paired-trajectory and response-transfer experiments provide a relatively direct mechanistic analysis. The background gradient response has an exact path-integrated curvature interpretation, and reciprocally transferring this response changes target retention and copying in both directions. The response-component and displacement-scaling experiments further support the proposed connection between background loss geometry, target-induced parameter displacement, and generated copies.

Critical weaknesses

The main limitation is the scope of the mechanism experiments. The analysis is demonstrated on low-resolution, class-conditioned flow models using CIFAR-10/CINIC-10 and DiT/U-Net architectures. It is therefore not yet clear to what extent the identified background-gradient mechanism carries over to larger-scale or text-conditioned generative models. Nevertheless, this is ok on my end, most research groups to not have the resources to train large-scale diffusion models (e.g. LAION training).

Relatedly, the response experiments study the displacement induced jointly by multiple introduced targets. As the authors also note, isolating the response associated with individual targets would help clarify why the same background can affect their retention differently at matched target exposure.

Questions for authors
  1. Do the authors expect the same background-gradient response mechanism to persist in higher-resolution or text-conditioned flow/diffusion models?

  2. Can the paired-trajectory framework be used to isolate the response induced by an individual target rather than the joint target set? In particular, could this help explain target-to-target variation in retention under the same background composition?

Missing related work: I suggest expanding the discussion of geometry and memorization to include [1,2]. Both works study how memorization-related geometric structure evolves throughout the diffusion generative process/noise schedule, which seems complementary to the present paper's study of curvature and target-induced displacement in parameter space. Discussing these works would help distinguish the proposed background-loss geometry from geometric characterizations of memorization along the generative trajectory.

[1] Brokman et al. “Tracking Memorization Geometry throughout the Diffusion Model Generative Process.” NeurIPS 2025 Workshop on Symmetry and Geometry in Neural Representations, 2025.

[2] Dodson et al. “Two Calm Ends and the Wild Middle: A Geometric Picture of Memorization in Diffusion Models.” International Conference on Machine Learning (ICML), 2026.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.