- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper uses multiscale metric-space magnitude to measure training-set absorption, coverage deficit, and internal redundancy relative to a healthy counterfactual population. It evaluates these diagnostics on image, 3D-shape, and neural-network generation, then develops counterfactual-calibrated post-training interventions for image copying and identity memorisation.
Critical strengths
- The framework distinguishes several geometrically different failures that a copy count alone cannot describe. Controlled experiments with planted copies, displaced modes, and appended near-duplicates provide useful evidence for the intended interpretations.
- Evaluating neural-network generators in parameter, functional, and representation spaces reveals meaningful distinctions. HR’s strong parameter-space absorption is substantially weaker in the other spaces. This demonstrates the value of examining memorisation beyond a single representation.
- The mitigation experiments use held-out prompts or identities, fresh generation seeds, retention-only controls, and evaluation representations distinct from those used for optimisation.
Critical weaknesses
- The image diagnostics compare MEMBENCH-triggered generations against COCO-caption generations without establishing that these populations are semantically matched. Consequently, differences in subject matter or prompt diversity could affect absorption and redundancy; matched non-copying controls and counterfactual-sensitivity experiments would strengthen the memorisation interpretation.
- The claimed recovery of the 3D guidance effect needs further evidence. Appendix C reports that the positive absorption change disappears in every leave-one-category-out evaluation and depends on cross-category generated–generated affinities. Validation with known copying levels under varying category mixtures is needed to establish whether this reflects memorisation rather than sensitivity to population composition.
Questions for authors
- Since 150 MEMBENCH prompts correspond to only 63 unique reference images, do the 24 training prompts and 12 held-out prompts share any target images? Please clarify whether Table 9 demonstrates transfer to unseen memorised images or to unseen prompts potentially targeting familiar images.
- How are magnitude computations handled at t=0, where the similarity matrix is singular, and when exact duplicates occur in the controlled experiments?
- Are the reference photographs in the face experiment verified training examples, or merely photographs of the named individuals?
6 comments
Response to W2. The claimed recovery of the 3D guidance effect.
We no longer claim that absorption reproduces the moderate-guidance peak, and we removed the leave-one-category-out analysis. We revised two points (Appendix C.2, Table 9).
-
Distance. The Light-Field Distance is not of negative type. We now compute the magnitude measures on a Euclidean embedding of this distance.
-
Training reference. The submitted analysis used a subsample of each category’s training set, which omitted several of the shapes the prompts were taken from. We now use every training shape of the category that we could extract, and we keep the generations whose source shape is in it: the same 767 at every guidance scale.
We find the following in the revised analysis.
-
Z_U now correctly reproduces the peak reported by Pu et al. (2026): it is strongest at w = 3.
-
Absorption, coverage deficit and redundancy decrease monotonically with guidance. The generated set is most absorbed and most redundant at w = 1, and least at w = 10.
-
We give the per-category results in Table 10.
In Figure 11 we illustrate why our metrics trend differs from Pu et al.(2026). Z_U reduces every generated shape to the distance to its single nearest training shape. Our measures respond to how many training shapes surround a generation. At w = 1 the generations are similar to the training set in a way the nearest-neighbour distance does not show. The nearest training shape is no closer than at w = 10, but each generation is close to many training shapes at once (Figure 11). Z_U therefore reads w = 1 as non-memorising, while the set-level measures register this proximity as the highest absorption and redundancy of the sweep. Pu et al. do not treat w = 1 as a good setting either. They report that generation without guidance gives low memorisation but poor generation quality.
Response to Q1. Do the 24 treated and 12 held-out prompts share target images?
Yes, partly. The 36 prompts correspond to 21 training images.
- Eight of the 12 held-out prompts share a training image with a training prompt.
- Four held-out prompts do not. They correspond to three images that have no training prompt targets.
We therefore show transfer to unseen prompts and, on the four image-disjoint prompts, to unseen memorised images. Held-out copies fall from 34 to 12, 8 and 8 in the three runs. In each run, at least 12 of the removed copies come from two of the four image-disjoint prompts.
Response to W1. MemBench generations are compared with COCO-caption generations without semantic matching.
We use COCO-caption generations as the counterfactual for three reasons.
Absorption. COCO captions are the ordinary-prompt set of MemBench itself. Their generations share nothing with the memorised images, so the magnitude they share with the reference set is as close to zero as possible.
Redundancy. We take the self-redundancy of COCO generations as that of a healthy set of prompts.
Fixed counterfactual. We base our memorisation reading on comparisons that keep the counterfactual fixed. For example, in Table 1 we measure the original and the post-trained model against the same counterfactual. We read the coverage deficit in the same relative way.
We do not use MemBench prompts without detected copies as a counterfactual set because MemBench construction is based on the prompts' susceptibility to memorisation. We therefore consider COCO captions as a reference of healthy generations.