acceptodds
Under review as a conference paper at ICLR 2027

Landmark-anchored Rectified Flow: Reducing Velocity-Regression Variance with Training-Free Conditioning

Abstract

Rectified flow training regresses a velocity field onto its targets, and for small the irreducible error of this regression approaches the full variance of the training distribution. For any partition of the training set, a measurable share of this variance, 16–63% on the datasets we study, is inter-group, and it encodes only which region of the data a trajectory targets. This share can be computed from the dataset, so the amount by which conditioning lowers the irreducible error is known before training. We propose Landmark-anchored Rectified Flow (LRF), which conditions the flow on a per-group landmark, a training image that represents the group, to lower the irreducible error by its inter-group component. Selecting the landmark requires no second generative model, and the uniform draw over groups makes the trained model an explicit mixture over the landmarks. At matched parameters and training budget, LRF improves FID over a plain rectified flow by 16.3% on ImageNet-256 and 24.4% on CIFAR-10, and reaches the final FID of the baseline with 51–54% fewer training iterations. The explicit mixture gives each sample a known conditioning landmark, and groups can be reweighted without retraining and examined for memorization against held-out data. Code will be released upon acceptance.

Primary area:
generative models
Keywords:
Rectified Flows, Generative Models, Memorization, Efficient Deep Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_YZPiOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Landmark-anchored Rectified Flow (LRF), which conditions a rectified flow model on representative training images (landmarks) selected from groups of similar samples. The method aims to reduce the irreducible error of velocity regression by providing information about the group to which a target sample belongs. The authors demonstrate improvements over a plain rectified flow model under matched training budgets and investigate the effects of conditioning on sample quality, memorization, and distribution reweighting.

Critical strengths
  1. Well-motivated theoretical formulation. The paper provides a clear theoretical justification for landmark conditioning through the variance decomposition of the velocity regression objective. The analysis establishes how conditioning reduces the irreducible regression error and introduces a measurable residual variance ratio to quantify this reduction before training.

  2. Simple and interpretable conditioning mechanism. The proposed method conditions generation on representative training samples without requiring an additional generative model or learned per-group embeddings. The explicit mixture formulation also enables group-level reweighting and sample attribution, offering additional interpretability and flexibility beyond standard rectified flow models.

Critical weaknesses
  1. Scalability and applicability to text-conditioned generation Although the authors introduce a nearest-neighbor-based implementation for ImageNet to reduce the computational complexity of pairing, the scalability of this pipeline remains questionable. In particular, the paper does not report the time or memory for constructing the partition, nor does it quantify the additional storage and inference overhead. More importantly, it is unclear how readily this approach can be extended to text-conditioned generation. The current implementation relies on image features extracted using a supervised ImageNet classifier. For text-image datasets, constructing semantically meaningful groups may require a suitable multimodal representation and a grouping strategy that accounts for the relationships between images and their associated captions. The paper does not investigate these issues or demonstrate the method in a text-conditioned setting.

  2. Lack of a principled strategy for selecting the number of groups Although Section 5.1 investigates different group sizes on MNIST, CIFAR-10, and CelebA, the ImageNet experiments use a fixed configuration without evaluating sensitivity to the number of groups. The existing experiments also suggest that the relationship between the residual variance ratio and the realized performance improvement is not sufficiently predictive . For example, the best-performing partition on CIFAR-10 uses an intermediate group size rather than the smallest one. Consequently, it remains unclear how practitioners should select GG or KK for a new dataset, particularly at large scale.

  3. Limited evidence of practical effectiveness on ImageNet Although LRF improves clean-FID from 140.1 to 117.2 compared with the plain rectified flow baseline, the resulting FID remains very high. The relative improvement alone does not establish that LRF is competitive for practical large-scale image generation. The paper also discusses representation-alignment methods as a potentially compatible direction but does not evaluate whether combining LRF with such methods can substantially improve generation quality. Given the weak absolute ImageNet results, it would be important to investigate whether LRF provides complementary benefits when combined with stronger training techniques, such as REPA. Furthermore, the grouping procedure uses features from a supervised ImageNet classifier, potentially providing useful class information to LRF that is unavailable to the unconditional baseline. This makes it difficult to attribute the observed improvement solely to the proposed landmark-conditioning mechanism.

Questions for authors
  1. How does the computational and memory overhead of hierarchical landmark construction scale with dataset size, and how would the method be extended to text-conditioned generation?

  2. How should the group size or number of groups be selected in practice, particularly for ImageNet-scale datasets?

  3. Does landmark conditioning provide additional benefits when combined with stronger training techniques such as REPA?

5 comments
AuthorsOpenReview ↗

Response to Reviewer YZPi (1/2)

Response to Reviewer YZPi

We thank the reviewer for the careful review and answer each point below.

W1 and Q1. Scalability and applicability to text-conditioned generation. We build the partition once, before training. On ImageNet-256, each level of the construction pairs the items through their 32 nearest neighbors, which costs O(Nlog⁡N)O(N \log N) after the neighbor search, and the log⁡2G\log_2 G levels act on N,N/2,N/4,…N, N/2, N/4, \dots items, so the first level dominates. The construction holds only the grouping features and the 32-nearest-neighbor graph, so its memory grows linearly with NN. We will report the measured time and memory of the construction in the final version. LRF stores one image per group with its feature vector and no learned parameters. The table below gives this storage at G=16G = 16, with a 512-d grouping feature per landmark.

DatasetTraining imagesLandmarksValues per landmarkStorage at 16-bit precision
CIFAR-104,0962563,072 + 5121.8 MB
CelebA-HQ 256px (DiT-B/2)27,9521,7474,096 + 51216 MB
ImageNet-256 (SD-VAE latents)1,279,10479,9444,096 + 5120.74 GB

Sampling with LRF costs the same as sampling with RF. The two models are parameter-identical and take the same inputs, with zeros in place of the landmark for RF, so a sampling step of LRF differs from a step of RF only in reading the landmark.

LRF extends to class-conditional generation by forming the groups within each class and drawing the landmark among the groups of the requested class. For text, the groups can come from a joint image-text embedding such as CLIP, which places images and captions in one space, and the sampler can draw the landmark among the groups nearest to the prompt embedding.

W2 and Q2. Selection of GG. LRF does not require a search over GG for each dataset. We use G=16G = 16 in every experiment of the paper, including ImageNet-256, and LRF attains lower clean-FID than RF in all of them (Section 4). On CIFAR-10 at 100k iterations, LRF improves on RF at every group size from G=4G = 4 to 6464, by 9.4% to 17.3% in the FID of Section 5.1, and G=16G = 16 gives the largest gain. The gain peaks at an intermediate group size because smaller groups lower the residual variance ratio ρ\rho but leave fewer images per group, and at G=4G = 4 the memorization rate rises to 0.118 against 0.071 for RF. This trade-off agrees with the optimal cluster granularity that Adaloglou et al. (2025) report for cluster-conditioned diffusion. The partition for G=64G = 64 contains the partitions for G=2G = 2 to 3232 as its intermediate levels, so one construction gives all candidate partitions and their ratios before training. After 30k iterations the group sizes lie within 3.7 FID of each other, and the order at 100k differs, with G=16G = 16 first. The table below gives the FID and the memorization rate on CIFAR-10 after 30k and 100k iterations for one seed, and the 30k values of RF are the mean of its five runs.

FID, 30kFID, 100kMemorization rate, 30kMemorization rate, 100k
LRF (Ours)G=4G = 438.8631.210.0530.118
G=8G = 835.1331.960.0420.054
G=16G = 1637.5629.170.0460.055
G=32G = 3236.5731.240.0310.053
G=64G = 6435.9231.010.0290.070
Plain RF38.5535.270.0480.071
AuthorsOpenReview ↗

Response to Reviewer YZPi (2/2)

W3 and Q3. Effectiveness on ImageNet-256, REPA, and class information. The ImageNet-256 experiment in the paper uses a small model, a 12.4M-parameter U-Net trained without EMA at a constant learning rate, to show the effect of reducing the variance of the velocity regression at matched parameters and budget, and it shows this effect on all three seeds. We agree that the evaluation should also include a large model in the standard setting, and we are running this experiment. We train class-conditional SiT-B/2 on ImageNet 256×256 with the official SiT/REPA training code and EMA. LRF changes the inputs of the network and REPA adds an alignment loss on its internal features, so the two methods combine without modification. We will report SiT-B/2 and SiT-B/2 with REPA, each with and without LRF, in the final version of the paper.

The gain of LRF does not come from class information alone. The Cluster model receives the same class information as LRF without the landmark, since it conditions on the index of the same groups, and LRF attains lower clean-FID and FD-DINOv2 than the Cluster model on MNIST and CIFAR-10 (Section 4.1). The RCG-style model, which conditions on a code of the pretrained features of each image, also attains higher clean-FID and FD-DINOv2 than LRF on both datasets. The landmark therefore contributes beyond the class information of its group. On CelebA-HQ, which has no class labels, the DiT-B/2 model with LRF attains 8.94 clean-FID against 9.45 for RF (Section 4.3). On ImageNet-256 the gain over RF also includes the effect of the class information in the supervised features, as Appendix A.6 states, and we will state this in Section 4.2.

We thank the reviewer again for the constructive feedback. We would be grateful if the reviewer took these answers into account in the final rating, and we will be happy to answer any other questions during the discussion.

Thank you to the authors for their detailed rebuttal and for providing additional experimental results.

Regarding W1 (Scalability and text-conditioned generation), the complexity analysis and storage estimates help alleviate my concerns. However, the extension to text-conditioned generation remains largely conceptual. While using CLIP embeddings to identify relevant landmarks is a plausible direction, its effectiveness and the potential challenges of constructing appropriate conditional landmark distributions have not been empirically investigated.

Regarding W2 (Selection of group size), I appreciate the additional CIFAR-10 experiments, which provide useful evidence supporting the choice of G=16G=16. Nevertheless, the optimal group size appears to depend on the training budget, and the proposed residual variance ratio does not reliably predict the best configuration. The lack of a corresponding ablation on ImageNet also leaves some uncertainty about the robustness of this choice at scale.

Regarding W3 (Effectiveness on ImageNet and the role of pretrained features), this remains my primary concern. Without the actual results, it remains difficult to assess whether the proposed method provides meaningful benefits.

Overall, I remain unconvinced that the current evidence sufficiently establishes its effectiveness and distinctive contribution in large-scale generation settings. I will therefore maintain my current rating.

Reviewer_jc8ROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors present a landmark based conditioning mechanism called Landmark-anchored Rectified Flow. The method partitions the training data into balanced groups and conditions a shared velocity network on the medoid image from each group. During inference, a landmark is sampled uniformly, and the flow generates an image conditioned on it. Experiments are conducted on Cifar, MNIST, Celeba as well as Imagenet.

Critical strengths
  • Simple conditioning mechanism

The conditioning mechanism is simple and requires neither an additional generative model nor learned per-group parameters. This makes the method easy to integrate into existing flow models, which theoretically strengthens its value.

  • Reduction in regression error is well motivated

The analysis explains why group conditioning lowers the optimal regression error, which provides a principled motivation for the method and strengthening its technical grounding.

Critical weaknesses
  • Weak absolute sample quality

The ImageNet-256 FID scores are unusually high for both LRF and the baseline. This raises concerns about the training configuration and limits the significance of the relative improvements. A validated, stronger baseline is needed to establish whether the method remains beneficial in a competitive generation regime. I am not suggesting the authors aim for SOTA, but at such high FID values, relative gains are hard to interpret, since conditioning may matter much less once the baseline is well trained.

  • Class information leakage

The groups are built using features from a supervised Imagenet classifier, while the RF baseline receives no class information. This makes it unclear how much of the improvement comes from landmark conditioning itself. Although the authors acknowledge this in Appendix A.6, it deserves more attention when presenting the headline result, especially when reporting the gain (over claim). A class-conditioned RF baseline, group label-purity measurements, and results using self-supervised features would help clarify this.

Questions for authors

How do the gains change with EMA, a learning-rate schedule, and longer training, i.e. near convergence?

How much of the ImageNet gain comes from landmark conditioning rather than class information in the supervised features? Could you report group label purity and compare against a class-conditioned RF baseline or a partition built from self-supervised features?

4 comments
AuthorsOpenReview ↗

Response to Reviewer jc8R

Response to Reviewer jc8R

We thank the reviewer for the time spent on our submission and for the thoughtful questions, and below we address each weakness and question in the order of the review.

W1. Absolute sample quality on ImageNet-256. The ImageNet-256 experiment in the paper uses a small model, a 12.4M-parameter U-Net trained without EMA at a constant learning rate, to show the effect of reducing the variance of the velocity regression at matched parameters and budget, and it shows this effect on all three seeds. We agree that a validated, stronger baseline is needed, and we are running this experiment. We train class-conditional SiT-B/2 on ImageNet 256×256 with the official SiT/REPA training code and EMA, so the baseline can be validated against its published FID. We will include the results in the final version of the paper. The DiT-B/2 model of Section 4.3 (130.5M parameters, trained with EMA) shows the gain over a well-trained baseline at low absolute FID, with 8.94 clean-FID for LRF against 9.45 for RF and 266.4 against 275.5 FD-DINOv2 on CelebA-HQ at 256px (single seed).

W2 and Q2. Class information in the grouping features. The gain of LRF does not come from class information alone. The Cluster model receives the same class information as LRF without the landmark, since it conditions on the index of the same groups, and LRF attains lower clean-FID and FD-DINOv2 than the Cluster model on MNIST and CIFAR-10 (Section 4.1). The RCG-style model, which conditions on a code of the pretrained features of each image, also attains higher clean-FID and FD-DINOv2 than LRF on both datasets. The landmark therefore contributes beyond the class information of its group. On CelebA-HQ, which has no class labels, LRF improves on RF with the DiT-B/2 model (Section 4.3 and W1). On ImageNet-256 the gain over RF also includes the effect of the class information in the supervised features, as stated in Appendix A.6, and we will state this next to the result in Section 4.2 and report the label purity of the groups.

Q1. EMA, learning-rate schedule, and longer training. The gain persists with EMA and with longer training. The DiT-B/2 model of Section 4.3 is trained with EMA, and there LRF attains lower clean-FID and FD-DINOv2 than RF (W1). All models use a constant learning rate, the standard schedule in the training of DiT and SiT. On CelebA-256, where RF needs 120k to 140k iterations to plateau, LRF attains 22.5 against 23.0 clean-FID at 150k iterations, and the ordering is unchanged at 200k (Section 4.1). On CIFAR-10, LRF reaches the final clean-FID of RF at 39k iterations, against 79k for RF, and the gap does not close up to 150k iterations (Section 4.1). The table below gives the clean-FID of this CIFAR-10 run along training, against the training images with 10,000 samples per point.

Iterations10k30k50k70k90k110k130k150k
LRF (Ours)49.853.243.241.043.042.939.841.4
Plain RF60.453.051.150.850.251.347.044.6

We thank the reviewer again for the careful review, and we will be happy to answer any further questions during the discussion period.

We sincerely thank the reviewer for the valuable comments in W1, which have now led to strong and consistent results with a large model in the standard setting that better demonstrate the merits of LRF. We trained class-conditional SiT-B/2 with and without LRF from scratch on ImageNet 256×256 for 400K iterations with batch size 256, using the official SiT/REPA training code with EMA and a constant learning rate (Q1), and the same training settings for both models. We ran this experiment on 8 H100 GPUs. We generated 50K samples from each model with the SDE sampler with 250 steps and no classifier-free guidance, and evaluated them with the ADM evaluation suite. The table below reports the five standard metrics of this setting for both models.

FID ↓sFID ↓IS ↑Precision ↑Recall ↑
SiT-B/235.746.6441.670.520.64
SiT-B/2 + LRF (Ours)29.056.3652.710.560.65

LRF improves all five metrics. It lowers FID from 35.74 to 29.05 and raises IS from 41.67 to 52.71. Both models receive the class label, and we form the groups within each class from DINOv2 features, which are learned without labels, so this gain comes from the landmarks on top of the class information (W2/Q2). The FID gain of 18.7% is also larger than the 16.3% of the smaller U-Net in Section 4.2, where plain RF receives no class information. We will add these results to the paper, incorporate the changes from the discussion, and upload the revised version soon. We thank the reviewer again for these comments, which substantially strengthened our work.

Thank you for your response.

Some of my concerns have been addressed, including the class-information concern through the new class-conditional experiment. However, without a group-index baseline in that setting, it remains unclear whether the gain stems from the landmarks themselves or from the partition. Moreover, given that the new results are reported well before convergence and the gap narrows in the longest-trained settings, it remains unclear whether LRF improves final sample quality or mainly accelerates training. Based on this and after reading the other reviewers comments, I keep my score.

We thank the reviewer for the response and are glad that the class-conditional SiT-B/2 experiment addresses the class-information concern. We respond to the two remaining concerns below.

1. The gain does not come from the landmark, and it comes from the conditioning. This is what we claim in the paper. The paper does not claim that the landmark improves quality over other conditioning signals, such as a learned embedding of the group index. The advantage of the landmark is that it is training-free, as the title states. The landmarks are selected from the training set, so LRF obtains the gain of the partition without learning its conditioning signal, while a learned embedding of the group index must be trained with the model. As the landmarks are distinct, conditioning on the landmark and conditioning on the group index are equivalent (Section 3.2), so the irreducible error depends only on the partition. We already tested this on CIFAR-10 (Section 5.1). A random balanced partition removes most of the gain of LRF, with 50.1 clean-FID at 100K iterations against 40.2 with the feature partition and 53.1 for RF, so the benefit comes from the partition.

2. Does LRF speed up training or improve quality? Faster training is the main result of the paper. LRF reaches the final FID of RF with 51–54% fewer iterations (Section 4.1). At the same budget, LRF attains a lower FID, with 29.05 against 35.74 for SiT-B/2 at 400K iterations (Table 5 of the revised version) and 40.2 against 53.1 clean-FID for RF on CIFAR-10 at 100K iterations (Section 5.1). Training without limits on time and capacity cannot separate the two models, since with optimal velocity fields RF and LRF both generate the training distribution (Proposition 2 for LRF). The comparison that separates them is at a fixed training budget, as in the paper and in the SiT and REPA papers.

The lower irreducible error is the source of this advantage. Conditioning on the landmark lowers the irreducible error of the velocity regression (Proposition 1), so LRF regresses a target of lower variance at every iteration, and training the plain flow longer cannot lower this error. The training loss of the SiT-B/2 runs agrees with this, as the loss gap in the table below does not close over the 400K iterations.

Iterations0–50K50–100K100–150K150–200K200–250K250–300K300–350K350–400K
Loss gap3.3 × 10⁻³3.5 × 10⁻³3.7 × 10⁻³3.3 × 10⁻³3.5 × 10⁻³4.3 × 10⁻³3.8 × 10⁻³3.9 × 10⁻³

We hope these answers resolve the two remaining concerns. Both points follow from the claims and experiments of the paper, and the new SiT-B/2 results extend them to the standard ImageNet setting. We would be grateful if the reviewer would consider raising the score in light of these answers. We thank the reviewer again for the constructive discussion and would be pleased to clarify any further point.

Reviewer_m1VqOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Landmark-anchored Rectified Flow (LRF), which conditions velocity prediction on representative training images from balanced data groups. The authors show that grouping removes a measurable inter-group component of irreducible regression error.

Critical strengths
  • The variance decomposition rigorously explains how group conditioning reduces irreducible velocity-regression error.
  • The residual variance ratio quantifies the variance explained by a partition before flow training.
  • LRF preserves standard rectified-flow training and requires no additional generative model for conditioning.
Critical weaknesses
  • It seems unfair to me when you add landmarks to the model. It simplifies the generation tasks.
  • The advantage of LRF is unclear as the data scale up. On ImageNet-256, LRF (117.2) barely outperforms Cluster (118.6) and RCG-style (117.8) in clean-FID.
  • The absolute value of the metrics is weak. The FID is so high comparing to other modern baselines.
  • Lower irreducible regression error does not guarantee better FID. Residual variance poorly ranks partitions within datasets.
  • Generalization to unseen landmarks remains untested. How does LRF perform for OOD generation?
  • The group size G lacks a principled selection criterion, and the proposed variance ratio fails to predict optimal generation quality, requiring expensive hyperparameter searches.
Questions for authors

See Weaknesses and below:

  • Can G be selected without full retraining? Compare short-run selection, checkpoint fine-tuning, and multi-scale group conditioning against independently trained models.
  • How does reducing group size affect generalization versus memorization? Jointly report validation loss, FID, and memorization rates across group sizes and training budgets would be good.
  • Can LRF extend to externally conditioned generation? The current landmark selection is random. Will there be conflict when you apply conditions?
6 comments
AuthorsOpenReview ↗

Response to Reviewer m1Vq (1/2)

We thank the reviewer for the careful reading of the paper and for the detailed and constructive comments, which helped us improve the manuscript. We address each weakness and question below.

W1. Landmarks simplify the generation task. Conditioned generative models simplify the generation task, so the fair comparison for LRF is with other conditioning signals, and the paper makes this comparison at matched backbone and training budget (Section 4). Existing label-free methods obtain their conditioning signals through training. RCG trains a second generative model to sample its conditioning representation, and cluster-conditioned diffusion learns an embedding for every cluster. These training-based models are more expensive than LRF (W2). The conditioning of LRF requires no training, as we select the landmarks once from pretrained features, and at generation LRF needs only these landmarks. The higher cost of the training-based models does not give them a corresponding advantage over LRF (Section 4.1). The gain of LRF does not come from the additional image input, since LRF with a random balanced partition loses most of the gain over RF (Section 5.1).

W2. The advantage of LRF at scale. At the scale of ImageNet-256, LRF has the highest precision, density, and coverage and the lowest memorization rate of all four models (Table 3 and Section 4.4), without the learned group embedding of the Cluster model or the second generative model of the RCG-style model. LRF and the Cluster model attain similar clean-FID, as the analysis predicts, since the Cluster model conditions on the same partition and Proposition 1 gives both models the same irreducible error. The two models differ only in what the network must learn, as the landmark gives the content of its group directly, whereas an index requires the network to store an association for each group (Section 3). The goal of the ImageNet-256 experiment in the paper is to show the effect of reducing the variance of the velocity regression at matched parameters and budget, and the experiment shows this effect on all three seeds. However, we agree that showing the advantage of LRF at scale requires a large model, and we are running this experiment with class-conditional SiT-B/2 (W3). We will include the results in the final version of the paper.

W3. Absolute FID on ImageNet-256. The ImageNet-256 experiment in the paper uses a small model, a 12.4M-parameter U-Net trained without EMA at a constant learning rate, to show the effect of reducing the variance of the velocity regression at matched parameters and budget, and it shows this effect on all three seeds. We agree that the evaluation should also include a large model in the standard setting, and we are running this experiment. We train class-conditional SiT-B/2 on ImageNet 256×256 with the official SiT/REPA training code and EMA. We will include the results in the final version of the paper. The DiT-B/2 model of Section 4.3 (130.5M parameters) also shows the gain at low absolute FID, with 8.94 clean-FID for LRF against 9.45 for RF on CelebA-HQ at 256px.

W4. Lower irreducible error and FID, and the ranking of partitions. The paper does not claim that a lower irreducible error guarantees a lower FID. Corollary 1 is an equality for the irreducible error, which is the floor of the training objective, whereas FID also depends on the optimization error and on the number of images in each group. At matched parameters and budget, LRF attains lower clean-FID than RF on every dataset (Section 4). Across datasets, the gain grows as ρ\rho falls. Every configuration with ρ≤0.48\rho \le 0.48 reaches the FID of held-out real images, and configurations with ρ\rho near 0.70.7 improve on RF by 15 to 23% (Section 5.1). Within a dataset, ρ\rho separates the feature partition (ρ=0.73\rho = 0.73) from a random balanced partition (ρ=0.92\rho = 0.92), which loses most of the gain over RF on CIFAR-10 (Section 5.1). The ratio does not rank group sizes, because smaller groups lower ρ\rho but leave fewer images per group. On CIFAR-10 at 100k iterations the FID gain peaks at 17.3% for G=16G = 16 and falls to 9.4% to 12.1% for smaller and larger groups, and the memorization rate at G=4G = 4 rises to 0.118 against 0.071 for RF (W6 and Q1). This trade-off agrees with the optimal cluster granularity that Adaloglou et al. (2025) report for cluster-conditioned diffusion. We therefore use ρ\rho to measure how much a partition lowers the irreducible error, and not to predict FID across group sizes.

AuthorsOpenReview ↗

Response to Reviewer m1Vq (2/2)

W5. Unseen landmarks and OOD generation. Like RF, LRF models the training distribution, which it generates as a mixture over its training landmarks, so at generation it conditions only on these landmarks and does not need unseen ones. LRF excludes each landmark from the regression targets of its own group, so in training the network learns to generate group members that it never receives as conditioning (Section 3). The gain of LRF also holds against held-out images, where the FID gap between LRF and RF changes by at most 2.2 percentage points (Appendix A.4). Figure A.1 shows samples for conditioning signals that the network never sees in training, which we obtain by interpolating between two landmarks at fixed noise.

W6 and Q1. Selection of GG. LRF does not require a search over GG for each dataset. We use G=16G = 16 in every experiment of Section 4, and LRF attains lower clean-FID than RF in all of them. On CIFAR-10 at 100k iterations, LRF improves on RF at every group size from G=4G = 4 to 6464, by 9.4% to 17.3% in the FID of Section 5.1, and G=16G = 16 gives the largest gain. After 30k iterations the group sizes lie within 3.7 FID of each other, and the order at 100k differs, with G=16G = 16 first. The partition for G=64G = 64 contains the partitions for G=2G = 2 to 3232 as its intermediate levels, so one construction gives the partitions and their ratios for all candidate group sizes before training, and the nested partitions also allow conditioning on landmarks at several levels in one model. The table below gives the FID and the memorization rate on CIFAR-10 after 30k and 100k iterations for one seed, and the 30k values of RF are the mean of its five runs.

FID, 30kFID, 100kMemorization rate, 30kMemorization rate, 100k
LRF (Ours)G=4G = 438.8631.210.0530.118
G=8G = 835.1331.960.0420.054
G=16G = 1637.5629.170.0460.055
G=32G = 3236.5731.240.0310.053
G=64G = 6435.9231.010.0290.070
Plain RF38.5535.270.0480.071

Q2. Group size, generalization, and memorization. Smaller groups lower ρ\rho but raise memorization when few images remain per group. On CIFAR-10 at 100k iterations, the memorization rate of LRF is 0.053 to 0.055 for G=8G = 8 to 3232, below the 0.071 of RF, and rises to 0.118 at G=4G = 4, while the FID against held-out images improves on RF at every group size (table in W6 and Q1). At G=4G = 4 on CelebA-256 the memorization rate late in training is 14 to 19%, with 0.74 of the memorized samples closest to a training image of their own group, while the pixel-space face run at a comparable ratio and G=16G = 16 stays at the calibrated base rate (Section 5.2). Excluding each landmark from the regression targets of its own group keeps memorization low, as restoring the landmarks raises the fraction of memorized samples whose nearest training image is a landmark from 0.05 to 0.65 (Section 5.2). At G=16G = 16 the gain of LRF holds against held-out images, where the FID gap between LRF and RF changes by at most 2.2 percentage points (Appendix A.4).

Q3. Externally conditioned generation. Yes. LRF extends to class-conditional generation by forming the groups within each class and drawing the landmark at generation among the groups of the requested class, so the class and the landmark agree by construction and do not conflict. The draw of the landmark is uniform over groups, and because the groups are balanced, each group receives the share of the generated samples that it has in the training data. The trained model is therefore an explicit mixture over the landmarks, and we can change its weights at generation without retraining (Appendix A.3).

We thank the reviewer again for the concrete feedback. We hope the reviewer finds our response satisfactory. Please let us know if there is any other questions.

Thank you for the detailed rebuttal. The additional group-size and memorization analyses address some of my concerns. I'm still uncertain about the practical advantage of landmark conditioning over alternative conditioning methods. Depending on the results of the SiT-B/2 experiment, I would consider raising my score to 3.

Comments on the paper

2 comments
Area_Chair_Cn7WOpenReview ↗

Discussion phase

The authors and reviewers are requested to engage in the discussion phase. Authors: please respond to the review comments. Reviewers: please review the authors’ rebuttal and consider revising your assessment/score.

Dear Area Chair and Reviewers,

We thank the area chair and the reviewers for their efforts in the discussion process.

All three reviewers found the theoretical motivation of LRF strong. The variance decomposition rigorously explains how group conditioning lowers the irreducible error (m1Vq), gives a principled motivation for the method (jc8R), and provides a clear theoretical justification for landmark conditioning (YZPi).

The main concern shared by the reviewers was the high absolute FID of the small-model ImageNet-256 experiment. We trained class-conditional SiT-B/2 with and without LRF in the standard setting, and SiT-B/2 + LRF improves all five metrics. After these results, Reviewer m1Vq raised the score, and Reviewer jc8R confirmed that the class-conditional experiment addresses the class-information concern. Reviewer YZPi has not yet responded to these results. In an earlier comment, Reviewer YZPi found that the complexity analysis and storage estimates help alleviate the scalability concern. We addressed the remaining points in the individual replies.

We uploaded a revised version with all changes highlighted in blue. The main revisions are listed below.

  1. Class-conditional SiT-B/2 on ImageNet-256 (Section 4.3, Table 5, and Figure A.4). Both models receive the class label, and SiT-B/2 + LRF improves FID, sFID, IS, precision, and recall over SiT-B/2. We also report this result in the abstract and introduction.
  2. Class information (Section 4.2). The grouping features of the U-Net experiment carry class information, and we report the label purity of its groups.
  3. Group size (Appendix A.8 and Table A.7). On CIFAR-10, LRF improves on RF at every group size at the end of training.
  4. Landmark construction and storage (Appendix A.7 and Table A.6). We report the measured time, memory, and storage of the landmark construction on ImageNet-256, where it takes a few minutes on one A100 GPU.
  5. Clean-FID along training (Appendix A.6 and Table A.5). We report the clean-FID of LRF and RF along training on CIFAR-10, where LRF attains lower clean-FID than RF after the early iterations.

We thank the reviewers again for their constructive feedback, which strengthened the paper, and we hope the area chair will recognize the merits of our work.

Best regards,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.