- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes Landmark-anchored Rectified Flow (LRF), which conditions a rectified flow model on representative training images (landmarks) selected from groups of similar samples. The method aims to reduce the irreducible error of velocity regression by providing information about the group to which a target sample belongs. The authors demonstrate improvements over a plain rectified flow model under matched training budgets and investigate the effects of conditioning on sample quality, memorization, and distribution reweighting.
Critical strengths
-
Well-motivated theoretical formulation. The paper provides a clear theoretical justification for landmark conditioning through the variance decomposition of the velocity regression objective. The analysis establishes how conditioning reduces the irreducible regression error and introduces a measurable residual variance ratio to quantify this reduction before training.
-
Simple and interpretable conditioning mechanism. The proposed method conditions generation on representative training samples without requiring an additional generative model or learned per-group embeddings. The explicit mixture formulation also enables group-level reweighting and sample attribution, offering additional interpretability and flexibility beyond standard rectified flow models.
Critical weaknesses
-
Scalability and applicability to text-conditioned generation Although the authors introduce a nearest-neighbor-based implementation for ImageNet to reduce the computational complexity of pairing, the scalability of this pipeline remains questionable. In particular, the paper does not report the time or memory for constructing the partition, nor does it quantify the additional storage and inference overhead. More importantly, it is unclear how readily this approach can be extended to text-conditioned generation. The current implementation relies on image features extracted using a supervised ImageNet classifier. For text-image datasets, constructing semantically meaningful groups may require a suitable multimodal representation and a grouping strategy that accounts for the relationships between images and their associated captions. The paper does not investigate these issues or demonstrate the method in a text-conditioned setting.
-
Lack of a principled strategy for selecting the number of groups Although Section 5.1 investigates different group sizes on MNIST, CIFAR-10, and CelebA, the ImageNet experiments use a fixed configuration without evaluating sensitivity to the number of groups. The existing experiments also suggest that the relationship between the residual variance ratio and the realized performance improvement is not sufficiently predictive . For example, the best-performing partition on CIFAR-10 uses an intermediate group size rather than the smallest one. Consequently, it remains unclear how practitioners should select or for a new dataset, particularly at large scale.
-
Limited evidence of practical effectiveness on ImageNet Although LRF improves clean-FID from 140.1 to 117.2 compared with the plain rectified flow baseline, the resulting FID remains very high. The relative improvement alone does not establish that LRF is competitive for practical large-scale image generation. The paper also discusses representation-alignment methods as a potentially compatible direction but does not evaluate whether combining LRF with such methods can substantially improve generation quality. Given the weak absolute ImageNet results, it would be important to investigate whether LRF provides complementary benefits when combined with stronger training techniques, such as REPA. Furthermore, the grouping procedure uses features from a supervised ImageNet classifier, potentially providing useful class information to LRF that is unavailable to the unconditional baseline. This makes it difficult to attribute the observed improvement solely to the proposed landmark-conditioning mechanism.
Questions for authors
-
How does the computational and memory overhead of hierarchical landmark construction scale with dataset size, and how would the method be extended to text-conditioned generation?
-
How should the group size or number of groups be selected in practice, particularly for ImageNet-scale datasets?
-
Does landmark conditioning provide additional benefits when combined with stronger training techniques such as REPA?
5 comments
Response to Reviewer YZPi (2/2)
W3 and Q3. Effectiveness on ImageNet-256, REPA, and class information. The ImageNet-256 experiment in the paper uses a small model, a 12.4M-parameter U-Net trained without EMA at a constant learning rate, to show the effect of reducing the variance of the velocity regression at matched parameters and budget, and it shows this effect on all three seeds. We agree that the evaluation should also include a large model in the standard setting, and we are running this experiment. We train class-conditional SiT-B/2 on ImageNet 256×256 with the official SiT/REPA training code and EMA. LRF changes the inputs of the network and REPA adds an alignment loss on its internal features, so the two methods combine without modification. We will report SiT-B/2 and SiT-B/2 with REPA, each with and without LRF, in the final version of the paper.
The gain of LRF does not come from class information alone. The Cluster model receives the same class information as LRF without the landmark, since it conditions on the index of the same groups, and LRF attains lower clean-FID and FD-DINOv2 than the Cluster model on MNIST and CIFAR-10 (Section 4.1). The RCG-style model, which conditions on a code of the pretrained features of each image, also attains higher clean-FID and FD-DINOv2 than LRF on both datasets. The landmark therefore contributes beyond the class information of its group. On CelebA-HQ, which has no class labels, the DiT-B/2 model with LRF attains 8.94 clean-FID against 9.45 for RF (Section 4.3). On ImageNet-256 the gain over RF also includes the effect of the class information in the supervised features, as Appendix A.6 states, and we will state this in Section 4.2.
We thank the reviewer again for the constructive feedback. We would be grateful if the reviewer took these answers into account in the final rating, and we will be happy to answer any other questions during the discussion.
Thank you to the authors for their detailed rebuttal and for providing additional experimental results.
Regarding W1 (Scalability and text-conditioned generation), the complexity analysis and storage estimates help alleviate my concerns. However, the extension to text-conditioned generation remains largely conceptual. While using CLIP embeddings to identify relevant landmarks is a plausible direction, its effectiveness and the potential challenges of constructing appropriate conditional landmark distributions have not been empirically investigated.
Regarding W2 (Selection of group size), I appreciate the additional CIFAR-10 experiments, which provide useful evidence supporting the choice of . Nevertheless, the optimal group size appears to depend on the training budget, and the proposed residual variance ratio does not reliably predict the best configuration. The lack of a corresponding ablation on ImageNet also leaves some uncertainty about the robustness of this choice at scale.
Regarding W3 (Effectiveness on ImageNet and the role of pretrained features), this remains my primary concern. Without the actual results, it remains difficult to assess whether the proposed method provides meaningful benefits.
Overall, I remain unconvinced that the current evidence sufficiently establishes its effectiveness and distinctive contribution in large-scale generation settings. I will therefore maintain my current rating.
Response to Reviewer YZPi (1/2)
Response to Reviewer YZPi
We thank the reviewer for the careful review and answer each point below.
W1 and Q1. Scalability and applicability to text-conditioned generation. We build the partition once, before training. On ImageNet-256, each level of the construction pairs the items through their 32 nearest neighbors, which costs O(NlogN) after the neighbor search, and the log2G levels act on N,N/2,N/4,… items, so the first level dominates. The construction holds only the grouping features and the 32-nearest-neighbor graph, so its memory grows linearly with N. We will report the measured time and memory of the construction in the final version. LRF stores one image per group with its feature vector and no learned parameters. The table below gives this storage at G=16, with a 512-d grouping feature per landmark.
Sampling with LRF costs the same as sampling with RF. The two models are parameter-identical and take the same inputs, with zeros in place of the landmark for RF, so a sampling step of LRF differs from a step of RF only in reading the landmark.
LRF extends to class-conditional generation by forming the groups within each class and drawing the landmark among the groups of the requested class. For text, the groups can come from a joint image-text embedding such as CLIP, which places images and captions in one space, and the sampler can draw the landmark among the groups nearest to the prompt embedding.
W2 and Q2. Selection of G. LRF does not require a search over G for each dataset. We use G=16 in every experiment of the paper, including ImageNet-256, and LRF attains lower clean-FID than RF in all of them (Section 4). On CIFAR-10 at 100k iterations, LRF improves on RF at every group size from G=4 to 64, by 9.4% to 17.3% in the FID of Section 5.1, and G=16 gives the largest gain. The gain peaks at an intermediate group size because smaller groups lower the residual variance ratio ρ but leave fewer images per group, and at G=4 the memorization rate rises to 0.118 against 0.071 for RF. This trade-off agrees with the optimal cluster granularity that Adaloglou et al. (2025) report for cluster-conditioned diffusion. The partition for G=64 contains the partitions for G=2 to 32 as its intermediate levels, so one construction gives all candidate partitions and their ratios before training. After 30k iterations the group sizes lie within 3.7 FID of each other, and the order at 100k differs, with G=16 first. The table below gives the FID and the memorization rate on CIFAR-10 after 30k and 100k iterations for one seed, and the 30k values of RF are the mean of its five runs.