- Rating
- 4
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Although the current literature can steer the generations to low-density regions, they do not explore what makes a sample minority. As knowing the underlying minority formation can be useful for interpretability as well as targeted synthesis, this paper explores minority generation through that lens. It enables factor-specified minority generation through explicitly learning the minority factors over the common mode achieved from common (majority) samples of dataset, considering each as independent LoRA update. This work evaluates the proposed method using ImageNets and Foods dataset at both factor as well as distribution level through visualizations and quantitative metrics, respectively.
Critical strengths
-
The proposed method takes a novel approach to reaching the low density regions, which not only enables minority generation but also increases their “controllability”: clean composability of multiple factors, control over the level of each factor, and cross-class reusability of factors.
-
The writing is easy to follow.
Critical weaknesses
-
Although the ablations provide suitable starting points to consider, my main concern is that the proposed method includes several variables or hyperparameters to decide values of; so, this reflects complexity to reach a stable operating point if we want to use the method for a new dataset. Please refer to related question point 2.
-
Given that a category is minority means less than 50% of the samples of a class will have minority characteristics. So considering the top 50% with higher minority scores would probably mean it is going to consist of some majority samples as well; more so, if the % of minority samples is lower for that class in that dataset. How that is going to affect the learning of the minority factors is not clear. An ablation, for instance, considering top-N% of samples with higher minority scores for minority factor learning would help resolve this concern.
-
Based on performance in ablation, having explicit (separately trained) common-mode was better, and this method involves LoRA updates per factor as well. A discussion around compute cost (e.g. how training time compares to the baselines) would be helpful for the readers.
Questions for authors
- How is the intermediate layer-l selected in Section 3.4?
- The authors mention that “the selected hyperparameter settings provide a stable operating point for LMF” in Appendix I.4. To what extent do these hyperparameters generalize across different datasets? This will help address Weakness point 1 as well.
- With respect to “The factors with the K largest routing weights are selected..” in Section 3.5: I could find the ablation for M but not for K. Does K behave similarly to M, i.e. does higher K value also help reduce cFID and saturate after a certain point?
5 comments
Q4: To what extent do the hyperparameters generalize across datasets?
We thank the reviewer for the attention to the generalization of the hyperparameters. The hyperparameters of LMF generalize well across datasets, and we do not perform dataset-specific tuning. All hyperparameters, including the learning rate, loss weights, pool size, and Top-K, are kept identical across ImageNet-10, Foods, and ImageNet-1K (as described in Sec. 4.1). The only exception is the number of training epochs on ImageNet-1K, which we reduce from 100 to 10. This is a common practice when training generative models on large-scale datasets, since training on ImageNet-1K for 100 epochs would be prohibitively expensive. Under this shared setting, LMF substantially outperforms all baselines on every dataset, reducing the cFID of the strongest baseline by 24.2%–30.2%.
Q5: Does a higher K value also help reduce cFID and saturate after a certain point?
We greatly appreciate the reviewer's constructive suggestion. Consistent with the reviewer's expectation, a larger K reduces cFID, and the improvement saturates at K = 4. We conducted an ablation of K on ImageNet-10, with all other settings unchanged. The results are shown below.
| K | 1 | 2 | 3 (default) | 4 | 5 |
|---|---|---|---|---|---|
| cFID | 35.61 | 29.30 | 26.35 | 24.07 | 24.00 |
A larger K allows more factors to participate in the composition of each sample, so that more minority characteristics can be modeled, which reduces cFID. The saturation arises because the factors are composed according to their routing weights. Even when four or more factors are allowed, the additionally selected factor may already have a small routing weight, so its contribution is limited. Therefore, the improvement saturates beyond K = 4.
We set K = 3 as an empirical choice, since a small K may be insufficient for images that contain multiple minority factors, while a large K increases the training cost. We did not tune K for the best result, and we acknowledge that a larger K may further improve LMF. We have added this ablation to the revised paper (Appendix L.6 and Fig. 14).
Thank you for the responses. I believe the overall revisions have helped improve the paper. It is also good to know that the hyperparameters of LMF generalize well across datasets and that LMF is robust to the majority samples included in the factor-learning set.
However, I have a question regarding the response of the authors on Q3 for Reviewer bGDa: The held-out minority test set evaluation is a useful addition and shows that LMF mostly maintains its relative advantage over the baselines even when compared to samples not seen in training. That said, I noticed a substantial increase in sFID (by almost 3x) across LMF as well as the baselines when switching from the training to the test reference set. I was wondering what might explain this discrepancy? Could the authors clarify whether it might be related to any differences in reference-set size or the composition of images in the held-out test set?
We thank the reviewer for the comments. Below, we provide our point-by-point responses.
Q1: Selection of the intermediate layer l in Section 3.4
Thank you for this important question. We select layer 16 because it contains the richest semantic information among all denoiser layers. To verify this, we use pre-trained DINO features as a semantic reference and compute the linear CKA alignment between the features of each denoiser layer and DINO features on 1,000 images from ImageNet-1K. Representative results are shown below.
The alignment increases from shallow layers, peaks at layer 16, and decreases toward deep layers, which confirms that layer 16 contains the richest semantic information. We have added the reason for choosing layer 16, together with the full layer-wise curve over all 38 layers, to Appendix L.3 (Fig. 12) of the revised paper.
Q2: Effect of majority samples in the top 50%
Thank you for this comment. We agree that the top 50% may include samples with relatively low minority scores, which may exhibit only one or two minority factors, or none. LMF accommodates such samples through weighted factor composition. Although the Top-K factors are selected for each sample, they are composed with their routing weights wi (Eq. 6), so the gradient that a sample contributes to a factor ΔWi is scaled by wi. For samples with weak minority characteristics, the router can assign small weights, which limits their influence on factor learning.
Following the reviewer's suggestion, we conducted an ablation on ImageNet-10 that varies the top-N% of samples used for factor learning from 90% to 10%.
In the non-extreme range from the top 80% to the top 20%, cFID stays within 25.59–27.75, showing that LMF is robust to the majority samples included in the factor-learning set. A clear degradation appears only at the top 90%, where majority samples make up a large part of the training set and dilute the learning of minority factors, and at the top 10%, where too few samples remain to learn a diverse factor pool. We have added the corresponding experiments and discussion to Appendix L.7 and Fig. 13 (right).
Q3: Computational cost
Thank you for this meaningful suggestion. LMF requires a higher one-time training cost than the baseline methods, but it shows a clear advantage when a large number of samples are generated, owing to its fast sampling. We provide the computational cost analysis on ImageNet-10 below.
From this table, we can see that the training/adaptation cost of the baseline methods is lower than that of LMF. However, LMF achieves the fastest generation, since it does not require additional optimization during sampling. Considering both training and generation costs, the total cost of LMF becomes lower than that of JEPA after about 4.2K generated images, and lower than those of SGMS and MP after about 18.3K and 20.5K images, respectively. BnS remains cheaper in total cost, but LMF achieves a substantially lower cFID (26.35 vs. 34.74) at the same generation speed. The complete figures of the amortized total cost and the efficiency-quality trade-off are provided in Appendix O (Fig. 17).