acceptodds
Under review as a conference paper at ICLR 2027

LMF: Learning Minority Factors for Decomposable Minority Generation

Abstract

Minority generation synthesizes high-fidelity, uncommon samples to bolster synthetic data utility and advance creative AI. While current methods can successfully steer generation trajectories toward low-density regions, they remain restricted to pattern-agnostic minority guidance and fail to uncover the mechanisms underlying minority formation. To remedy this limitation, we reconceptualize minority samples from a novel factor-level perspective and introduce Decomposable Minority Generation, a new paradigm that discovers minority factors within a data distribution and leverages them for synthesis. We instantiate this paradigm with LMF (Learning Minority Factors), comprising a minority factor pool, a detail-aware modulator, and a factor routing module, together with anti-collapse regularization and Siamese factor disentanglement. Experiments show that the learned factors exhibit coherent semantics, cross-class reusability, continuous controllability, and compositionality, supporting pattern-specified minority generation and interpretation of minority characteristics. Quantitatively, LMF achieves more faithful minority distribution matching than prior methods across all benchmarks, reducing cFID and sFID by up to 30.2% and 34.2%, respectively. Furthermore, we demonstrate the utility of LMF generated minority samples beyond generation through improved downstream visuo-motor control.

Primary area:
generative models
Keywords:
minority sample generation, diffusion models

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_SuoHOpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Although the current literature can steer the generations to low-density regions, they do not explore what makes a sample minority. As knowing the underlying minority formation can be useful for interpretability as well as targeted synthesis, this paper explores minority generation through that lens. It enables factor-specified minority generation through explicitly learning the minority factors over the common mode achieved from common (majority) samples of dataset, considering each as independent LoRA update. This work evaluates the proposed method using ImageNets and Foods dataset at both factor as well as distribution level through visualizations and quantitative metrics, respectively.

Critical strengths
  1. The proposed method takes a novel approach to reaching the low density regions, which not only enables minority generation but also increases their “controllability”: clean composability of multiple factors, control over the level of each factor, and cross-class reusability of factors.

  2. The writing is easy to follow.

Critical weaknesses
  1. Although the ablations provide suitable starting points to consider, my main concern is that the proposed method includes several variables or hyperparameters to decide values of; so, this reflects complexity to reach a stable operating point if we want to use the method for a new dataset. Please refer to related question point 2.

  2. Given that a category is minority means less than 50% of the samples of a class will have minority characteristics. So considering the top 50% with higher minority scores would probably mean it is going to consist of some majority samples as well; more so, if the % of minority samples is lower for that class in that dataset. How that is going to affect the learning of the minority factors is not clear. An ablation, for instance, considering top-N% of samples with higher minority scores for minority factor learning would help resolve this concern.

  3. Based on performance in ablation, having explicit (separately trained) common-mode was better, and this method involves LoRA updates per factor as well. A discussion around compute cost (e.g. how training time compares to the baselines) would be helpful for the readers.

Questions for authors
  1. How is the intermediate layer-l selected in Section 3.4?
  2. The authors mention that “the selected hyperparameter settings provide a stable operating point for LMF” in Appendix I.4. To what extent do these hyperparameters generalize across different datasets? This will help address Weakness point 1 as well.
  3. With respect to “The factors with the K largest routing weights are selected..” in Section 3.5: I could find the ablation for M but not for K. Does K behave similarly to M, i.e. does higher K value also help reduce cFID and saturate after a certain point?
5 comments

We thank the reviewer for the comments. Below, we provide our point-by-point responses.

Q1: Selection of the intermediate layer l in Section 3.4

Thank you for this important question. We select layer 16 because it contains the richest semantic information among all denoiser layers. To verify this, we use pre-trained DINO features as a semantic reference and compute the linear CKA alignment between the features of each denoiser layer and DINO features on 1,000 images from ImageNet-1K. Representative results are shown below.

Layer148121416172024283238
CKA0.1290.1790.2310.3320.3430.3850.3840.3600.3760.2580.2140.191

The alignment increases from shallow layers, peaks at layer 16, and decreases toward deep layers, which confirms that layer 16 contains the richest semantic information. We have added the reason for choosing layer 16, together with the full layer-wise curve over all 38 layers, to Appendix L.3 (Fig. 12) of the revised paper.

Q2: Effect of majority samples in the top 50%

Thank you for this comment. We agree that the top 50% may include samples with relatively low minority scores, which may exhibit only one or two minority factors, or none. LMF accommodates such samples through weighted factor composition. Although the Top-K factors are selected for each sample, they are composed with their routing weights wiw_i (Eq. 6), so the gradient that a sample contributes to a factor ΔWi\Delta W_i is scaled by wiw_i. For samples with weak minority characteristics, the router can assign small weights, which limits their influence on factor learning.

Following the reviewer's suggestion, we conducted an ablation on ImageNet-10 that varies the top-N% of samples used for factor learning from 90% to 10%.

Factor-learning setcFID
top 90%30.13
top 80%27.75
top 70%27.13
top 60%27.60
top 50% (default)26.35
top 40%26.97
top 30%26.76
top 20%25.59
top 10%28.93

In the non-extreme range from the top 80% to the top 20%, cFID stays within 25.59–27.75, showing that LMF is robust to the majority samples included in the factor-learning set. A clear degradation appears only at the top 90%, where majority samples make up a large part of the training set and dilute the learning of minority factors, and at the top 10%, where too few samples remain to learn a diverse factor pool. We have added the corresponding experiments and discussion to Appendix L.7 and Fig. 13 (right).

Q3: Computational cost

Thank you for this meaningful suggestion. LMF requires a higher one-time training cost than the baseline methods, but it shows a clear advantage when a large number of samples are generated, owing to its fast sampling. We provide the computational cost analysis on ImageNet-10 below.

MethodTraining (GPU-h)Generation (s / image)Training + 30K images (GPU-h)cFID
SGMS30.007.3691.3340.87
BnS30.001.5242.6734.74
MP30.006.7486.1743.10
JEPA30.0027.12256.0044.90
LMF (Ours)59.721.5272.3926.35

From this table, we can see that the training/adaptation cost of the baseline methods is lower than that of LMF. However, LMF achieves the fastest generation, since it does not require additional optimization during sampling. Considering both training and generation costs, the total cost of LMF becomes lower than that of JEPA after about 4.2K generated images, and lower than those of SGMS and MP after about 18.3K and 20.5K images, respectively. BnS remains cheaper in total cost, but LMF achieves a substantially lower cFID (26.35 vs. 34.74) at the same generation speed. The complete figures of the amortized total cost and the efficiency-quality trade-off are provided in Appendix O (Fig. 17).

Q4: To what extent do the hyperparameters generalize across datasets?

We thank the reviewer for the attention to the generalization of the hyperparameters. The hyperparameters of LMF generalize well across datasets, and we do not perform dataset-specific tuning. All hyperparameters, including the learning rate, loss weights, pool size, and Top-K, are kept identical across ImageNet-10, Foods, and ImageNet-1K (as described in Sec. 4.1). The only exception is the number of training epochs on ImageNet-1K, which we reduce from 100 to 10. This is a common practice when training generative models on large-scale datasets, since training on ImageNet-1K for 100 epochs would be prohibitively expensive. Under this shared setting, LMF substantially outperforms all baselines on every dataset, reducing the cFID of the strongest baseline by 24.2%–30.2%.

Q5: Does a higher K value also help reduce cFID and saturate after a certain point?

We greatly appreciate the reviewer's constructive suggestion. Consistent with the reviewer's expectation, a larger K reduces cFID, and the improvement saturates at K = 4. We conducted an ablation of K on ImageNet-10, with all other settings unchanged. The results are shown below.

K123 (default)45
cFID35.6129.3026.3524.0724.00

A larger K allows more factors to participate in the composition of each sample, so that more minority characteristics can be modeled, which reduces cFID. The saturation arises because the factors are composed according to their routing weights. Even when four or more factors are allowed, the additionally selected factor may already have a small routing weight, so its contribution is limited. Therefore, the improvement saturates beyond K = 4.

We set K = 3 as an empirical choice, since a small K may be insufficient for images that contain multiple minority factors, while a large K increases the training cost. We did not tune K for the best result, and we acknowledge that a larger K may further improve LMF. We have added this ablation to the revised paper (Appendix L.6 and Fig. 14).

Thank you for the responses. I believe the overall revisions have helped improve the paper. It is also good to know that the hyperparameters of LMF generalize well across datasets and that LMF is robust to the majority samples included in the factor-learning set.

However, I have a question regarding the response of the authors on Q3 for Reviewer bGDa: The held-out minority test set evaluation is a useful addition and shows that LMF mostly maintains its relative advantage over the baselines even when compared to samples not seen in training. That said, I noticed a substantial increase in sFID (by almost 3x) across LMF as well as the baselines when switching from the training to the test reference set. I was wondering what might explain this discrepancy? Could the authors clarify whether it might be related to any differences in reference-set size or the composition of images in the held-out test set?

Reviewer_bGDaOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Learning Minority Factors (LMF), a framework for decomposable minority generation. Rather than treating minority generation as pattern-agnostic guidance toward low-density regions, the paper models a minority sample as a common-mode sample augmented with a composition of reusable minority factors. LMF represents these factors as a pool of LoRA updates and uses a detail-aware diffusion representation and a sparse routing module to select sample-specific factors. Anti-collapse regularization encourages broader factor utilization, while Siamese factor disentanglement encourages different factor compositions to produce distinct effects. Experiments on ImageNet and Food-101 suggest that the learned factors exhibit interpretable semantics, cross-class reuse, continuous strength control, and compositionality. LMF also obtains substantially improved cFID, sFID, and recall against the constructed minority reference distributions, and the generated samples provide modest improvements in downstream visuo-motor control and image classification.

Critical strengths

The problem is novel and interesting, extending minority generation toward interpretable and controllable factors. The method is clearly designed, and the qualitative results on factor control and composition are convincing. LMF achieves strong quantitative improvements across multiple datasets, supported by useful ablation studies.

Critical weaknesses

The paper defines the top 50% of images by reconstruction error as minority samples. However, these images may simply be difficult for SD1.5 to reconstruct rather than genuinely rare. LMF also appears to be trained on the same minority distribution used for FID evaluation, which may favor the proposed method. In addition, factor interpretability and disentanglement are mainly supported by selected qualitative examples rather than a systematic evaluation of all factors. The downstream experiments do not report variance or statistical significance.

Questions for authors

Are the minority reference images used for FID evaluation also used to train LMF? Could the authors evaluate on a held-out test set? Why is the top 50% defined as the minority subset? Are the results stable under other thresholds? How many of the 64 factors have clear and non-redundant semantics? Can the authors directly show that activating a learned factor increases the minority level of generated samples?

6 comments

We thank the reviewer for the comments. Below, we provide our point-by-point responses.

Q1: Do high reconstruction errors reflect genuine rarity?

A1: We thank the reviewer for raising this concern. High reconstruction errors are indeed closely related to genuine rarity. Prior work [1] has theoretically established a connection between diffusion reconstruction error and data likelihood through an ELBO-derived lower bound, suggesting that low-density samples tend to exhibit larger reconstruction errors. To further validate this relationship, we conduct an additional experiment on the cocktail shaker class of ImageNet-1K. We extract the DINO and ViT features of all samples, rank the samples by their distances to the class centroid in each feature space, and divide them into ten percentile groups. The table below reports the mean and standard deviation of the minority scores in each group.

Distance-to-Center PercentileMinority Score (DINO)Minority Score (ViT)
0–10% (Nearest)0.3737 ± 0.10820.3771 ± 0.1086
10–20%0.4025 ± 0.11310.3978 ± 0.1236
20–30%0.3995 ± 0.12620.4087 ± 0.1231
30–40%0.4151 ± 0.12760.4314 ± 0.1278
40–50%0.4399 ± 0.14390.4439 ± 0.1317
50–60%0.4525 ± 0.13590.4475 ± 0.1363
60–70%0.4619 ± 0.13420.4719 ± 0.1393
70–80%0.4639 ± 0.13530.4760 ± 0.1475
80–90%0.4739 ± 0.14850.4967 ± 0.1410
90–100% (Farthest)0.5579 ± 0.11460.4997 ± 0.1448

As shown, the minority score generally increases with the distance to the class centroid in both feature spaces. In particular, the farthest 10% of samples exhibit substantially higher minority scores than the nearest 10%, increasing from 0.3737 to 0.5579 in DINO and from 0.3771 to 0.4997 in ViT. The full curves of the minority score versus the distance-to-center percentile are provided in Appendix E (Fig. 8).

Q2: Reference Set Selection

A2: We appreciate the reviewer's attention to the evaluation protocol. We clarify that the minority reference images used for FID evaluation are indeed selected from the training set. However, we ensure a fair comparison across all methods.

The minority reference images are not exclusively seen by LMF during training. Although existing minority generation baselines primarily modify the sampling process, their underlying generators have also been trained or adapted on the target dataset, as stated in Appendix C.1, which includes the same minority reference images. Therefore, both LMF and all baseline methods have been exposed to these reference samples during training and are evaluated using the identical reference set, ensuring a fair comparison under a consistent evaluation protocol.

Moreover, using training images as FID references is a well-established practice in both minority generation and general generative modeling. For example, JEPA [2] selects minority reference images from the generator's training set for FID evaluation, and we follow a similar evaluation setting. Other works, including DDPM [3], ADM [4], and JiT [5], also select their reference sets from the training set.

References

[1] Diffusion Reconstruction-based Data Likelihood Estimation for Core-Set Selection (AAAI 2026).

[2] Beyond Generative Priors: Minority Sampling with JEPA-Guided Diffusion (ICML 2026).

[3] Denoising Diffusion Probabilistic Models (NeurIPS 2020).

[4] Diffusion Models Beat GANs on Image Synthesis (NeurIPS 2021).

[5] Back to Basics: Let Denoising Generative Models Denoise (CVPR 2026).

Q3: Could the authors evaluate on a held-out test set?

A3: We appreciate the reviewer's suggestion for this valuable additional experiment. We select 1,000 minority samples from the original dataset as a held-out test set, ensuring that these images are excluded from the training of both LMF and all baseline generators. We then evaluate LMF and all baseline methods on ImageNet-10 using this held-out minority reference set.

The quantitative results are presented below:

MethodcFIDsFIDRecallPrecisionCLIP ScorePick Score
BnS50.58228.840.14900.636080.1520.36
SGMS62.04236.480.13800.379579.1720.24
MP57.73226.670.18460.487978.9220.38
JEPA56.85236.300.18900.281178.5420.31
LMF40.58226.380.19900.605879.7820.24

As shown, LMF achieves the best cFID, sFID, and Recall, reducing cFID by 19.8% compared to the second-best method. This shows that LMF does not merely fit the training images, but also generalizes to minority samples not seen during training. We have included the held-out evaluation results and discussion in Appendix J and Table 6.

Q4: Why is the top 50% defined as the minority subset? Are the results stable under other thresholds?

Thank you for this question. We would like to clarify that the paper does not define the top 50% as the minority subset. The top 50% is used as the training set for minority factor discovery, as stated in Sec. 4.1 and Appendix A.1. We choose 50% so that the training set contains sufficient samples while still covering the minority samples. We acknowledge that the top 50% may include some common samples with weak minority characteristics. LMF accommodates such samples through weighted factor composition. The selected factors are composed with their routing weights wiw_i (Eq. 6), so the gradient that a sample contributes to a factor ΔWi\Delta W_i is scaled by wiw_i. For samples with weak minority characteristics, the router can assign small weights, which limits their influence on factor learning.

Following the reviewer's suggestion, we conducted a new experiment on ImageNet-10 that varies the factor-learning set from the top 90% to the top 10% of samples ranked by minority score.

Factor-learning setcFID
top 90%30.13
top 80%27.75
top 70%27.13
top 60%27.60
top 50% (default)26.35
top 40%26.97
top 30%26.76
top 20%25.59
top 10%28.93

In the non-extreme range from the top 80% to the top 20%, cFID stays within 25.59–27.75, showing that LMF is robust to the choice of threshold. Performance degrades only in extreme settings. When the top 90% is used, the large proportion of common samples dilutes the learning of minority factors. When only the top 10% is used, too few samples remain for training. We have added the corresponding experiments and discussion to Appendix L.7 and Fig. 13 (right).

Q5: How many of the 64 factors have clear and non-redundant semantics?

Thank you for this question. We address semantic clarity and non-redundancy separately.

Semantic clarity. Most factors have clear semantics, with 85.9%, 81.3%, and 68.8% of the factors receiving a clear semantic label on ImageNet-10, Foods, and ImageNet-1K, respectively. These numbers are counted from the post-hoc interpretation results, where GPT-5.6 Sol assigns a semantic label to each factor by comparing the images generated before and after activating it, and returns "Ambiguous semantics" when no clear semantic meaning can be identified. The detailed counts over all 64 factors on each dataset are shown below.

DatasetClear semanticsAmbiguous semantics
ImageNet-1055 / 64 (85.9%)9 / 64 (14.1%)
Foods52 / 64 (81.3%)12 / 64 (18.8%)
ImageNet-1K44 / 64 (68.8%)20 / 64 (31.3%)

Although most factors have clear semantics, the number of ambiguous factors on ImageNet-1K is relatively higher than that on ImageNet-10 and Foods. This is because ImageNet-1K contains the largest number of classes, and these classes differ substantially from one another. As a result, the same factor may be retrieved by many classes, so its visual effect varies across them and its semantics is harder to identify clearly.

However, ambiguous semantics does not mean that a factor is ineffective, since some minority patterns are inherently difficult to describe in natural language. To verify this, we activate each ambiguous factor individually and compare the minority score of generated samples before and after activation, using 20 matched pairs with identical prompts and seeds per factor. Factor off denotes the average minority score of samples generated without the factor, and Factor on denotes the average score after activating it.

Dataset# Ambiguous factorsFactor offFactor onGainFactors with increased score
ImageNet-1090.50000.5149+0.0149 (+3.0%)7 / 9
Foods120.47460.5049+0.0302 (+6.4%)12 / 12
ImageNet-1K200.52900.5547+0.0257 (+4.9%)16 / 20

As shown in the table, most ambiguous factors still increase the minority score (35 of 41 in total). These results confirm that the vast majority of the learned factors, including those without a clear semantic label, help raise the minority level of generated samples. We have included this discussion in Appendix K.

Non-redundancy. We evaluate the non-redundancy of the factors through the pairwise subspace similarity of the learned LoRA updates for all 64 factors at early, intermediate, and late layers. We have reported these results in Appendix G (Fig. 9) of our original submission. The off-diagonal similarities remain low in both the input and output subspaces, showing that each factor occupies a largely distinct parameter subspace and that the factors do not collapse onto shared update directions. This confirms that the learned factors are non-redundant.

Reviewer_QWK7OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Decomposable Minority Generation, a paradigm that models minority samples as compositions of discovered minority factors. The proposed LMF framework learns a pool of LoRA-based factors through a detail-aware modulator and a factor routing module, incorporating anti-collapse regularization and Siamese factor disentanglement. Experiments on ImageNet-1K and Foods demonstrate qualitatively coherent, reusable, and compositional factors. LMF substantially improves minority distribution matching over existing methods, achieving up to 30.2% lower cFID and 34.2% lower sFID.

Critical strengths
  1. The paper introduces an interesting perspective on minority generation by decomposing minority samples into reusable factors, providing a more interpretable and controllable approach to minority generation.
  2. The proposed framework is well-designed, combining a minority factor pool, factor routing, and regularization techniques to learn distinct minority factors.
  3. The experimental results are promising. The proposed LMF achieves substantial improvements in cFID, sFID, and Recall across multiple benchmarks.
Critical weaknesses
  1. The definition of minority factor is not clear enough. For example, in some categories, factor X may be very common; in others, it might be relatively rare. The minority factors should depend on a specific data distribution or category, not the visual concept itself.
  2. The main improvements from the proposed LMF might come from minority-specific training instead of decomposition. Existing methods focus on only modifying the sampling process on trained generative models, but LMF additionally trains a LoRA, factor pool, and router using the minority data. Training on minority data may itself reduce the FID.
  3. The evaluation mainly measures how well the model fits the minority distribution in the training data, but not its ability to generalize to unseen minority patterns. The results can be strengthened by constructing a held-out minority test set to evaluate performance on images not seen during training.
Questions for authors
  1. Does the advantage of LMF come from the discovery of meaningful minority factors, or from the addition of conditional parameter adaptation? How does LMF compare against a minority-specific LoRA fine-tuning baseline under matched training data and trainable parameters?
  2. Can the authors show how the minority score changes before and after factor activation?
  3. Why are the top 50% samples with higher minority scores used for minority factor learning? It seems unreasonable to me that half of the samples are referred to as the minority. I suggest adding experiments using different thresholds like 25%, 10%, or even 5%.
  4. Can LMF generate minority modes that were not seen during training by leveraging the factors it has learned, or does it primarily recombine existing, already-discovered minority modes?
4 comments

We thank the reviewer for the comments. Below, we provide our point-by-point responses.

Q1: The definition of minority factor

Thank you for this insightful comment. We agree that minority factors should depend on the data distribution of each category.

In Eq. 4, a minority sample is decomposed as xm≈xc⊕{fi1,…,fiK}x^m \approx x^c \oplus \{f_{i_1}, \dots, f_{i_K}\}. As stated in Line 200, xcx^c denotes the common sample of a specific class. Each factor fikf_{i_k} is therefore defined with respect to the same class and describes how the sample deviates from the common mode of that class. We acknowledge that the definition of minority factors was not stated clearly enough. Following the reviewer’s comment, we added a clarification in Lines 201--203 to explicitly state that minority factors are class-dependent.

Q2: Does the improvement come from minority-specific training or from decomposition? Comparison with a matched LoRA baseline

Thank you for raising this important point. First, we would like to clarify that the improvement of LMF comes from decomposition rather than from additional trainable parameters or minority-specific adaptation alone. In our ablation study (Tab. 2a), we already include a LoRA fine-tuning baseline under matched training data and trainable parameters, which is referred to as the static pool in the original paper. We reproduce Tab. 2a below with revised method names and two additional columns for better illustration.

Methods# ParametersDecompositioncFIDsFIDDenCovPreRec
LoRA Fine-tune×1✗44.9181.670.36980.54870.48800.0860
LoRA Fine-tune×64✗45.4777.380.40860.59430.56630.1087
Decomposable (LMF)×64✓26.3576.720.52310.87870.59090.1903

Compared with the matched LoRA fine-tuning baseline (second row), LMF improves all six metrics, reducing the cFID from 45.47 to 26.35 and raising the Coverage from 0.5943 to 0.8787. Without decomposition, the performance drops substantially even under the same training data and trainable parameters, which shows that decomposition is the key to the improvement. In the original paper, this ablation was described under the framework of the minority factor pool, which may have caused ambiguity. We have revised this part to highlight this experiment, renamed the baselines, and emphasized that it is an ablation of decomposition.

Second, although prior methods mainly focus on sampling intervention, all of them are applied to a generator that is first adapted to the target dataset using the same training data (Appendix C.1), which means that they have also seen the minority samples. In addition, all methods are evaluated on the same reference set. In other words, the baselines and LMF use the same data for both generator training and evaluation, which ensures a fair comparison.

Q3: Can the authors show how the minority score changes before and after factor activation?

Thank you for this suggestion. We compare the minority scores of the samples generated in our distribution-level evaluation (Tab. 1 and Tab. 4) with and without the learned factors, under identical prompts and initial noise. Factor off denotes the minority score computed on images generated without any learned factor, while Factor on (LMF) denotes the minority score computed on images generated with the learned minority factors activated. A higher score indicates stronger minority characteristics.

DatasetFactor offFactor on (LMF)Gain
ImageNet-100.51860.6218+0.1032 (+19.8%)
Foods0.50210.5814+0.0793 (+15.7%)
ImageNet-1K0.56160.6545+0.0929 (+16.4%)

From this table, we can see that activating the learned factors consistently increases the minority score on all three datasets, by 15.7%–19.8%, directly showing that the learned factors raise the minority level of generated samples. We have included this comparison in Appendix I and Fig. 10.

Q4: Why are the top 50% samples with higher minority scores used for minority factor learning?

Thank you for this question. We would like to clarify that the paper does not define the top 50% as minority samples. These samples are used as the training set for minority factor discovery, as stated in Sec. 4.1 and Appendix A.1. We acknowledge that the top 50% may include some common images with weak minority characteristics. LMF accommodates such samples through weighted factor composition. The selected factors are composed with their routing weights wiw_i (Eq. 6), so the gradient that a sample contributes to a factor ΔWi\Delta W_i is scaled by wiw_i. For samples with weak minority characteristics, the router can assign small weights, which limits their influence on factor learning.

Following the reviewer's suggestion, we conducted a new experiment on ImageNet-10 that varies the factor-learning set thresholds from the top 90% to the top 5% of samples ranked by minority score.

Factor-learning setcFID
top 90%30.13
top 80%27.75
top 70%27.13
top 60%27.60
top 50% (default)26.35
top 40%26.97
top 30%26.76
top 20%25.59
top 10%28.93
top 5%33.73

In the non-extreme range from the top 80% to the top 20%, cFID stays within 25.59–27.75, showing that LMF is robust to the choice of threshold. Performance degrades only in extreme settings. In particular, when only the top 5% of samples is used, cFID increases to 33.73, since too few samples remain for training. When the top 90% is used, cFID also increases to 30.13, since the large proportion of common images dilutes the learning of minority factors. We therefore use the 50% split as a balanced, tuning-free default rather than a deliberately selected optimum. We have added the corresponding experiments and discussion to Appendix L.7 and Fig. 13 (right).

Q5: Can LMF generate minority modes that were not seen during training?

Thank you for this interesting question. In our formulation (Eq. 4), a minority sample is formed by augmenting the common sample with a composition of minority factors, so we regard a minority mode as the pattern formed by such a factor composition. Under this definition, LMF can generate minority modes that do not exist in the training set, as long as the corresponding factor composition does not occur in the training data. However, if a minority mode refers to a minority factor that never appears in the training data, LMF cannot generate it, since our method cannot create factors that are absent from the training data.

Q6: Evaluation on held-out minority samples

Thank you for this valuable suggestion. To evaluate generalization to unseen minority samples, we construct a held-out minority test set on ImageNet-10. We select 1,000 minority samples from the original dataset according to the minority score (Eq. 3), and exclude them from the training of both LMF and all baseline generators. All methods are then evaluated against this held-out minority reference set.

MethodcFIDsFIDRecallPrecisionCLIPScorePickScore
SGMS62.04236.480.13800.379579.1720.24
BnS50.58228.840.14900.636080.1520.36
MP57.73226.670.18460.487978.9220.38
JEPA56.85236.300.18900.281178.5420.31
LMF (Ours)40.58226.380.19900.605879.7820.24

LMF achieves the lowest cFID and sFID and the highest Recall on the held-out set, reducing the cFID of the strongest baseline from 50.58 to 40.58. This shows that LMF does not merely fit the training images, but also generalizes to minority samples not seen during training. We have included the held-out evaluation results and discussion in Appendix J and Table 6.

Thank you for the detailed responses. The additional clarification and experiments have addressed most of my concerns. The response to Q5 is somewhat conceptual and does not provide empirical evidence that LMF can successfully generate meaningful unseen factor combinations. But I think this remains an open question rather than a major limitation. Overall, I will raise my rating.

We sincerely thank you for your positive feedback. We are glad that our additional clarifications and experiments have addressed most of your concerns. We agree that exploring unseen factor combinations is an interesting open question, and we will include it as a direction for future work in the revised paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.