LMF: Learning Minority Factors for Decomposable Minority Generation

Minority generation synthesizes high-fidelity, uncommon samples to bolster synthetic data utility and advance creative AI. While current methods can successfully steer generation trajectories toward low-density regions, they remain restricted to pattern-agnostic minority guidance and fail to uncover the mechanisms underlying minority formation. To remedy this limitation, we reconceptualize minority samples from a novel factor-level perspective and introduce Decomposable Minority Generation, a new paradigm that discovers minority factors within a data distribution and leverages them for synthesis. We instantiate this paradigm with LMF (Learning Minority Factors), comprising a minority factor pool, a detail-aware modulator, and a factor routing module, together with anti-collapse regularization and Siamese factor disentanglement. Experiments show that the learned factors exhibit coherent semantics, cross-class reusability, continuous controllability, and compositionality, supporting pattern-specified minority generation and interpretation of minority characteristics. Quantitatively, LMF achieves more faithful minority distribution matching than prior methods across all benchmarks, reducing cFID and sFID by up to 30.2% and 34.2%, respectively. Furthermore, we demonstrate the utility of LMF generated minority samples beyond generation through improved downstream visuo-motor control.