Omnimodal Dataset Distillation via High-Order Proxy Alignment

Dataset distillation compresses large-scale datasets into compact synthetic sets while preserving training performance, but existing methods are largely restricted to single-modal or bimodal settings. Extending dataset distillation to scenarios involving more than two modalities, i.e., Omnimodal Dataset Distillation, remains underexplored and challenging due to increased heterogeneity and complex cross-modal interactions. In this work, we propose HoPA, a unified framework for omnimodal dataset distillation, which captures high-order cross-modal alignments via a compact proxy. By abstracting omnimodal alignment with a shared similarity structure, our method avoids the combinatorial complexity of pairwise modality modeling and enables scalable joint distillation across heterogeneous modalities. Theoretical analysis from the spectral perspective reveals the rationality of our proposed method. Extensive experiments on various benchmarks demonstrate that the proposed method achieves superior compression–performance trade-offs compared to existing competitors. For reproducibility, the source code is now available at https://anonymous.4open.science/r/HoPA-7C5E/.