- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces "Omnimodal Dataset Distillation", extending multimodal dataset distillation beyond the conventional bimodal setting to scenarios involving three or more modalities. The authors propose HoPA (High-order Proxy Alignment), which represents the joint relationships among multiple modalities using a compact instance-level spectral proxy: the principal right singular vector of the multimodal representation matrix. This construction is motivated by the optimal rank-1 approximation of the corresponding within-instance Gram matrix. The proxy is incorporated into a trajectory-matching framework through two complementary objectives: a within-instance spectral alignment objective that promotes the leading singular direction, and a between-instance similarity-preservation objective based on proxy similarities and a learnable instance-level similarity matrix. The paper further provides theoretical analyses of relational fidelity and local proxy stability. Experiments on five datasets covering three- and four-modality configurations show that HoPA generally matches or modestly improves average retrieval performance over adapted pairwise distillation baselines, while reducing total distillation time relative to repeatedly distilling individual modality pairs. The paper also evaluates cross-architecture transfer, objective ablations, pairwise-versus-omniwise distillation, proxy rank, and SVD's computational overhead.
Critical strengths
The main strengths of the paper are that (i) it formulates a timely and underexplored problem, extending multimodal dataset distillation beyond two modalities, and (ii) it proposes a simple, unified mechanism for handling multiple modalities, with theoretical analysis closely tied to the design.
First, the paper addresses a meaningful and relatively unexplored setting. Existing multimodal dataset-distillation methods focus on two modalities and pairwise alignment, whereas many current multimodal systems involve video, audio, text, depth, IMU, and other modalities. Defining omnimodal dataset distillation and handling three or more modalities within a single joint framework is a useful step. The evaluation spans video–audio–text, RGB–depth–text, and a four-modality setting with IMU.
Second, HoPA's core idea is simple and conceptually appealing, and its theory is directly connected to the design. Rather than maintaining separate constraints for every modality pair, HoPA summarizes each instance's dominant joint structure with a single spectral proxy: the principal right singular vector. This proxy aggregates modality embeddings based on their agreement with the shared direction, avoiding repeated pairwise optimization. The theoretical analysis relates the spectral objective to energy concentration in the leading direction, bounds the error of replacing all modality-pair similarities with the proxy similarity, and characterizes the proxy's sensitivity through the leading spectral gap. Together, these results provide a principled motivation for the rank-1 proxy and clarify when it is expected to work well.
Critical weaknesses
My main concerns are that (i) the experimental protocol is close to saturation, so the reported improvements are too small to establish the method's benefit, and (ii) the methodological contribution is limited, while the central claims about higher-order structure and scalability are not directly tested. Together, these issues lead to my current rating.
First, the experimental protocol leaves very little room to distinguish methods. The experiments freeze pretrained ImageBind encoders, optimize only modality-specific projection heads, and synthesize embeddings rather than raw multimodal inputs. Under this setting, a random coreset already recovers most of the full-data performance: on MSR-VTT, Random with 500 samples reaches 21.2 Avg. R@1 versus 22.6 for full data (about 94%), and on VGGSound-S it reaches 12.8 versus 13.7 (about 93%). Several distillation baselines even fall below Random, such as MTT-VL and TESLA on Ego4D-S and RepBlend on SUN-SPOT. Within this narrow margin, HoPA's improvements are small and often within the standard deviation. It ties LoRS on MSR-VTT with 100 samples (21.6 vs. 21.6), and other margins are only 0.1–0.4 points (e.g., 13.4 vs. 13.3 on VGGSound-S, 23.54 vs. 23.25 on SUN-SPOT). This is consequential because the evaluation cannot clearly separate the quality of the distilled data from run-to-run variation. It also provides limited evidence for the broader claim of general omnimodal dataset distillation beyond a fixed pretrained embedding space.
Second, the methodological contribution appears limited, and the claimed advantages of the omnimodal formulation are not directly verified. Lemma 1 and the nonnegativity assumption are adopted from Liu et al. (2026). That work already optimizes the dominant singular value of the multimodal representation matrix through a softmax loss over singular values, together with instance-wise contrastive regularization on the leading eigenvectors. HoPA's within-instance objective L_M is essentially this loss, and its use of the leading singular vector as an instance-level proxy closely follows the same idea. The between-instance objective explicitly follows LoRS. The main contribution therefore appears to be combining these existing components inside trajectory matching. At the same time, the evaluation does not directly test the two benefits that motivate this combination. All downstream metrics are pairwise retrieval scores, so it remains unclear whether HoPA preserves information that is useful only when three or more modalities are used jointly. The pairwise-versus-omniwise comparison (Table 9) does not resolve this either, because its pairwise variants often fall below Random (e.g., 18.11 vs. 21.2 on MSR-VTT with 500 samples), suggesting they may not be well tuned. In addition, the paper examines scalability with the number of modalities only on a single four-modality dataset, without a controlled experiment that varies modality cardinality on the same instances. This is consequential because the main motivation for the proposed formulation remains unsupported.
Questions for authors
- Clarification on the scope of the proposed distillation setting and the cross-architecture protocol. The experiments use pretrained ImageBind encoders, freeze the backbone networks, optimize only modality-specific projection heads, and directly synthesize modality embeddings rather than raw multimodal inputs. Could the authors clarify whether the intended contribution should primarily be interpreted as embedding-level omnimodal dataset distillation within a pretrained multimodal representation space, or whether HoPA is intended to support the broader claim of general omnimodal dataset distillation?
Given that a random coreset already recovers roughly 90–94% of full-data performance under this protocol, could the authors also evaluate a less saturated setting? For example, more of the network could be trained, or a setting with a larger gap between Random and Full could be used.
Relatedly, in the cross-architecture experiment, how exactly are the synthesized ImageBind embeddings used to train the projection heads of LanguageBind and OmniBind, given that these models have different embedding spaces? Have the authors also evaluated HoPA when the source representation space itself is changed, for example by performing distillation independently with LanguageBind or OmniBind instead of ImageBind? Such an experiment would help determine whether the method is robust to the choice of pretrained multimodal backbone or relies substantially on the alignment geometry provided by ImageBind.
- Empirical evidence for scalability with the number of modalities. Avoiding the combinatorial growth of pairwise modality interactions is a central motivation of HoPA, yet most experiments involve three modalities and the four-modality evaluation is restricted to Ego4D-S. Could the authors provide a controlled experiment in which modality cardinality is systematically varied while keeping the underlying data fixed?
For example, Ego4D-S already contains video, audio, text, and IMU modalities. A comparison using subsets with k = 2, 3, and 4 modalities from the same underlying instances would allow reporting retrieval performance, wall-clock time, peak memory, and total synthetic parameter count as a function of k. Such a result would directly test whether the practical advantage of omniwise distillation becomes more pronounced as modality cardinality increases. It would also be helpful to explain why HoPA underperforms LoRS on several IMU-related directions in Table 8.
- Empirical validation of the assumptions behind the spectral proxy. The relational-fidelity analysis assumes that the within-instance Gram matrix is entrywise nonnegative, while the approximation bound becomes favorable when the spectral residual is small. The local-stability result additionally depends on a sufficiently separated leading singular direction. Could the authors report empirical statistics for these quantities? In particular, it would be useful to report:
the fraction of negative off-diagonal entries in the within-instance Gram matrices; the distribution of the spectral residual ρ_i = 1 - σ_{1,i}² / k; the distribution of the leading spectral gap γ_i = σ_{1,i}² - σ_{2,i}².
Ideally, these statistics could be reported separately for real-data representations used to construct expert trajectories and for synthetic representations during distillation. This would help establish whether the regimes assumed by the theoretical analysis actually occur in practice.
- Evaluation requiring joint use of three or more modalities. The paper motivates HoPA as a method for preserving high-order structure among more than two modalities, but all reported downstream metrics are pairwise cross-modal retrieval scores. Could the authors provide an evaluation in which three or more modalities must be used jointly at inference time? For example, this could involve retrieval with a multimodal query, a fusion-based classification or recognition task, or another downstream setting where complementary evidence from three or more modalities is required simultaneously. Such an experiment would help distinguish the benefit of preserving genuinely joint multimodal structure from improvements in the average quality of pairwise alignments.
In addition, regarding Table 9, the 3-pair, T-bind, and V-bind variants often perform below the Random coreset. Could the authors clarify how these variants were tuned and whether they include L_M? Without this, it is difficult to attribute the gap to the omniwise formulation itself.
- Clarification on budget matching and efficiency. Appendix C.2 states that, for a budget of N, each pairwise baseline independently synthesizes N samples for every modality pair, whereas HoPA synthesizes N multimodal tuples in a single joint run. Could the authors report the corresponding total number of optimized synthetic parameters or modality embeddings, storage requirements, peak memory, and total compute for each method under the current protocol?
The present setup appears to give pairwise approaches a larger aggregate synthetic-data budget while also requiring multiple independent distillation runs. Per run, HoPA appears slower than the baselines (e.g., 42.8 minutes versus about 26 minutes per LoRS run on MSR-VTT with 100 samples). An equal-total-storage, equal-synthetic-parameter, or equal-compute comparison would therefore help separate the intrinsic efficiency of the omniwise formulation from the efficiency gained simply by avoiding repeated runs of a bimodal method.
- Positioning relative to prior omnimodal alignment and multimodal condensation methods. The within-instance objective L_M and the use of the leading singular vector as an instance-level representation closely follow Liu et al. (2026), and L_wBCE follows LoRS. Could the authors clarify which components of HoPA are new relative to these works, beyond applying them within trajectory matching for dataset distillation?
The paper also claims to be the first dataset-distillation framework explicitly targeting datasets with more than two modalities. It would be helpful to position this claim relative to closely related multimodal condensation methods such as ImageBindDC (Min et al., AAAI 2026). ImageBindDC also performs condensation in a shared multimodal representation space and introduces uni-modal, cross-modal, and joint-modal alignment objectives, although its experiments are limited to bimodal combinations (audio–image, depth–text, and audio–text). Could the authors clarify whether HoPA's central novelty claim is specifically the simultaneous joint distillation of k > 2 modalities, rather than joint-modal condensation more generally?
11 comments
Q1: The scope of the experiments.
Clarification on the scope of the proposed distillation setting and the cross-architecture protocol.
HoPA is a general framework for omnimodal dataset distillation; it's not tied to ImageBind or restricted to embedding-level distillation. HoPA’s spectral proxy and trajectory-matching objective operate on normalized modality representations of a common dimension. Different encoders can supply these representations. For computational efficiency, our main experiments follow RepBlend’s embedding-level protocol. We freeze the ImageBind backbones, train modality-specific projection heads, and optimize the modality embeddings. In Table 3, we optimize video frames, audio filterbanks, and text hidden states through ImageBind, then reconstruct inputs for the target models. We will distinguish these two protocols explicitly in the revision.
Could the authors also evaluate a less saturated setting?
We repeated the MSR-VTT experiment at with frozen ImageBind backbones and randomly initialized heads. We use random initialization for expert-buffer training; the resulting trajectories are used for distillation, and downstream evaluation trains newly initialized heads. LoRS is distilled independently for the three modality pairs. Each seed's test recall is averaged over six retrieval directions; the table below reports mean ± standard deviation over three seeds:
| Training data | R@1 (%) | R@5 (%) | R@10 (%) |
|---|---|---|---|
| Random | 2.306 ± 0.136 | 8.229 ± 0.516 | 13.319 ± 0.719 |
| LoRS | 4.507 ± 0.118 | 14.979 ± 0.435 | 23.792 ± 0.235 |
| HoPA | 5.097 ± 0.142 | 16.722 ± 0.167 | 25.924 ± 0.335 |
| Full | 19.972 ± 0.223 | 43.028 ± 0.173 | 54.500 ± 0.327 |
With randomly initialized heads, Random reached 11.5% of full-data R@1 on MSR-VTT, compared with roughly 90% to 94% in the main tables. In this setting, HoPA achieved higher mean R@1, R@5, and R@10 than both Random and LoRS. These results show that HoPA's benefit extends beyond the near-saturated pretrained-head regime.
Relatedly, in the cross-architecture experiment, how exactly are the synthesized ImageBind embeddings used to train the projection heads of LanguageBind and OmniBind, given that these models have different embedding spaces?
In Table 3, LanguageBind and OmniBind receive reconstructed video, audio, and text. Each model computes its own embeddings from those inputs. In this experiment on VGGSound-S with 200 synthetic instances, we optimize preprocessed video frames, audio filterbanks, and text hidden states through ImageBind. To transfer them to target models, we reconstruct each modality: video frames are denormalized, clipped, quantized, and stored as FFV1 video; audio is approximately reconstructed using inverse normalization, a mel pseudo-inverse, and 32 Griffin-Lim iterations; text is retrieved as the nearest sentence from the training corpus using the optimized hidden states. Each target model then applies its own preprocessing, tokenizer, and frozen encoders to these inputs before training its projection heads. Because each target model encodes the reconstructed inputs itself, transfer requires no coordinate mapping between its embedding space and ImageBind’s.
Have the authors also evaluated HoPA when the source representation space itself is changed, for example by performing distillation independently with LanguageBind or OmniBind instead of ImageBind?
We repeated the MSR-VTT experiment with LanguageBind as the source backbone, using the same random-head protocol. We evaluated on MSR-VTT at with three seeds. The results are as follows:
| Training data | R@1 (%) | R@5 (%) | R@10 (%) |
|---|---|---|---|
| Random | 1.410 ± 0.043 | 4.729 ± 0.130 | 7.924 ± 0.387 |
| LoRS | 3.236 ± 0.052 | 10.826 ± 0.249 | 17.028 ± 0.264 |
| HoPA | 3.611 ± 0.345 | 11.701 ± 1.244 | 18.361 ± 1.622 |
| Full | 11.299 ± 1.565 | 25.764 ± 2.764 | 33.465 ± 2.849 |
With LanguageBind as the source backbone, HoPA achieved mean R@1 of 3.61, compared with 3.24 for LoRS and 1.41 for Random. We observed the same ordering with ImageBind.
Q2: Empirical evidence for scalability with the number of modalities.
Could the authors provide a controlled experiment in which modality cardinality is systematically varied while keeping the underlying data fixed?
We kept the Ego4D-S instances fixed and varied the number of modalities from two to four at , measuring the mean R@1 and total distillation time. The results demonstrate the accuracy–time trade-off as modality cardinality increases:
| k | Modalities | HoPA R@1 | LoRS R@1 | HoPA time (min) | LoRS total (min) | Time ratio |
|---|---|---|---|---|---|---|
| 2 | VT | 16.50 ± 0.00 | 15.52 ± 1.72 | 15.08 | 11.70 | 0.78× |
| 3 | VTA | 10.40 ± 0.33 | 8.87 ± 0.62 | 19.58 | 33.31 | 1.70× |
| 3 | VTI | 7.80 ± 0.46 | 7.22 ± 0.78 | 19.72 | 35.40 | 1.80× |
| 4 | VTAI | 6.42 ± 0.12 | 6.05 ± 0.35 | 22.31 | 68.52 | 3.07× |
With fixed instances, HoPA achieves higher mean R@1 than LoRS across all modality cardinalities. At , each method needs one distillation run. HoPA takes longer because it computes SVD operations as shown in Table 11. At and , the independent LoRS runs take longer in total than the single HoPA run. This reflects the structural difference: HoPA performs one joint distillation storing embeddings, while independent pairwise distillation requires runs storing embeddings in aggregate.
Why does HoPA underperform LoRS on several IMU-related directions in Table 8?
By Lemma 2, . A smaller coefficient derived from SVD gives the modality less weight in the proxy. We measured the modality-specific components of , averaged over all synthetic instances at representative expert-buffer epochs:
| Expert epoch | ||||
|---|---|---|---|---|
| 0 | 0.6104 ± 0.0003 | 0.4391 ± 0.0003 | 0.5177 ± 0.0003 | 0.2965 ± 0.0005 |
| 2 | 0.5760 ± 0.0005 | 0.4587 ± 0.0021 | 0.3872 ± 0.0053 | 0.2275 ± 0.0064 |
| 4 | 0.5804 ± 0.0008 | 0.4737 ± 0.0014 | 0.3977 ± 0.0061 | 0.2208 ± 0.0025 |
| 6 | 0.5833 ± 0.0001 | 0.4790 ± 0.0014 | 0.4022 ± 0.0035 | 0.2199 ± 0.0071 |
| 10 | 0.5855 ± 0.0006 | 0.4843 ± 0.0009 | 0.4110 ± 0.0025 | 0.2106 ± 0.0009 |
IMU had the smallest mean component of at every epoch we measured. By Lemma 2, the shared proxy gives less weight to its cross-instance relations. This may help explain HoPA’s weaker performance on several IMU retrieval directions in Table 8.
Q3: Empirical validation of the assumptions behind the spectral proxy.
Could the authors report empirical statistics for these quantities?
We measured the three quantities on MSR-VTT using the same fixed ImageBind projection heads for real instances and the synthetic set.
| Representations | Negative off-diagonal Gram entries | Mean residual | Mean leading gap |
|---|---|---|---|
| Real | 1.79% | 0.5061 | 0.6083 |
| Synthetic | 8.33% | 0.4279 | 0.9431 |
These statistics show that Most instances in both sets satisfy the entrywise nonnegativity condition in Theorem 3. The leading singular direction captures a larger share of spectral energy () in synthetic instances, averaging 57.2% versus 49.4% in real instances. Synthetic instances also have a larger mean leading gap (), indicating greater separation between the first two squared singular values on average.
Thank you for recognizing the importance of extending dataset distillation beyond two modalities, the simplicity of a unified proxy mechanism, and the theoretical analysis tied to its design.
W1: Concern about the experimental protocol.
We repeated the MSR-VTT experiment with randomly initialized projection heads, using ImageBind and LanguageBind separately as source backbones. In both settings, Random reached about 12% of full-data R@1, and HoPA outperformed independently distilled LoRS (Q1). In the transfer experiment, LanguageBind and OmniBind each used their own frozen encoders to process inputs distilled through ImageBind and shown in Table 3. These results show that HoPA’s retrieval advantage persists with randomly initialized heads and that the method applies beyond ImageBind’s embedding space.
Even in the near-saturated pretrained-head setting, HoPA achieves comparable retrieval performance at a lower total distillation cost. At matched embedding storage on MSR-VTT, one HoPA run achieves comparable mean R@1 to three independent LoRS runs with 37% less total distillation time (Q5).
W2: Contribution, higher-order structure, and scalability.
HoPA jointly distills k>2 modalities by learning relations between complete multimodal instances. For each pair of synthetic instances, HoPA matches the similarity of their spectral proxies to a single entry in a learned target matrix. This allows the same instance-level relation to supervise all k modality branches in one distillation run. We explain how this formulation adapts PMRL’s spectral alignment and LoRS’s bimodal similarity learning in Q6.
To test whether this joint supervision benefits retrieval when multiple modalities are used together, we evaluated queries that combine two modalities to retrieve a third (Q4). HoPA achieves a mean joint-query R@1 of 30.05, compared with 25.87 for 3-pair at the same triplet budget and 27.87 for the frozen model. We tuned each Table 9 variant independently using the same tuning procedure as for HoPA. Under this procedure, 3-pair still scored below Random on MSR-VTT at N=500 (Q4).
We also varied the number of modalities on the same Ego4D-S instances (Q2). HoPA takes longer than independent LoRS at two modalities, where both require one run. At three modalities, LoRS takes 1.70–1.80 times as long in aggregate, rising to 3.07 times at four modalities. HoPA also achieves higher mean R@1 in each tested configuration.
We will clarify the relation with PMRL and LoRS in the revision. We will also report the joint-query results and the controlled comparison across modality counts to show where this design improves joint alignment and where it saves distillation time.