acceptodds
Under review as a conference paper at ICLR 2027

Omnimodal Dataset Distillation via High-Order Proxy Alignment

Abstract

Dataset distillation compresses large-scale datasets into compact synthetic sets while preserving training performance, but existing methods are largely restricted to single-modal or bimodal settings. Extending dataset distillation to scenarios involving more than two modalities, i.e., Omnimodal Dataset Distillation, remains underexplored and challenging due to increased heterogeneity and complex cross-modal interactions. In this work, we propose HoPA, a unified framework for omnimodal dataset distillation, which captures high-order cross-modal alignments via a compact proxy. By abstracting omnimodal alignment with a shared similarity structure, our method avoids the combinatorial complexity of pairwise modality modeling and enables scalable joint distillation across heterogeneous modalities. Theoretical analysis from the spectral perspective reveals the rationality of our proposed method. Extensive experiments on various benchmarks demonstrate that the proposed method achieves superior compression–performance trade-offs compared to existing competitors. For reproducibility, the source code is now available at https://anonymous.4open.science/r/HoPA-7C5E/.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
dataset distillation, omnimodal alignment, trajectory matching, spectral analysis

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_o82EOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces "Omnimodal Dataset Distillation", extending multimodal dataset distillation beyond the conventional bimodal setting to scenarios involving three or more modalities. The authors propose HoPA (High-order Proxy Alignment), which represents the joint relationships among multiple modalities using a compact instance-level spectral proxy: the principal right singular vector of the multimodal representation matrix. This construction is motivated by the optimal rank-1 approximation of the corresponding within-instance Gram matrix. The proxy is incorporated into a trajectory-matching framework through two complementary objectives: a within-instance spectral alignment objective that promotes the leading singular direction, and a between-instance similarity-preservation objective based on proxy similarities and a learnable instance-level similarity matrix. The paper further provides theoretical analyses of relational fidelity and local proxy stability. Experiments on five datasets covering three- and four-modality configurations show that HoPA generally matches or modestly improves average retrieval performance over adapted pairwise distillation baselines, while reducing total distillation time relative to repeatedly distilling individual modality pairs. The paper also evaluates cross-architecture transfer, objective ablations, pairwise-versus-omniwise distillation, proxy rank, and SVD's computational overhead.

Critical strengths

The main strengths of the paper are that (i) it formulates a timely and underexplored problem, extending multimodal dataset distillation beyond two modalities, and (ii) it proposes a simple, unified mechanism for handling multiple modalities, with theoretical analysis closely tied to the design.

First, the paper addresses a meaningful and relatively unexplored setting. Existing multimodal dataset-distillation methods focus on two modalities and pairwise alignment, whereas many current multimodal systems involve video, audio, text, depth, IMU, and other modalities. Defining omnimodal dataset distillation and handling three or more modalities within a single joint framework is a useful step. The evaluation spans video–audio–text, RGB–depth–text, and a four-modality setting with IMU.

Second, HoPA's core idea is simple and conceptually appealing, and its theory is directly connected to the design. Rather than maintaining separate constraints for every modality pair, HoPA summarizes each instance's dominant joint structure with a single spectral proxy: the principal right singular vector. This proxy aggregates modality embeddings based on their agreement with the shared direction, avoiding repeated pairwise optimization. The theoretical analysis relates the spectral objective to energy concentration in the leading direction, bounds the error of replacing all modality-pair similarities with the proxy similarity, and characterizes the proxy's sensitivity through the leading spectral gap. Together, these results provide a principled motivation for the rank-1 proxy and clarify when it is expected to work well.

Critical weaknesses

My main concerns are that (i) the experimental protocol is close to saturation, so the reported improvements are too small to establish the method's benefit, and (ii) the methodological contribution is limited, while the central claims about higher-order structure and scalability are not directly tested. Together, these issues lead to my current rating.

First, the experimental protocol leaves very little room to distinguish methods. The experiments freeze pretrained ImageBind encoders, optimize only modality-specific projection heads, and synthesize embeddings rather than raw multimodal inputs. Under this setting, a random coreset already recovers most of the full-data performance: on MSR-VTT, Random with 500 samples reaches 21.2 Avg. R@1 versus 22.6 for full data (about 94%), and on VGGSound-S it reaches 12.8 versus 13.7 (about 93%). Several distillation baselines even fall below Random, such as MTT-VL and TESLA on Ego4D-S and RepBlend on SUN-SPOT. Within this narrow margin, HoPA's improvements are small and often within the standard deviation. It ties LoRS on MSR-VTT with 100 samples (21.6 vs. 21.6), and other margins are only 0.1–0.4 points (e.g., 13.4 vs. 13.3 on VGGSound-S, 23.54 vs. 23.25 on SUN-SPOT). This is consequential because the evaluation cannot clearly separate the quality of the distilled data from run-to-run variation. It also provides limited evidence for the broader claim of general omnimodal dataset distillation beyond a fixed pretrained embedding space.

Second, the methodological contribution appears limited, and the claimed advantages of the omnimodal formulation are not directly verified. Lemma 1 and the nonnegativity assumption are adopted from Liu et al. (2026). That work already optimizes the dominant singular value of the multimodal representation matrix through a softmax loss over singular values, together with instance-wise contrastive regularization on the leading eigenvectors. HoPA's within-instance objective L_M is essentially this loss, and its use of the leading singular vector as an instance-level proxy closely follows the same idea. The between-instance objective explicitly follows LoRS. The main contribution therefore appears to be combining these existing components inside trajectory matching. At the same time, the evaluation does not directly test the two benefits that motivate this combination. All downstream metrics are pairwise retrieval scores, so it remains unclear whether HoPA preserves information that is useful only when three or more modalities are used jointly. The pairwise-versus-omniwise comparison (Table 9) does not resolve this either, because its pairwise variants often fall below Random (e.g., 18.11 vs. 21.2 on MSR-VTT with 500 samples), suggesting they may not be well tuned. In addition, the paper examines scalability with the number of modalities only on a single four-modality dataset, without a controlled experiment that varies modality cardinality on the same instances. This is consequential because the main motivation for the proposed formulation remains unsupported.

Questions for authors
  1. Clarification on the scope of the proposed distillation setting and the cross-architecture protocol. The experiments use pretrained ImageBind encoders, freeze the backbone networks, optimize only modality-specific projection heads, and directly synthesize modality embeddings rather than raw multimodal inputs. Could the authors clarify whether the intended contribution should primarily be interpreted as embedding-level omnimodal dataset distillation within a pretrained multimodal representation space, or whether HoPA is intended to support the broader claim of general omnimodal dataset distillation?

Given that a random coreset already recovers roughly 90–94% of full-data performance under this protocol, could the authors also evaluate a less saturated setting? For example, more of the network could be trained, or a setting with a larger gap between Random and Full could be used.

Relatedly, in the cross-architecture experiment, how exactly are the synthesized ImageBind embeddings used to train the projection heads of LanguageBind and OmniBind, given that these models have different embedding spaces? Have the authors also evaluated HoPA when the source representation space itself is changed, for example by performing distillation independently with LanguageBind or OmniBind instead of ImageBind? Such an experiment would help determine whether the method is robust to the choice of pretrained multimodal backbone or relies substantially on the alignment geometry provided by ImageBind.

  1. Empirical evidence for scalability with the number of modalities. Avoiding the combinatorial growth of pairwise modality interactions is a central motivation of HoPA, yet most experiments involve three modalities and the four-modality evaluation is restricted to Ego4D-S. Could the authors provide a controlled experiment in which modality cardinality is systematically varied while keeping the underlying data fixed?

For example, Ego4D-S already contains video, audio, text, and IMU modalities. A comparison using subsets with k = 2, 3, and 4 modalities from the same underlying instances would allow reporting retrieval performance, wall-clock time, peak memory, and total synthetic parameter count as a function of k. Such a result would directly test whether the practical advantage of omniwise distillation becomes more pronounced as modality cardinality increases. It would also be helpful to explain why HoPA underperforms LoRS on several IMU-related directions in Table 8.

  1. Empirical validation of the assumptions behind the spectral proxy. The relational-fidelity analysis assumes that the within-instance Gram matrix is entrywise nonnegative, while the approximation bound becomes favorable when the spectral residual is small. The local-stability result additionally depends on a sufficiently separated leading singular direction. Could the authors report empirical statistics for these quantities? In particular, it would be useful to report:

the fraction of negative off-diagonal entries in the within-instance Gram matrices; the distribution of the spectral residual ρ_i = 1 - σ_{1,i}² / k; the distribution of the leading spectral gap γ_i = σ_{1,i}² - σ_{2,i}².

Ideally, these statistics could be reported separately for real-data representations used to construct expert trajectories and for synthetic representations during distillation. This would help establish whether the regimes assumed by the theoretical analysis actually occur in practice.

  1. Evaluation requiring joint use of three or more modalities. The paper motivates HoPA as a method for preserving high-order structure among more than two modalities, but all reported downstream metrics are pairwise cross-modal retrieval scores. Could the authors provide an evaluation in which three or more modalities must be used jointly at inference time? For example, this could involve retrieval with a multimodal query, a fusion-based classification or recognition task, or another downstream setting where complementary evidence from three or more modalities is required simultaneously. Such an experiment would help distinguish the benefit of preserving genuinely joint multimodal structure from improvements in the average quality of pairwise alignments.

In addition, regarding Table 9, the 3-pair, T-bind, and V-bind variants often perform below the Random coreset. Could the authors clarify how these variants were tuned and whether they include L_M? Without this, it is difficult to attribute the gap to the omniwise formulation itself.

  1. Clarification on budget matching and efficiency. Appendix C.2 states that, for a budget of N, each pairwise baseline independently synthesizes N samples for every modality pair, whereas HoPA synthesizes N multimodal tuples in a single joint run. Could the authors report the corresponding total number of optimized synthetic parameters or modality embeddings, storage requirements, peak memory, and total compute for each method under the current protocol?

The present setup appears to give pairwise approaches a larger aggregate synthetic-data budget while also requiring multiple independent distillation runs. Per run, HoPA appears slower than the baselines (e.g., 42.8 minutes versus about 26 minutes per LoRS run on MSR-VTT with 100 samples). An equal-total-storage, equal-synthetic-parameter, or equal-compute comparison would therefore help separate the intrinsic efficiency of the omniwise formulation from the efficiency gained simply by avoiding repeated runs of a bimodal method.

  1. Positioning relative to prior omnimodal alignment and multimodal condensation methods. The within-instance objective L_M and the use of the leading singular vector as an instance-level representation closely follow Liu et al. (2026), and L_wBCE follows LoRS. Could the authors clarify which components of HoPA are new relative to these works, beyond applying them within trajectory matching for dataset distillation?

The paper also claims to be the first dataset-distillation framework explicitly targeting datasets with more than two modalities. It would be helpful to position this claim relative to closely related multimodal condensation methods such as ImageBindDC (Min et al., AAAI 2026). ImageBindDC also performs condensation in a shared multimodal representation space and introduces uni-modal, cross-modal, and joint-modal alignment objectives, although its experiments are limited to bimodal combinations (audio–image, depth–text, and audio–text). Could the authors clarify whether HoPA's central novelty claim is specifically the simultaneous joint distillation of k > 2 modalities, rather than joint-modal condensation more generally?

11 comments

Thank you for recognizing the importance of extending dataset distillation beyond two modalities, the simplicity of a unified proxy mechanism, and the theoretical analysis tied to its design.

W1: Concern about the experimental protocol.

We repeated the MSR-VTT experiment with randomly initialized projection heads, using ImageBind and LanguageBind separately as source backbones. In both settings, Random reached about 12% of full-data R@1, and HoPA outperformed independently distilled LoRS (Q1). In the transfer experiment, LanguageBind and OmniBind each used their own frozen encoders to process inputs distilled through ImageBind and shown in Table 3. These results show that HoPA’s retrieval advantage persists with randomly initialized heads and that the method applies beyond ImageBind’s embedding space.

Even in the near-saturated pretrained-head setting, HoPA achieves comparable retrieval performance at a lower total distillation cost. At matched embedding storage on MSR-VTT, one HoPA run achieves comparable mean R@1 to three independent LoRS runs with 37% less total distillation time (Q5).

W2: Contribution, higher-order structure, and scalability.

HoPA jointly distills k>2k>2 modalities by learning relations between complete multimodal instances. For each pair of synthetic instances, HoPA matches the similarity of their spectral proxies to a single entry in a learned target matrix. This allows the same instance-level relation to supervise all kk modality branches in one distillation run. We explain how this formulation adapts PMRL’s spectral alignment and LoRS’s bimodal similarity learning in Q6.

To test whether this joint supervision benefits retrieval when multiple modalities are used together, we evaluated queries that combine two modalities to retrieve a third (Q4). HoPA achieves a mean joint-query R@1 of 30.05, compared with 25.87 for 3-pair at the same triplet budget and 27.87 for the frozen model. We tuned each Table 9 variant independently using the same tuning procedure as for HoPA. Under this procedure, 3-pair still scored below Random on MSR-VTT at N=500N=500 (Q4).

We also varied the number of modalities on the same Ego4D-S instances (Q2). HoPA takes longer than independent LoRS at two modalities, where both require one run. At three modalities, LoRS takes 1.70–1.80 times as long in aggregate, rising to 3.07 times at four modalities. HoPA also achieves higher mean R@1 in each tested configuration.

We will clarify the relation with PMRL and LoRS in the revision. We will also report the joint-query results and the controlled comparison across modality counts to show where this design improves joint alignment and where it saves distillation time.

Q1: The scope of the experiments.

Clarification on the scope of the proposed distillation setting and the cross-architecture protocol.

HoPA is a general framework for omnimodal dataset distillation; it's not tied to ImageBind or restricted to embedding-level distillation. HoPA’s spectral proxy and trajectory-matching objective operate on normalized modality representations of a common dimension. Different encoders can supply these representations. For computational efficiency, our main experiments follow RepBlend’s embedding-level protocol. We freeze the ImageBind backbones, train modality-specific projection heads, and optimize the modality embeddings. In Table 3, we optimize video frames, audio filterbanks, and text hidden states through ImageBind, then reconstruct inputs for the target models. We will distinguish these two protocols explicitly in the revision.

Could the authors also evaluate a less saturated setting?

We repeated the MSR-VTT experiment at N=100N=100 with frozen ImageBind backbones and randomly initialized heads. We use random initialization for expert-buffer training; the resulting trajectories are used for distillation, and downstream evaluation trains newly initialized heads. LoRS is distilled independently for the three modality pairs. Each seed's test recall is averaged over six retrieval directions; the table below reports mean ± standard deviation over three seeds:

Training dataR@1 (%)R@5 (%)R@10 (%)
Random2.306 ± 0.1368.229 ± 0.51613.319 ± 0.719
LoRS4.507 ± 0.11814.979 ± 0.43523.792 ± 0.235
HoPA5.097 ± 0.14216.722 ± 0.16725.924 ± 0.335
Full19.972 ± 0.22343.028 ± 0.17354.500 ± 0.327

With randomly initialized heads, Random reached 11.5% of full-data R@1 on MSR-VTT, compared with roughly 90% to 94% in the main tables. In this setting, HoPA achieved higher mean R@1, R@5, and R@10 than both Random and LoRS. These results show that HoPA's benefit extends beyond the near-saturated pretrained-head regime.

Relatedly, in the cross-architecture experiment, how exactly are the synthesized ImageBind embeddings used to train the projection heads of LanguageBind and OmniBind, given that these models have different embedding spaces?

In Table 3, LanguageBind and OmniBind receive reconstructed video, audio, and text. Each model computes its own embeddings from those inputs. In this experiment on VGGSound-S with 200 synthetic instances, we optimize preprocessed video frames, audio filterbanks, and text hidden states through ImageBind. To transfer them to target models, we reconstruct each modality: video frames are denormalized, clipped, quantized, and stored as FFV1 video; audio is approximately reconstructed using inverse normalization, a mel pseudo-inverse, and 32 Griffin-Lim iterations; text is retrieved as the nearest sentence from the training corpus using the optimized hidden states. Each target model then applies its own preprocessing, tokenizer, and frozen encoders to these inputs before training its projection heads. Because each target model encodes the reconstructed inputs itself, transfer requires no coordinate mapping between its embedding space and ImageBind’s.

Have the authors also evaluated HoPA when the source representation space itself is changed, for example by performing distillation independently with LanguageBind or OmniBind instead of ImageBind?

We repeated the MSR-VTT experiment with LanguageBind as the source backbone, using the same random-head protocol. We evaluated on MSR-VTT at N=100N=100 with three seeds. The results are as follows:

Training dataR@1 (%)R@5 (%)R@10 (%)
Random1.410 ± 0.0434.729 ± 0.1307.924 ± 0.387
LoRS3.236 ± 0.05210.826 ± 0.24917.028 ± 0.264
HoPA3.611 ± 0.34511.701 ± 1.24418.361 ± 1.622
Full11.299 ± 1.56525.764 ± 2.76433.465 ± 2.849

With LanguageBind as the source backbone, HoPA achieved mean R@1 of 3.61, compared with 3.24 for LoRS and 1.41 for Random. We observed the same ordering with ImageBind.

Q2: Empirical evidence for scalability with the number of modalities.

Could the authors provide a controlled experiment in which modality cardinality is systematically varied while keeping the underlying data fixed?

We kept the Ego4D-S instances fixed and varied the number of modalities from two to four at N=100N=100, measuring the mean R@1 and total distillation time. The results demonstrate the accuracy–time trade-off as modality cardinality increases:

kModalitiesHoPA R@1LoRS R@1HoPA time (min)LoRS total (min)Time ratio
2VT16.50 ± 0.0015.52 ± 1.7215.0811.700.78×
3VTA10.40 ± 0.338.87 ± 0.6219.5833.311.70×
3VTI7.80 ± 0.467.22 ± 0.7819.7235.401.80×
4VTAI6.42 ± 0.126.05 ± 0.3522.3168.523.07×

With fixed instances, HoPA achieves higher mean R@1 than LoRS across all modality cardinalities. At k=2k=2, each method needs one distillation run. HoPA takes longer because it computes SVD operations as shown in Table 11. At k=3k=3 and k=4k=4, the independent LoRS runs take longer in total than the single HoPA run. This reflects the structural difference: HoPA performs one joint distillation storing kNkN embeddings, while independent pairwise distillation requires k(k−1)/2k(k−1)/2 runs storing k(k−1)Nk(k−1)N embeddings in aggregate.

Why does HoPA underperform LoRS on several IMU-related directions in Table 8?

By Lemma 2, σ1v1=∑m[u1]mzm\sigma_1 \mathbf{v}_1=\sum_m [\mathbf{u}_1]_m \mathbf{z}_m. A smaller coefficient derived from SVD gives the modality less weight in the proxy. We measured the modality-specific components of u1\textbf{u}_1, averaged over all synthetic instances at representative expert-buffer epochs:

Expert epochuV\mathbf{u}_VuT\mathbf{u}_TuA\mathbf{u}_AuI\mathbf{u}_I
00.6104 ± 0.00030.4391 ± 0.00030.5177 ± 0.00030.2965 ± 0.0005
20.5760 ± 0.00050.4587 ± 0.00210.3872 ± 0.00530.2275 ± 0.0064
40.5804 ± 0.00080.4737 ± 0.00140.3977 ± 0.00610.2208 ± 0.0025
60.5833 ± 0.00010.4790 ± 0.00140.4022 ± 0.00350.2199 ± 0.0071
100.5855 ± 0.00060.4843 ± 0.00090.4110 ± 0.00250.2106 ± 0.0009

IMU had the smallest mean component of u1u_1 at every epoch we measured. By Lemma 2, the shared proxy gives less weight to its cross-instance relations. This may help explain HoPA’s weaker performance on several IMU retrieval directions in Table 8.

Q3: Empirical validation of the assumptions behind the spectral proxy.

Could the authors report empirical statistics for these quantities?

We measured the three quantities on MSR-VTT using the same fixed ImageBind projection heads for real instances and the synthetic set.

RepresentationsNegative off-diagonal Gram entriesMean residual ρ\rhoMean leading gap γ\gamma
Real1.79%0.50610.6083
Synthetic8.33%0.42790.9431

These statistics show that Most instances in both sets satisfy the entrywise nonnegativity condition in Theorem 3. The leading singular direction captures a larger share of spectral energy (1−ρ1-\rho) in synthetic instances, averaging 57.2% versus 49.4% in real instances. Synthetic instances also have a larger mean leading gap (γ=σ12−σ22\gamma=\sigma_1^2-\sigma_2^2), indicating greater separation between the first two squared singular values on average.

Reviewer_wyrtOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper extends dataset distillation to settings with more than two modalities. HoPA represents each instance by the leading singular direction of its modality embeddings, then uses this proxy for joint trajectory matching with learned similarity targets. Experiments on five datasets suggest that a shared distilled set can retain much of the full-data retrieval performance at a lower cost than separate pairwise distillation.

Critical strengths

S1. The paper addresses a useful gap in extending bimodal distillation to several modalities. Distilling each modality pair separately creates multiple synthetic sets and does not directly provide one common set for joint training. HoPA instead maintains shared synthetic instances across modality branches for joint distillation.

S2. The proxy design connects within-instance alignment with discrimination between instances. The leading singular direction gives the modality branches a common instance-level reference, while learned similarity targets allow the supervision between instances to adapt during trajectory matching.

S3. The experiments support both the retrieval quality and the practical efficiency of joint distillation. Evaluation covers five datasets, including a four-modality setting. With 500 instances on MSR-VTT, average R@1 is 22.1 versus 22.6 with full data, while runtime is 57.6 minutes versus 120.4 for LoRS under the reported protocol. Comparisons with 3-pair, T-bind, and V-bind variants also help assess the joint objective.

Critical weaknesses

W1. The information loss introduced by the rank-1 proxy is not sufficiently characterized. Modalities can contain useful complementary information outside their dominant shared direction. The rank-2 ablation favors rank-1 on the tested datasets, but does not explain when the discarded directions are unimportant.

W2. The theoretical justification depends on conditions whose applicability during optimization remains unclear. The approximation relies on nonnegative within-instance similarities and a small spectral residual, while stability requires a separated leading singular value. Normalization alone does not ensure these conditions, particularly early in synthetic-data optimization. Appendix B.2 addresses the sign choice, but the behavior when the other conditions fail still needs explanation.

W3. The experiments do not yet establish reliable transfer beyond the reported frozen-encoder setting. Most results use frozen encoders and trainable projection heads, and the cross-architecture protocol does not clearly explain how ImageBind-derived synthetic embeddings enter LanguageBind and OmniBind. Table 3 also lacks variability estimates; for example, LanguageBind R@1 rises only from 5.87 to 5.93 over random data.

Questions for authors

Q1. How does the rank-1 proxy handle useful modality-specific information that is not aligned with the shared direction? What does the rank-2 comparison reveal about this trade-off?

Q2. Are the nonnegativity, residual, and eigengap conditions usually met during training? What happens to optimization when the leading singular values are close?

Q3. What exactly is transferred to each target model in Table 3, and how are the feature spaces matched? Please also report the variation across runs.

4 comments

Thank you for recognizing the value of extending distillation beyond two modalities with one shared synthetic set. We also appreciate your comments on the proxy design and the experiments that support retrieval quality and practical efficiency.

W1 / Q1: Rank-1 choice

How does the rank-1 proxy handle useful modality-specific information that is not aligned with the shared direction?

What Rank-1 compresses is the cross-instance score, not the synthetic representations. HoPA retains a full synthetic embedding for each modality and matches each branch to its own expert trajectory (Eq. 4). A modality-specific feature can therefore influence distillation through that branch’s parameter updates even when it leaves v1v_1 and the cross-instance score unchanged. The score aggregates the kk modalities by the components of u1u_1, which by Lemma 2 are the modality weights; a weakly aligned modality enters the proxy with a smaller weight and has its cross-instance relations supervised less directly.

The theory quantifies when the proxy score is a good substitute. In the fully aligned case (App. B.2, Eq. 16), every cross-instance modality pair has the same similarity, which the proxy reproduces exactly. Theorem 3 extends this to partial alignment under the nonnegative assumption: if ρi=1−σ1,i2/k\rho_i=1-\sigma_{1,i}^2/k is the fraction of instance ii's energy outside v1v_1, then the theorem guarantees the average absolute error of the proxy score is at most 2ρi+2ρj\sqrt{2\rho_i}+\sqrt{2\rho_j}.

What does the rank-2 comparison reveal about this trade-off?

For an instance with kk unit-norm modality embeddings, the squared singular values sum to kk. Retaining the second singular direction therefore captures an additional fraction σ22/k\sigma_2^2/k of the representation energy. Table 10 tests whether incorporating this additional within-instance variation into cross-instance supervision helps retrieval. Across all nine dataset–budget settings, the tested rank-2 proxy yielded lower mean R@1 than rank-1. One possible explanation is that the second direction may give weight to modality-specific details or noise that do little to distinguish matching from non-matching instances.

W2 / Q2: Theoretical conditions

Are the nonnegativity, residual, and eigengap conditions usually met during training?

We measured the three quantities on MSR-VTT using the same fixed ImageBind projection heads for real instances and the synthetic set.

RepresentationsNegative off-diagonal Gram entriesMean residual ρ\rhoMean leading gap γ\gamma
Real1.79%0.50610.6083
Synthetic8.33%0.42790.9431

These statistics show that Most instances in both sets satisfy the entrywise nonnegativity condition in Theorem 3. The leading singular direction captures a larger share of spectral energy (1−ρ1-\rho) in synthetic instances, averaging 57.2% versus 49.4% in real instances. Synthetic instances also have a larger mean leading gap (γ=σ12−σ22\gamma=\sigma_1^2-\sigma_2^2), indicating greater separation between the first two squared singular values on average.

What happens to optimization when the leading singular values are close?

When σ1\sigma_1 and σ2\sigma_2 are close, a small update to the modality embeddings can substantially rotate the proxy direction v1v_1. Gradients propagated through v1v_1 can consequently be amplified. Proposition 4 quantifies this sensitivity through the squared spectral gap γ=σ12−σ22\gamma=\sigma_1^2-\sigma_2^2. For γ>0\gamma>0, the local sensitivity of v1v_1 is bounded by 2σ1/γ2\sigma_1/\gamma.

To see how the spectral loss acts on this gap, consider one instance with σ1>σ2\sigma_1>\sigma_2. Let pj=exp⁡(σj/τ)/∑ℓexp⁡(σℓ/τ)p_j=\exp(\sigma_j/\tau)/\sum_\ell\exp(\sigma_\ell/\tau). The instance’s contribution to Eq. 5 is

ℓM=−log⁡p1=log⁡∑ℓexp⁡(σℓ/τ)−σ1τ.\ell_M=-\log p_1=\log\sum_\ell \exp(\sigma_\ell/\tau)-\frac{\sigma_1}{\tau}.

Differentiating with respect to the singular values gives

\frac{\partial\ell_M}{\partial\sigma_j}=\frac{p_j}{\tau}>0\quad(j\ge 2).$$ Thus, increasing $\sigma_1$ and decreasing $\sigma_2$ both lower the loss. Among the nonleading singular values, $\sigma_2$ has the largest positive derivative because it has the largest softmax weight. Since the normalized modality embeddings have fixed total spectral energy, consider a small transfer of energy $\varepsilon$ from the second direction to the first that preserves their ordering. This lowers $\ell_M$ and widens the gap to $$(\sigma_1^2+\varepsilon)-(\sigma_2^2-\varepsilon)=\gamma+2\varepsilon.$$ The spectral loss therefore favors the same separation that makes the proxy direction less sensitive to perturbations in *Proposition 4*.

W3 / Q3: Table 3 protocol and variation

What exactly is transferred to each target model in Table 3, and how are the feature spaces matched?

In Table 3, we transfer reconstructed video, audio, and text to the target models. It's not the same as what we do in the main experiment. In this experiment on VGGSound-S with 200 synthetic instances, we optimize preprocessed video frames, audio filterbanks, and text hidden states through ImageBind. To transfer them to target models, we reconstruct each modality: video frames are denormalized, clipped, quantized, and stored as FFV1 video; audio is approximately reconstructed using inverse normalization, a mel pseudo-inverse, and 32 Griffin-Lim iterations; text is retrieved as the nearest sentence from the training corpus using the optimized hidden states. Each target model then applies its own preprocessing, tokenizer, and frozen encoders to these inputs before training its projection heads. This pipeline allows the distilled set to enter each target model's native feature space without requiring direct coordinate matching across ImageBind, LanguageBind, and OmniBind embedding spaces.

Please also report the variation across runs.

We independently repeated the VGGSound-S cross-architecture evaluation and recomputed all three settings. For Random and Distilled, we report the mean and standard deviation over three runs. In this evaluation, LanguageBind achieved a mean R@1 of 5.63 with distilled inputs and 5.41 with randomly sampled real inputs.

ModelMethodR@1R@5R@10
ImageBindZero-shot12.425032.633344.4167
ImageBindRandom12.5333 ± 0.090133.5639 ± 0.101845.2083 ± 0.3114
ImageBindDistilled12.6361 ± 0.304633.5778 ± 0.263845.2528 ± 0.0668
LanguageBindZero-shot4.941719.858329.5333
LanguageBindRandom5.4056 ± 0.072821.4556 ± 0.379532.2944 ± 0.4385
LanguageBindDistilled5.6278 ± 0.145621.5694 ± 0.113832.3194 ± 0.2694
OmniBindZero-shot12.741735.058348.2167
OmniBindRandom12.7944 ± 0.024135.2278 ± 0.111048.4917 ± 0.1561
OmniBindDistilled13.0194 ± 0.214135.5028 ± 0.128148.6472 ± 0.0347

We will clarify the transfer protocol in the paper and update Table 3 to report the mean and standard deviation across three runs.

Dear Reviewer wyrt,

We have properly addressed all your concerns and provided clarifications on all confusing concepts. Could you please kindly re-evaluate our paper based on the current situation? If you have any further questions, we are also very glad to discuss them.

Thanks!

Authors

Dear Reviewer wyrt,

As today is the final day of the rebuttal period, we would like to kindly follow up on our responses. We have carefully addressed your concerns and provided additional clarifications. We would greatly appreciate it if you could review our responses and reconsider your assessment. If any concerns remain, we would be happy to discuss them further.

Thanks!

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.