acceptodds
Under review as a conference paper at ICLR 2027

Beyond Modality Level Competition: Frequency-Resolved Spectral Regulation for Multimodal Learning

Abstract

Modality competition is a persistent challenge in multimodal learning, where dominant modalities can suppress the effective learning of weaker ones. Existing methods typically assess and mitigate competition at the modality-level, overlooking heterogeneous competitive states within latent representations. By examining latent representations from a frequency-domain perspective, we find that modality preference varies substantially across spectral regions and can even reverse across frequency bands during optimization, exposing competitive heterogeneity that modality-level assessment can obscure and uniform regulation cannot effectively address. Motivated by this observation, we propose SpecGuide, a frequency-resolved method that mitigates modality competition at the spectral level. Specifically, SpecGuide derives band-wise preference from class-conditioned spectral discriminative margins, uses the resulting preference maps to guide adaptive spectral shaping, and encourages frequency-wise complementarity across modality-specific spectral responses. Extensive experiments on six benchmarks demonstrate that SpecGuide consistently achieves state-of-the-art performance, delivers gains across different backbones, and remains robust to missing and noise-corrupted modalities.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Imbalanced Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_oKqxOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies modality competition in multimodal learning from a frequency-domain perspective. The authors observe that modality preference is not necessarily consistent across different frequency bands, which may be overlooked by existing methods that assess and regulate modalities as a whole. Based on this observation, they propose SpecGuide, which estimates modality preference separately for different frequency regions and uses it to guide spectral modulation. The method also introduces a complementarity loss to encourage different modalities to capture complementary frequency information.

The authors provide theoretical analysis to motivate frequency-wise regulation and evaluate the method on six multimodal benchmarks. The results show improvements over existing methods across all six datasets. Ablation studies examine the effects of the proposed components, and additional experiments investigate different fusion strategies, modality corruption, and missing modalities.

Critical strengths
  1. The observation that modality preference can vary or even reverse across frequency bands is interesting. Most existing methods treat modality competition as a global issue, while this paper shows that the situation can be quite different within the same representation. This gives a clear motivation for studying modality competition at a finer granularity.
  2. The experimental evaluation is fairly convincing. SpecGuide outperforms the compared methods on all six datasets, and the ablation results show how the individual components contribute to the final performance. The comparisons with different modulation strategies are particularly useful, as they suggest that the gains come from preference-guided regulation rather than simply applying frequency-domain processing.
Critical weaknesses
  1. The assumption behind Eq. (3) needs further justification. The decomposition of global discriminability assumes weak correlations between frequency bands. However, the spectral descriptors in Eq. (50) are normalized using the total energy across all bands, which introduces dependencies between them. Although the theoretical and implemented descriptors are not explicitly identical, the paper does not establish whether the approximation holds for the actual FMP features.
  2. The formulation of PGSS is somewhat restrictive. In Eqs. (12)–(13), the sigmoid produces modulation weights between 0 and 1, meaning that spectral amplitudes can only be suppressed rather than enhanced. It is unclear why this design is sufficient for balancing modality competition. The authors should explain this choice and compare it with modulation functions that allow both amplification and suppression.
  3. The motivation for frequency-wise modality competition is not fully supported. The analysis in Section 2 uses Fisher discriminability, while FMP estimates modality preference through class-conditioned spectral prototypes. Both measure class separability, but neither directly shows whether one modality suppresses the learning of another. The observed preference reversals could simply reflect differences in discriminative feature learning. Moreover, the fixed radial partition with thresholds of 0.25 and 0.55 is not sufficiently justified. Other partitioning strategies, such as directional or sector-based regions, may produce different preference patterns. These issues make it difficult to determine whether the proposed frequency-wise competition is a general phenomenon or partly a result of the chosen spectral representation.
  4. The contribution and adaptability of frequency-wise preference guidance remain unclear. In Table 5, PGSS alone improves accuracy from 62.90% to 75.27%, while adding FMP only increases it to 76.21%. The full model reaches 77.96%, but the authors do not include a PGSS + FCG variant to isolate the contribution of FMP. Furthermore, FMP uses sample-wise preferences derived from ground-truth labels during training, whereas inference relies on averaged preference statistics. This means the preference guidance cannot adapt to individual test samples, even when modality quality changes. Together, these concerns make it difficult to assess whether frequency-wise preference estimation provides a substantial benefit beyond spectral modulation itself.
Questions for authors
  1. Eq. (3) assumes weak cross-band correlations, but the normalized spectral descriptors in Eq. (50) depend on the energy of all frequency bands. Can the authors clarify the relationship between the theoretical descriptors and those used in FMP? Have the cross-band covariance assumptions been verified on the actual learned representations?
  2. In Eqs. (12)–(13), PGSS uses sigmoid-based weights that can only attenuate spectral amplitudes. Why is suppression-only modulation preferred when the goal is to mitigate modality competition? Would allowing both amplification and suppression change the results, particularly for weaker modalities?
  3. How do the authors distinguish frequency-wise modality competition from differences in class discriminability across frequency bands? The current preference estimates do not directly measure whether one modality suppresses another during optimization. Also, how sensitive are the observed preference reversals to the fixed radial partition in Eqs. (47)–(48)? Would directional or alternative frequency partitions produce similar observations?
  4. Could the authors provide the PGSS + FCG result without FMP to better isolate the contribution of frequency-wise preference estimation? Additionally, how much performance is lost when sample-wise training preferences are replaced by averaged preference statistics at inference? This comparison would help determine whether FMP provides useful adaptive guidance beyond the spectral shaping performed by PGSS.
1 comment

Dear Reviewer oKqx,

Thank you again for your valuable comments and suggestions.

As the discussion period will close in approximately 48 hours, we would sincerely appreciate it if you could review our rebuttal and the additional experiments and clarifications provided in the revised manuscript.

We would be particularly grateful to know whether our responses have adequately addressed your concerns or if any issues remain that we could clarify before the discussion deadline.

We highly value your feedback and would welcome the opportunity for further discussion.

Thank you very much for your time and consideration.

Best regards,

The Authors

Reviewer_ydPmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SpecGuide, a frequency-resolved framework for mitigating modality competition in multimodal learning. By identifying heterogeneous modality preferences across frequency bands, the method introduces preference-guided spectral modulation and frequency-wise complementarity regularization. Although the proposed method achieves promising results across multiple benchmarks, concerns remain regarding the connection between its motivating observations and the proposed regulation mechanism.

Critical strengths

S1. The paper is well-written and logically organized. The observation of heterogeneous modality preferences across frequency bands provides an interesting perspective on modality competition.

S2. Extensive experiments across six benchmarks, including ablation and robustness studies, demonstrate the effectiveness and applicability of the proposed method across different multimodal settings.

Critical weaknesses

W1. Although the paper identifies heterogeneous modality preferences across frequency bands, it lacks direct analysis of how these competitive states change after applying SpecGuide. This weakens the connection between the motivating observations and the claimed mitigation of frequency-wise modality competition.

W2. The performance gains appear to be primarily driven by PGSS, while the additional improvement from FMP is relatively limited (Table 5), raising questions about the necessity of frequency-wise preference guidance.

W3. Despite evaluations on multiple benchmarks, comparisons with closely related frequency-domain multimodal methods remain limited, making it difficult to fully assess the methodological advantages of SpecGuide over recent spectral approaches.

Questions for authors

Q1. Can the authors demonstrate how SpecGuide changes frequency-wise modality competition before and after regulation?

Q2. Given the dominant gains from PGSS, can the authors further justify the necessity of FMP?

1 comment

Dear Reviewer ydPm,

Thank you again for your valuable comments and suggestions.

As the discussion period will close in approximately 48 hours, we would sincerely appreciate it if you could review our rebuttal and the additional experiments and clarifications provided in the revised manuscript.

We would be particularly grateful to know whether our responses have adequately addressed your concerns or if any issues remain that we could clarify before the discussion deadline.

We highly value your feedback and would welcome the opportunity for further discussion.

Thank you very much for your time and consideration.

Best regards,

The Authors

Reviewer_r3AeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces region-wise modality imbalance, which, compared with previous modality/sample-wise methods, is at a finer scale. It mitigates modality competition at the spectral level from a frequency-domain perspective. Experiments show its efficiency.

Critical strengths
  1. The motivation of region-wise modality imbalance is new and at a finer scale than previous works.
  2. It produces consistent effects on six datasets.
Critical weaknesses
  1. The theory mainly supports fine-grained regulation, but fine-grained regulation also has other implementation methods. It has not been proven that Fourier frequency bands are superior to fine-grained regulation methods such as spatial, channel, or other subspace partitioning. That is, why must it be done in the frequency domain? What is its theoretical necessity?
  2. It lacks a spatial attention control with the same granularity and the same preference guidance to prove the necessity and advantage of operating in the frequency domain.
  3. There are too few comparison methods; works from 2026 are absent, and the advantage is not large. The advantage in the comparison in Table 4 is also not obvious.
  4. In addition, the ablation experiment proves that phase is ineffective, so does this theory have a loophole?
Questions for authors

“Since such regulation acts through parameter updates, its influence on latent representations is indirect and takes effect only in subsequent forward passes, limiting its ability to correct competitive states already present within the current representations.” Is this statement too subjective? How can one prove that this indirect approach will not learn better representations?

1 comment

Dear Reviewer r3Ae,

Thank you again for your valuable comments and suggestions.

As the discussion period will close in approximately 48 hours, we would sincerely appreciate it if you could review our rebuttal and the additional experiments and clarifications provided in the revised manuscript.

We would be particularly grateful to know whether our responses have adequately addressed your concerns or if any issues remain that we could clarify before the discussion deadline.

We highly value your feedback and would welcome the opportunity for further discussion.

Thank you very much for your time and consideration.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.