acceptodds
Withdrawn from ICLR 2027

Feature-Pullback: Rethinking Multimodal Imbalance Through the Geometry of Feature Space

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Modality imbalance prevents multimodal models from fully exploiting heterogeneous information and has become a major issue that limits joint learning performance. Existing studies reduce disparities between modalities by regulating gradients, learning speeds, or contributions of the modality. These methods expect modalities to receive the same optimization signal but rarely examine the feature-space updates induced by this signal in modality representation spaces. To this end, we revisit modality imbalance from the perspective of featurespace geometry. We empirically show that optimization signals undergo modality-specific matrix-valued responses as they propagate through the encoders, inducing feature updates with different directions, scales, and reachability. First, a controlled modality-manifold experiment shows that changing only the geometry of modality representations yields an 8.86% performance improvement. Further local optimization analysis reveals that task signals must pass through modality-specific feature response operators before being converted into feature updates. It also establishes a correspondence between response distortion in feature space and pullback geometry in parameter space. Based on this insight, we propose Feature-Pullback, which uses feature probes and layer-wise Kronecker structures to obtain a scalable approximation. It corrects the backward updates of each modality without altering the forward network. Experiments on four audio-visual and image-text benchmarks show that the method consistently improves modality learning and fusion performance and achieves new state-of-the-art results on all benchmarks.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Learning, Modality Imbalance, Representation Learning, Feature Space

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_yGe5OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper investigates multimodal learning imbalance from the perspective of feature-space geometry. The authors argue that conventional approaches primarily focus on regulating gradients, learning rates, or modality contributions, while overlooking how parameter updates are translated into effective representation updates. The method is evaluated on four audio-visual and image-text benchmarks: Kinetics-Sound, CREMA-D, Sarcasm Detection, and Twitter-15. The authors report improvements over the selected multimodal rebalancing baselines across eight evaluation metrics. Additional analyses investigate unimodal branch performance, different fusion strategies, feature probe counts, and computational overhead.

Critical strengths
  1. The paper provides an interesting perspective by analyzing multimodal imbalance in terms of effective representation updates rather than merely comparing gradient magnitudes or relative modality performance.

  2. The proposed method reports improvements across four datasets covering audio, video, image, and text modalities. In addition to fusion accuracy, the paper investigates the performance of individual modality branches, which is relevant to its discussion of modality under-optimization.

  3. The narrative is clear and the language is well-articulated.

Critical weaknesses
  1. The experimental evaluation is not sufficiently representative of recent developments in multimodal imbalance learning. There is just only one method AMSS and AMSS+ (2025), several highly relevant peer-reviewed methods published in 2025–2026 are missing. Given these omissions, the paper's claim of achieving new state-of-the-art results on all four benchmarks is not sufficiently substantiated.

  2. Table 1 contains accuracy values that cannot be obtained from a single Top-1 evaluation under the stated test set sizes in APPENDIX. For example: CREMA-D: 76.93% with 744 test samples. The reported values do not satisfy this discrete constraint, even after rounding to two decimal places. They could arise from averaging multiple runs, evaluation splits, or nonstandard aggregation procedures. However, the paper does not specify the number of runs, report standard deviations, or sufficiently describe the aggregation protocol. Similar irregularities occur in some baseline entries, suggesting that the evaluation protocol requires clarification more generally.

  3. The baseline results in Table 1 appear to be identical to those reported in AMSS for 14 overlapping methods. However, the authors do not explicitly clarify whether these results were reproduced independently or directly adopted from the prior work.

  4. The computational overhead is considerable. Figure 3(c) reports 205.8 seconds per epoch for FP, compared with 41.6 seconds for Concat and 94.3 seconds for AMSS+. Thus, FP requires approximately 4.95 times the training time of Concat and 2.18 times that of AMSS+, with 46.25% additional GPU memory. The identical reported inference FLOPs do not imply comparable training computation. No matched-training-budget comparison is provided.

  5. The presentation of empirical evidence could be improved. Figure 4 compares FP primarily with AMSS+ on CREMA-D, although MMPareto achieves a higher accuracy in Table 1.

  6. The Introduction and Related Work sections provide an overly simplified discussion of existing multimodal imbalance learning methods. Moreover, the literature coverage is not sufficiently up to date, with several relevant works published in 2025–2026 being overlooked. This weakens the timeliness and completeness of the literature review and makes it difficult to clearly assess the novelty and positioning of Feature-Pullback relative to recent advances.

  7. The experimental section directly presents comparisons with numerous baseline methods in Table 1 without systematically introducing these methods or explaining the rationale behind their selection. Although some approaches are briefly mentioned in the Related Work section, the experimental evaluation lacks a clear description of the compared baselines, their relevant characteristics, and the experimental settings used for comparison. This makes it difficult to assess the representativeness and fairness of the comparisons and weakens the clarity and reproducibility of the experimental evaluation.

Questions for authors

Pease see weaknesses.

Reviewer_79ZNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates modality imbalance from the perspective of feature-space geometry and proposes Feature-Pullback (FP) to correct feature updates through Jacobian-based preconditioning. Experiments on four multimodal benchmarks demonstrate improvements over existing rebalancing methods.

Critical strengths
  1. The paper provides an interesting geometric perspective on modality imbalance, offering an alternative to conventional gradient-based rebalancing methods.

  2. FP achieves strong results across four benchmarks and different fusion strategies.

Critical weaknesses
  1. Existing methods typically perform gradient-based optimization in parameter space, whereas FP employs Euclidean steepest descent in feature space. The motivation for this choice remains insufficiently justified, and the paper lacks a theoretical explanation of why this optimization principle is particularly effective in alleviating modality imbalance.

  2. The related work focuses primarily on modality rebalancing methods, with limited discussion of existing second-order optimization approaches. The distinction between FP and methods such as K-FAC remains unclear. Consequently, it is difficult to determine whether the performance improvements arise from general optimization benefits or a mechanism specifically addressing modality imbalance.

Questions for authors
  1. Could the authors provide direct evidence that FP reduces feature-response distortion and brings actual feature updates closer to the theoretically desired updates?

  2. Given the substantial training overhead of FP, could the authors clarify whether this cost can be reduced and whether FP maintains its performance advantages under comparable training-time or computational budgets?

Reviewer_F8qUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies modality imbalance through the Jacobian-induced transformation of loss gradients into feature updates. It proposes a pullback-based gradient conditioner, approximating feature-sensitivity metrics with random probes and layerwise Kronecker factors. Experiments on four audio-visual and image-text datasets report improvements against standard fusion and modality-rebalancing baselines.

Critical strengths
  • The paper formulates a coherent feature-space target and connects it to an implementable preconditioner via Jacobian-based analysis, random probes, and Kronecker factors. The local derivations are generally well motivated, and the algorithm does not increase inference parameters or FLOPs.
  • Table 1 reports notable gains across four datasets, with evidence of improvements in both fused and individual modality branches (Figs. 2 and 4). The manuscript also evaluates fusion compatibility and reports training-time and memory costs (Fig. 3), which helps assess practicality.
Critical weaknesses
  • The Jacobian response Jᵀ, inverse-Jacobian projection, and Kronecker gradient conditioning closely relate to known NTK/function-space optimization, Gauss-Newton, and K-FAC ideas (Eqs. 6, 10, 13). The paper insufficiently distinguishes its new feature-probe metric from these methods, and Table 1 lacks structured-optimizer comparisons.
  • The claimed causal link between representation-response geometry and modality underlearning is not directly measured: Fig. 1 changes forward representations as well as backward behavior, and no response-spectrum or task-gradient alignment analysis validates the proposed mechanism. The gains on unimodal training (Fig. 3b) also admit the alternative explanation that FP is a generally useful preconditioner rather than a specific solution to modality imbalance.
Questions for authors

Q1. Which technical component is new relative to NTK/function-space dynamics, Jacobian-based Gauss-Newton, K-FAC, and the 2026 Error Whitening analysis (arXiv:2605.11316)? Can Table 1 include K-FAC/Shampoo-type comparators using the same backbones and training conditions? Q2. Can the authors ablate the activation factor A and feature-sensitivity factor S^f separately in Eq. (13)? Please also measure how closely the implemented updates match the projected feature-space reference derived in Sec. 3.2. Q3. Can Fig. 1 isolate backward response changes from the effect of modifying the forward representations with M_m? As currently designed, the manifold comparison does not establish a geometry-only causal explanation. Q4. Table 1 should report multiple seeds and uncertainty intervals, and compare optimization methods at equal total wall-clock budgets. FP requires substantially more time and GPU memory per epoch (Fig. 3c; Appendix Table 3). Q5. Can the paper measure the feature-response spectra, effective rank or condition number, and alignment between task gradients and actual feature updates for each modality? Correlating these measurements with branch underlearning in Figs. 2 and 4 would test the central mechanistic claim. Q6. Can FP be applied to only the audio or only the video encoder, and can gains be stratified by the initial severity of modality imbalance? Q7. If a modality receives a negligible or zero feature-level task gradient, preconditioning alone cannot introduce new supervision for that branch (Sec. 3.2). Can the authors clarify which forms of modality under-learning are expected to be corrected by FP and which remain outside its scope?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.