acceptodds
Under review as a conference paper at ICLR 2027

Balancing Multimodal Learning via Functional Progress

Abstract

Multimodal learning often suffers from modality imbalance, where the joint optimization process is dominated by a single modality. Existing methods typically estimate modality imbalance from score disparities derived from prediction uncertainty or optimization statistics. However, due to distinct prediction uncertainty and learning dynamics across modalities, direct comparison of such scores may misinterpret intrinsic modality differences as progress gaps, leading to biased imbalance estimation. In this paper, we propose Function-Space Guided Multimodal Optimization (FGMO), which leverages a function-space progress signal to assess modality-wise optimization progress and coordinate optimization across modalities to alleviate modality imbalance. Specifically, we introduce Functional Progress Estimation (FPE) to measure each modality’s update-induced function-space response and calibrate it against a loss-aligned unimodal reference, producing a comparable progress signal. Based on this signal, Functional Response Control (FRC) redistributes modality-level function-space budgets and realizes the target responses through tensor-wise learning-rate adjustment. Theoretical analysis establishes a one-step target-contraction property of FRC under bounded controller-state mismatch, and extensive experiments demonstrate the effectiveness of FGMO across multiple multimodal benchmarks.

Primary area:
optimization
Keywords:
multimodal learning, modality imbalance, function-space optimization, relative functional progress, adaptive optimization

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Yr6SOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes FGMO, which introduces function progress estimation to produce a comparable progress signal (the RPF component) for each modality and adjusts the learning rate accordingly (the FRC component) to alleviate the modality imbalance problem. Theoretical analysis shows that FRC reduces modality optimization discrepancy.

Critical strengths
  1. The paper identifies a key research gap in existing modality imbalance estimation methods, which ignore the inherent differences in prediction uncertainty and learning dynamics across modalities. The teaser figure (Figure 1) vividly demonstrates and compares these aspects, making the motivation and advantages of the proposed model clear and intuitive.

  2. Rethinking modality imbalance in the function space and introducing a more comparable signal for modality rebalancing is both novel and interesting.

Critical weaknesses
  1. About the Experiment: The statistical reliability of the improvements in Table 1 is unclear. Several gains over the strongest baselines are modest(e.g, Sarcasm, NVGesture), yet variability is reported only for FGMO. Without corresponding baseline statistics, it is difficult to determine whether these gains consistently exceed run-to-run variation.

  2. About the Method: The interpretation of function-space response as optimization progress needs stronger justification. Equation (3) measures the magnitude of logit changes but does not capture whether those changes reduce the loss or improve predictive performance. A large response may reflect an unhelpful update, while a small response may occur near convergence.

Questions for authors
  1. Can you report variability for the strongest baselines under matched training seeds, and analyse the significance of the performance differences?

  2. Is RFP strongly correlated with subsequent loss reduction or performance gains? Please provide quantitative evidence supporting this relationship.

  3. Figure 3 shows that more frequent adjustments, such as K=5, lead to lower performance. I am really interested about the causes this degradation.

  4. The fusion method studied in this paper is late fusion, where unimodal logits are averaged. Can the proposed method be extended to early- or intermediate-fusion architectures, such as attention-based fusion methods?

8 comments
AuthorsOpenReview ↗

W1 & Q1. Statistical Reliability of the Modest Improvements and Baseline Variability

Thank you for your careful evaluation of our work and for your positive assessment of the motivation, function-space perspective, and RFP design. We respond to each of your questions below.

We added a paired statistical analysis using five matched training seeds and combined it with experiments across fusion architectures. These additions validate FGMO’s performance advantage from the perspectives of statistical reliability and architectural generalization.

A. Statistical Reliability under Matched Training Seeds

To determine whether FGMO’s performance improvements are affected by random initialization, we select the strongest existing baseline in terms of accuracy on each of six datasets and compare it with FGMO using five matched random seeds. All methods use the same backbone, initialization strategy, data split, training budget, and model-selection rule, while retaining each baseline’s original optimization mechanism.

We report the mean and standard deviation for each method, the paired accuracy improvement and its standard deviation, and a two-sided paired tt-test with a 95% confidence interval.

Table 1: Statistical comparison with the strongest accuracy baseline across six datasets.

DatasetStrongest Baseline (Mean ±\pm STD, %)FGMO (Mean ±\pm STD, %)Paired Δ\DeltaAcc. ±\pm SD (%)95% CI (%)pp-value
CREMA-DAUG 85.18±0.3185.18 \pm 0.3185.86 ±\pm 0.24+0.68±0.17+0.68 \pm 0.17[0.47,0.89][0.47, 0.89]0.000864
KSoundsRGM 73.14±0.2973.14 \pm 0.2975.70 ±\pm 0.23+2.56±0.20+2.56 \pm 0.20[2.31,2.81][2.31, 2.81]0.000009
VGGSoundAUG 52.98±0.2052.98 \pm 0.2053.44 ±\pm 0.15+0.46±0.13+0.46 \pm 0.13[0.30,0.62][0.30, 0.62]0.001381
TwitterAMSS+ 75.62±0.3375.62 \pm 0.3376.25 ±\pm 0.24+0.63±0.18+0.63 \pm 0.18[0.41,0.85][0.41, 0.85]0.001439
SarcasmAUG 85.13±0.1985.13 \pm 0.1985.49 ±\pm 0.15+0.36±0.12+0.36 \pm 0.12[0.21,0.51][0.21, 0.51]0.002570
NVGestureAUG 84.96±0.2784.96 \pm 0.2785.46 ±\pm 0.21+0.50±0.16+0.50 \pm 0.16[0.30,0.70][0.30, 0.70]0.002207

FGMO improves performance on all six datasets, with paired accuracy gains ranging from 0.36% to 2.56%. Even on Sarcasm, where the gain is relatively small, FGMO produces a stable positive improvement, showing that its advantage does not depend on a particular dataset or random seed. In addition, none of the 95% confidence intervals for the six paired improvements contain zero, and every pp-value is below 0.0083. The improvements therefore remain statistically significant after Bonferroni correction for multiple comparisons. These results demonstrate the statistical reliability of FGMO’s improvements over the strongest baseline on each dataset.

B. Performance across Different Fusion Architectures

Our method also outperforms existing baselines across different fusion architectures. In Appendix C.8 of the revised manuscript, “Generalization Across Fusion Architectures,” we added experiments with Early Fusion and an Intermediate Fusion architecture based on bidirectional Cross-Attention. We compare Vanilla, RGM, AUG, and FGMO on KSounds and Sarcasm. The complete experimental setup and further discussion are provided in our response to Q4.

Table 2: Performance comparison across different fusion architectures.

FusionMethodKS Acc. (%)KS mAP (%)Sar. Acc. (%)Sar. F1 (%)
EarlyVanilla64.6871.1381.7280.98
RGM71.1577.1283.2082.57
AUG71.6977.9483.7483.10
FGMO73.6380.0284.4383.86
IntermediateVanilla66.6473.0182.5181.87
RGM72.3479.0584.1783.52
AUG72.8779.6684.8184.26
FGMO74.8781.8985.3985.01
LateVanilla64.5571.6582.8682.01
RGM73.0980.8884.4983.97
AUG72.6379.0185.1084.58
FGMO75.6582.1085.4784.86

FGMO outperforms existing baselines in fused prediction under both Early and Intermediate Fusion, maintaining consistent advantages in accuracy, mAP, and F1. Thus, FGMO not only improves multimodal learning under the original Late Fusion design but also effectively coordinates optimization under feature-level concatenation and attention-based cross-modal interaction, further demonstrating the method’s clear advantages.

AuthorsOpenReview ↗

W2. Justification for Interpreting Function-Space Response as Optimization Progress

We apologize for the confusion. The following sentence in the original manuscript may have suggested that the function-space response itself is the optimization progress:

“FGMO characterizes modality-wise optimization progress by estimating the function-space response and calibrating it against a loss-aligned unimodal reference to obtain the function-space progress signal.”

This sentence appears in Section 3.2, Lines 189–191. What we intended to express is that the function-space response is the basic measurement of the function change induced by the current update, whereas RFP, obtained after calibration against the loss-aligned unimodal reference, is the signal used to characterize relative modality-wise optimization progress.

Thank you for this careful feedback. We revised the sentence to:

“FGMO estimates the function-space response and calibrates it against a loss-aligned unimodal reference to obtain RFP, which characterizes modality-wise optimization progress.”

The change is highlighted in red in the revised manuscript. Thank you again for helping us clarify this statement.

AuthorsOpenReview ↗

Q2. Quantitative Evidence on the Correlation Between RFP and Subsequent Loss Reduction or Performance Gains (PART 1/2)

Thank you for this important suggestion. To further verify whether RFP reflects the modality learning state and whether it is associated with subsequent optimization outcomes, we added two quantitative experiments to Appendix C.11 of the revised manuscript, “RFP Discrepancy and Learning Outcomes.” They examine the relationship of the RFP Gap with current unimodal prediction performance and with subsequent validation-loss reduction.

A. Relationship between RFP and Modality Prediction Performance

On the Sarcasm dataset, we compare independently trained unimodal models (Unimodal), Vanilla, a model without RFP normalization, and FGMO under controller gains λ∈{0.1,0.25,0.5,1.0,1.5,2.0,2.5,3.5,4.0}\lambda \in \{0.1, 0.25, 0.5, 1.0, 1.5, 2.0, 2.5, 3.5, 4.0\}. For each setting, we record the RFP Gap, the accuracies of the Image and Text branches, and the final fused accuracy to analyze the relationship between the RFP Gap and unimodal and multimodal prediction performance.

Table 3: Relationship between RFP discrepancy and modality-specific accuracy on Sarcasm.

MethodRFP Gap ↓\downarrowImage Acc. (%) ↑\uparrowText Acc. (%) ↑\uparrowMulti Acc. (%) ↑\uparrow
Unimodal–71.8181.36–
Vanilla0.8269.0780.6682.86
w/o RFP0.5869.7881.3283.48
λ=0.1\lambda=0.10.3271.1482.2185.32
λ=0.25\lambda=0.250.2970.9482.2585.26
λ=0.5\lambda=0.50.2371.3582.5185.39
λ=1.0\lambda=1.00.0971.6582.8185.47
λ=1.5\lambda=1.50.2471.1982.5585.15
λ=2.0\lambda=2.00.4470.9782.3284.88
λ=2.5\lambda=2.50.5970.2081.7184.41
λ=3.5\lambda=3.51.0069.3080.8282.97
λ=4.0\lambda=4.01.3168.7780.1082.48

The RFP Gap is strongly negatively correlated with the accuracies of the Image branch, Text branch, and fused prediction. As the RFP Gap decreases, both unimodal branches improve and the fused prediction improves accordingly. In particular, the initially under-optimized Image branch gradually approaches its independently trained performance, while the Text branch also benefits further. A Spearman correlation analysis across controller gains supports the same trend. Thus, the RFP Gap reflects not only relative optimization differences between modalities but also actual unimodal and fused prediction performance.

Reviewer_zkRcOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes FGMO to address modality imbalance through reference-calibrated function-space measurements. It estimates update-induced output changes and normalizes them against loss-aligned unimodal references. A KL regularized controller then reallocates modality budgets through tensor wise learning rates. Experiments cover six benchmarks, controlled interventions, and component ablations. The theoretical analysis establishes conditional one-step contraction of the assigned response targets.

Critical strengths
  1. Comparing each modality with its own loss-aligned reference accounts for modality-specific response scales, offering a useful alternative to directly comparing confidence or gradient magnitudes.

  2. FGMO achieves the best reported results across six datasets, including a 2.56-point accuracy gain on KSounds. Controlled interventions and Table 9 further isolate the value of function-space measurements under shared calibration and intervention rules.

Critical weaknesses
  1. FLeRM already introduces function-space estimation and reference-based learning-rate matching. Table 2 lacks a comparison that retains FPE and tensor-wise calibration while replacing the stateful controller with fixed shares or a memoryless rule. The contribution of the proposed allocation mechanism therefore remains unclear.

2, The full computational trade-off is unclear. Each unimodal reference requires 100 training epochs, but Figure 5(b) excludes this cost. Although references can be reused, neither end-to-end costs nor comparisons under matched total compute are reported, limiting assessment of practical value.

Questions for authors

1, Do Tables 4 and 6 evaluate budget realization and target contraction before or after the clipping operations in Appendix C.2? How frequently is clipping active? This affects how directly the theoretical analysis describes the implemented controller.

5 comments
AuthorsOpenReview ↗

W1. Unclear Contribution of the Proposed Allocation Mechanism (Stateful Controller vs. Fixed/Memoryless)

Thank you for this important suggestion. To isolate the contribution of FRC’s modality-budget allocation mechanism, we added an FRC Controller Ablation (Table 3) to Section 4.3 of the revised manuscript. We hold FPE, the loss-aligned unimodal reference, and tensor-wise learning-rate calibration fixed, and vary only the strategy used to allocate modality-level response budgets. This design isolates the performance gain attributable to the FRC control mechanism.

Specifically, we compare three strategies: (1) Fixed Allocation, which fixes the modality-budget share at 0.3, 0.5, or 0.7; (2) Memoryless Allocation, which dynamically allocates the budget using only the current RFP without retaining the previous allocation state; and (3) Stateful FRC, which uses the KL-regularized update in Eq. 6 to allocate the budget dynamically based on the current RFP and the budget share from the preceding adjustment event. All strategies use the same model, optimizer, reference trajectory, training setup, and tensor-wise learning-rate calibration.

Table 1: FRC controller ablation on Sarcasm and KSounds (accuracy, %).

StrategySarcasm MultiImageTextKSounds MultiAudioVideo
Fixed (0.3)82.9170.3480.1172.1051.2754.68
Fixed (0.5)83.2070.8681.8873.5451.7856.69
Fixed (0.7)83.4369.3281.5970.3952.6255.34
Memoryless84.8671.2882.4574.8252.7157.42
Stateful FRC85.4771.6582.8175.6553.1959.91

Dynamic budget allocation outperforms fixed allocation on both datasets, showing that different datasets require different modality-budget configurations and that preset fixed shares cannot adapt adequately to changing modality-optimization requirements during training. Moreover, Stateful FRC improves multimodal accuracy over Memoryless Allocation by 0.61% and 0.83% on Sarcasm and KSounds, respectively, while simultaneously boosting every unimodal branch. Thus, allocating budgets relying solely on the current RFP is insufficient; incorporating the historical allocation state via KL regularization better coordinates the continuous optimization process and yields additional performance gains.

In summary, when FPE, reference calibration, and tensor-wise learning-rate implementation are identical, Stateful FRC still achieves the best unimodal and multimodal performance. This confirms that FRC’s independent contribution lies not only in dynamically allocating the modality budget but also in using historical state to provide continuous and stable closed-loop budget control, thereby further improving multimodal optimization.

AuthorsOpenReview ↗

W2. Unclear Full Computational Trade-off and Comparisons under Matched Total Compute (PART 1/2)

Thank you for raising the issue of computational cost. Following your suggestion, we added three experiments to Appendix C.9, “Computational Cost Analysis,” to evaluate FGMO from the perspectives of full training cost, performance under the same total compute, and sensitivity to reference length.

A. End-to-End Computational Cost

We first measure the two components of FGMO’s computational cost: offline Reference construction and online joint training. Specifically, we train each modality independently for 100 epochs, record the training loss and tensor-wise function-space response, and construct the complete Reference. We then measure the cost of joint training, including periodic FPE estimation and FRC control operations. All experiments use the same hardware environment as the main experiments, and computation is measured in GPU-hours.

When the same Reference is reused across five independent joint-training runs, we compute the average cost per run as

Cavg=Cjoint+Cref5.C_{\mathrm{avg}} = C_{\mathrm{joint}} + \frac{C_{\mathrm{ref}}}{5}.

The complete results are shown below.

Table 2: End-to-end computational cost across datasets (GPU-hours).

DatasetRef.JointTotal5-Run Avg.
CREMA-D1.202.183.382.42
KSounds3.054.457.505.06
Sarcasm2.302.955.253.41

Although constructing the reference introduces an additional offline cost, the exact same reference trajectory can be reused across independent joint-training runs for the same dataset and model configuration, effectively amortizing its construction cost. The cost of a single run therefore differs substantially from the average cost when the Reference is reused across multiple runs.

B. Matched Total Compute

To determine whether FGMO’s advantage comes from additional computational budget, we conduct comparisons under matched total compute. We first use FGMO’s complete end-to-end cost—Reference construction plus online joint training—as the common GPU-hour budget on each of the three datasets. While preserving their original optimization mechanisms, RGM and AUG are allowed to train longer and adjust their learning-rate schedules so that they consume the same total compute. FGMO follows its original configuration, including Reference construction and joint training. Each method is evaluated at the end of the last training epoch permitted by its budget, and we report final accuracy.

Table 3: Performance under matched total computational budgets.

DatasetBudget (GPU-h)RGM Acc. (%)AUG Acc. (%)FGMO Acc. (%)
CREMA-D3.3884.1085.3885.89
KSounds7.5074.0273.4675.65
Sarcasm5.2584.9685.2885.47

Even when the full offline reference-construction cost is accounted for, and RGM and AUG are granted extended training horizons to match the total compute, FGMO still achieves the highest accuracy across all three datasets. This confirms that FGMO’s performance advantage is not merely a byproduct of increased computation; rather, it stems from the effective reference-calibrated function-space progress estimation and closed-loop optimization.

AuthorsOpenReview ↗

W2. Unclear Full Computational Trade-off and Comparisons under Matched Total Compute (PART 2/2)

C. Reference Length Sensitivity

Finally, to examine whether Reference-construction cost can be reduced further, we compare Reference training lengths of 25, 50, 75, and 100 epochs on CREMA-D. Each shorter Reference is constructed from the corresponding prefix of the complete 100-epoch independent-training trajectory. The joint-training network, optimizer, learning-rate schedule, controller configuration, and number of optimization steps remain identical; only Reference length changes. We measure Reference-construction cost, total end-to-end cost, and final fused accuracy.

Table 4: Sensitivity to Reference training length on CREMA-D.

Reference EpochsRef. Cost (GPU-h)Total Cost (GPU-h)Accuracy (%)
250.312.4984.91
500.632.8185.63
750.893.0785.82
1001.203.3885.89

Longer References improve performance, but with diminishing returns. In particular, a 50-epoch Reference preserves most of the performance gain, while extending it to 100 epochs produces only a limited additional accuracy improvement. Reference length therefore offers a flexible trade-off between computational cost and performance, and resource-constrained settings do not always require the longest reference trajectory.

Together, the three experiments validate FGMO’s computational efficiency from the perspectives of cost accounting, computational fairness, and reference-construction efficiency. The Reference-construction cost can be amortized through reuse; FGMO still outperforms existing baselines after counting the full offline cost and matching total compute; and a shorter Reference retains most of the performance gain. FGMO’s advantage is therefore not built on extra computation, and the method offers a practical computational-efficiency benefit.

Reviewer_Rg1KOpenReview ↗
Rating
3(+1)
Confidence
4(+1)
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This work proposes a different score to quantify modality imbalance. The score is a function-space progress signal, which is used to assess modality-wise optimization progress and align optimization across modalities. The experiments show that it can achieve superior performance over some previous scores.

Critical strengths
  1. This paper tries to address a difficult problem. It is hard to theoretically find a truly accurate score. In fact, such a score may not necessarily exist, because the causes and generation process of multimodal imbalance are themselves unknown, and each score is only an observation from a local perspective.
  2. The work is basically complete in terms of metric design, regulation method, and corresponding validation experiments, and it is easy to follow.
  3. The experiments verify its effectiveness.
Critical weaknesses
  1. A key experiment is missing: in my understanding, the implementation route of this paper for modality balance is still to encourage each modality to learn better rather than to pursue a balance between the performance of the two modalities. For this reason, a score that can reflect the learning situation of each modality itself should be designed. Therefore, the change in unimodal learning effect should be analyzed in focus. Theoretically, after regulation by this score, each modality should learn more sufficiently, the unimodal prediction results should be better, and the unimodal performance should be better than that obtained with other scores.
  2. The argumentation and experiments on why the old score fails and the mechanism by which the new score solves the failure cause are still insufficient.
  3. The improvement shown by the experimental results is limited; for example, in Table 1 relative to AUG, the improvement on many datasets is <1%.
  4. For the ablation experiment in Table 2, I would rather see the effect of adding a, b, c, and d one by one, rather than ablating only one design separately, especially whether the unimodal effect after combining a and b is better than the unimodal effect with only a.
  5. The three-modality experiment does not use the more mainstream MOSEI and MOSI.
  6. The comparison is unfair: in the end, the regulation effect of each score is achieved through the learning rate, but some scores were not originally intended to intervene in the learning rate.
Questions for authors
  1. “cross-modal score gaps poorly calibrated to the actual rebalancing demand.” What exactly is the actual rebalancing demand?
  2. What relationship does this paper ultimately aim to achieve between unimodal and multimodal training? Is the author pursuing absolute modality balance or relative modality balance? This method encourages each modality to learn sufficiently, which is actually relative balance, that is, giving play to the upper limit of each modality, which may intensify the imbalance between modalities. Therefore, the author needs to clarify the balance objective and prove that narrowing the response difference after reference normalization indeed corresponds to reducing harmful modality suppression, rather than merely balancing a custom statistic.
14 comments
AuthorsOpenReview ↗

W1. Missing Key Experiment on Unimodal Learning Effect(PART 1/2)

Thank you for your careful evaluation of our work. We first address your concern regarding unimodal learning effects, clarifying our relative-balance objective, the motivation behind RFP, and the corresponding experimental evidence.

A. Absolute Balance or Relative Balance

The multimodal imbalance addressed in this work primarily concerns the under-optimization of non-dominant modalities during joint training. Therefore, as the reviewer noted, our objective is relative balance. The central goal of modality rebalancing is to alleviate optimization competition during joint training so that every modality can be learned sufficiently, rather than to eliminate the inherent differences among modalities. In fact, from an ensemble learning perspective, the inherent differences among modalities provide complementary information that enhances fusion performance. An ideal modality rebalancing method should thus preserve modality diversity while fully exploiting the learning potential of each modality, thereby facilitating more effective optimization during joint training and ultimately improving multimodal fusion. This forms the core motivation for assessing modality imbalance through relative optimization progress.

B. A Score That Reflects the Learning State of Each Modality

Regarding the reviewer’s suggestion that “a score that can reflect the learning situation of each modality itself should be designed,” RFP is precisely the signal designed to assess the optimization progress of each modality relative to its own learning state. Instead of directly comparing absolute scores across modalities, RFP calibrates each modality’s update-induced function-space response during joint training against its own independently trained loss-aligned unimodal reference. Since the update direction during joint training is determined by the overall objective combining both unimodal and fusion losses, it inherently reflects cross-modal interactions in fusion training. Meanwhile, the unimodal reference provides a baseline to gauge the modality’s relative learning state. Consequently, RFP captures both cross-modal interactions during joint training and the intrinsic learning dynamics of each individual modality.

AuthorsOpenReview ↗

W1. Missing Key Experiment on Unimodal Learning Effect(PART 2/2)

C. Unimodal Performance Experiments

Following the reviewer’s suggestion, we added a unimodal performance analysis to Section 4.5(Lines 462-470) of the revised manuscript. This includes the accuracy and loss dynamics of the Audio and Video branches in Figure 4, as well as a comparison of unimodal performance among score-based rebalancing methods in Table 4, evaluated alongside Vanilla and strong baselines. The main experimental results also report the unimodal and multimodal prediction performance of each method. These experiments verify, at both the unimodal learning and final fusion levels, that FGMO effectively promotes sufficient learning across every modality.

C.1 Unimodal Learning Dynamics Compared with the Strongest Baseline and Vanilla (Figure 4)

We first compare the unimodal Audio and Video training losses and validation accuracies of Vanilla, AUG, and FGMO on KSounds. As illustrated in the newly added Figure 4, FGMO achieves lower training losses and higher validation accuracies for both modalities. The Video branch, in particular, exhibits a more pronounced accuracy advantage over Vanilla and AUG throughout the training process. These results confirm that FGMO’s coordinated optimization not only enhances fused prediction but also promotes sufficient learning in both unimodal branches.

C.2 Comparison with Score-Based Rebalancing Methods (Table 4)

To further determine whether FGMO provides better unimodal performance than existing scoring systems, we added Table 4 to the revised manuscript. It compares five representative score-based modality rebalancing methods: OGM, PMR, AGM, SMV, and RGM. These methods construct modality scores from predictive contribution, prototype prediction, historical information strength, Shapley modality value, and accumulated gradient energy, respectively, and then use their native rebalancing mechanisms to regulate training. Vanilla serves as the reference without rebalancing. On KSounds and Sarcasm, we report the accuracy of each unimodal branch and the fused prediction from the jointly trained model. The complete results are shown below. These results demonstrate that FGMO improves the predictive ability of each individual modality while simultaneously advancing fusion performance beyond existing score-based rebalancing methods.

Table 2: Modality-specific and multimodal accuracy on KSounds and Sarcasm. Unimodal accuracy is evaluated from each modality branch during joint training. The best and second-best results are highlighted in bold and underlined, respectively.

MethodKSounds (Acc.) — AudioKSounds (Acc.) — VideoKSounds (Acc.) — MultiSarcasm (Acc.) — ImageSarcasm (Acc.) — TextSarcasm (Acc.) — Multi
Vanilla.4992.4798.6455.6907.8066.8286
OGM.4970.4930.6582.6955.8105.8360
PMR.4990.5170.6675.6969.8131.8310
AGM.5042.5238.6791.6991.8149.8306
SMV.5128.5426.6900.7012.8176.8418
RGM.5218‾\underline{.5218}.5683‾\underline{.5683}.7309‾\underline{.7309}.7042‾\underline{.7042}.8239‾\underline{.8239}.8449‾\underline{.8449}
FGMO (Ours).5319.5991.7565.7165.8281.8547
AuthorsOpenReview ↗

W2 & Q1. Insufficient Argumentation on the Failure of Old Scores and the Actual Rebalancing Demand

First, we would like to clarify that the issue with existing scores is their susceptibility to bias, rather than indicating that the old score fundamentally fails. Existing methods successfully capture modality learning states from various perspectives and improve multimodal performance to a certain degree. Our central claim is that existing scores exhibit comparison biases that hinder their ability to accurately guide modality rebalancing during specific training stages, making it difficult to consistently yield positive gains. We also believe that a score has no explicit ground truth. A score is not the final objective. What matters is whether it can guide subsequent training effectively and ultimately improve model performance. This is the practical requirement that a rebalancing score should satisfy. At the same time, we sincerely apologize for any confusion caused by the sentence in the Introduction. We have revised the phrasing in Lines 44-48 of the Introduction, and these modifications are highlighted in red.

With this clarification, our analysis of why previous scoring systems are biased and how the new system corrects this bias focuses on cross-modal score comparability. Existing methods generally compare the raw scores of different modalities directly. However, different modalities naturally exhibit distinct prediction uncertainties and learning dynamics, making their raw scores not directly comparable. Consequently, cross-modal score gaps may not fully reflect differences in their relative optimization states, which may lead to suboptimal rebalancing decisions and limit subsequent performance improvements.

To address this limitation, we introduce unimodal references and compare each modality’s joint-training state with its independently trained reference, thereby mitigating the influence of inherent modality-specific response scales. To empirically verify this, we replaced the original experiment in the Introduction—which mapped scores to learning rates through a common rule—with a controlled intervention experiment that preserves the native rebalancing mechanisms of RGM and SMV. The detailed setting is described in the fourth paragraph of the Introduction. The Original Score retains a clear Audio–Video score gap late in training even when the corresponding intervention already produces negative gains. The Relative Original Score reduces this gap and improves the gains from intervention. This finding demonstrates that normalizing the score relative to a reference helps correct its bias.

We further introduce a function-space perspective that directly captures the model-output change induced by the current optimization update, effectively reducing measurement biases stemming from prediction uncertainty and heterogeneous parameterizations. By calibrating each modality’s current function-space response against its loss-aligned unimodal reference, we derive RFP, a comparable progress signal that characterizes modality-wise optimization progress relative to its independently trained reference. In Figure 1, the RFP gap gradually narrows and eventually reverses over the course of training, and RFP-guided interventions consistently yield positive gains at most checkpoints.

By comparing different scores and their corresponding intervention gains at successive stages of training, these experiments reveal how cross-modal scoring bias arises and how it affects practical rebalancing. They further show that reference normalization alleviates comparison bias caused by inherent modality differences, while function-space measurement improves the characterization of relative optimization progress. Together, these changes enable the score to guide modality rebalancing more reliably across all training stages.

In summary, existing scoring systems exhibit a degree of cross-modal comparison bias. RFP directly addresses this issue by integrating unimodal reference calibration with function-space measurement to provide a more comparable signal for relative optimization progress across modalities. We also clarify the practical requirement for a score: whether it can guide subsequent rebalancing effectively and ultimately improve both unimodal and multimodal performance. In pursuit of this goal, we redesigned the controlled intervention experiment in the Introduction, correcting the unfairness of the original setup. The trajectories of different scores and their corresponding performance gains further reveal how cross-modal scoring bias arises, how it affects practical rebalancing, and how RFP progressively corrects it through reference normalization and function-space measurement. These results robustly validate the advantages of RFP for guiding modality rebalancing.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.