- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper proposes FGMO, which introduces function progress estimation to produce a comparable progress signal (the RPF component) for each modality and adjusts the learning rate accordingly (the FRC component) to alleviate the modality imbalance problem. Theoretical analysis shows that FRC reduces modality optimization discrepancy.
Critical strengths
-
The paper identifies a key research gap in existing modality imbalance estimation methods, which ignore the inherent differences in prediction uncertainty and learning dynamics across modalities. The teaser figure (Figure 1) vividly demonstrates and compares these aspects, making the motivation and advantages of the proposed model clear and intuitive.
-
Rethinking modality imbalance in the function space and introducing a more comparable signal for modality rebalancing is both novel and interesting.
Critical weaknesses
-
About the Experiment: The statistical reliability of the improvements in Table 1 is unclear. Several gains over the strongest baselines are modest(e.g, Sarcasm, NVGesture), yet variability is reported only for FGMO. Without corresponding baseline statistics, it is difficult to determine whether these gains consistently exceed run-to-run variation.
-
About the Method: The interpretation of function-space response as optimization progress needs stronger justification. Equation (3) measures the magnitude of logit changes but does not capture whether those changes reduce the loss or improve predictive performance. A large response may reflect an unhelpful update, while a small response may occur near convergence.
Questions for authors
-
Can you report variability for the strongest baselines under matched training seeds, and analyse the significance of the performance differences?
-
Is RFP strongly correlated with subsequent loss reduction or performance gains? Please provide quantitative evidence supporting this relationship.
-
Figure 3 shows that more frequent adjustments, such as K=5, lead to lower performance. I am really interested about the causes this degradation.
-
The fusion method studied in this paper is late fusion, where unimodal logits are averaged. Can the proposed method be extended to early- or intermediate-fusion architectures, such as attention-based fusion methods?
8 comments
W2. Justification for Interpreting Function-Space Response as Optimization Progress
We apologize for the confusion. The following sentence in the original manuscript may have suggested that the function-space response itself is the optimization progress:
“FGMO characterizes modality-wise optimization progress by estimating the function-space response and calibrating it against a loss-aligned unimodal reference to obtain the function-space progress signal.”
This sentence appears in Section 3.2, Lines 189–191. What we intended to express is that the function-space response is the basic measurement of the function change induced by the current update, whereas RFP, obtained after calibration against the loss-aligned unimodal reference, is the signal used to characterize relative modality-wise optimization progress.
Thank you for this careful feedback. We revised the sentence to:
“FGMO estimates the function-space response and calibrates it against a loss-aligned unimodal reference to obtain RFP, which characterizes modality-wise optimization progress.”
The change is highlighted in red in the revised manuscript. Thank you again for helping us clarify this statement.
Q2. Quantitative Evidence on the Correlation Between RFP and Subsequent Loss Reduction or Performance Gains (PART 1/2)
Thank you for this important suggestion. To further verify whether RFP reflects the modality learning state and whether it is associated with subsequent optimization outcomes, we added two quantitative experiments to Appendix C.11 of the revised manuscript, “RFP Discrepancy and Learning Outcomes.” They examine the relationship of the RFP Gap with current unimodal prediction performance and with subsequent validation-loss reduction.
A. Relationship between RFP and Modality Prediction Performance
On the Sarcasm dataset, we compare independently trained unimodal models (Unimodal), Vanilla, a model without RFP normalization, and FGMO under controller gains . For each setting, we record the RFP Gap, the accuracies of the Image and Text branches, and the final fused accuracy to analyze the relationship between the RFP Gap and unimodal and multimodal prediction performance.
Table 3: Relationship between RFP discrepancy and modality-specific accuracy on Sarcasm.
| Method | RFP Gap | Image Acc. (%) | Text Acc. (%) | Multi Acc. (%) |
|---|---|---|---|---|
| Unimodal | – | 71.81 | 81.36 | – |
| Vanilla | 0.82 | 69.07 | 80.66 | 82.86 |
| w/o RFP | 0.58 | 69.78 | 81.32 | 83.48 |
| 0.32 | 71.14 | 82.21 | 85.32 | |
| 0.29 | 70.94 | 82.25 | 85.26 | |
| 0.23 | 71.35 | 82.51 | 85.39 | |
| 0.09 | 71.65 | 82.81 | 85.47 | |
| 0.24 | 71.19 | 82.55 | 85.15 | |
| 0.44 | 70.97 | 82.32 | 84.88 | |
| 0.59 | 70.20 | 81.71 | 84.41 | |
| 1.00 | 69.30 | 80.82 | 82.97 | |
| 1.31 | 68.77 | 80.10 | 82.48 |
The RFP Gap is strongly negatively correlated with the accuracies of the Image branch, Text branch, and fused prediction. As the RFP Gap decreases, both unimodal branches improve and the fused prediction improves accordingly. In particular, the initially under-optimized Image branch gradually approaches its independently trained performance, while the Text branch also benefits further. A Spearman correlation analysis across controller gains supports the same trend. Thus, the RFP Gap reflects not only relative optimization differences between modalities but also actual unimodal and fused prediction performance.
W1 & Q1. Statistical Reliability of the Modest Improvements and Baseline Variability
Thank you for your careful evaluation of our work and for your positive assessment of the motivation, function-space perspective, and RFP design. We respond to each of your questions below.
We added a paired statistical analysis using five matched training seeds and combined it with experiments across fusion architectures. These additions validate FGMO’s performance advantage from the perspectives of statistical reliability and architectural generalization.
A. Statistical Reliability under Matched Training Seeds
To determine whether FGMO’s performance improvements are affected by random initialization, we select the strongest existing baseline in terms of accuracy on each of six datasets and compare it with FGMO using five matched random seeds. All methods use the same backbone, initialization strategy, data split, training budget, and model-selection rule, while retaining each baseline’s original optimization mechanism.
We report the mean and standard deviation for each method, the paired accuracy improvement and its standard deviation, and a two-sided paired t-test with a 95% confidence interval.
Table 1: Statistical comparison with the strongest accuracy baseline across six datasets.
FGMO improves performance on all six datasets, with paired accuracy gains ranging from 0.36% to 2.56%. Even on Sarcasm, where the gain is relatively small, FGMO produces a stable positive improvement, showing that its advantage does not depend on a particular dataset or random seed. In addition, none of the 95% confidence intervals for the six paired improvements contain zero, and every p-value is below 0.0083. The improvements therefore remain statistically significant after Bonferroni correction for multiple comparisons. These results demonstrate the statistical reliability of FGMO’s improvements over the strongest baseline on each dataset.
B. Performance across Different Fusion Architectures
Our method also outperforms existing baselines across different fusion architectures. In Appendix C.8 of the revised manuscript, “Generalization Across Fusion Architectures,” we added experiments with Early Fusion and an Intermediate Fusion architecture based on bidirectional Cross-Attention. We compare Vanilla, RGM, AUG, and FGMO on KSounds and Sarcasm. The complete experimental setup and further discussion are provided in our response to Q4.
Table 2: Performance comparison across different fusion architectures.
FGMO outperforms existing baselines in fused prediction under both Early and Intermediate Fusion, maintaining consistent advantages in accuracy, mAP, and F1. Thus, FGMO not only improves multimodal learning under the original Late Fusion design but also effectively coordinates optimization under feature-level concatenation and attention-based cross-modal interaction, further demonstrating the method’s clear advantages.