- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper aims to address the challenge of long-tailed concept distributions in multimodal data: the frequency and representation compactness of the same concept may differ substantially across visual-only, text-only, and joint views. To address this issue, the paper proposes Pre-decoupled Multimodal-aware Long-tail Learning (PMLL), which consists of three components: modality-decoupled re-weighting, rare word reinforcement, and adaptive task-aware optimization. The experiments are extensive, covering multiple datasets, backbone models, and tasks.
Critical strengths
-
Decomposing multimodal long-tail distributions into unimodal and multimodal observation views is an interesting idea that better reflects the practical characteristics of multimodal learning.
-
The proposed method jointly considers occurrence frequency, representation compactness, rare words, and multi-task optimization dynamics. These components address the problem from multiple perspectives and are reasonably motivated.
-
The experiments are extensive and cover diverse datasets, models, and tasks.
Critical weaknesses
-
The paper suffers from unclear presentation, with several key concepts and experimental quantities insufficiently defined or explained. Examples include the computation of the "modality-inconsistent long-tail degree" in Figure 1, the definition or selection criterion of "rare words" in the proposed method, and the meaning of "Overall avg relative increase for PMLL" in the experiments. These issues substantially hinder the readability of the work and indicate that the current manuscript is not yet ready for publication.
-
While the overall perspective and motivation are interesting, the experimental results do not provide sufficient evidence to support the central claims. In particular, the proposed method does not appear to clearly resolve the challenge identified by the paper. Further investigation is needed to explain why jointly considering multiple observation views provides no practical improvement over the conventional use of a single aggregated frequency.
Questions for authors
-
The results in Tables 1 and 2 show that the proposed method performs comparably to, and in some cases worse than, existing approaches. This weakens the empirical support for both the proposed perspective and the corresponding solution.
-
How is "Overall avg relative increase for PMLL" defined? Is it the relative improvement over the best-performing baseline? A precise definition is necessary for correctly interpreting the reported results.
-
The "modality-inconsistent long-tail degree" in Figure 1 also lacks a detailed description of how it is computed. In addition, I recommend using vector graphics for the figures to improve readability and avoid visual distortion.
-
The paper highlights missing cross-modal evidence as an important scenario, yet this core setting is not explicitly discussed in relation to Eq. (1). Moreover, how is "rare" defined or determined in rare word mining? I also recommend that the authors carefully review the definitions and mathematical formulations throughout the paper, particularly the distinction between vectors and scalars, to improve clarity and readability.
6 comments
Response to the Question 2, 3 and 4
Q2 (Metric definition):
A2: This metric is indeed intended to quantify the relative improvement achieved by PMLL over the strongest competing method. As request, we have added a more detailed explanation of its calculation in Section 4.2. Furthermore, to complement this metric and provide a more intuitive perspective on the magnitude of the improvements, we have additionally introduced a new GR metric to measure the ratio between improvement of PMLL and that of the strongest competing method. This provides readers with a clearer and more intuitive understanding of the performance gains achieved by PMLL.
Q3 (Figures issue)
A3: We have added further explanations to the figure caption to clarify the definition of this metric, including its underlying basis and a brief description of how it is calculated. However, since its calculation involves the frequency-based and compactness-based long-tailedness measurements introduced in Section 2.2.1, we have chosen not to include the detailed equations directly in the figure caption. Introducing these formulations too early would make the presentation repetitive and potentially disrupt the reading flow. Instead, we provide a concise explanation in the caption while retaining the detailed formulations in Section 2.2.1.
Furthermore, We have replaced all figures in the manuscript with PDF versions, which improves their visual clarity and the overall readability and presentation quality of the paper.
Q4 (Equation explanation and refinements)
A4: First, we would like to clarify that we do not consider the inconsistency in long-tailed distributions across modalities to necessarily stem from missing cross-modal evidence. Rather, the evidence provided by different modalities may inherently exhibit discrepancies. Our goal is to leverage information from different modalities to characterize a more realistic long-tailed distribution and provide more effective supervision, rather than simply focusing on completing missing evidence across multiple views. Moreover, although we adopt a weighting-based approach, more accurate long-tailed distribution information can also provide a better basis for multi-view evidence completion. To make this distinction clearer to readers, we have added relevant explanations after Eq. (1) in Section 2.2.1.
In the revised manuscript provided with this rebuttal, we have added further descriptions clarifying that rare words are identified based on their occurrence frequencies in the training captions. This is essentially analogous to the way the rarity of class-label words is determined. The definitions and mathematical formulations are also carefully reviewed and revised throughout the manuscript. In particular, we have corrected the mathematical notation and symbol definitions to ensure a consistent distinction between vectors and scalars.
Addtional response for Q1
Addtional respond for Q1 (Performance improvements)
A5: To further validate the robustness of PMLL’s improvements, we conducted additional experiments with different random seeds on COCO and VG and report the corresponding mean and standard deviation in A.6 (shown in Table b and c).
Table b: Results of joint captioning and classification on COCO over different random seeds. Results for CIDEr, Polos, KLE, Tail, and Avg are reported as mean ± standard deviation. The optimal values in important columns are bolded.
| Metric / Method | BLEU | CIDEr | R-L | SPICE | C-S | R-C-S | P-S++ | R-P-S++ | Polos | KLE | Head | Mid | Tail | Avg |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Softmax | 0.477 | 3.373 ± 0.005 | 0.622 | 0.566 | 0.793 | 0.846 | 0.580 | 0.708 | 0.783 ± 0.001 | 4.352 ± 0.002 | 0.863 | 0.796 | 0.754 ± 0.001 | 0.800 ± 0.001 |
| Focal | 0.442 | 3.070 ± 0.004 | 0.595 | 0.539 | 0.791 | 0.843 | 0.579 | 0.707 | 0.776 ± 0.001 | 4.048 ± 0.006 | 0.851 | 0.774 | 0.713 ± 0.002 | 0.773 ± 0.001 |
| CLA | 0.455 | 3.145 ± 0.023 | 0.607 | 0.550 | 0.792 | 0.844 | 0.580 | 0.706 | 0.776 ± 0.001 | 4.100 ± 0.018 | 0.864 | 0.792 | 0.743 ± 0.005 | 0.794 ± 0.001 |
| BS | 0.434 | 2.929 ± 0.005 | 0.591 | 0.534 | 0.791 | 0.842 | 0.578 | 0.704 | 0.768 ± 0.000 | 3.928 ± 0.004 | 0.846 | 0.791 | 0.755 ± 0.002 | 0.793 ± 0.000 |
| LinUpper | 0.468 | 3.287 ± 0.019 | 0.614 | 0.560 | 0.792 | 0.845 | 0.579 | 0.707 | 0.779 ± 0.000 | 4.258 ± 0.009 | 0.858 | 0.801 | 0.753 ± 0.002 | 0.799 ± 0.003 |
| TUL | 0.477 | 3.383 ± 0.004 | 0.622 | 0.567 | 0.793 | 0.846 | 0.580 | 0.709 | 0.783 ± 0.000 | 4.357 ± 0.007 | 0.861 | 0.797 | 0.752 ± 0.000 | 0.799 ± 0.001 |
| Gala | 0.477 | 3.376 ± 0.004 | 0.622 | 0.567 | 0.793 | 0.846 | 0.580 | 0.708 | 0.783 ± 0.001 | 4.355 ± 0.002 | 0.863 | 0.797 | 0.753 ± 0.001 | 0.799 ± 0.001 |
| TR | 0.468 | 3.292 ± 0.002 | 0.616 | 0.561 | 0.792 | 0.845 | 0.579 | 0.707 | 0.780 ± 0.000 | 4.177 ± 0.013 | 0.849 | 0.794 | 0.753 ± 0.000 | 0.794 ± 0.001 |
| Bal | 0.478 | 3.393 ± 0.009 | 0.623 | 0.569 | 0.793 | 0.847 | 0.580 | 0.708 | 0.783 ± 0.000 | 4.362 ± 0.008 | 0.863 | 0.800 | 0.755 ± 0.002 | 0.801 ± 0.000 |
| PMLL | 0.479 | 3.402 ± 0.004 | 0.623 | 0.568 | 0.793 | 0.847 | 0.580 | 0.709 | 0.784 ± 0.001 | 4.371 ± 0.004 | 0.860 | 0.797 | 0.763 ± 0.003 | 0.802 ± 0.000 |
Table c: Results of joint captioning and classification on VG over different random seeds.
| Metric / Method | BLEU | CIDEr | R-L | SPICE | C-S | R-C-S | P-S++ | R-P-S++ | Polos | KLE | Head | Mid | Tail | Avg |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Softmax | 0.381 | 1.712 ± 0.004 | 0.487 | 0.453 | 0.793 | 0.838 | 0.577 | 0.699 | 0.797 ± 0.001 | 3.358 ± 0.001 | 0.729 | 0.633 | 0.463 ± 0.002 | 0.593 ± 0.002 |
| Focal | 0.355 | 1.533 ± 0.009 | 0.465 | 0.436 | 0.790 | 0.835 | 0.577 | 0.698 | 0.790 ± 0.001 | 3.169 ± 0.002 | 0.675 | 0.568 | 0.405 ± 0.000 | 0.534 ± 0.000 |
| CLA | 0.357 | 1.501 ± 0.002 | 0.466 | 0.431 | 0.791 | 0.834 | 0.578 | 0.697 | 0.788 ± 0.000 | 3.093 ± 0.010 | 0.729 | 0.624 | 0.446 ± 0.001 | 0.584 ± 0.001 |
| BS | 0.072 | 0.250 ± 0.243 | 0.097 | N/A | 0.671 | 0.659 | 0.505 | 0.552 | 0.348 ± 0.128 | 1.165 ± 0.388 | 0.682 | 0.584 | 0.423 ± 0.007 | 0.549 ± 0.006 |
| LinUpper | 0.363 | 1.572 ± 0.001 | 0.470 | 0.437 | 0.793 | 0.835 | 0.577 | 0.697 | 0.790 ± 0.000 | 3.195 ± 0.009 | 0.728 | 0.626 | 0.455 ± 0.002 | 0.588 ± 0.002 |
| TUL | 0.382 | 1.717 ± 0.004 | 0.487 | 0.453 | 0.793 | 0.838 | 0.577 | 0.699 | 0.797 ± 0.001 | 3.357 ± 0.001 | 0.727 | 0.630 | 0.459 ± 0.001 | 0.591 ± 0.002 |
| Gala | 0.368 | 1.594 ± 0.021 | 0.475 | 0.443 | 0.794 | 0.837 | 0.577 | 0.698 | 0.793 ± 0.001 | 3.241 ± 0.007 | 0.723 | 0.630 | 0.459 ± 0.005 | 0.589 ± 0.005 |
| TR | 0.371 | 1.626 ± 0.003 | 0.480 | 0.443 | 0.793 | 0.836 | 0.577 | 0.698 | 0.792 ± 0.000 | 3.167 ± 0.017 | 0.726 | 0.633 | 0.471 ± 0.007 | 0.594 ± 0.006 |
| Bal | 0.383 | 1.723 ± 0.001 | 0.488 | 0.454 | 0.793 | 0.839 | 0.577 | 0.699 | 0.797 ± 0.000 | 3.372 ± 0.002 | 0.727 | 0.631 | 0.461 ± 0.000 | 0.592 ± 0.002 |
| PMLL | 0.382 | 1.724 ± 0.003 | 0.488 | 0.453 | 0.793 | 0.839 | 0.578 | 0.700 | 0.797 ± 0.001 | 3.372 ± 0.004 | 0.725 | 0.633 | 0.470 ± 0.002 | 0.595 ± 0.002 |
The results show that PMLL consistently achieves the best overall classification accuracy while maintaining the strongest or highly competitive captioning performance across both datasets. Its small standard deviations further demonstrate that these improvements remain stable under different random seeds. Although the gains on some individual metrics are modest, their consistency across tasks, datasets, and random seeds provides additional empirical support for both the modality-inconsistent long-tail perspective and the proposed solution.
Response to the Question 1
We sincerely appreciate your careful reading and constructive suggestions, which have greatly helped us improve the technical clarity and readability of our manuscript.
Q1 (Performance improvements)
A1: Since our work focuses on the modality-inconsistent long-tailed learning problem in multimodal generation, our objective is to robustly improve model performance across different multimodal subtasks. Accordingly, our experiments jointly optimize the two representative tasks, captioning and classification. However, these tasks rely on different types of features and supervision, which makes it challenging for PMLL to achieve the best performance across all individual metrics.
To better evaluate the overall improvements, we introduced the Overall avg relative increase score in Section 4.2 to measure the overall relative gains of PMLL over the strongest competing method (0.1, 0.4, 0.2 for three datasets with same setting). However, we also recognize that these relatively modest gains alone may not fully demonstrate the advantages of PMLL. Therefore, in the revised manuscript provided with this rebuttal, we additionally calculate the relative improvements of different methods over the Softmax baseline and introduce the GR metric in Section 4.2 to quantify the improvement achieved by PMLL compared with the strongest competing method. Across different experimental settings and datasets, PMLL consistently achieves better results. Notably, on the more challenging VG dataset, the improvements achieved by PMLL are 6× and 3.91× those of the strongest competing method under the two respective settings.
Furthermore, to better understand why different methods exhibit inconsistent improvements across tasks, we conducted additional single-task training experiments in A.7 (as shown in Table a below). The results show that GALA and BAL, which are designed for token generation, are better suited to the captioning task, whereas TR, which is designed for class generation, is better suited to the classification task. However, their performance on the respective other tasks is relatively poor. These findings suggest that different multimodal generation subtasks require different forms of supervision, and improvements on an individual task metric alone do not necessarily indicate overall superiority in multimodal generation. In contrast, although PMLL does not achieve the best performance on either task individually, it maintains strong performance on both tasks simultaneously.
Table a: Comparison of single generation tasks, including captioning, classification (with hallucination performance) on COCO. Method/M means employing our modality-decoupled weighting computed using multi-modal frequency, instead of joint weighting.
Notably, after incorporating our proposed modality-decoupled weighting mechanism, GALA, BAL, and TR all achieve further improvements on their respective advantageous tasks. This provides strong evidence for the effectiveness of our proposed modality-decoupled long-tailed learning strategy.