acceptodds
Under review as a conference paper at ICLR 2027

Rethinking Long-Tail Learning in Vision-Language Models: From Single Distribution to Modality-Coupled Distributions

Abstract

Vision-language models (VLMs) have achieved remarkable progress in multimodal processing and generation, yet still exhibit systematic failures in object hallucination and rare-concept understanding. Prior studies have shown that the concept distribution in multimodal data is markedly long-tailed, resulting in imbalanced downstream performance. However, existing long-tail methods mainly employ a single frequency statistic to define long-tail distribution, which cannot accurately reflect the effective long-tail distribution in multimodal scenarios. For example, the same concept may exhibit markedly different scarcity across modalities, leaving some concepts are frequent in text but weakly grounded visually. Moreover, shared multimodal representation spaces entangle long-tail biases across modalities, making it difficult to decouple modality-specific effects, causing incomplete rare-concept expression. We refer to this problem as modality-inconsistent long-tail. To address this, we propose a pre-decoupled multimodal-aware long-tail learning framework (PMLL), which introduces modality-decoupled reweighting to model multimodal long-tail distributions, rare-word reinforcement to improve rare token expression, and adaptive task-aware optimization to balance multi-task contribution. Furthermore, a keyword-decoupled long-tail evaluation metric is introduced to discover the detailed learning performance of informative tokens, while a metric-general co-occurrence-aware long-tail mining protocol is developed to reduce co-occurrence-induced bias in estimating long-tail distributions. Overall, we provide a unified framework for understanding, mitigating, and evaluating modality-inconsistent long-tail effects in VLM learning. Extensive experiments demonstrate the effectiveness of the PMLL under modality-inconsistent long-tail.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Long-tail learning, vision-language models, multimodal dataset, loss reweighting

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_wH2ZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper aims to address the challenge of long-tailed concept distributions in multimodal data: the frequency and representation compactness of the same concept may differ substantially across visual-only, text-only, and joint views. To address this issue, the paper proposes Pre-decoupled Multimodal-aware Long-tail Learning (PMLL), which consists of three components: modality-decoupled re-weighting, rare word reinforcement, and adaptive task-aware optimization. The experiments are extensive, covering multiple datasets, backbone models, and tasks.

Critical strengths
  1. Decomposing multimodal long-tail distributions into unimodal and multimodal observation views is an interesting idea that better reflects the practical characteristics of multimodal learning.

  2. The proposed method jointly considers occurrence frequency, representation compactness, rare words, and multi-task optimization dynamics. These components address the problem from multiple perspectives and are reasonably motivated.

  3. The experiments are extensive and cover diverse datasets, models, and tasks.

Critical weaknesses
  1. The paper suffers from unclear presentation, with several key concepts and experimental quantities insufficiently defined or explained. Examples include the computation of the "modality-inconsistent long-tail degree" in Figure 1, the definition or selection criterion of "rare words" in the proposed method, and the meaning of "Overall avg relative increase for PMLL" in the experiments. These issues substantially hinder the readability of the work and indicate that the current manuscript is not yet ready for publication.

  2. While the overall perspective and motivation are interesting, the experimental results do not provide sufficient evidence to support the central claims. In particular, the proposed method does not appear to clearly resolve the challenge identified by the paper. Further investigation is needed to explain why jointly considering multiple observation views provides no practical improvement over the conventional use of a single aggregated frequency.

Questions for authors
  1. The results in Tables 1 and 2 show that the proposed method performs comparably to, and in some cases worse than, existing approaches. This weakens the empirical support for both the proposed perspective and the corresponding solution.

  2. How is "Overall avg relative increase for PMLL" defined? Is it the relative improvement over the best-performing baseline? A precise definition is necessary for correctly interpreting the reported results.

  3. The "modality-inconsistent long-tail degree" in Figure 1 also lacks a detailed description of how it is computed. In addition, I recommend using vector graphics for the figures to improve readability and avoid visual distortion.

  4. The paper highlights missing cross-modal evidence as an important scenario, yet this core setting is not explicitly discussed in relation to Eq. (1). Moreover, how is "rare" defined or determined in rare word mining? I also recommend that the authors carefully review the definitions and mathematical formulations throughout the paper, particularly the distinction between vectors and scalars, to improve clarity and readability.

6 comments
AuthorsOpenReview ↗

Response to the Question 1

We sincerely appreciate your careful reading and constructive suggestions, which have greatly helped us improve the technical clarity and readability of our manuscript.

Q1 (Performance improvements)

A1: Since our work focuses on the modality-inconsistent long-tailed learning problem in multimodal generation, our objective is to robustly improve model performance across different multimodal subtasks. Accordingly, our experiments jointly optimize the two representative tasks, captioning and classification. However, these tasks rely on different types of features and supervision, which makes it challenging for PMLL to achieve the best performance across all individual metrics.

To better evaluate the overall improvements, we introduced the Overall avg relative increase score in Section 4.2 to measure the overall relative gains of PMLL over the strongest competing method (0.1, 0.4, 0.2 for three datasets with same setting). However, we also recognize that these relatively modest gains alone may not fully demonstrate the advantages of PMLL. Therefore, in the revised manuscript provided with this rebuttal, we additionally calculate the relative improvements of different methods over the Softmax baseline and introduce the GR metric in Section 4.2 to quantify the improvement achieved by PMLL compared with the strongest competing method. Across different experimental settings and datasets, PMLL consistently achieves better results. Notably, on the more challenging VG dataset, the improvements achieved by PMLL are 6× and 3.91× those of the strongest competing method under the two respective settings.

Furthermore, to better understand why different methods exhibit inconsistent improvements across tasks, we conducted additional single-task training experiments in A.7 (as shown in Table a below). The results show that GALA and BAL, which are designed for token generation, are better suited to the captioning task, whereas TR, which is designed for class generation, is better suited to the classification task. However, their performance on the respective other tasks is relatively poor. These findings suggest that different multimodal generation subtasks require different forms of supervision, and improvements on an individual task metric alone do not necessarily indicate overall superiority in multimodal generation. In contrast, although PMLL does not achieve the best performance on either task individually, it maintains strong performance on both tasks simultaneously.

Table a: Comparison of single generation tasks, including captioning, classification (with hallucination performance) on COCO. Method/M means employing our modality-decoupled weighting computed using multi-modal frequency, instead of joint weighting.

Metric / MethodBLEUR-LCIDErSPICEC-SR-C-SP-S++R-P-S++PolosKLETailAvgTail HR (%)Avg HR (%)
Softmax0.4920.6323.5430.5740.7930.8470.5790.7090.7834.4550.7370.79024.64216.806
Focal0.4400.5923.0440.5310.7930.8440.5800.7070.7734.0220.6210.70520.19213.722
CLA0.4720.6213.3110.5600.7920.8440.5790.7060.7774.2470.7090.77922.49916.738
BS0.4500.6043.1060.5490.7910.8430.5780.7050.7704.1370.7410.79025.87816.901
LinUpper0.4920.6333.5430.5740.7930.8470.5790.7090.7834.4590.7330.78824.80516.767
TUL0.4920.6323.5430.5740.7930.8470.5790.7080.7834.4530.7320.78824.40616.650
Gala0.4920.6333.5540.5750.7920.8470.5790.7090.7834.4610.7350.79024.40916.734
TR0.4810.6263.3970.5680.7930.8460.5800.7080.7834.3380.7480.79425.57317.166
Bal0.4930.6343.5430.5750.7930.8470.5790.7090.7844.4650.7150.77622.97515.540
PMLL0.4920.6333.5450.5740.7930.8470.5790.7080.7834.4560.7430.79424.65616.707
Bal/M0.4930.6343.5430.5750.7930.8480.5800.7090.7854.465N/AN/AN/AN/A
Gala/M0.4920.6333.5550.5750.7920.8470.5790.7090.7834.463N/AN/AN/AN/A
TR/MN/AN/AN/AN/AN/AN/AN/AN/AN/AN/A0.7490.79524.54517.164

Notably, after incorporating our proposed modality-decoupled weighting mechanism, GALA, BAL, and TR all achieve further improvements on their respective advantageous tasks. This provides strong evidence for the effectiveness of our proposed modality-decoupled long-tailed learning strategy.

AuthorsOpenReview ↗

Response to the Question 2, 3 and 4

Q2 (Metric definition):

A2: This metric is indeed intended to quantify the relative improvement achieved by PMLL over the strongest competing method. As request, we have added a more detailed explanation of its calculation in Section 4.2. Furthermore, to complement this metric and provide a more intuitive perspective on the magnitude of the improvements, we have additionally introduced a new GR metric to measure the ratio between improvement of PMLL and that of the strongest competing method. This provides readers with a clearer and more intuitive understanding of the performance gains achieved by PMLL.

Q3 (Figures issue)

A3: We have added further explanations to the figure caption to clarify the definition of this metric, including its underlying basis and a brief description of how it is calculated. However, since its calculation involves the frequency-based and compactness-based long-tailedness measurements introduced in Section 2.2.1, we have chosen not to include the detailed equations directly in the figure caption. Introducing these formulations too early would make the presentation repetitive and potentially disrupt the reading flow. Instead, we provide a concise explanation in the caption while retaining the detailed formulations in Section 2.2.1.

Furthermore, We have replaced all figures in the manuscript with PDF versions, which improves their visual clarity and the overall readability and presentation quality of the paper.

Q4 (Equation explanation and refinements)

A4: First, we would like to clarify that we do not consider the inconsistency in long-tailed distributions across modalities to necessarily stem from missing cross-modal evidence. Rather, the evidence provided by different modalities may inherently exhibit discrepancies. Our goal is to leverage information from different modalities to characterize a more realistic long-tailed distribution and provide more effective supervision, rather than simply focusing on completing missing evidence across multiple views. Moreover, although we adopt a weighting-based approach, more accurate long-tailed distribution information can also provide a better basis for multi-view evidence completion. To make this distinction clearer to readers, we have added relevant explanations after Eq. (1) in Section 2.2.1.

In the revised manuscript provided with this rebuttal, we have added further descriptions clarifying that rare words are identified based on their occurrence frequencies in the training captions. This is essentially analogous to the way the rarity of class-label words is determined. The definitions and mathematical formulations are also carefully reviewed and revised throughout the manuscript. In particular, we have corrected the mathematical notation and symbol definitions to ensure a consistent distinction between vectors and scalars.

AuthorsOpenReview ↗

Addtional response for Q1

Addtional respond for Q1 (Performance improvements)

A5: To further validate the robustness of PMLL’s improvements, we conducted additional experiments with different random seeds on COCO and VG and report the corresponding mean and standard deviation in A.6 (shown in Table b and c).

Table b: Results of joint captioning and classification on COCO over different random seeds. Results for CIDEr, Polos, KLE, Tail, and Avg are reported as mean ± standard deviation. The optimal values in important columns are bolded.

Metric / MethodBLEUCIDErR-LSPICEC-SR-C-SP-S++R-P-S++PolosKLEHeadMidTailAvg
Softmax0.4773.373 ± 0.0050.6220.5660.7930.8460.5800.7080.783 ± 0.0014.352 ± 0.0020.8630.7960.754 ± 0.0010.800 ± 0.001
Focal0.4423.070 ± 0.0040.5950.5390.7910.8430.5790.7070.776 ± 0.0014.048 ± 0.0060.8510.7740.713 ± 0.0020.773 ± 0.001
CLA0.4553.145 ± 0.0230.6070.5500.7920.8440.5800.7060.776 ± 0.0014.100 ± 0.0180.8640.7920.743 ± 0.0050.794 ± 0.001
BS0.4342.929 ± 0.0050.5910.5340.7910.8420.5780.7040.768 ± 0.0003.928 ± 0.0040.8460.7910.755 ± 0.0020.793 ± 0.000
LinUpper0.4683.287 ± 0.0190.6140.5600.7920.8450.5790.7070.779 ± 0.0004.258 ± 0.0090.8580.8010.753 ± 0.0020.799 ± 0.003
TUL0.4773.383 ± 0.0040.6220.5670.7930.8460.5800.7090.783 ± 0.0004.357 ± 0.0070.8610.7970.752 ± 0.0000.799 ± 0.001
Gala0.4773.376 ± 0.0040.6220.5670.7930.8460.5800.7080.783 ± 0.0014.355 ± 0.0020.8630.7970.753 ± 0.0010.799 ± 0.001
TR0.4683.292 ± 0.0020.6160.5610.7920.8450.5790.7070.780 ± 0.0004.177 ± 0.0130.8490.7940.753 ± 0.0000.794 ± 0.001
Bal0.4783.393 ± 0.0090.6230.5690.7930.8470.5800.7080.783 ± 0.0004.362 ± 0.0080.8630.8000.755 ± 0.0020.801 ± 0.000
PMLL0.4793.402 ± 0.0040.6230.5680.7930.8470.5800.7090.784 ± 0.0014.371 ± 0.0040.8600.7970.763 ± 0.0030.802 ± 0.000

Table c: Results of joint captioning and classification on VG over different random seeds.

Metric / MethodBLEUCIDErR-LSPICEC-SR-C-SP-S++R-P-S++PolosKLEHeadMidTailAvg
Softmax0.3811.712 ± 0.0040.4870.4530.7930.8380.5770.6990.797 ± 0.0013.358 ± 0.0010.7290.6330.463 ± 0.0020.593 ± 0.002
Focal0.3551.533 ± 0.0090.4650.4360.7900.8350.5770.6980.790 ± 0.0013.169 ± 0.0020.6750.5680.405 ± 0.0000.534 ± 0.000
CLA0.3571.501 ± 0.0020.4660.4310.7910.8340.5780.6970.788 ± 0.0003.093 ± 0.0100.7290.6240.446 ± 0.0010.584 ± 0.001
BS0.0720.250 ± 0.2430.097N/A0.6710.6590.5050.5520.348 ± 0.1281.165 ± 0.3880.6820.5840.423 ± 0.0070.549 ± 0.006
LinUpper0.3631.572 ± 0.0010.4700.4370.7930.8350.5770.6970.790 ± 0.0003.195 ± 0.0090.7280.6260.455 ± 0.0020.588 ± 0.002
TUL0.3821.717 ± 0.0040.4870.4530.7930.8380.5770.6990.797 ± 0.0013.357 ± 0.0010.7270.6300.459 ± 0.0010.591 ± 0.002
Gala0.3681.594 ± 0.0210.4750.4430.7940.8370.5770.6980.793 ± 0.0013.241 ± 0.0070.7230.6300.459 ± 0.0050.589 ± 0.005
TR0.3711.626 ± 0.0030.4800.4430.7930.8360.5770.6980.792 ± 0.0003.167 ± 0.0170.7260.6330.471 ± 0.0070.594 ± 0.006
Bal0.3831.723 ± 0.0010.4880.4540.7930.8390.5770.6990.797 ± 0.0003.372 ± 0.0020.7270.6310.461 ± 0.0000.592 ± 0.002
PMLL0.3821.724 ± 0.0030.4880.4530.7930.8390.5780.7000.797 ± 0.0013.372 ± 0.0040.7250.6330.470 ± 0.0020.595 ± 0.002

The results show that PMLL consistently achieves the best overall classification accuracy while maintaining the strongest or highly competitive captioning performance across both datasets. Its small standard deviations further demonstrate that these improvements remain stable under different random seeds. Although the gains on some individual metrics are modest, their consistency across tasks, datasets, and random seeds provides additional empirical support for both the modality-inconsistent long-tail perspective and the proposed solution.

Reviewer_bXhXOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the long-tail problem in VLM fine-tuning. The authors argue that existing methods characterize concept rarity using a single aggregate frequency, overlooking differences in the rarity of the same concept across visual and textual modalitie, a phenomenon they term the “modality-inconsistent long tail.” To address this issue, they propose PMLL, a framework that reweights token-level losses through three components: (i) modality-decoupled reweighting based on concept frequencies and feature compactness estimated using a teacher VLM from image-only, text-only, and joint image–text perspectives; (ii) increased emphasis on rare words in captions; and (iii) adaptive optimization that dynamically balances the captioning and classification tasks. The authors also introduce KLE, a keyword-weighted evaluation metric, and a co-occurrence-aware protocol for mining tail concepts. Experiments are conducted on COCO, CC3M, and VG using LoRA fine-tuning, with Qwen2.5-VL-32B as the teacher and Qwen2.5-VL-3B as the student. Additional experiments examine alternative student and teacher models, evaluate VQA performance, and provide ablation studies. Overall, the results show that PMLL performs on par with or slightly better than existing long-tail methods in the joint captioning and classification setting.

Critical strengths
  1. I find the problem the paper tackles quite interesting. Previous long-tail methods mostly rely on a single overall frequency and don’t consider that the same concept can be rare in one modality but common in another. The authors pull out this “modality-inconsistent long-tail” issue and analyze it, which I think is a fresh angle, and it connects nicely to real VLM issues like hallucination and missed objects.

  2. I also like the idea behind the method. On one side it looks at data statistics (frequency and feature compactness), and on the other it looks at how the model is learning, adjusting weights based on token difficulty. Combining the two for loss reweighting gives consistent gains across several datasets and models. The design is a bit complex (a little "engineering", not very graceful for me), but the idea is still worth taking note of.

Critical weaknesses

1. The method feels over-engineered, and the effectiveness of each component is not well validated.

The problem itself is interesting, but the solution is quite heavily engineered. The framework stacks several mechanisms (MLR, tail compensation, RWR, ATO) and introduces many hyperparameters (λV,T,J\lambda_{V,T,J}, ηcls\eta_{cls}, ηcap\eta_{cap}, ρ\rho, the number of bins BB, momentum μ\mu, etc.), many of which are “empirically set”. Overall it reads more like a set of heuristic designs. The paper also lacks a clear and complete ablation showing that each component is actually necessary. For example, in Tab. 3, adding ATO barely changes anything (CIDEr 3.405 vs. 3.406, classification Avg 0.802 vs. 0.802), the tail compensation term is never ablated on its own, and most hyperparameters come with no sensitivity analysis.

2. The core claim of “modality decoupling” is not really supported by the experiments.

The main selling point is to distinguish long-tail distributions across modalities. However, changing λ\lambda from uniform to 0.8 on the joint view leaves results essentially unchanged, and the ablation is missing the key control of using only joint-view statistics. As a result, it is unclear whether the gains come from modality decoupling, or simply from teacher-derived statistics plus compactness-based weighting. In addition, the motivating analysis in Fig. 1 shows only a fairly loose correlation, with no correlation coefficients reported.

3. The improvements are very small, which limits how convincing the conclusions are.

In the main experiments, PMLL’s gains over the strongest baseline are mostly in the range of 0.001 to 0.003, yet results are from a single seed, with only one additional seed on COCO. It is hard to tell whether these gains go beyond random variation. Moreover, under InternVL3.5-4B (Tab. 2) and the Gemma teacher (Tab. 4), TR actually outperforms PMLL on classification, which does not match the claim that PMLL is “consistently” the best. Also, although the paper is motivated by hallucination, it does not evaluate on dedicated hallucination benchmarks such as CHAIR or POPE.

Questions for authors

1.On ablations and hyperparameters (related to Weaknesses 1 and 2):

Could the authors provide a more complete ablation, including (a) a control using only joint-view statistics (i.e., λJ=1\lambda_J=1) to verify whether modality decoupling itself brings gains; (b) results with the tail compensation term removed; and (c) a sensitivity analysis for η\eta, ρ\rho, BB, and μ\mu? Also, since adding ATO barely changes the results, how is its necessity justified?

2.On statistical significance (related to Weakness 3):

Could the authors report the mean and standard deviation over multiple seeds on all three datasets? How do the authors interpret TR outperforming PMLL under the InternVL and Gemma settings? In addition, how exactly is the “Overall avg relative increase” in the tables computed?

3. On the fairness of the experimental setup (clarification):

During student training, are the targets the original dataset annotations, or captions and labels generated by the teacher? Do all baselines use exactly the same training targets? Also, since CC3M has no object category annotations, how are its classification labels obtained? If PMLL and the baselines are trained on different targets, part of the improvement may come from distillation rather than from the reweighting itself.

9 comments
AuthorsOpenReview ↗

Response to the Question 1

we sincerely thank you for taking the time to review our paper and provide these valuable comments.

Q1 (ablations and hyperparameters)

A1: We have conducted a more comprehensive set of ablation experiments with seed and data seed set to 0 (as shown in Table a below), and updated Section 4.3 and A.8 accordingly. These experiments include comparisons between frequency-based and compactness-based weighting, as well as additional analyses with the tail compensation term removed, to verify whether modality separation itself brings actual benefits.

Table a: Ablation and sensitivity analysis of PMLL. MLR, RWR, and ATO denote modality-decoupled long-tail re-weighting, rare word reinforcement, and adaptive task-aware optimization, respectively. † denotes the full configuration of PMLL (highlighted in light orange in the original table). The best results within each block are shown in bold. A/B denote the weighting types used by MLR for frequency and compactness.

The results show that modality separation consistently brings improvements when applied to either frequency-based or compactness-based weighting. The tail compensation term likewise makes its intended contribution. In addition, our sensitivity analyses across multiple hyperparameters indicate that PMLL is relatively insensitive to these hyperparameters, and we have selected robust default values.

Regarding ATO, its ablation results indicate that although its performance gains are relatively modest, it remains an important component for improving the overall multi-task performance of the model. Specifically, incorporating ATO leads to improvements in tail-class accuracy and the semantic features of generated captions. Furthermore, our current experiments only consider two tasks. When more multimodal generation subtasks are jointly considered, balancing the contributions of different tasks will become even more important, further highlighting the potential value of ATO.

AuthorsOpenReview ↗

Table a: Ablation and sensitivity analysis of PMLL.

Table a: Ablation and sensitivity analysis of PMLL. MLR, RWR, and ATO denote modality-decoupled long-tail re-weighting, rare word reinforcement, and adaptive task-aware optimization, respectively. † denotes the full configuration of PMLL (highlighted in light orange in the original table). The best results within each block are shown in bold. A/B denote the weighting types used by MLR for frequency and compactness.

MLR/ViewRWRATOλV\lambda_VλT\lambda_TλJ\lambda_JBBμ\muρ\rhoηcls\eta_{\mathrm{cls}}ηcap\eta_{\mathrm{cap}}BLEUCIDErR-LSPICEPolosKLETailAvg.
MLR settings
×××------300.750.500.50.10.4773.3750.6220.5660.7834.3450.7550.800
Joint/-××0.250.250.50300.750.500.50.10.4763.3670.6220.5660.7834.3400.7580.801
Multi/-××0.250.250.50300.750.500.50.10.4773.3740.6220.5660.7834.3450.7580.801
Joint/Joint××0.250.250.50300.750.500.50.10.4763.3690.6220.5660.7834.3380.7580.801
Multi/Joint××0.250.250.50300.750.500.50.10.4773.3760.6220.5660.7834.3440.7580.801
Joint/Multi××0.250.250.50300.750.500.50.10.4763.3740.6220.5660.7834.3430.7580.801
✓××0.250.250.50300.750.500.50.10.4773.3760.6220.5660.7834.3470.7580.801
Component
✓✓×0.250.250.50300.750.500.50.10.4803.3980.6240.5670.7834.3610.7580.802
✓†✓✓0.250.250.50300.750.500.50.10.4793.3920.6230.5670.7834.3640.7600.802
✓✓✓0.330.330.34300.750.500.50.10.4793.3900.6230.5670.7834.3630.7600.801
✓✓✓0.100.100.80300.750.500.50.10.4793.3860.6230.5670.7834.3600.7600.801
ATO: BB
✓✓✓0.250.250.50100.750.500.50.10.4793.3920.6230.5670.7834.3650.7610.801
✓✓✓0.250.250.50200.750.500.50.10.4793.3870.6230.5670.7834.3600.7600.801
✓✓✓0.250.250.50400.750.500.50.10.4793.3890.6230.5670.7834.3620.7600.802
✓✓✓0.250.250.50600.750.500.50.10.4793.3960.6230.5670.7834.3660.7610.802
ATO: μ\mu
✓✓✓0.250.250.50300.000.500.50.10.4793.3860.6230.5670.7834.3620.7600.801
✓✓✓0.250.250.50300.500.500.50.10.4793.3890.6240.5670.7834.3600.7600.801
✓✓✓0.250.250.50300.900.500.50.10.4793.3860.6230.5670.7834.3640.7600.802
✓✓✓0.250.250.50300.990.500.50.10.4793.3910.6230.5670.7834.3640.7600.801
RWR: ρ\rho
✓✓✓0.250.250.50300.750.100.50.10.4773.3770.6220.5660.7834.3470.7600.802
✓✓✓0.250.250.50300.750.300.50.10.4793.3930.6230.5670.7834.3550.7590.801
✓✓✓0.250.250.50300.750.800.50.10.4803.4090.6240.5670.7844.3760.7600.801
✓✓✓0.250.250.50300.751.000.50.10.4813.4110.6240.5680.7844.3850.7590.801
coefficients
✓✓✓0.250.250.50300.750.500.50.00.4793.3890.6230.5670.7834.3620.7600.801
✓✓✓0.250.250.50300.750.500.00.10.4793.3950.6230.5670.7834.3670.7600.801
✓✓✓0.250.250.50300.750.500.50.50.4793.3980.6230.5680.7844.3670.7590.801
✓✓✓0.250.250.50300.750.500.10.10.4793.3910.6230.5670.7834.3650.7600.801
AuthorsOpenReview ↗

Response to the Question 2

Q2 (statistical significance)

A2: To better address this concern, we conducted additional single-task experiments (as shown in Table b below) in A.7 to investigate why PMLL does not achieve the best performance across all individual metrics. Our primary objective is to improve the overall performance of multimodal tasks on multimodal long-tailed datasets. Accordingly, our previous experiments focused on jointly optimizing multiple subtasks. The single-task results reveal that some competing methods are specifically tailored to the captioning task, whereas TR is better suited to the classification task, which explains their superior performance on certain individual metrics. In contrast, PMLL aims to achieve consistent and stable improvements in overall performance across multiple multimodal generation tasks through joint optimization. Therefore, it is not necessarily expected to outperform all competing methods on every individual metric.

Table b: Comparison of single generation tasks, including captioning, classification (with hallucination performance) on COCO. Method/M means employing our modality-decoupled weighting computed using multi-modal frequency, instead of joint weighting. The optimal values in important columns are bolded.

Metric / MethodBLEUR-LCIDErSPICEC-SR-C-SP-S++R-P-S++PolosKLETailAvgTail HR (%)Avg HR (%)
Softmax0.4920.6323.5430.5740.7930.8470.5790.7090.7834.4550.7370.79024.64216.806
Focal0.4400.5923.0440.5310.7930.8440.5800.7070.7734.0220.6210.70520.19213.722
CLA0.4720.6213.3110.5600.7920.8440.5790.7060.7774.2470.7090.77922.49916.738
BS0.4500.6043.1060.5490.7910.8430.5780.7050.7704.1370.7410.79025.87816.901
LinUpper0.4920.6333.5430.5740.7930.8470.5790.7090.7834.4590.7330.78824.80516.767
TUL0.4920.6323.5430.5740.7930.8470.5790.7080.7834.4530.7320.78824.40616.650
Gala0.4920.6333.5540.5750.7920.8470.5790.7090.7834.4610.7350.79024.40916.734
TR0.4810.6263.3970.5680.7930.8460.5800.7080.7834.3380.7480.79425.57317.166
Bal0.4930.6343.5430.5750.7930.8470.5790.7090.7844.4650.7150.77622.97515.540
PMLL0.4920.6333.5450.5740.7930.8470.5790.7080.7834.4560.7430.79424.65616.707
Bal/M0.4930.6343.5430.5750.7930.8480.5800.7090.7854.465N/AN/AN/AN/A
Gala/M0.4920.6333.5550.5750.7920.8470.5790.7090.7834.463N/AN/AN/AN/A
TR/MN/AN/AN/AN/AN/AN/AN/AN/AN/AN/A0.7490.79524.54517.164

Nevertheless, our additional weighting-replacement experiments further demonstrate the effectiveness of the proposed modality-decoupled mechanism. Specifically, applying our proposed weighting strategy consistently brings further improvements to other methods, including on the tasks where they already demonstrate superior performance. Furthermore, the experiments with different random seeds in A.6 provide additional evidence of the effectiveness and robustness of PMLL.

Moreover, we have added the calculation details of the Overall avg relative Increase score in Section 4.2. We have also introduced an additional GR metric to quantify the improvement achieved by PMLL over the baseline Softmax relative to that achieved by the strongest competing method, where the relative improvements of all methods are first calculated with respect to Softmax. The results demonstrate that PMLL delivers consistent and substantial improvements across multimodal tasks.

Reviewer_1bpHOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that a concept can be rare in images but common in text, so one overall frequency count can miss where a vision language model needs more help. It proposes PMLL to weight training by modality and tests it on COCO, CC3M, and Visual Genome, with the clearest gains in tail class recognition.

Critical strengths
  1. I like that the paper points out a simple issue: a concept can be common in text but rare in images, so one frequency count does not tell the whole story. This matters because the ablation shows that accounting for those differences improves tail class accuracy on COCO, giving the main idea some direct support.
Critical weaknesses
  1. The gains seem modest. PMLL improves some tail-class results, but it does not consistently outperform the other methods, so the overall benefit is unclear.

  2. The ablation does not make clear what accounts for the gains. Is it the separation of modalities, feature compactness, or the teacher model? Since modality separation is the paper’s main claim, I would like to see its contribution isolated more clearly.

  3. The evaluation is fairly narrow. The datasets share the same 80 concepts, and the method relies on a large teacher model to construct the weights. It is unclear whether the results would hold for open-vocabulary concepts or at a larger scale.

Questions for authors

In Table 3, how much of the tail class gain comes from separating the modalities, and how much comes from the compactness term? A controlled comparison would really help me judge the paper’s main claim.

5 comments
AuthorsOpenReview ↗

Response to the Question

Thank you for taking the time to review our paper and for raising this valuable question.

Q1 (Modality-decoupled Ablation strategy)

A1: We agree with your suggestion that the contributions of frequency and compactness should be disentangled to demonstrate that the observed gains come from our proposed modality-decoupled weighting strategy, rather than simply from introducing additional compactness-based weights. To address this concern, we conducted a more comprehensive ablation study with seed and data seed set to 0, as presented in Section 4.3 (as shown in Table a below), and added a more detailed analysis of the results.

Table a: Ablation and sensitivity analysis of PMLL. MLR, RWR, and ATO denote modality-decoupled long-tail re-weighting, rare word reinforcement, and adaptive task-aware optimization, respectively. † denotes the full configuration of PMLL (highlighted in light orange in the original table). The best results within each block are shown in bold. A/B denote the weighting types used by MLR for frequency and compactness.

MLR/ViewRWRATOλV\lambda_VλT\lambda_TλJ\lambda_JBBμ\muρ\rhoηcls\eta_{\mathrm{cls}}ηcap\eta_{\mathrm{cap}}BLEUCIDErR-LSPICEPolosKLETailAvg.
MLR settings
×××------300.750.500.50.10.4773.3750.6220.5660.7834.3450.7550.800
Joint/-××0.250.250.50300.750.500.50.10.4763.3670.6220.5660.7834.3400.7580.801
Multi/-××0.250.250.50300.750.500.50.10.4773.3740.6220.5660.7834.3450.7580.801
Joint/Joint××0.250.250.50300.750.500.50.10.4763.3690.6220.5660.7834.3380.7580.801
Multi/Joint××0.250.250.50300.750.500.50.10.4773.3760.6220.5660.7834.3440.7580.801
Joint/Multi××0.250.250.50300.750.500.50.10.4763.3740.6220.5660.7834.3430.7580.801
✓××0.250.250.50300.750.500.50.10.4773.3760.6220.5660.7834.3470.7580.801
Component
✓✓×0.250.250.50300.750.500.50.10.4803.3980.6240.5670.7834.3610.7580.802
✓†✓✓0.250.250.50300.750.500.50.10.4793.3920.6230.5670.7834.3640.7600.802
✓✓✓0.330.330.34300.750.500.50.10.4793.3900.6230.5670.7834.3630.7600.801
✓✓✓0.100.100.80300.750.500.50.10.4793.3860.6230.5670.7834.3600.7600.801

Specifically, our results demonstrate that modality-decoupled weighting provides more effective long-tailed supervision. Both frequency-based and compactness-based modality-decoupled weighting improve the overall generation performance, while combining the two achieves the best results. Furthermore, the slightly better performance of frequency-based weighting compared with compactness-based weighting provides additional evidence supporting our motivation for modality separation.

Moreover, we would like to clarify the motivation behind introducing the compactness term. In realistic and complex multimodal long-tailed learning scenarios, we argue that the number of samples alone may not adequately reflect the degree of long-tailedness across different classes. Some classes may contain a relatively large number of samples, but these samples can be highly redundant and therefore provide insufficient additional supervision. This motivates us to incorporate the compactness term to better characterize the effective long-tail supervision available for each class.

Furthermore, the ablation results indicate that the proposed modality-decoupled weighting strategy improves captioning performance but does not yield gains in classification. Therefore, we conduct additional single-task experiments in A.7 to further investigate its impact on tail-class performance. In the single-task experiments, we conducted additional weighting-adjustment experiments to investigate whether the compared methods could further improve their already strong performance on individual tasks with our modality-decoupled weighting strategy(as shown in Table b below). Specifically, after replacing their original weighting schemes with our proposed frequency-based modality-decoupled weighting, these methods achieve further improvements over their original single-task results. These findings provide additional support for our argument and further demonstrate the effectiveness of the proposed weighting strategy.

AuthorsOpenReview ↗

Response to the Weakness 1

Table b: Comparison of single generation tasks, including captioning, classification (with hallucination performance) on COCO. Method/M means employing our modality-decoupled weighting computed using multi-modal frequency, instead of joint weighting. The optimal values in important columns are bolded.

Metric / MethodBLEUR-LCIDErSPICEC-SR-C-SP-S++R-P-S++PolosKLETailAvgTail HR (%)Avg HR (%)
Softmax0.4920.6323.5430.5740.7930.8470.5790.7090.7834.4550.7370.79024.64216.806
Focal0.4400.5923.0440.5310.7930.8440.5800.7070.7734.0220.6210.70520.19213.722
CLA0.4720.6213.3110.5600.7920.8440.5790.7060.7774.2470.7090.77922.49916.738
BS0.4500.6043.1060.5490.7910.8430.5780.7050.7704.1370.7410.79025.87816.901
LinUpper0.4920.6333.5430.5740.7930.8470.5790.7090.7834.4590.7330.78824.80516.767
TUL0.4920.6323.5430.5740.7930.8470.5790.7080.7834.4530.7320.78824.40616.650
Gala0.4920.6333.5540.5750.7920.8470.5790.7090.7834.4610.7350.79024.40916.734
TR0.4810.6263.3970.5680.7930.8460.5800.7080.7834.3380.7480.79425.57317.166
Bal0.4930.6343.5430.5750.7930.8470.5790.7090.7844.4650.7150.77622.97515.540
PMLL0.4920.6333.5450.5740.7930.8470.5790.7080.7834.4560.7430.79424.65616.707
Bal/M0.4930.6343.5430.5750.7930.8480.5800.7090.7854.465N/AN/AN/AN/A
Gala/M0.4920.6333.5550.5750.7920.8470.5790.7090.7834.463N/AN/AN/AN/A
TR/MN/AN/AN/AN/AN/AN/AN/AN/AN/AN/A0.7490.79524.54517.164

W1 (Insufficient performance improvements)

A2: First, we would like to clarify that the reason our method does not achieve the best results across all individual metrics is related to the joint learning objective of our work. Specifically, captioning and classification are both important tasks in multimodal generation, and our method jointly trains the model to accommodate both tasks. Therefore, PMLL aims to achieve consistent and stable improvements across these tasks, rather than improving one task at the expense of the other.

To better evaluate the overall improvements, we introduced the Overall avg Relative Increase score in Section 4.2 to approximately measure the relative performance gains of PMLL over the strongest competing method (0.1, 0.4, 0.2 for three datasets with same setting). Furthermore, in the revised manuscript provided with this rebuttal, we additionally introduce the GR metric to quantify the ratio of relative improvement of PMLL to that of the strongest competing method.(1.29×, 1.23×, 6.00× for three datasets with same setting). The results demonstrate that PMLL substantially improves the overall performance of multimodal generation tasks. In addition, the VQA results reported in Section 4.2 provide further evidence of PMLL's effectiveness in improving the understanding ability of image-text data.

Furthermore, the single-task experiments in A.7 provide additional insights into the performance differences among the compared methods (as shown in Table b above). Specifically, the superior performance of some methods on individual task metrics can be attributed to their task-specific improvements, which are tailored to individual tasks. In contrast, our method aims to robustly improve the overall performance across multimodal generation tasks. Moreover, our additional weighting replacement experiments demonstrate the necessity of addressing the proposed modality-inconsistent problem, which is prevalent across different subtasks in multimodal generation.

AuthorsOpenReview ↗

Response to the Weakness 3

W3 (Narrow evaluation)

A3: To address this concern, we have added an extended-vocabulary evaluation using the 100-category label set of CIFAR-100 (as shown in Table c below), as reported in Section 4.4. Furthermore, we specifically control the teacher-model scale to examine whether the improvements originate from the proposed weighting mechanism rather than from stronger teacher supervision. Qwen3.5-4B is used to generate the training-set ground-truth captions and class annotations, Qwen2.5-32B is used exclusively to construct the test-set references, and Qwen2.5-3B serves as the student model. Therefore, the training supervision and modality-specific weight construction do not rely on a large-scale teacher model.

Table c: Results of VLM fine-tuning on CoCo using label set of CIFAR-100. The optimal values in important columns are bolded.

MethodBLEUCIDErR-LSPICEC-SR-C-SP-S++R-P-S++PolosKLETailAvg
Softmax0.4583.5700.6380.5880.7830.8450.5740.7070.7513.6550.2940.533
Focal0.4103.0930.6020.5410.7840.8420.5750.7050.7393.3090.2780.495
CLA0.4643.6470.6500.5900.7810.8450.5730.7070.7573.8120.1980.460
BS0.4483.4660.6320.5830.7840.8440.5750.7070.7513.6470.4250.589
LinUpper0.4503.5380.6290.5800.7800.8430.5710.7050.7493.6180.2980.538
TUL0.4603.5950.6400.5900.7830.8450.5740.7070.7523.6550.2930.531
Gala0.4593.5830.6400.5890.7830.8450.5750.7070.7523.6630.2940.532
TR0.4353.4000.6190.5740.7800.8420.5700.7030.7373.4560.3580.573
Bal0.4683.6800.6470.5960.7830.8460.5740.7080.7573.7410.2890.531
PMLL0.4733.6990.6540.6110.7830.8460.5750.7090.7603.8450.3810.587

The results show that PMLL achieves the best performance on 8 out of 10 captioning metrics while maintaining competitive classification accuracy (58.67%), close to the best baseline (58.95%). It also achieves a 2.7% overall average relative increase score with a GR of 2.12$\times$. These findings suggest that the improvements are not simply a consequence of employing a substantially larger teacher model. Instead, they further support the effectiveness of our modality-decoupled weighting mechanism in exploiting modality-specific long-tailed information under a broader concept space.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Rebuttal summary

In this rebuttal, we first refined the equation, figure formatting, and explanations of key terms of the paper to improve readability.

We then conducted four additional sets of experiments to address the reviewers’ questions and weeknesses.

  1. More detailed ablation studies to address whether the proposed modal-inconsistent decoupled weights are effective and whether the framework is sensitive to its hyperparameters. The results show that the proposed weights improve the model’s performance on multiple task objectives simultaneously. They also show that the framework has low sensitivity to its hyperparameters and that we selected the most stable default settings.

  2. Single-task experiments to address the reviewers’ concerns about modest gains and the lack of improvement across multiple metrics. The results show that jointly optimizing multiple objectives in multimodal generation is challenging since different objectives require different directions of optimization. Consequently, competing methods perform differently across metrics. Differently, our method improves performance across the objectives, which explains why the gains on individual metrics may appear modest. In particular, existing methods achieve overall improvements simply by adopting our decoupled weights, further demonstrating the importance of the problem and the effectiveness of the proposed method.

  3. Experiments assessing dependence on the teacher model and concept set to address the concern that the method might lose effectiveness with a larger concept set or a smaller teacher model. The results show that the method remains effective with larger concept sets and yields gains even with a very small teacher model, proving the robustness and effectiveness of the proposed method.

  4. Experiments across multiple random seeds to address the reviewers’ concern about whether the method can deliver robust gains. The results show that the method delivers consistent and strong improvements across seeds.

Overall, we have addressed every question and weakness raised by the reviewers through our responses and revised manuscript, and demonstrated the importance of the problem and the effectiveness of the proposed method.

Dear Reviewers,

The rebuttal period is coming to an end. Please read the authors' responses and update your reviews if appropriate. It would be helpful to share your thoughts as soon as possible to allow enough time for a productive discussion. Thank you!

Best regards, Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.