acceptodds
Under review as a conference paper at ICLR 2027

Beyond Feature Shift: Measuring and Mitigating Output Distribution Collapse in Cross-Domain VQA

Abstract

Although vision-language models (VLMs) achieve strong performance on general-domain benchmarks, their effectiveness often degrades substantially when applied to domain-specific visual question answering (VQA) tasks such as medical imaging, remote sensing, and art understanding. Existing studies mainly attribute this degradation to feature-level distribution mismatch and focus on representation alignment. In this work, we show that such explanations are insufficient to characterize the failure mechanisms of pretrained VLMs under domain shift. Through a systematic study across six domain-specific VQA benchmarks and multiple VLM architectures, we identify a previously overlooked phenomenon termed output distribution collapse, characterized by prediction distribution collapse toward dominant answer prototypes, confidence miscalibration, and decision-margin instability. To quantify this phenomenon, we introduce an entropy-based Collapse Index (CI) that measures the severity of predictive distribution degeneration. We show that CI consistently exhibits a strong negative correlation with cross-domain VQA accuracy across datasets and architectures. Motivated by these findings, we propose Distribution Collapse Stabilization (DCS), a lightweight output-space learning framework that jointly promotes distribution dispersion, reliability consistency, and decision separability. Extensive experiments on six benchmarks and ten representative VLM backbones demonstrate consistent improvements, yielding an average gain of 4.1% over pretrained baselines while outperforming other adaptation methods. Our findings identify output distribution collapse as a previously underexplored failure mechanism in cross-domain VQA and establish output-space stabilization as a promising alternative to conventional feature-space adaptation.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
vision-language models, visual question answering, output distribution collapse

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 2 reviews, above 9% of the others.
Reviewer_8hfkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that cross-domain VQA failures partly stem from output distribution collapse, measured by a Collapse Index CI=1−H(p^)/log⁡C\mathrm{CI} = 1 - H(\hat{p}) / \log C on the dataset-level prediction histogram p^\hat{p}. It proposes DCS, which trains with L=LCE+λ(t)(Ldisp+αLrel+βLsep)\mathcal{L} = \mathcal{L}_{\mathrm{CE}} + \lambda(t) \big( \mathcal{L}_{\mathrm{disp}} + \alpha \mathcal{L}_{\mathrm{rel}} + \beta \mathcal{L}_{\mathrm{sep}} \big), combining a per-sample entropy floor, a penalty (max⁡cpc−py)2(\max_c p_c - p_y)^2, and a logit margin. Across six datasets and ten backbones, DCS improves over CE fine-tuning by about 4 points on average and lowers both CI and ECE.

Critical strengths

Examining the predicted-answer distribution, rather than only feature shift, is a reasonable and cheap diagnostic for closed-vocabulary VQA under domain shift: CI\mathrm{CI} and top-kk concentration are model-agnostic and easy to compute. The experiments are broad, covering six domains (medical, remote sensing, art) and ten backbones from ViLT to Qwen3-VL and InternVL3.5, and the gains over CE fine-tuning are consistent in direction (e.g., CLIP on VQA-RAD: 66.7→72.866.7 \rightarrow 72.8), at about 0.60.6\\% extra training cost and no inference cost.

Critical weaknesses
  1. The loss does not target what the metric measures, and "collapse" has no reference point. CI\mathrm{CI} is defined on the marginal prediction distribution over the dataset, whereas Ldisp\mathcal{L}_{\mathrm{disp}} raises the entropy of each individual prediction. These are different quantities: a model can be confident on every sample and still have a diverse marginal. Forcing per-sample entropy above Hmin⁡=0.3log⁡CH_{\min} = 0.3 \log C works against confident correct predictions, so it is unclear why this term should fix marginal collapse, or why the lower ECE (e.g., 0.302→0.1680.302 \rightarrow 0.168) is not simply the result of pushing the model toward under-confidence. Lrel\mathcal{L}_{\mathrm{rel}} is zero whenever the prediction is correct, and Lsep\mathcal{L}_{\mathrm{sep}} with a log-sum-exp over negatives is close to a hinge version of CE, so their specific role is unclear. More fundamentally, the paper never reports the CI\mathrm{CI} of the ground-truth test answers. In VQA-RAD and SLAKE many closed questions are yes/no, so a low-entropy prediction histogram may largely match the true answer distribution. Methods that directly correct marginal prediction bias (e.g., logit adjustment or prior correction under label shift) are not discussed, and no feature-level measurement supports the "beyond feature shift" framing.

  2. Some results are inconsistent or implausible. In Table 7, temperature scaling raises accuracy over CE (VQA-RAD 66.7→68.566.7 \rightarrow 68.5, OVQA 70.1→72.370.1 \rightarrow 72.3, SemArt 26.4→27.526.4 \rightarrow 27.5). Post-hoc temperature scaling divides all logits by one constant and cannot change the arg⁡max⁡\arg\max, so accuracy should be identical to CE. Either a different procedure was run or the table contains errors, which lowers my confidence in the other comparisons. The average gain is reported as 4.14.1\\% (abstract), 4.24.2\\% (Sec. 4.4), and ≈4.0\approx 4.0\\% (large VLMs); the †\dagger and ∗\ast markers in Table 1 are not explained; no seeds or variances are reported; and the fine-tuning setup for LLaVA, BLIP-2, Qwen3-VL, and InternVL3.5 (full vs. LoRA, how the closed answer space is attached, how it fits on V100s) is missing. The SemArt "SOTA" claim (31.431.4 vs. 22.422.4 for BAN) compares a modern backbone with much older methods.

Questions for authors
  1. What is the CI\mathrm{CI} of the ground-truth test answers on each dataset? Could you also break down VQA-RAD and SLAKE into closed and open questions?
  2. How exactly was temperature scaling applied in Table 7, and why does it change accuracy?
  3. How does DCS compare with a regularizer on the batch-level marginal entropy H(pˉbatch)H(\bar{p}_{\text{batch}}), and with logit adjustment using the training answer prior?
  4. How were the large VLMs fine-tuned, and what is the answer head? Are FSA and DCS applied identically to them?
  5. What is the variance across seeds, and what correlation coefficient supports the claimed link between CI\mathrm{CI} and accuracy?
Reviewer_9LRyOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses the problem of performance degradation in cross-domain visual question answering (VQA) which is practically important. It introduces a Collapse Index, defined as normalized entropy deficiency of the empirical distribution of predicted answers, and proposes Distribution Collapse Stabilization, comprising an entropy-floor penalty, a confidence ground-truth-probability consistency loss, a logit-margin penalty, and an adaptive regularization schedule. Experiments cover six conventional VLM backbones across six datasets, with additional results for four large VLMs on four datasets. The tables report improvements over the baseline models, FSA, and several standard regularizers. The paper also reports reductions in ECE and output concentration, together with negative CI-accuracy correlations.

Critical strengths
  1. Examining answer distributions alongside accuracy is a worthwhile direction. The paper goes beyond reporting a single performance metric and attempts to connect prediction concentration, calibration, and decision behavior.
  2. The paper reports consistent performance improvements over the backbone baselines and the FSA adaptation method across multiple datasets and model architectures. The breadth of the evaluation is a notable strength and suggests that the method’s practical benefits are not confined to a single model or benchmark.
  3. The proposed method addresses cross-domain adaptation through additional loss terms rather than architectural modifications, offering a straightforward approach that does not require new architectural changes to the model.
  4. The paper also includes component-wise ablations, comparisons with standard regularization methods, and results reported as mean and standard deviation over three runs for selected settings. These analyses provide useful evidence for assessing the method’s components and the consistency of its reported gains.
Critical weaknesses
  1. CI measures prediction concentration relative to a uniform distribution without accounting for the ground-truth answer distribution. Consequently, even a highly accurate model could receive a high collapse score on a dataset with strongly imbalanced answer frequencies. This limits its ability to distinguish pathological prediction collapse from concentration that appropriately reflects the underlying data. Evaluating CI under controlled label imbalance and comparing predicted and ground-truth answer distributions would strengthen its diagnostic value.
  2. CI is computed from the dataset-level distribution of predicted answer categories, whereas the dispersion loss encourages higher entropy in each example’s predicted probability distribution. These are distinct quantities increasing uncertainty within individual predictions does not necessarily increase the diversity of predicted answers across the dataset. A model could produce relatively high-entropy probabilities while still selecting the same answer for every input. The paper therefore needs a stronger justification connecting the proposed loss to the phenomenon measured by CI.
  3. The reported negative correlation between CI and accuracy is informative, but it does not establish that mitigating output distribution collapse explains the performance improvements rather than more general regularization benefits. Although the comparisons with standard regularizers are useful, they do not fully resolve this distinction.
Questions for authors
  1. How does the CI tell the difference between a model's answers naturally leaning toward the correct prior versus a harmful collapse?
  2. What happens under controlled class imbalance and candidate-vocabulary changes?
  3. What theoretical or empirical evidence links the per-example entropy floor to the entropy of the hard predicted-label histogram?
  4. How is the reliability loss justified as calibration-oriented despite being zero for uniform predictions? How does the separability loss differ in effect from a true-label probability penalty?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.