- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper argues that cross-domain VQA failures partly stem from output distribution collapse, measured by a Collapse Index on the dataset-level prediction histogram . It proposes DCS, which trains with , combining a per-sample entropy floor, a penalty , and a logit margin. Across six datasets and ten backbones, DCS improves over CE fine-tuning by about 4 points on average and lowers both CI and ECE.
Critical strengths
Examining the predicted-answer distribution, rather than only feature shift, is a reasonable and cheap diagnostic for closed-vocabulary VQA under domain shift: and top- concentration are model-agnostic and easy to compute. The experiments are broad, covering six domains (medical, remote sensing, art) and ten backbones from ViLT to Qwen3-VL and InternVL3.5, and the gains over CE fine-tuning are consistent in direction (e.g., CLIP on VQA-RAD: ), at about extra training cost and no inference cost.
Critical weaknesses
-
The loss does not target what the metric measures, and "collapse" has no reference point. is defined on the marginal prediction distribution over the dataset, whereas raises the entropy of each individual prediction. These are different quantities: a model can be confident on every sample and still have a diverse marginal. Forcing per-sample entropy above works against confident correct predictions, so it is unclear why this term should fix marginal collapse, or why the lower ECE (e.g., ) is not simply the result of pushing the model toward under-confidence. is zero whenever the prediction is correct, and with a log-sum-exp over negatives is close to a hinge version of CE, so their specific role is unclear. More fundamentally, the paper never reports the of the ground-truth test answers. In VQA-RAD and SLAKE many closed questions are yes/no, so a low-entropy prediction histogram may largely match the true answer distribution. Methods that directly correct marginal prediction bias (e.g., logit adjustment or prior correction under label shift) are not discussed, and no feature-level measurement supports the "beyond feature shift" framing.
-
Some results are inconsistent or implausible. In Table 7, temperature scaling raises accuracy over CE (VQA-RAD , OVQA , SemArt ). Post-hoc temperature scaling divides all logits by one constant and cannot change the , so accuracy should be identical to CE. Either a different procedure was run or the table contains errors, which lowers my confidence in the other comparisons. The average gain is reported as (abstract), (Sec. 4.4), and (large VLMs); the and markers in Table 1 are not explained; no seeds or variances are reported; and the fine-tuning setup for LLaVA, BLIP-2, Qwen3-VL, and InternVL3.5 (full vs. LoRA, how the closed answer space is attached, how it fits on V100s) is missing. The SemArt "SOTA" claim ( vs. for BAN) compares a modern backbone with much older methods.
Questions for authors
- What is the of the ground-truth test answers on each dataset? Could you also break down VQA-RAD and SLAKE into closed and open questions?
- How exactly was temperature scaling applied in Table 7, and why does it change accuracy?
- How does DCS compare with a regularizer on the batch-level marginal entropy , and with logit adjustment using the training answer prior?
- How were the large VLMs fine-tuned, and what is the answer head? Are FSA and DCS applied identically to them?
- What is the variance across seeds, and what correlation coefficient supports the claimed link between and accuracy?