acceptodds
Withdrawn from ICLR 2027

Stop Describing, Start Answering: Reducing Description Reliance in Vision-Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Existing Vision-Language Models (VLMs) have an undesirable habit in visual question answering: models often rely on image descriptions, first describing the visual content and then answering the question. This describe-then-answer pattern can improve accuracy, but it also produces redundant tokens and increases latency. When forced to answer directly, VLMs give shorter but often less accurate answers. We call this gap description reliance. To reduce this reliance without generating descriptions at inference, we propose Internalization, a two-stage training method that uses caption-derived answer supervision to train a compact image-question prior for direct answering. Stage I aligns the new prior module with the base model distribution through logits-wise warm-up, and Stage II injects prior tokens into hidden states through preference training, using the Stage I model as a teacher to reduce distribution drift. We further introduce InternVQA, a benchmark that measures description reliance across image settings, task types, and answer lengths. Experiments on Qwen3-VL and LLaVA-OneVision show that Internalization raises the average direct-answer score from to and shrinks the caption-direct gap from to judge points while increasing the caption score from to . It also achieves aggregate retention across ten public benchmarks and shortens end-to-end inference time by by generating fewer tokens, showing that caption-derived supervision can improve direct answering through compact prior tokens instead of generated descriptions.

Primary area:
foundation or frontier models, including LLMs
Keywords:
vision-language models, visual question answering, description reliance, direct answering, implicit prior, preference optimization, inference efficiency, benchmark, post-training

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_5cMxOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies description reliance in vision-language models. Generating an image description before answering a visual question often improves VQA accuracy, but also increases inference cost. The authors propose Internalization, a two-stage post-training method that transfers the benefit of caption-assisted answering into a compact latent prior, enabling direct answering. The paper also introduces InternVQA to measure the performance gap between direct answering and caption-then-answering.

Critical strengths

1、The paper addresses a practically important problem. It clearly shows the gap between direct and caption-assisted answering, while reducing this reliance can directly lower inference cost. 2、The evaluation is comprehensive, covering multiple VLM families and model scales, together with extensive ablations, public benchmarks, efficiency analysis, multiple evaluators, and human evaluation. This provides strong support for the main empirical findings.

Critical weaknesses

1、The current experiments do not fully establish that the gains come from "description internalization." Caption-then-answering adds not only an explicit description, but also extra autoregressive computation and self-generated context. Stage II further contrasts caption-assisted and unconstrained answers, so the training signal also reflects differences in answer quality and directness. It is therefore unclear whether explicit descriptions are the key factor. 2、The InternVQA judge knows which answer comes from direct answering and which comes from caption-then-answering. This may introduce evaluator bias. A cleaner setup would hide the inference mode and randomize answer order.

Questions for authors

1、Would similar gains remain if the chosen answers came from a strong teacher without an explicit caption-then-answer stage? This would help distinguish description internalization from more general stronger-inference distillation. 2、Could the authors re-evaluate the main InternVQA results with the inference mode hidden and the answer order randomized?

Reviewer_GMQHOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors proposes Internalization, a two-stage training method that enables VLMs to answer VQA task directly without generating intermediate long description of the original images. Specifically, the method introduces logits-wise warm-up and implicit prior injection. The authors also propose a new benchmark InternVQA to evaluate the proposed method.

Critical strengths
  • The paper addresses a practical problem of decreasing unnecessary image caption descriptions which improving direct answer effeciency;
  • The paper conducts substantial experiments across four VLM models, and also provides detailed analysis experiments.
Critical weaknesses
  • The reliability of InternVQA is not fully validated. The performance gap between description-then-answering and direct answering may result from additional computation or different prompting strategies rather than description reliance alone;
  • The method causes performance decrease on some public VQA benchmarks. For example, Qwen3-VL-2B drops from 77.40 to 56.92 on ChartQA, raising concerns;
Questions for authors
  • How do the paper distinguish description reliance from effect of additional computation and different prompting strategies?

  • Can the authors explain the performance degradation on ChartQA after applying Internalization?

  • Can the authors compare against a simpler baseline that directly finetunes VLM using the same supervision, preference optimization, and KL regularization, without introducing additional prior modules?

Reviewer_RnSvOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper looks at a habit common in VLMs: they often describe the image before answering a question. That description improves accuracy, but it also costs tokens and time. The authors call the score gap between caption-then-answer and direct answering “description reliance.” To close it, they propose Internalization, a two-stage method that adds a small module at the last decoder layer:

a) Stage I: pools image and question tokens from the final hidden states using per-sample PCA and learns a correction to the output logits.

b) Stage II: turns that representation into 16 “prior tokens,” injects them into the hidden states through gated attention, and trains with a preference loss. The chosen response is the answer part of a caption-then-answer output, and the rejected response is an unconstrained answer. A KL term ties the model to Stage I.

They also introduce InternVQA, 1.5K questions scored by GPT-5-mini in both modes. On Qwen3-VL-2B/8B and LLaVA-OneVision-0.5B/7B, the average direct score goes from 40.2 to 63.1, and the gap shrinks from 16.6 to 4.0 points. The method keeps 96.5% aggregate performance on ten public benchmarks and cuts inference time by 24%.

Critical strengths
  1. A practical, well-motivated problem. Describe-then-answer is a real cost in deployed VLMs, and framing it as a measurable gap between two inference modes is simple and reusable.

  2. Broad comparison. The paper compares against ten baselines from five families on four backbones, reports public-benchmark retention and wall-clock time, and includes a human study and cross-family judges. On paper, the coverage is good.

Critical weaknesses
  1. The consistency of the baseline scores in Table 1 needs explaining. The base models vary a lot across subsets (for example, LLaVA-OV-0.5B ranges from −30.1 to 62.0). Several baselines, though, score almost the same in all 16 subsets: PromptCap on LLaVA-OV-0.5B stays between 54.4 and 55.7, PAI between 85.5 and 86.5, and OPERA on Qwen3-VL-8B between 79.5 and 80.4. Since the subsets differ in size, task type, and base-model gap, I would have expected more variation. There may be a reason I’m missing, but the baseline comparison is central to the paper. A short explanation of how the subset scores were computed, or per-item outputs, would help me trust it.

  2. It would help to show more clearly that the gains come from the prior tokens and not from distillation alone. The prior is computed from the final-layer hidden states and injected back into that same layer, so it seems to work much like a last-layer adapter. Stage I also fine-tunes the last decoder layer and the LM head on 280K pairs, half of them labeled by GPT-5. The SFT control is described as using “the same chosen answers.” If that means it didn’t get the GPT-5 Stage I data, the comparison may favor the proposed method. The evaluation is also fairly close to the training setup: GPT-5 generates the training questions from the same ten task categories as InternVQA and labels half the answers, and GPT-5-mini judges without reference answers. Since the caption score also goes up (56.8 → 67.1), part of the improvement may come from general GPT-5 distillation rather than from internalizing descriptions. Finally, aggregate retention looks strong, but some individual tasks drop noticeably, for example ChartQA on Qwen3-VL-2B falls by 20.5 points.

Questions for authors
  1. Why are the baseline scores in Table 1 nearly constant across all subsets? Can you release per-item judge scores for the baselines?

  2. Is the SFT baseline trained on exactly the same data as your method, including the 140K GPT-5 Stage I answers, with the same trainable parameters? What does a plain context-distillation baseline get (fine-tuning the same layers on the answer part of caption-then-answer outputs, with no prior module)?

  3. Since the prior comes from the same final-layer states it is injected into, what does it add over a LoRA or adapter of matching size on the last layer?

  4. How does the method do on held-out VQA data that GPT-5 didn’t generate, with reference-based scoring?

  5. How were the baselines run (prompts, checkpoints, training data for NaPO and Visual-RFT)? How were Cap scores obtained for the decoding-only methods?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.