acceptodds
Under review as a conference paper at ICLR 2027

Revisiting Caption Gains When the Image Remains Available

Abstract

In a multimodal conversation, an assistant generates a generic caption to summarize an image and retain it for later visual questions, even when the original image remains available at answer time. Whether such captions help beyond simply placing text in the conversation remains unclear. By generating one question-blind generic caption per model–image pair for seven vision-language models and testing 20 conditions on natural-image and medical-image question-answering datasets, we measure how much of the gain from caption history over a minimal exchange with a fixed, image-independent acknowledgement comes from the image description. To isolate the description's contribution, we compare each frozen caption with length-matched neutral text in the same assistant-history format. Neutral text alone accounts for much of this gain on the natural-image dataset, where replacing it with the caption adds only a small further gain and placing the same text alongside the question yields a caption–neutral difference of similar size. On the medical-image dataset, the caption adds no clear gain over neutral text in history, although on that dataset the average gain of neutral history over the minimal exchange and the average inline caption–neutral difference are each driven by one or two models. We further examine image placement and longer conversations to probe how these comparisons depend on the input sequence. With the original image available, a caption's description adds at most a small average gain beyond neutral text on either dataset. An image-removal replay shows that the captions carry answer-relevant content: when the image is removed, captions preserve most of the accuracy that length-matched neutral text loses on both datasets. A caption's value for later questions must therefore be understood in relation to continued access to the image.

Primary area:
datasets and benchmarks
Keywords:
vision-language models, evaluation methodology, length-matched control, visual question answering, conversation history

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ncuuOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The goal of this paper is to study how image information, in the form of a caption, is propagated through a text conversation with a multimodal model which retains history. 

The paper conducts an experiment on two datasets with natural and medical images respectively by comparing the resulting conversations with seven different VLMs, weighing the contribution from image captions versus neutral text.

In the experiment, various ways of incorporating the caption and history are explored. Results study how image and caption inclusions affects model performance.

Critical strengths

Interesting to study how visual information is used in caption form for multimodal conversations across a suite of models.

Critical weaknesses

Lack of examples and visualizations. It would help to see a concrete example in the introduction. Currently it is not clear what problem the paper is tackling and at what steps the image and caption information are included. It would help to show a real example with a visual query, conversation history, task, and model output.

The experimental setup in Section 3.2 needs more clarification and motivation. The contrasts and their significance could be explained better and currently read as a list of different settings. Are they critical to the experiment setup, or included for completeness? There are 20 different conditions listed and the different settings and this is a bit confusing for the reader to keep track of. It could be helpful to focus on the cases which lead to interesting insights. Sections 3 and 4 could also benefit from some concrete examples of the input history, query, image, and caption, and model outputs showing the change under the different conditions and experimental settings.

Limited and unclear takeaways. Although there are some insights about using image captions vs neutral text, it is not clear how the findings in the paper should change how we train or interact with multimodal models in the future.

Questions for authors

Section 4.2 reports many quantitative percentage point changes but could add more high level analysis. Why some of the trends so different for each dataset? Why is neutral text better on average for SLAKE?

For the query, what kinds of questions are being answered? And what do the answers look like? What tasks are being solved?

What are some example of captions? How long are the captions, and what level of detail do they cover? What does the conversation history look like? How do the results change as the caption quality or detail increases?

What happens if you add the image at every step in the conversation history?

How were the datasets Visual7W and SLAKE selected? Especially since SLAKE is much smaller. Are the different results because of the different image domains, different queries/answers, dataset sizes, or something else? Where any other datasets studied?

1 comment

Thank you for the detailed reading. Your comments shaped the main presentation changes.

R3.1

  • Response: We agree. §1 ¶1 now opens with the question: a stored caption can replace an image that is no longer supplied, but does it add to an image that remains available? Fig. 1 shows a real image, its frozen caption, the matched neutral text, and their positions in the conversation. Table 4 gives one logged item with its four stored answers; Fig. A.1 and Appendix A.2 list all messages and prompts.
  • Change: §1 ¶1; Fig. 1; Table 4; §1 ¶4; Fig. A.1; Appendix A.2.

R3.2

  • Response: We rewrote §3.2 around the core comparison: eight primary conditions cross caption or length-matched neutral text with history or inline delivery while the image remains attached to the question. The other twelve conditions are grouped by role (no-text baseline, image placement, diagnostics) and drawn in Fig. A.1 as supporting controls.
  • Change: §3.1 ¶1; §3.2 ¶1; Fig. A.1.

R3.3

  • Response: We evaluate fixed models, so we refrain from training advice. For evaluating multimodal conversational memory, our findings yield three takeaways: first, keeping generic captions in history while the image remains available gives at most a small average gain over neutral text, as item-level gains and losses largely cancel (§4.3); second, an image-free caption test is not sufficient evidence of added value with the image, as most items where captions beat neutral text without the image are solved by both once the image is present (§4.3, exploratory); third, in diagnostic single-fact edits for three models, a wrong caption fact hurts accuracy far more inline than in history (§4.4).
  • Change: §5 ¶1 and ¶3.

R3.4

  • Response: On SLAKE, neutral text is not clearly better in history (ΔH −0.26 pp [−1.44, +0.96]). The inline difference (−1.69 pp) reflects negative contrasts for InternVL3.5 (−7.51 pp) and InternVL2-8B (−6.21 pp), the latter almost entirely from repeated body-part questions, and does not hold under the second judge. SLAKE has only 94 distinct question strings across 333 images, so one question type can drive a model-level effect.
  • Change: §4.2 ¶3; §4.5 ¶1.
  • Evidence: Table 2; §4.2 ¶3; §4.1 (Datasets).

R3.5

  • Response: Both datasets are short-answer VQA scored against a reference answer. Visual7W asks what, where, when, who, why, and how questions on everyday photos, e.g., "What are all the people wearing?" ("Jackets."). SLAKE asks about organ position, size, color, and count on CT, MRI, and X-ray images, e.g., "What is the organ on the right side of this image?" ("Spleen").
  • Change: §4.1 (Datasets); Table 4.

R3.6

  • Response: Examples are in Fig. 1, Table 1, and Table 4. Each caption is one short, question-blind sentence averaging 14.8–25.5 words per model on Visual7W and 17.1–31.0 on SLAKE. History is one exchange: a fixed text-only request, then the caption or neutral text as the assistant reply (Fig. A.1). To test detail, we asked three models for detailed captions (three to five sentences on objects, attributes, counts, and spatial relations) on 500 new images: without the image, accuracy rises by +4.89 pp over paper captions, whereas with the image ΔH changes by +0.12 pp [−0.74, +1.00]. In this run, detailed captions preserve more accuracy without the image but do not raise the history contrast with it (exploratory, after a preregistered check failed; Appendix H).
  • Change: §4.1 (Models); Table E.1; Fig. A.1; Appendix H.
  • Evidence: §4.1 (Models); Appendix H.

R3.7

  • Response: Thank you; we tested this on 1,000 Visual7W images (three models), attaching the image to every user message in 3- and 6-exchange conversations (72,000 answers across three image placements). The caption–neutral contrast is +0.03 pp [−0.72, +0.80] and +0.07 pp [−0.68, +0.81], close to placing the image once beside the question (+0.09 and +0.03 pp). With the image only in the first message, it is −1.01 and −0.94 pp; the every-message minus first-message difference is +1.04 pp [+0.07, +2.04] and +1.01 pp [+0.01, +2.03] (Holm p 0.067; intervals include zero with gpt-4o), so a change is not established. We report this run as exploratory. In the one-exchange both condition, accuracy does not clearly change (−0.10 pp [−0.43, +0.25] on Visual7W, −0.26 pp [−1.20, +0.69] on SLAKE).
  • Change: Table 2 (Q3); Appendix C.
  • Evidence: Table 2; Appendix C.

R3.8

  • Response: Both datasets were chosen in an earlier phase requiring short reference answers, object or organ questions, and local images. We piloted AI2D and PathVQA: qualifying AI2D questions ask about diagram relations, and many PathVQA references are descriptive fragments that short answers cannot match. Domain, question style, and size differ together, so dataset differences cannot be attributed to one factor. The main item-level findings replicate on 500 new Visual7W images.
  • Change: §4.1 (Datasets); §4.5 ¶3.
  • Evidence: §4.5 ¶3.
Reviewer_vDyaOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper empirically investigates whether providing pre-generated, generic image captions improves the visual question-answering (VQA) performance of Vision-Language Models (VLMs) when the original image remains accessible. To isolate the effect of the caption's semantic content from prompt length, the authors compare frozen captions against length-matched, image-independent neutral text across various prompt placements. Through extensive experiments spanning seven VLMs, two datasets, and 20 distinct conditions—including multi-turn conversations and image-removal ablations—the authors demonstrate that generic captions offer negligible performance gains over neutral text when the image is present. Conversely, captions significantly preserve answering capabilities when the image is removed. Ultimately, the work draws an important distinction between the visual information a caption retains and its marginal utility when the model has direct access to the visual input.

Critical strengths
  1. More rigorous evaluation of caption gains: Comparing caption-assisted answering only with direct answering can misattribute context effects to the caption’s visual content. By using frozen captions and length-matched neutral text within each format, the paper provides a cleaner baseline. This is a useful contribution to evaluating visual memory, even when the resulting gains are small.
  2. Strong validation via image removal: When images are removed, captions consistently lose less accuracy than neutral text. This confirms that the captions contain answer-relevant information, despite offering little added benefit when images remain available. Consistent results across both datasets and all seven models make this finding more convincing than an isolated negative result.
Critical weaknesses
  1. The contribution beyond prior work is not sufficiently clear. Length-matched neutral-text controls have already been used in GEASS. Extending these comparisons to conversational history is useful, but the paper does not clearly establish which existing understanding its findings revise or what additional insight emerges from the new setting. The concern is not the small gains themselves, but the significance of the conclusions drawn from them.

  2. The analysis does not sufficiently explain the main findings. A small average gain could mean that captions rarely affect answers, or that benefits on some questions are offset by harms on others. Despite observing substantial model-level differences, the paper does not systematically distinguish these possibilities or connect them to the question-dependent effects reported in GEASS. This leaves unclear when caption content helps subsequent answers and limits the practical implications of the study.

Questions for authors
  1. Could the authors systematically analyze the mechanisms behind the main findings (e.g., whether small average gains reflect rare prediction changes or offsetting positive and negative effects), and explain how these insights can guide the practical use of captions in multi-turn conversations?
  2. Beyond average accuracy, under what specific conditions do captions substantially outperform neutral text (or vice versa)? Could the authors break down these differences by question type and caption content?
2 comments

Thank you for the thoughtful review. Your two points shaped the main new analysis in the revision and helped sharpen our problem statement and contributions: we ask whether keeping a generic caption in multi-turn history helps VQA when the original image remains available, and we show why most of the image-free caption advantage is redundant once the image is present (R2.1, exploratory) and how offsetting item-level gains and losses produce the small average (R2.2–R2.3).

R2.1

  • Response: We agree that the length-matched neutral control is not new, and §2 now credits GEASS for it. The inference our work tests is that, because a caption captures answer-relevant details (as shown when the image is absent), keeping it in conversation history will also improve answers when the image remains attached. Our contribution is pairing the same frozen captions against neutral controls in history and inline delivery, both with and without the image, and linking each item across the two. This shows that a caption's value without the image mostly does not carry over: of the items the caption gets right but neutral text misses without the image, 83.65% (Visual7W) and 89.23% (SLAKE) are answered correctly by both inputs once the image is present (in the history condition). We added this item-linked analysis in response to review and mark it as exploratory.
  • Change: §1 (contributions); §2; §4.3 (Table 3, Fig. 3).
  • Evidence: Table 3.

R2.2

  • Response: Both readings apply in a specific sense: correctness changes on a modest share of items, and within that share gains and losses nearly cancel. With the image, correctness differs between caption and neutral inputs on 5.51% (history) and 9.79% (inline) of Visual7W items, yet the averages are only +0.35 and +0.36 pp. Splitting by an automatic label of whether the caption supplies the answer reproduces part of the GEASS pattern in conversational history: +1.21 pp [+0.61, +1.82] for captions labeled as supplying it and −1.58 pp [−2.71, −0.50] for partial ones, and +0.02 pp for absent ones (Visual7W history); these labels are not validated against human judgments.
  • Change: §4.3; Appendix D.
  • Evidence: Table 3; Appendix D (Automatic caption-support labels).

R2.3

  • Response: The decomposition in R2.2 directly informs how captions should be evaluated and used in multi-turn conversations. First, for evaluation: because most of the image-free caption advantage is redundant with the image and item-level gains and losses cancel in the mean, we suggest that benchmarks of conversational memory report the caption–neutral contrast both with and without the image alongside item-level gain and loss rates. Second, for delivery: in single-fact edits for three models, a wrong caption fact placed beside the question lowers accuracy much more than the same fact in the history (harm difference +17.27 pp [+10.13, +24.60]; unclear for Gemma-4-12B-it). These edits are question-aware diagnostic stimuli, not natural captions. We do not recommend one delivery format in general.
  • Change: §4.4; §5 ¶1.
  • Evidence: §4.4 ¶2.

R2.4

  • Response: By caption content, see R2.2. By delivery, wrong caption content hurts more inline (R2.3). By dataset, the largest SLAKE effects come from a few models and the repeated body-part questions (§4.2). By question type, in a post hoc breakdown we have now added: after Holm correction, no history contrast differs from zero, and two inline contrasts do, Visual7W "when" questions (+7.57 pp [+5.31, +9.95]) and SLAKE position questions (−3.44 pp [−5.62, −1.35]). The "when" gap comes mainly from neutral text beside the question: its accuracy on these questions drops from 40.1% in history to 31.2% inline, with more answers such as "unknown", and only 1 of 20 sampled captions states a time of day. Question types are not randomly assigned, so these are not effects of question content.
  • Change: Appendix D (Automatic caption-support labels; Question types); §4.2 ¶3.
  • Evidence: Appendix D (Question types).

I lean toward a Clear Reject on technical grounds. The paper does not make a convincing case for why the research question matters, the methodology needs further development, and the experiments provide insufficient evidence for the main claims. These concerns remain after reading the authors’ response.

Reviewer_L4mdOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

this paper asks whether a generic caption stored in a conversation adds anything beyond merely having text in that slot, when the original image is still available at answer time. It generates one question-blind caption per model-image pair for seven VLMs, freezes it, and compares it with length-matched, image-independent neutral text in the same message format, in history and inline delivery, across 20 conditions on Visual7W (2,900 images) and SLAKE (333 images). Image-removal replay checks whether the captions carry answer-relevant content. The main finding is that with the image available the caption’s description adds at most a small average gain over neutral text (+0.35 on Visual7W, -0.26 pp on SLAKE in history), while neutral text alone reproduces much of the point estimate of the caption-history gain. So the image-removal replay shows that captions do carry content.

Critical strengths
  • design isolates the right comparison. a frozen, question-blind caption is compared with length-matched neutral text in the same message role and format, with the image fixed, across seven models, 20 conditions, and two delivery formats, plus an image-removal replay. This kind of controled study or analysis is much needed to separate “the description helps” from “having text helps,” and it is reusable beyond this paper.

  • the result is useful and honestly reported With the image available, the caption adds only about 0.35-0.36 on Visual7W and is negative on SLAKE (-0.26 in history, -1.69 inline).. Neutral text alone reproduces much of the caption-history point estimate (+0.60 of +0.95 on Visual7W). And, the image-removal replay shows the captions do carry content (11.33 vs 40.48 loss). This is a useful caution for visual-memory and agent design, and the paper states its limits rather than overselling.

Critical weaknesses
  • the judge is unvalidated, and the main effect is at judge-noise scale. Accuracy is measured by gpt-4o-mini with a YES/NO prompt, with no human agreement rate, no second judge, and no sensitivity analysis, while the headline Visual7W history contrast is +0.35 [0.03, 0.67] about 10 images out of 2,900. Because the measurement instrument is the weakest link, a condition-correlated judge bias (for example, rewarding answers that paraphrase the caption) could produce or erase the effect. Thus, human-labelling a stratified sample and re-estimating the contrast is needed.

  • "neutral text accounts for much of the gain" decomposition is not statistically separated, and external validity is narrow. The bridge minus H difference is +0.25 [-0.21, +0.70], so the data do not establish that neutral text accounts for more of the gain than the caption. SLAKE has 333 images and only 94 distinct question strings, and there is one caption, one question per image, and greedy decoding. The most interesting interpretive claim is therefore a point estimate, and the scope is narrower than the abstract suggests.

Questions for authors
  1. What is the human agreement rate with the gpt-4o-mini judge on a stratified sample ?
  2. does the +0.35 Visual7W history contrast survive when the sample is restricted to answers where the judge agrees with humans? A second judge would also help.
  3. Is the bridge -ΔH difference statistically distinguishable from zero on either dataset?
  4. Does ΔH change if the caption is generated with a different prompt or by a different captioner, and with 2-3 questions / image instead of one?
2 comments

Thank you for the careful review and concrete suggestions on validating the judge. In brief, we test whether a generic caption kept in the conversation adds to an image that is still attached, beyond length-matched neutral text in the same slot. Under the second judge and on new images, the caption adds at most a small average gain; the human audit does not settle this contrast, and captions written by gpt-4o give about 1 pp more (exploratory).

R1.1

  • Response: We agree that the judge needed validation. The revision adds a second judge (gpt-4o) on every stored answer and a blinded two-rater audit (R1.3). Under gpt-4o, the Visual7W history contrast ΔH is +0.21 pp [−0.10, +0.53], so we now read the caption gain as at most small, not as a reliable positive effect. Regarding condition-correlated bias, the judge never sees the caption, the neutral text, or the condition, so condition differences reach it only through the answer strings. In the audit, the correct answers the judges miss are slightly more frequent for caption than for neutral inputs.
  • Change: §4.1 (judge paragraph); §4.5 ¶1–¶2; Appendix F.
  • Evidence: §4.5 ¶1; §4.1 (judge paragraph).

R1.2

  • Response: We agree with this critique. We removed the claim that neutral text accounts for much of the gain (intervals in R1.5), and the Visual7W bridge also does not hold under gpt-4o; the revision instead centers the contribution on the direct caption-versus-neutral contrast paired with and without the image (§4.3). To widen the scope, we added a replication on 500 new Visual7W images (four of the seven models) with up to three questions per image, and SLAKE results weighted by question string. One caption per image and greedy decoding remain limitations (§5).
  • Change: Abstract; §4.2 ¶2; §4.5 ¶3; Appendix D; §5.
  • Evidence: §4.5 ¶3; Appendix D.

R1.3

  • Response: On a blinded sample of 210 model–image bundles (762 distinct answers), gpt-4o-mini agrees with the two raters on 87.2% (κ 0.744) and 87.4% (κ 0.747) of answer rows without an UNSURE label under the strict rule, and the raters agree on 88.7% (κ 0.768) of the 693 items neither marked UNSURE. The judge is somewhat stricter than either rater. One rater is an author, and disagreements were not adjudicated.
  • Change: §4.5 ¶2; Appendix F.
  • Evidence: Appendix F (Table F.1).

R1.4

  • Response: Thank you for this suggestion; we ran it on the audit sample as a post hoc check. Keeping only Visual7W answers on which gpt-4o-mini and both raters agree leaves 56 of the 68 bundles with complete labels, and the sampled history contrast with the image goes from +4.41 pp [0.00, +10.29] to +3.57 pp [0.00, +8.93], so the sign does not change, although keeping only agreed answers favors answers that are easy to judge. The three judgments disagree about equally often on caption and neutral answers (16.2% and 17.7%). This sample is too small to test +0.35 pp itself, so we add that with gpt-4o the contrast is +0.21 pp [−0.10, +0.53] and regenerating the answers gives +0.24 pp [−0.08, +0.56]. The +0.35 pp interval excludes zero only under the main judge and run, and we describe it as at most a small gain.
  • Change: Appendix F (Answers on which all three judgments agree); §4.5 ¶1; §5 (limitations paragraph).
  • Evidence: Appendix F (Answers on which all three judgments agree); §4.5 ¶1; §5 ¶2.

R1.5

  • Response: No. Bridge minus ΔH is +0.25 pp [−0.21, +0.70] on Visual7W and +1.46 pp [−0.26, +3.13] on SLAKE. Both pointwise 95% intervals include zero, and §4.2 now states this.
  • Change: §4.2 ¶2.
  • Evidence: §4.2 ¶2.

R1.6

  • Response: Thank you; we have now run both. On 500 new Visual7W images (1,362 questions, three models), a detailed caption request changes ΔH relative to the paper request by +0.12 pp [−0.74, +1.00], although these captions preserve more accuracy without the image (+4.89 pp). Captions from gpt-4o instead of the answering model raise ΔH by +0.97 pp [+0.27, +1.67]. A preregistered check required the new captions to raise image-free accuracy by 3 pp on 200 images first; neither met it, so we report this run as exploratory. For more questions per image, on the 500 new Visual7W images with up to three questions each, ΔH is +0.41 pp [−0.23, +1.05] (+0.91 pp [+0.25, +1.58] with gpt-4o), and the other main findings replicate.
  • Change: §4.5 ¶3; Appendix H (Caption request and captioner); §5.
  • Evidence: §4.5 ¶3; Appendix H (Caption request and captioner).

Thank you for the thorough and concrete response. My concerns are resolved. My rating is not changed

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.