- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The goal of this paper is to study how image information, in the form of a caption, is propagated through a text conversation with a multimodal model which retains history. The paper conducts an experiment on two datasets with natural and medical images respectively by comparing the resulting conversations with seven different VLMs, weighing the contribution from image captions versus neutral text.
In the experiment, various ways of incorporating the caption and history are explored. Results study how image and caption inclusions affects model performance.
Critical strengths
Interesting to study how visual information is used in caption form for multimodal conversations across a suite of models.
Critical weaknesses
Lack of examples and visualizations. It would help to see a concrete example in the introduction. Currently it is not clear what problem the paper is tackling and at what steps the image and caption information are included. It would help to show a real example with a visual query, conversation history, task, and model output.
The experimental setup in Section 3.2 needs more clarification and motivation. The contrasts and their significance could be explained better and currently read as a list of different settings. Are they critical to the experiment setup, or included for completeness? There are 20 different conditions listed and the different settings and this is a bit confusing for the reader to keep track of. It could be helpful to focus on the cases which lead to interesting insights. Sections 3 and 4 could also benefit from some concrete examples of the input history, query, image, and caption, and model outputs showing the change under the different conditions and experimental settings.
Limited and unclear takeaways. Although there are some insights about using image captions vs neutral text, it is not clear how the findings in the paper should change how we train or interact with multimodal models in the future.
Questions for authors
Section 4.2 reports many quantitative percentage point changes but could add more high level analysis. Why some of the trends so different for each dataset? Why is neutral text better on average for SLAKE?
For the query, what kinds of questions are being answered? And what do the answers look like? What tasks are being solved?
What are some example of captions? How long are the captions, and what level of detail do they cover? What does the conversation history look like? How do the results change as the caption quality or detail increases?
What happens if you add the image at every step in the conversation history?
How were the datasets Visual7W and SLAKE selected? Especially since SLAKE is much smaller. Are the different results because of the different image domains, different queries/answers, dataset sizes, or something else? Where any other datasets studied?
Thank you for the detailed reading. Your comments shaped the main presentation changes.
R3.1
R3.2
R3.3
R3.4
R3.5
R3.6
R3.7
bothcondition, accuracy does not clearly change (−0.10 pp [−0.43, +0.25] on Visual7W, −0.26 pp [−1.20, +0.69] on SLAKE).R3.8