- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2(+1)
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This study analyzes the effect of input configuration on the performance of multi-image VLMs. The study proposes Visual Context Composition (VCC), which renders instructions and labels as images and groups or separates input images, rather than providing text and images separately. The study compares three input configurations, Joint, Role, and Example, across 5 VLMs and 4 benchmark families. The results show that Example improves fine-grained classification accuracy by 5.37 pp on average, while Role improves peak accuracy on VL-ICL by 12.59 pp. Additionally the study uses controlled experiments on synthetic data and visual representation analysis to examine information propagation and performance changes across different input configurations.
Critical strengths
- The study addresses a practical research question concerning the effect of input configuration on the performance of multi-image VLMs.
- The study conducted extensive experiments across 5 VLMs and multiple benchmarks and achieved substantial performance gains in some settings.
Critical weaknesses
Major Weaknesses
-
The experimental results do not sufficiently support the paper's core claim. The core claim of the paper is that joint visual encoding can effectively represent relationships between images and text. However, the Joint configuration did not achieve the best performance on any of the benchmark families. In fine-grained classification, the Example configuration, which encodes each reference separately, achieved the best performance. Although the visual representation analysis shows that information propagates within the visual encoder, the study does not demonstrate that this explains the performance improvements observed on the benchmarks.
-
It is unclear where the performance gains come from, and the main experimental results seem sensitive to the input configurations. The effects of rendering, grouping, and input structure are not sufficiently isolated. In particular, the study does not evaluate how much performance improvement can be achieved by rendering only the labels on reference images in the main benchmarks. The Role configuration shows a performance drop of 3.09 pp in fine-grained classification, but gains of 4.11 ~ 5.87 pp in the controlled experiments. The ranking of the grouping configurations also changes.
-
The techniques used in this paper, such as text rendering, image grouping, and visual labeling, have already been explored in conventional studies. In particular, the paper does not sufficiently discuss the following related studies:
- SoM [1]: Adds visual labels to regions in an image.
- VisInContext [2]: Renders text as images for multimodal ICL.
- IG-VLM [3]: Combines multiple images into a single grid without training.
The paper does not clearly explain how the proposed method differs from these approaches, and direct experimental comparisons are also lacking. Thus, it is unclear what novel technical contributions the proposed method offers beyond existing input formatting techniques. The paper also does not provide a general composition principle for applying VCC to new tasks.
[1] Yang, Jianwei, et al. "Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v." arXiv preprint arXiv:2310.11441 (2023).
[2] Wang, Alex Jinpeng, et al. "Leveraging visual tokens for extended text contexts in multi-modal learning." Advances in Neural Information Processing Systems 37 (2024): 14325-14348.
[3] Kim, Wonkyun, et al. "An image grid can be worth a video: Zero-shot video question answering using a vlm." IEEE Access 12 (2024): 193057-193075.
- The paper selects the best grouping based on the evaluation results for each benchmark family and reports the main performance using results from the same datasets. Thus, the reported performance gains may be affected by selection bias. The study should select the configuration on a separate validation set and evaluate it on an independent test set. Although the paper evaluates transfer across tasks in Table 3, this does not fully address the selection bias in Table 2.
Minor Weaknesses
-
Fig. 1 presents Joint as a representative VCC configuration, but Joint was not selected as the best configuration in the main experiments. This may confuse readers.
-
The paper does not sufficiently discuss the additional computational costs and inference latency caused by rendering and reorganizing the inputs as images.
Questions for authors
- What is the rationale for rendering instructions as images rather than providing them as regular text?