acceptodds
Under review as a conference paper at ICLR 2027

Visual Context Composition: Rendering and Grouping Evidence for Multi-Image VLMs

Abstract

Multi-image tasks require vision-language models (VLMs) to relate images, labels, and instructions. Text can enter the language decoder directly or pass through the visual encoder when rendered in an image. Placing multiple elements in one image also allows them to be visually encoded together. We introduce Visual Context Composition (VCC), a training-free method that controls these processing routes while keeping images with their associated labels. Across five VLMs and four benchmark families, the selected VCC input configurations improve average task scores over baseline inputs. Giving each labeled example its own image performs best on average for fine-grained classification; separating task roles, such as instructions, references, and queries, performs best in the other benchmark families. Fine-grained gains persist for two models when individual image sizes or total input pixel counts are matched in separate comparisons. With rendered content held fixed, changing which task roles share an image also changes accuracy. We hypothesize that joint visual encoding represents task-relevant relationships. Assigned labels can be read from reference-image features. Token-removal experiments further show that information propagated from a region to other visual tokens contributes to predictions even after that region's own tokens are removed.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Vision-language models, Multimodal in-context learning, Multi-image understanding, Visual prompting, Representation analysis

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_kdyxOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2(+1)
Presentation
2
Soundness
2
Read review
Paper summary

This study analyzes the effect of input configuration on the performance of multi-image VLMs. The study proposes Visual Context Composition (VCC), which renders instructions and labels as images and groups or separates input images, rather than providing text and images separately. The study compares three input configurations, Joint, Role, and Example, across 5 VLMs and 4 benchmark families. The results show that Example improves fine-grained classification accuracy by 5.37 pp on average, while Role improves peak accuracy on VL-ICL by 12.59 pp. Additionally the study uses controlled experiments on synthetic data and visual representation analysis to examine information propagation and performance changes across different input configurations.

Critical strengths
  1. The study addresses a practical research question concerning the effect of input configuration on the performance of multi-image VLMs.
  2. The study conducted extensive experiments across 5 VLMs and multiple benchmarks and achieved substantial performance gains in some settings.
Critical weaknesses

Major Weaknesses

  1. The experimental results do not sufficiently support the paper's core claim. The core claim of the paper is that joint visual encoding can effectively represent relationships between images and text. However, the Joint configuration did not achieve the best performance on any of the benchmark families. In fine-grained classification, the Example configuration, which encodes each reference separately, achieved the best performance. Although the visual representation analysis shows that information propagates within the visual encoder, the study does not demonstrate that this explains the performance improvements observed on the benchmarks.

  2. It is unclear where the performance gains come from, and the main experimental results seem sensitive to the input configurations. The effects of rendering, grouping, and input structure are not sufficiently isolated. In particular, the study does not evaluate how much performance improvement can be achieved by rendering only the labels on reference images in the main benchmarks. The Role configuration shows a performance drop of 3.09 pp in fine-grained classification, but gains of 4.11 ~ 5.87 pp in the controlled experiments. The ranking of the grouping configurations also changes.

  3. The techniques used in this paper, such as text rendering, image grouping, and visual labeling, have already been explored in conventional studies. In particular, the paper does not sufficiently discuss the following related studies:

  • SoM [1]: Adds visual labels to regions in an image.
  • VisInContext [2]: Renders text as images for multimodal ICL.
  • IG-VLM [3]: Combines multiple images into a single grid without training.

The paper does not clearly explain how the proposed method differs from these approaches, and direct experimental comparisons are also lacking. Thus, it is unclear what novel technical contributions the proposed method offers beyond existing input formatting techniques. The paper also does not provide a general composition principle for applying VCC to new tasks.

[1] Yang, Jianwei, et al. "Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v." arXiv preprint arXiv:2310.11441 (2023).

[2] Wang, Alex Jinpeng, et al. "Leveraging visual tokens for extended text contexts in multi-modal learning." Advances in Neural Information Processing Systems 37 (2024): 14325-14348.

[3] Kim, Wonkyun, et al. "An image grid can be worth a video: Zero-shot video question answering using a vlm." IEEE Access 12 (2024): 193057-193075.

  1. The paper selects the best grouping based on the evaluation results for each benchmark family and reports the main performance using results from the same datasets. Thus, the reported performance gains may be affected by selection bias. The study should select the configuration on a separate validation set and evaluate it on an independent test set. Although the paper evaluates transfer across tasks in Table 3, this does not fully address the selection bias in Table 2.

Minor Weaknesses

  1. Fig. 1 presents Joint as a representative VCC configuration, but Joint was not selected as the best configuration in the main experiments. This may confuse readers.

  2. The paper does not sufficiently discuss the additional computational costs and inference latency caused by rendering and reorganizing the inputs as images.

Questions for authors
  1. What is the rationale for rendering instructions as images rather than providing them as regular text?
Reviewer_G1GFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Visual Context Composition (VCC), a training-free method that controls which text is rendered into images and how task elements are grouped for visual encoding. Its contributions are a unified input-composition framework, evidence of task-dependent grouping preferences, and analyses of contextual information in visual representations. Evaluation across five VLMs and four benchmark families reports improved average performance with selected configurations.

Critical strengths

S1. Clear motivation and a coherent empirical investigation. The paper starts from an interesting observation: changing how the same task evidence is presented can substantially affect VLM performance. It develops this observation into a well-structured investigation by distinguishing rendering scope from image grouping. Evaluation across multiple models and task families, supported by targeted controls, gives the observation substance beyond isolated examples. This is consequential because it establishes input composition as a practical, training-free intervention and provides a useful framework for studying how VLMs process multimodal context.

Critical weaknesses

W1. Limited conceptual novelty. The paper explicitly distinguishes VCC from PixelWorld, CollagePrompt, and MIRB through joint control of rendering scope and image grouping. Nevertheless, the underlying operations—rendering text alongside images and reorganizing visual inputs—closely resemble established approaches. While the systematic investigation is valuable, the conceptual advance appears incremental, and the distinct benefit over closely related input-composition strategies is not sufficiently established.

W2. Limited generalizable insight and broader significance. The results primarily demonstrate task- and model-dependent improvements from input formatting. Despite the within-family transfer experiments, the paper does not establish a reliable composition-selection principle for unseen settings. Moreover, the representation analyses do not explain the natural-task gains, as acknowledged in the limitations. Consequently, the work provides limited guidance on how multimodal models should represent or integrate contextual evidence, constraining its broader significance beyond the evaluated input configurations.

Questions for authors

See weaknesses.

Reviewer_suEFOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how rendering text into images and grouping task elements changes multi-image VLM performance without training. It compares Joint, Role and Example configurations across five models and four benchmark families. Further experiments test grouping transfer, image size and pixel budgets, grouping with rendered content fixed, visual-feature readout, and information propagation through retained visual tokens.

Critical strengths
  • The empirical controls are careful and transparently reported.
  • The paper establishes a useful practical result: one-shot Example gains persist under size/pixel controls.
Critical weaknesses
  • Frankly speaking, the writing and figure representation needs lots of improvements ...
  • Some headline gains need a more precise account of baseline comparability. For example, I see gains on Qwen3.5's VL-ICL, but pooled logs report many output-limit and parsing failures.
Questions for authors
  • Are the Appendix E MuirBench examples the actual inference inputs? Will the gains change when the Baseline receives identical wording, option ordering and image indices as Role?
  • Please break down output-limit and unparsed responses by model, task and condition.
  • Will you please redraw and replace the figures in the future?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.