acceptodds
Under review as a conference paper at ICLR 2027

Rethinking Multi-Image Re-Representation in Multi-Image Understanding

Abstract

Multi-image understanding requires MLLMs not only to recognise the content of individual images, but also to organise visual evidence distributed across them. We study this problem through multi-image re-representation, viewing prompted Chain-of-Thought reasoning and agentic visual tool use as different ways of re-organising visual evidence during reasoning. We introduce Mosaic, a general-purpose multi-image visual harness that enables an MLLM to actively construct visual intermediates with ten composable image operations. We compare five re-representation settings on existing multi-image benchmarks and on MosaicBench, a new grounding-focused benchmark for fine-grained multi-image understanding. Our experiments show that the relative benefits of textual and visual re-representation are strongly task-dependent. Visual re-representation is particularly effective for tasks requiring precise visual evidence, including hypothesis testing, precision comparison, and orientation-sensitive reasoning, while tasks dominated by higher-level semantic content show smaller or less consistent gains. Building on this finding, we train MosaicAgent-8B to use Mosaic with reinforcement learning using only accuracy and format rewards. Without demonstration trajectories or rewards for specific tool-use, the agent learns to compose visual operations over multiple steps and exhibits diverse problem-solving patterns unpromptedly. Our code and data will be made public.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
vision language models, agentic AI, multimodal learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_e9tqOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies multi-image re-representation in multi-image understanding and introduces a unified perspective that views textual re-representation and visual re-representation as two approaches for organizing visual evidence. The authors design the Mosaic visual harness, which supports various composable image operations, and construct the MosaicBench benchmark to evaluate different re-representation settings. Furthermore, the paper trains MosaicAgent-8B with GRPO-based reinforcement learning to learn multi-step visual tool usage. Experimental results show that visual re-representation provides advantages on tasks requiring fine-grained visual comparison, orientation-sensitive reasoning, and spatial relationship understanding.

Critical strengths
  1. The paper introduces the perspective of multi-image re-representation, viewing textual and visual re-representation as different ways of organizing visual evidence. Through systematic experiments, it examines the performance of different re-representation settings across multi-image understanding tasks, revealing the task-dependent benefits of visual re-representation and providing valuable empirical insights into the role of visual intermediates in multimodal reasoning.
  2. The paper introduces a visual harness with composable image operations and a grounding-focused benchmark. Mosaic supports multi-step visual operations and the reuse of visual intermediates through a persistent image workspace, while MosaicBench covers visual challenges such as orientation, precision comparison, hypothesis testing, and spatial reference, providing a useful benchmark for evaluating fine-grained multi-image understanding.
Critical weaknesses

1.The experimental comparison of different re-representation settings involves confounding factors. Appendix A.2 shows that the tool mode not only provides access to visual tools but also includes image sizes, tool and canvas conventions, and explicit planning and reflection requirements, which are absent in the no-tool mode. Although the authors acknowledge these differences, they do not provide controlled experiments that isolate the effects of tool access from prompt differences. Furthermore, visual tool use introduces additional inference overhead, while comparisons under matched computational budgets remain insufficient. Consequently, the performance gains cannot be fully attributed to visual re-representation itself, weakening the conclusions. 2. The technical novelty and cross-task generalization of the method require further validation. Mosaic primarily integrates existing image operations, while MosaicAgent-8B employs standard GRPO with accuracy and format rewards, leaving its distinct technical contributions over existing visual tool-based reasoning methods insufficiently demonstrated. Moreover, the training data selection favors examples that benefit from tool use, potentially limiting generalization. The performance gains are mainly concentrated on MosaicBench and M4Bench, with smaller or inconsistent gains on Mantis, BLINK, and MMIU, leaving broader cross-task generalization insufficiently validated.

Questions for authors
  1. When comparing different re-representation settings, could the authors provide controlled experiments that match general reasoning instructions, image metadata, and inference budgets to better isolate the benefits of visual re-representation from prompt differences?
  2. The training suite is selected based on the outcomes of tool-enabled rollouts and a no-tool run. Could the authors provide comparisons with randomly sampled or unfiltered training data to assess the impact of this selection strategy on model performance and generalization across benchmarks?
  3. The paper reports a substantial increase in tool calls after reinforcement learning. Could the authors provide comparisons of end-to-end inference latency, token usage, and computational cost, and further analyze the trade-off between additional inference overhead and accuracy improvements?
3 comments

Thank you for your constructive feedback!

W1 & Q1: controlled experiments We added a matched-prompt no-tool reference across five benchmarks (Section 4.1, Table 1). V-Re² improves over the matched reference from 37.3% to 47.2% on MosaicBench and 48.0% to 51.8% on M4Bench for Qwen3-VL-8B, and from 42.3% to 63.3% and 49.8% to 61.8% for MosaicAgent-8B. We also make the additional inference budget explicit: total tool calls increase from 1,830 to 3,037 (1.66x) after training (Section 5.2, Figure 4), and V-Re² raises MosaicAgent-8B accuracy from approximately 47% to 62% with higher token consumption (Section 5.2, Figure 6). The matched no-tool scores are also lower than the original T-Re² scores, confirming that performance depends on combinatorial factors including the prompt and the model's de-facto action space rather than on independently additive components. The controlled results therefore rule out the original prompt difference as the sole explanation while preserving the paper's task-dependent conclusion.

Table 1 is reproduced below (accuracy, %). T-Re² is the original no-tool setting; T-Re²† is the matched-prompt no-tool control.

ModelSettingMosaicBenchM4BenchBLINKMantisMMIU
Qwen3-VL-8BT-Re²45.2 ± 4.351.3 ± 1.264.4 ± 0.481.9 ± 1.061.0
Qwen3-VL-8BT-Re²†37.348.061.577.056.0
Qwen3-VL-8BV-Re²47.2 ± 1.551.8 ± 0.363.6 ± 0.279.7 ± 1.656.6
MosaicAgent-8BT-Re²46.0 ± 1.251.3 ± 1.463.7 ± 0.482.8 ± 0.459.4
MosaicAgent-8BT-Re²†42.349.863.577.457.5
MosaicAgent-8BV-Re²63.3 ± 1.161.8 ± 1.462.8 ± 1.081.6 ± 1.257.2

W2: novelty and generalisation

  • better performance and general application: Our core contribution lies in systematically comparing established multi-image inference approaches within a unified framework and introducing a new general-purpose multi-image visual harness. MosaicAgent maintains its performance on traditional multi-image and visual understanding datasets and achieves substantial gains on demanding datasets.
  • the simplicity of training methods: Our minimal GRPO training design for this new multi-image setting demonstrates the generalisability of our method. We find that MosaicAgent learns heuristic problem-solving skills for multi-image tasks and this training recipe can be easily adapted to other tasks;
  • novel mechanism analysis: most importantly, our paper and the revision explain why textual CoT is insufficient for some tasks and why visual re-representation performs better.: visual re-representation can externalise implicit visual reasoning with tools and effectively organise visual evidence explicitly compared to textual reasoning in token space. This clarifies the distinction between textual and visual re-representation in visual understanding tasks.

Q2: Training data selection: Our rollout-based selection is not post hoc selection of favourable evaluation cases; it constructs an effective training distribution for group-relative agentic RL. With eight tool-enabled rollouts, examples for which all rollouts succeed or all fail provide no group-relative signal from answer accuracy, because their within-group accuracy rewards are identical. This is the same motivation as DAPO[1] and others[2]’s dynamic sampling, which removes prompt groups with accuracy 0 or 1 to avoid zero-advantage policy updates. Our evaluation data are independently constructed after excluding overlap with the training suite at the sample, source-provenance, and image-similarity levels.

Q3: inference overheads: The revision quantifies inference costs through two comparisons. Total tool calls increase by a factor of 1.66 after training (Section 5.2, Figure 4). The accuracy-token comparison shows that V-Re² raises MosaicAgent-8B accuracy from approximately 47% to 62% with more token consumption (Section 5.2, Figure 6). The trained policy uses more inference and obtains a substantially larger accuracy return on MosaicBench. The additional inference budget therefore yields a substantially larger accuracy return with the trained policy.

The accuracy–token comparison in Fig. 6 is summarised below. Values are approximate readings from the plot; k denotes thousands of tokens. These are the Fig. 6 measurements, reported separately from Table 1.

ModelSettingToken consumption (k)Accuracy (%)
Qwen3-VL-8B-InstructT-Re²≈5≈44
Qwen3-VL-8B-InstructV-Re²≈36≈47
MosaicAgent-8BT-Re²≈3≈47
MosaicAgent-8BV-Re²≈52≈62
Metric (Fig. 4)Before RLAfter RLRatio
Total tool calls1,8303,0371.66×

[1] Dapo: An open-source llm reinforcement learning system at scale, 2025 [2] Online Difficulty Filtering for Reasoning Oriented Reinforcement Learning, 2026

Thank you for the detailed responses and additional experiments. The authors have addressed my concerns. I maintain my original rating.

Dear reviewer,

Thanks again for your valuable feedback and positive acknowledgment.

Reviewer_tijJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Mosaic, a general-purpose framework that enables MLLMs to actively reorganize multi-image evidence through ten composable visual operations. Experiments show that visual re-representation benefits tasks requiring precise visual grounding, while its effectiveness varies across task types. The authors further develop MosaicAgent-8B using reinforcement learning with only accuracy and format rewards, demonstrating emergent multi-step tool use and diverse problem-solving strategies without explicit tool-use supervision.

Critical strengths
  1. Multi-image re-representation is formulated for five re-representation settings.

  2. A multi-image visual harness, called Mosaic, is proposed to enable an MLLM to actively construct visual intermediates across multiple images.

  3. The experimental results show partial effectiveness of the proposed method.

Critical weaknesses
  1. What is visual re-representation requires to be clearly explained.

  2. The components and working mechanism of the proposed multi-image visual harness is unclear. The writing of the paper requires to be significantly improved.

  3. Regarding experimental results, it seems Mosaic is not so useful, with only 0.004 improvements (0.728 vs. 0.724) in Mantis Overall for the method GPT-4o-mini + Mosaic

Questions for authors

See above

1 comment
  1. W1: Definition of visual re-representation We define multi-image re-representation as constructing a variable-length intermediate record to organise visual evidence for answering the question (Section 3.1, Definition 1). The distinction is therefore precise: textual re-representation is a form of textual CoT in which MLLMs reason over multiple images autoregressively, whereas visual re-representation externalises image-related reasoning through the provided tools and processes explicitly constructed visual representations.

  2. W2: Mosaic components and mechanism: Mosaic is an agentic visual understanding environment with a persistent image workspace where models can call external multi-image tools to process images to help answer the questions. For the first time, we extend the range of available visual tools from single-image operations, such as zooming and cropping in the Think-with-Image line of work, to the multi-image setting (Section 3.2, Figure 1). This makes Mosaic's role explicit: it enables the model to construct and reuse visual intermediates over successive calls rather than treating tool use as a single isolated action.

We extend our discussion of the underlying mechanism of Mosaic in our revision. We formulate textual and visual re-representation as two ways of organising visual evidence: textual re-representation changes how evidence is expressed in language, whereas visual re-representation can also change how it is presented. We added two complementary analyses.

  • (A) Tool usage distribution: We calculate the tool usage distribution in different tasks. We find that agents use different tools according to task-specific heuristics (Fig. 7): for example, cropping exposes fine details in resolution tasks, rotation and flipping change orientation, and pixel differencing makes subtle changes explicit. This shows that models can proactively construct useful visual re-representations to make visual evidence explicit when answering questions. Similarly, successful trajectories use operations aligned with the task’s visual requirements more often than failed trajectories: cropping for resolution (+17 points), pixel differencing for precision comparison (+11), and collage/overlay for hypothesis testing (+16).
  • (B) Rubric-based trajectory analysis: We define a five-dimensional rubric to assess the effectiveness of reasoning in textual space (T-Re$^2$) and visual space (V-Re$^2$)). This includes coverage (required visual evidence is explicitly stated), fidelity (visual attributes are described accurately), binding (evidence is assigned to the correct image/entity), relations (spatial, directional, and cross-image correspondences are represented correctly), and sufficiency (stated evidence distinguishes the answer from alternatives). As shown in Figure 8, textual re-representation fails to explicitly and effectively identify the evidence during reasoning.
MetricT-Re²V-Re²Δ
Correct & sufficient22.941.4+18.5
Coverage72.988.6+15.7
Fidelity20.740.7+20.0
Binding68.672.9+4.3
Relations33.647.1+13.5
Sufficiency71.485.7+14.3
Answer accuracy46.462.1+15.7

GPT-6.1 Sol assesses the final explicit textual evidence. The five rubric dimensions report the percentage receiving the highest score (2). Correct & sufficient requires evidence to be both accurate and adequate to resolve the task; sufficiency alone is conditional on the stated facts being true. Answer accuracy uses the recorded final responses. All 140 trajectories remain in each denominator, including unassessable ratings.

  1. Performance on Mantis: * Our claim is explicitly task-dependent rather than a claim of uniform benchmark improvement.* For the same GPT-4o-mini comparison, Mosaic changes Mantis from 72.4% to 72.8% (+0.4), but improves MosaicBench from 32.5% to 42.5% (+10.0) and M4Bench from 33.4% to 42.2% (+8.8) (Section 5.1, Table 2). At the task level, Qwen3-VL-8B improves on M4Bench Detailed Difference from 46.1% under T-Re² to 59.9% under V-Re², while all three evaluated models perform worse with V-Re² than T-Re² on State Comparison (Section 4.1, Figure 2). These results directly support the paper's conclusion that visual re-representation is particularly effective for tasks requiring precise visual evidence, while tasks dominated by higher-level semantic content show smaller or inconsistent gains.
Reviewer_Ne1hOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

To improve multi-image understanding, the authors propose Mosaic, a general-purpose multi-image visual harness that enables the model to perform ten composable image operations, including cropping, rotation, stitching, and overlaying, when answering questions. The processed images are retained and remain available for further observation and processing. Experiments show that the effectiveness of this method is task-dependent. For problems that require precise observation, such as detecting subtle differences, determining spatial orientation, and verifying puzzle configurations, processing the images is usually helpful. However, for problems that mainly rely on semantic understanding, the benefits are limited, and additional image processing can sometimes hurt accuracy. The authors also train MosaicAgent-8B with reinforcement learning, using rewards based only on answer accuracy and format compliance, without providing demonstrations of tool use. The model gradually learns to perform multi-step visual operations. On MosaicBench, performance increases from 45.2% for the base model with text-only reasoning to 63.3% for MosaicAgent-8B.

Critical strengths
  1. This paper systematically compares five multi-image re-representation settings. Textual re-representation, online visual re-representation, and prefabricated visual re-representation are compared within a unified framework. The results show that the benefits of visual re-representation are task-dependent and help distinguish between the construction and use of visual representations.
  2. The paper introduces a useful grounding-focused benchmark, MosaicBench. It covers six categories of visual challenges and provides a testbed for studying fine-grained comparison, geometric reasoning, and spatial reference capabilities.
Critical weaknesses
  1. The paper does not clearly explain how individual image operations causally contribute to multi-image understanding. Although the overall motivation and narrative are clear, the experimental details and causal analyses supporting the central claims are not fully developed. The paper introduces Mosaic to allow the model to construct and reuse intermediate image views, thereby examining when visual re-representation is more useful than textual reasoning. However, further analysis is needed to determine why these operations are effective and what drives the observed gains.
  2. Of the 32 candidate task types, 17 involve only a single source image. Although the paper counts transformed views derived from a single image as multi-image re-representation, this does not necessarily show that the model can align information across independent source images, preserve object identity, or perform multi-view fusion. The paper should report results separately for tasks involving one, two, and three or more independent source images, and include questions that can only be answered with at least two independent source images. Statistics from tool-use dependency graphs should focus on genuine cross-source fusion operations, rather than treating multiple views derived from a single image as evidence of multi-image understanding.
  3. Some aspects of the experimental design are not fully controlled. For example, the tool-enabled setting requires the model to plan before each tool call and reflect afterward, whereas the no-tool setting does not have the same requirements. Although the authors acknowledge this asymmetry, it makes it difficult to attribute the gains solely to the visual tools themselves.
Questions for authors
  1. It would be useful to report results separately for tasks involving one, two, and three or more independent source images. The paper could also include questions that require at least two independent source images to answer.
  2. Further analysis could clarify why these operations are effective and what drives the observed gains.
  3. The presentation could be made more concise and clear, and some experiments could use stricter controls.
1 comment

Thanks for your constructive feedback.

  1. W1 & Q2: Mechanism analysis: We formulate textual and visual re-representation as two ways of organising visual evidence: textual re-representation changes how evidence is expressed in language, whereas visual re-representation can also change how it is presented. We added two complementary analyses.
  • (A) Tool usage distribution: We calculate the tool usage distribution in different tasks. We find that agents use different tools according to task-specific heuristics (Fig. 7): for example, cropping exposes fine details in resolution tasks, rotation and flipping change orientation, and pixel differencing makes subtle changes explicit. This proves that the mechanism lies in that models can proactively construct useful visual re-representation to make visual evidence explicit to answer the questions.
    Samely, successful trajectories use operations aligned with the task's visual requirement more often than failed trajectories: cropping for resolution (+17 points), pixel differencing for precision comparison (+11), and collage/overlay for hypothesis testing (+16).
  • (B) Rubric-based trajectory analysis: We define a 5-dimension rubrics to analysis if reasoning in textual space (T-Re) and visual space (V-Re) effectively. This includes coverage (required visual evidence is explicitly stated), fidelity (visual attributes are described accurately), binding (evidence is assigned to the correct image/entity), relations (spatial, directional, and crossimage correspondences are represented correctly), and sufficiency (stated evidence distinguishes the answer from alternatives). As shown in Figure 8, textual re-representation fail to explicitly and effectively identify the evidence in its textual reasoning.
MetricT-Re²V-Re²Δ
Correct & sufficient22.941.4+18.5
Coverage72.988.6+15.7
Fidelity20.740.7+20.0
Binding68.672.9+4.3
Relations33.647.1+13.5
Sufficiency71.485.7+14.3
Answer accuracy46.462.1+15.7

GPT-6.1 Sol assesses the final explicit textual evidence. The five rubric dimensions report the percentage receiving the highest score (2). Correct & sufficient requires evidence to be both accurate and adequate to resolve the task; sufficiency alone is conditional on the stated facts being true. Answer accuracy uses the recorded final responses. All 140 trajectories remain in each denominator, including unassessable ratings.

  1. W2 & Q1: Source image number: We report accuracy by the number of source images. MosaicAgent’s improvements over T-Re are +7.9, +27.2, and +22.5 percentage points for one, two, and three-or-more inputs, respectively. Although MosaicBench is designed as a grounding-focused visual understanding benchmark rather than an exclusively multi-image dataset, cross-image visual understanding is essential in its multi-source tasks, and MosaicAgent performs well in this demanding setting.
Evaluation groupQwen3-VL-8B-Instruct T-ReQwen3-VL-8B-Instruct V-ReMosaicAgent T-ReMosaicAgent V-Re
1 initial input image41.242.147.455.3
2 initial input images56.160.649.476.7
≥3 initial input images†12.522.530.052.5
Independent-source change tasks‡40.055.042.557.5
  1. W3 & Q3: Controlled prompt: We added a matched-prompt no-tool reference for Qwen3-VL-8B and MosaicAgent-8B across five benchmarks (Section 4.1, Table 1). For Qwen3-VL-8B, V-Re² exceeds the matched no-tool reference on all five benchmarks, including 37.3% to 47.2% on MosaicBench (+9.9) and 48.0% to 51.8% on M4Bench (+3.8); for MosaicAgent-8B, the corresponding gains are 42.3% to 63.3% (+21.0) and 49.8% to 61.8% (+12.0), while BLINK and MMIU remain essentially unchanged (-0.7 and -0.3). The matched no-tool scores are also lower than the original T-Re² scores, confirming that performance depends on combinatorial factors including the prompt and the model's de-facto action space rather than on independently additive components. The controlled results therefore rule out the original prompt difference as the sole explanation while preserving the paper's task-dependent conclusion.

Comments on the paper

1 comment

Dear reviewers and AC,

Thanks for your constructive feedback and affirmations of our work on:

  • a novel perspective on multi-image re-representation and multi-image understanding;
  • systematic comparison and evaluation of re-representation settings;
  • valuable findings and insights for multi-image understanding;
  • MosaicAgent as a valuable harness and agentic model;
  • MosaicBench as a useful new testbed for visual understanding;

We summarise the changes made during the rebuttal below for your reference and have highlighted them in blue in the revised manuscript:

  1. Controlled experiments. We added a matched-prompt no-tool comparison for Qwen3-VL-8B and MosaicAgent-8B across five benchmarks (Table 1).

  2. Task-dependent tool routing distributions. We analysed tool use across tasks in relation to task-specific visual heuristics and compared tool-use distribution between successful and failed trajectories (Section 5.2, Fig. 7).

  3. MosaicBench motivation and positioning. We clarified that MosaicBench evaluates task-driven and grounding-focused construction: visual evidence may be distributed across source images or reorganised through newly constructed intermediate views (Section 4.2).

  4. Accuracy–token consumption analysis. We added an accuracy–token-consumption comparison on MosaicBench and discussed the accuracy gain obtained with additional inference tokens by MosaicAgent-8B (Section 5.2, Fig. 6).

  5. Rubric-based trajectory analysis. We added a GPT-6.1 Sol assessment comparing T-Re$^2$ and V-Re$^2$ trajectories across coverage, fidelity, binding, relations, and sufficiency (Section 5.2, Fig. 8) to identify the mechanism of re-representation in textual and visual forms.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.