acceptodds
Under review as a conference paper at ICLR 2027

Timage: A Generative Text-in-Image Paradigm for Fine-Tuning Vision-Language Models

34 views

Abstract

Multimodal Large Language Models (MLLMs) often lose track of the right image regions during fine-grained spatial reasoning, because a textual query rarely carries any explicit geometric anchor into the pixel domain. Prevailing remedies either rewire the model's weights or pad the prompt with verbose instructions, yet neither reliably pins the language to the correct visual coordinates without eroding the backbone's general competence. We introduce Timage, a paradigm that recasts multimodal understanding as an alignment problem solved *at the input*: the query is drawn, as a typeset overlay, onto the image itself. The placement and appearance of this overlay are produced by a *Constrained Schrödinger Bridge* (cSB), an entropic optimal-transport sampler that factorizes layout synthesis into two coupled stochastic stages. The first stage, *Region Search*, transports noise toward query-aligned image zones while obeying a hard occlusion barrier that protects salient foreground content; the second stage, *Appearance Shaping*, sizes the glyphs through an "ink-budget" regularizer so that the rendered text stays legible and visually balanced. The resulting overlay behaves as an explicit attention beacon that channels the model's focus along spatial semantics. On the VMCBench and MME suite, Timage paired with a modest 7B backbone clearly overtakes far larger proprietary systems as well as parameter-tuned baselines. The study positions deliberate input reconstruction as a powerful, architecture-neutral lever for strengthening multimodal reasoning.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Multimodal Reasoning, Schr\"odinger Bridge, Multimodal Fine-Tuning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Co2gOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents Timage, a simple yet effective way to rethink multimodal reasoning: instead of feeding text questions and images into a model separately, it renders the question directly onto the image itself.

To place the text naturally, Timage uses a Constrained Schrödinger Bridge (CSB) to solve two practical layout challenges: finding empty space that avoids covering key foreground objects, and adjusting the font style and visibility to fit the scene.

At inference time, the underlying Multimodal Large Language Model (MLLM) remains completely frozen and takes only the composite image as input. Benchmarks on VMCBench and MME show that a frozen 7B model using Timage not only beats fine-tuned baselines (LoRA and full fine-tuning), but also outperforms much larger commercial models like GPT-4o.

Critical strengths

Transforming multimodal alignment from a model-internal fusion/fine-tuning problem into an input-level image rendering formulation is conceptually innovative. Feeding only the composite image into a frozen backbone bridges the spatial-semantic gap using the model's native OCR and visual grounding capabilities.

The use of a CSB with projections offers an elegant mathematical treatment of the inherently one-to-many layout generation problem, appropriately balancing soft cross-attention semantic pull against hard foreground obstacle avoidance.

The approach is inherently human-interpretable. Users can visually inspect exactly where the model's textual cues are anchored relative to scene contents.

Critical weaknesses

Time cost: Inlaying text requires multiple pre-processing modules: foreground instance segmentation, saliency detection, a cross-attention extraction forward pass, and an iterative 20-step CSB sampling process. This poses a significant hurdle for deployments.

Upstream vision quality:The admissible mask fundamentally relies on the reliability of the external instance segmenter and saliency estimator. In cluttered scenes where off-the-shelf segmenters fail, the non-occlusion barrier could fail and occlude critical target details.

Long and multi-turn prompts:The experiments primarily focus on short VQA queries. In real-world applications requiring complex multi-turn dialogs, few-shot in-context demonstrations, or long instructions, rendering everything onto the image without causing visual clutter or massive font reduction remains unproven.

Baselines: Models tested and evaluation suits are not up to date. More like 2025 paper.

Questions for authors

Please see weaknesses.

8 comments
AuthorsOpenReview ↗

Q1: Time cost

We thank the reviewer for raising concerns regarding the computational efficiency and practical deployment of Timage. We evaluate its efficiency from two aspects.

(1) Inference Time Breakdown.

We profile the complete Timage inference pipeline, including instance segmentation, saliency estimation, query-relevance extraction, cSB sampling, text rendering, and downstream MLLM inference. As shown in Table 1, query-relevance extraction and downstream MLLM inference account for the majority of the total latency. Importantly, auxiliary visual information is computed only once for each image–question pair and reused across subsequent layout sampling steps, avoiding redundant preprocessing overhead.

(2) Accuracy–Latency Trade-off.

We further evaluate the effects of different cSB refinement steps on inference efficiency and task performance under the K=1 setting. As shown in Table 2, reducing the sampling steps from 20 to 10 and 5 lowers the total latency to 1,750 ms and 1,705 ms, respectively, while maintaining accuracies of 85.9% and 84.2%. The 20-step configuration achieves the highest accuracy of 87.3% among the evaluated settings, while fewer steps provide practical alternatives for latency-sensitive applications.

Moreover, Timage uses the relatively compact Qwen2.5-VL-7B backbone, avoiding the computational requirements of substantially larger MLLMs. Combined with one-time auxiliary feature extraction and flexible cSB sampling, this design provides a practical balance between model size, inference efficiency, and task performance. The sampling budget can be adjusted according to deployment requirements without changing the underlying model architecture or training procedure.

Table 1. Inference Time Breakdown of Timage (K=1)

ComponentOperationLatency (ms)Percentage (%)
Instance SegmentationMask R-CNN1206.52
Saliency EstimationU²-Net452.45
Query-Relevance ExtractionQwen2.5-VL Attention56030.43
cSB Sampling20 Refinement Steps1809.78
Text RenderingQuery Overlay Synthesis50.27
Downstream InferenceQwen2.5-VL-7B93050.54
TotalComplete Timage Pipeline (K=1)1840100.00

Table 2. Accuracy–Latency Trade-off under Different cSB Sampling Steps (K=1)

cSB StepsAvg. Acc. (%)cSB Latency (ms)Total Latency (ms)
584.2451705
1085.9901750
2087.31801840
3086.52701930
5084.14502110
AuthorsOpenReview ↗

Q2: Robustness to Imperfect Upstream Visual Estimates

We evaluate Timage's robustness to upstream visual estimation errors through three complementary experiments, covering instance segmentation, saliency estimation, and their joint effects under different scene complexities. All robustness experiments use K=5.

(1) Instance Segmentation Robustness.

We perturb Mask R-CNN masks through erosion, dilation (5% and 10% area changes), and random instance dropping (10%, 20%, and 30%). As shown in Table 1, accuracy decreases from 87.7% to 86.1% under 30% instance dropping, indicating robustness to moderate segmentation errors.

(2) Saliency Estimation Robustness.

We perturb U²-Net saliency maps using Gaussian noise (σ = 0.03, 0.06, 0.10), Gaussian blur (σ = 1.0 and 2.0 pixels), and threshold shifts (±0.05 and ±0.10). As shown in Table 2, accuracy remains at 86.3% under the strongest noise and 86.6% under the largest threshold shift.

(3) Joint Robustness across Scene Complexities.

We group images by foreground object count: simple (1–2), moderate (3–10), and crowded (more than 10), using the original Mask R-CNN predictions. As shown in Table 3, joint perturbations reduce accuracy by 0.7, 1.1, and 1.5 percentage points, respectively, suggesting greater sensitivity to upstream errors in crowded scenes.

Additionally, Fig. 4 provides qualitative examples under complex visual conditions and incomplete query-relevance information, complementing these quantitative results.

Table 1. Robustness to Instance Segmentation Perturbations (%) (K=5)

SettingPerturbation StrengthAvg. Acc. (%)
Original Mask0%87.7
Mask Erosion5%87.4
Mask Erosion10%86.9
Mask Dilation5%87.6
Mask Dilation10%87.1
Instance Drop10%87.3
Instance Drop20%86.7
Instance Drop30%86.1

Table 2. Robustness to Saliency Estimation Perturbations (%) (K=5)

SettingPerturbation StrengthAvg. Acc. (%)
Original Saliency0%87.7
Gaussian Noiseσ = 0.0387.4
Gaussian Noiseσ = 0.0686.9
Gaussian Noiseσ = 0.1086.3
Gaussian Blurσ = 1.0 px87.5
Gaussian Blurσ = 2.0 px86.8
Threshold Shift±0.0587.2
Threshold Shift±0.1086.6

Table 3. Joint Robustness across Scene Complexities (%) (K=5)

SceneForeground ObjectsAuxiliary MapsAvg. Acc. (%)
Simple1–2Original90.4
Simple1–2Jointly Perturbed89.7
Moderate3–10Original87.6
Moderate3–10Jointly Perturbed86.5
Crowded>10Original85.2
Crowded>10Jointly Perturbed83.7
AuthorsOpenReview ↗

Q3: Applicability to Long Instructions and Multi-turn Visual Interactions

We appreciate the reviewer’s suggestion to further discuss the applicability of Timage to longer instructions and more complex visual interaction scenarios.

(1) Handling Long Instructions. Timage incorporates a query-adaptive Ink Budget that adjusts text rendering according to instruction length and available visual space. As described in Section 3.2 and illustrated in Fig. 5, this mechanism supports adaptive font scaling and multi-line rendering for longer instructions. We also evaluate its performance under different query lengths in Table 9 of Appendix D.3. Timage achieves 87.7% accuracy on short, keyword-based queries and 87.3% on longer textual descriptions, showing that the rendering mechanism can accommodate longer instructions without a substantial drop in performance.

(2) Multi-turn Visual Interactions. Timage introduces a generative visual input reformulation paradigm, in which textual instructions are transformed into visual content and rendered directly onto the original image. The resulting image is then provided to the frozen MLLM, rather than supplying the instruction through a separate text input. This changes how task instructions are presented to the model, rather than simply adding visual markers to guide its attention.

Extending this paradigm to multi-turn interactions introduces a different challenge: the model needs to account for dialogue context and semantic dependencies across turns, rather than treating each instruction independently. In particular, relevant information from previous turns needs to be considered when reformulating the visual input for a subsequent query. We consider this an important direction for extending the proposed input paradigm and will discuss it in the future work section.

(3) Few-shot Scenarios. Timage may also be extended to few-shot settings. In this case, the input reformulation process would need to account for not only the current instruction but also task-relevant demonstrations. Selecting representative examples and organizing their information within the available visual space, while maintaining readability, are important considerations for this extension. We will discuss this direction in the future work section.

We will incorporate these discussions into the limitations and future work section to clarify the current scope of Timage and its potential extensions to more complex visual understanding scenarios.

Reviewer_zifjOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents Timage, which renders the text query onto the reference image to improve MLLM performance. It optimizes both the layout and style of the rendered text. A Constrained Schrödinger Bridge (cSB) generator uses query-relevance heatmaps and segmentation/saliency information to guide placement, while an ink-budget regularizer controls font size. The generator is trained with flow matching and downstream task supervision, and the resulting image is fed to a frozen MLLM without a separate textual query. Extensive experiments are conducted across multiple benchmarks and MLLM backbones.

Critical strengths
  1. An intuitive formulation of the text-layout problem. The method explicitly considers query relevance, non-occlusion, and readability when determining where and how to render the query, addressing practical challenges in constructing text-in-image prompts.
  2. Broad empirical coverage. The evaluation spans multiple benchmarks and MLLM backbones, with comparisons against heuristic placement, deterministic layout optimization, and parameter adaptation methods.
Critical weaknesses

The paper does not sufficiently substantiate its central claim that rendering the text query into the image is a key source of the performance gains. Timage uses query-conditioned relevance maps and segmentation/saliency masks to determine text placement, which may itself convey useful spatial guidance. Prior work, API [1], shows that query-guided heatmap overlays can improve MLLM performance while retaining the textual query. Although Table 7 provides some evidence for direct text overlay, the paper lacks a matched control that uses the same relevance maps to highlight relevant regions while retaining the textual query. Consequently, the experiments do not sufficiently distinguish the contribution of text-in-image rendering from that of auxiliary spatial guidance.

[1] Attention Prompting on Image for Large Vision-Language Models, ECCV 2024.

Questions for authors
  1. Can the authors compare Timage with an image-plus-text control using a non-textual marker at the same cSB-selected location, under matched preprocessing and inference budgets, to isolate the contribution of rendering the query itself?
  2. In Section 3.2 (Building the Energy Function), could the authors specify the auxiliary modules used and how the segmentation masks, saliency maps, and query-relevance heatmaps are generated? This would clarify what auxiliary information is introduced when rendering the query into the image.
5 comments
AuthorsOpenReview ↗

Q1: Comparison with Non-textual Visual Prompting Strategies

We thank the reviewer for this valuable suggestion. We agree that comparing against non-textual visual markers can help distinguish the contribution of explicit query embedding from that of spatial guidance.

Quantitative Comparison. We conduct controlled comparisons between Timage and three non-textual visual prompting strategies: Rectangle Outline, Corner Brackets, and Circular Marker. For these baselines, the original query is retained as textual input, while the corresponding visual marker is placed at the same cSB-selected location used by Timage. All methods use the same frozen Qwen2.5-VL-7B backbone, image resolution, preprocessing pipeline, sampling budget, number of model calls, and majority-voting protocol.

As shown in Table R1, the original Image-plus-Text baseline achieves 79.1% accuracy, while Rectangle Outline, Corner Brackets, and Circular Marker improve it to 80.3%, 80.7%, and 81.4%, respectively. In comparison, Timage achieves 87.7%, outperforming the strongest non-textual marker baseline by 6.3 percentage points. These results indicate that spatial markers provide some benefits, but explicit visual embedding of query semantics offers additional improvements beyond spatial guidance alone.

Qualitative Comparison. We further visualize representative examples from the MME test set in Fig. 6. Under the same original images and queries, non-textual markers provide only spatial cues, whereas Timage directly renders the complete query into the image, jointly conveying linguistic semantics and spatial guidance. These examples illustrate how explicit text embedding provides task-relevant information that cannot be fully conveyed by non-textual markers alone.

Table R1. Comparison with Non-textual Visual Prompting Strategies (%)

MethodQuery InputVisual MarkerPlacement StrategyAvg. Acc. (%)
Image + Text (Baseline)Original TextNoneStandard Input79.1
Text + RectangleOriginal TextRectangle OutlinecSB-guided80.3
Text + Corner BracketsOriginal TextCorner BracketscSB-guided80.7
Text + Circular MarkerOriginal TextCircular MarkercSB-guided81.4
Timage (Ours)Image-embedded TextRendered QuerycSB-guided87.7
AuthorsOpenReview ↗

Q2: Clarification of Auxiliary Modules and Information Extraction for Energy Construction

We thank the reviewer for this helpful suggestion.

we extract three complementary types of information using pretrained models:

(1) Instance Segmentation Masks.

We employ a pretrained Mask R-CNN to detect and segment foreground objects in the input image, producing instance-level binary masks (Oi(O_i). We then take the union of all instance masks to construct a unified object-protection mask, identifying the image regions occupied by foreground objects.

(2) Visual Saliency Maps.

We use a pretrained U$^2$-Net to generate dense saliency maps, which are resized to the original image resolution and normalized to the range ([0,1]([0,1]). These maps capture visually prominent regions and complement the information provided by instance-level object segmentation.

(3) Query-Relevance Heatmaps.

We extract query-to-visual attention weights from the frozen Qwen2.5-VL-7B backbone. Specifically, we use the last four Transformer layers and all attention heads. For each layer, attention weights from all valid query tokens to visual tokens are first averaged across attention heads. The resulting visual-token relevance scores are then averaged across the four Transformer layers.

Subsequently, the aggregated scores are reshaped into a two-dimensional spatial map according to the visual-token grid specified by the image-grid metadata. The resulting map is upsampled to the original image resolution using bilinear interpolation and normalized via min-max normalization. This produces the query-conditioned relevance heatmap (Hq(H_q), which identifies visual regions relevant to the input instruction.

These three sources of information play complementary roles in the energy function. The instance segmentation masks and visual saliency maps jointly construct the spatial candidate region (Mcand(M_{\mathrm{cand}}), preventing text placement from obscuring important visual content. In contrast, the query-relevance heatmap (Hq(H_q) contributes to the semantic placement energy (Esem(E_{\mathrm{sem}}), guiding text placement toward visual regions relevant to the current query.

Thank you for the clarifications and the additional experiments. Table R1 addresses my question about markers placed at the same location, but it remains unclear whether directly providing the same query-relevance map alongside the textual query would yield comparable gains. The method also relies on pretrained segmentation and saliency modules, raising questions about its practicality in settings where these modules are less reliable. On balance, I maintain my original rating of Weak Reject.

Reviewer_G7ePOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper proposes Timage, an input-space adaptation framework that renders a natural-language query directly into an image before passing the composite image to a frozen vision-language model. The downstream model does not receive a separate textual copy of the query. A learned generator determines the overlay’s placement and appearance using query-conditioned relevance, a non-occlusion mask derived from segmentation and saliency, and an adaptive ink-budget constraint. Layout generation is formulated as a constrained Schrödinger Bridge (cSB), implemented through flow-matching training and projected stochastic updates, with additional supervision from the frozen model’s answer loss. At inference, predictions from multiple sampled layouts can be aggregated by majority voting. The paper evaluates the method on VMCBench and four selected MME subtasks, with additional experiments across backbones, benchmarks, rendering conditions, and languages. It reports improvements over conventional prompting, alternative visual prompts, and parameter adaptation. The supplementary material also includes learned deterministic placement, constrained stochastic baselines, and single-sample comparisons intended to isolate the contribution of distributional layout generation.

Critical strengths
  1. A practically appealing approach to adapting frozen VLMs. Learning where and how to render a readable query provides a concrete way to modify the model’s visual input while leaving its backbone unchanged. The joint consideration of semantic relevance, non-occlusion, and readability addresses meaningful limitations of arbitrary text placement. The resulting prompts are directly inspectable, which makes the approach potentially useful as an adaptation interface.
  2. The evaluation includes relevant attempts to isolate the proposed components. The paper goes beyond comparisons with the original backbone: Appendix B includes learned deterministic placement, stochastic placement with projection, constrained diffusion, and single-sample cSB. These controls address important alternative explanations for the reported gains, particularly whether they arise from learning a placement function or from test-time ensembling. The additional backbone and benchmark experiments also attempt to establish a broader scope of applicability. This experimental coverage is a strength, although its conclusions depend on resolving the reporting and protocol issues below.
Critical weaknesses
  1. The central empirical results are inconsistent, and the training protocol is insufficiently specified. Several reported aggregate accuracies do not reconcile with the category scores under the official VMCBench evaluator. That evaluator averages over the 20 constituent datasets, with 7 General, 6 Reasoning, 2 OCR, and 5 Doc&Chart datasets. Consequently, the overall score is[A=\frac{7G+6R+2O+5D}{20}.] Using the category values in the submission, the Qwen2.5-VL-7B baseline aggregates to approximately 81.6, whereas Table 4 reports 79.1. Similarly, the (K=1) Timage results aggregate to approximately 86.3, whereas Tables 5 and 6 report 87.3. The (K=5) category scores do aggregate to approximately 87.7. These discrepancies materially affect the claimed improvements. There is also an internal inconsistency: Table 3 reports 86.0 for the full semantic, mask, and style configuration without ensembling, implying a 1.7-point ensemble gain, while Tables 5 and 6 report 87.3 for (K=1), implying only 0.4 points. No difference in experimental settings is explained. Furthermore, Section 4 states that cSB is trained on “official training splits,” but does not identify the datasets, sample counts, or split definitions. The public VMCBench release specifies 1,000 development examples and 8,018 test examples, without a training split. Training on source-dataset splits may be a valid explanation, but it needs to be stated explicitly, together with overlap controls and the supervision budgets used for the adaptation baselines. These omissions prevent assessing whether the reported comparisons and generalization claims are supported.

  2. The implemented algorithm is not adequately connected to the claimed constrained Schrödinger Bridge, and essential implementation steps are missing. Appendix A.3 defines a path-space KL-minimizing bridge with prescribed endpoint distributions. However, Equations (22)–(25) train a velocity field using straight-line conditional flow matching, while Equations (6)/(19) subsequently use that field directly as the drift of an SDE with added Brownian noise and projection. In general, a probability-flow velocity cannot simply be used as a stochastic drift without changing the marginal evolution; the relationship normally involves a score-dependent correction. The submission does not establish why its particular training objective and noisy projected sampler approximate the stated bridge or preserve the intended terminal distribution. I appreciate that the authors explicitly describe their implementation as an approximation. My concern is the missing justification for this particular approximation, rather than the absence of an exact solver. The implementation is also insufficiently specified at crucial points. Section 3.3 defers the Langevin pseudo-target sampler to the supplement, but the supplement does not provide its concrete update rule or sampling configuration. The reproducibility statement mentions straight-through gradient estimation, yet the estimator and the gradient path through text rendering are not defined. These details determine how task supervision actually updates placement and appearance. This concern is especially consequential because prior Prompt-in-Image work already investigates rendering instructions into images and removing the separate sample-specific text input. The distinctive contribution therefore rests substantially on the proposed learned layout mechanism and its justification.

Questions for authors
  1. Please provide the evaluator version and per-dataset accuracies underlying the main comparisons and ablations. In particular, please explain the discrepancies between the category-derived overall scores and the reported averages, and between the (K=1) results in Table 3 and Tables 5–6. Which results should be used to assess the gains from ensembling and the comparisons against parameter adaptation and alternative placement methods?
  2. Please identify the source datasets, splits, and sample counts for cSB and the trained baselines, along with how overlap with evaluation images and questions was controlled. For the cross-backbone and cross-benchmark experiments, was the same trained generator transferred, or were separate generators trained or adapted? This distinction is important for interpreting the generalization claims.
  3. Please explain how the velocity learned in Equation (24) relates to the stochastic drift in Equation (19), including the endpoint coupling, diffusion schedule, and any correction used in the implementation. Under what assumptions does the resulting process approximate the objective in Equation (15) or the terminal distribution in Equation (12)?
7 comments
AuthorsOpenReview ↗

Q1-1: Clarification of Random Seeds and Accuracy Reporting

We sincerely thank the authors for carefully reviewing our manuscript and providing several constructive comments.

Table 1 reports category-level accuracies averaged over three independent runs (Seeds 1–3), whereas the overall accuracies in the other tables are obtained using Seed 1. We have clarified this distinction in the revised captions and summarized the seed configurations in Table R1.

Table R2 compares the multi-seed and Seed-1 overall accuracies of Qwen2.5-VL-7B, Timage (K=1), and Timage (K=5). Multi-seed results are reported as Mean ± Std, while single-seed results correspond to Seed 1.

For Table 6, the four category-level accuracies are averaged over Seeds 1–3 and reported as Mean ± Std, whereas Avg. Acc. corresponds to Seed 1. We have clarified this reporting protocol and provide the individual seed results in Table R3 for verification.

Table R1. Random-seed configurations

ModelTableSeeds
Qwen2.5-VL-7BTable 11, 2, 3
Qwen2.5-VL-7BTable 41
Timage (K=1)Table 11, 2, 3
Timage (K=1)Table 5/61
Timage (K=5)Table 11, 2, 3
Timage (K=5)Table 5/61

Table R2. Overall accuracy under different settings (%)

Model3 SeedsSeed 1
Qwen2.5-VL-7B81.679.1
Timage (K=1)86.387.3
Timage (K=5)87.787.7

Table R3. Category-level accuracy across three seeds (%)

ModelCategorySeed 1Seed 2Seed 3Mean ± Std
Qwen2.5-VL-7BGeneral83.087.486.485.6 ± 2.3
Reasoning65.969.569.268.2 ± 2.0
OCR92.297.196.695.3 ± 2.7
Doc&Chart84.188.087.486.5 ± 2.1
Timage (K=1)General90.890.788.890.1 ± 1.1
Reasoning75.175.072.374.1 ± 1.6
OCR98.998.795.897.8 ± 1.7
Doc&Chart92.392.088.791.0 ± 2.0
Timage (K=5)General91.592.190.391.3 ± 0.9
Reasoning75.577.074.975.8 ± 1.1
OCR98.699.797.398.5 ± 1.2
Doc&Chart92.693.491.892.6 ± 0.8
AuthorsOpenReview ↗

Q1-2: Per-Dataset Results across 20 VMCBench Subsets

To provide a comprehensive comparison, we report per-dataset accuracies across all 20 VMCBench subsets in Tables S1. All results are averaged over three independent runs. Open-weight models use Seeds 1–3, while API-based models use three independent API evaluations. Table S1 presents comparison methods.

Table S1. Per-dataset accuracy for comparison methods (%)

All results are averaged over three independent runs.

ModelVQAv2OK-VQAMMVetVizWizA-OKVQAMMStarSEED-BenchMath-VisionGQAMMMUReal-WorldQAMath-VistaScience-QAOCR-VQAText-VQADoc-VQAInfo-VQAChart-QATable-VQA-BenchAI2D
Qwen2-VL-72B89.988.984.689.789.689.087.878.475.171.571.968.969.895.797.992.589.692.285.490.8
GPT-4o85.583.184.387.387.283.086.065.366.566.467.366.469.595.197.787.279.783.880.684.2
Claude-3.5-Sonnet79.682.782.881.479.683.179.961.962.961.564.561.964.193.193.786.689.580.581.185.3
Molmo-72B84.883.781.680.482.482.485.065.967.462.675.664.164.095.094.479.983.981.884.175.8
Qwen2.5-VL-14B90.488.287.590.889.190.492.269.378.268.569.977.971.896.296.892.090.290.187.689.6
Cambrian-34B80.883.883.386.384.484.782.666.062.166.266.670.564.096.594.972.774.867.475.775.9
VILA1.5-40B82.181.580.682.983.483.883.265.162.869.265.464.864.593.892.660.771.767.370.367.0
CogVLM2-19B77.178.182.276.179.177.376.855.058.158.949.055.557.191.093.679.572.070.868.871.9
Qwen3-8B88.288.184.190.486.683.690.172.970.072.568.871.075.697.195.786.988.891.389.491.1
Qwen2.5-VL-7B (Base)89.087.285.984.882.187.782.564.770.964.372.067.869.595.495.287.581.886.186.790.4
Qwen2-VL-7B81.480.185.786.086.187.185.162.865.159.562.759.466.796.296.681.981.078.179.979.6
Molmo-7B-D73.272.574.870.570.576.674.353.452.758.356.551.760.491.292.270.872.068.577.272.0
Cambrian-8B77.078.577.577.878.679.176.853.256.057.852.758.560.290.391.766.561.471.462.065.7
Phi-3-Vision71.672.876.175.875.973.373.254.655.858.852.156.360.890.990.366.674.475.577.974.6
LLaVA-1.5-7B66.265.462.963.263.459.264.945.346.542.344.644.644.973.574.532.034.937.734.036.4
Timage (K=1)89.894.086.992.090.788.089.373.775.674.073.572.675.298.297.490.992.290.790.291.0
Timage (K=5)90.988.791.893.193.091.090.677.473.172.771.982.876.999.098.091.792.492.694.691.7
AuthorsOpenReview ↗

Q1-3: Per-Dataset Ablation Results

We report per-dataset accuracies for all nine placement and ablation methods across 20 VMCBench subsets in Table S2. Results are averaged over three independent runs (Seeds 1–3).

Table S2. Per-dataset accuracy (%) across 20 subsets.

ModelVQAv2OK-VQAMMVetVizWizA-OKVQAMMStarSEED-BenchMath-VisionGQAMMMUReal-WorldQAMath-VistaScience-QAOCR-VQAText-VQADoc-VQAInfo-VQAChart-QATable-VQA-BenchAI2D
Base89.087.285.984.882.187.782.564.770.964.372.067.869.595.495.287.581.886.186.790.4
Direct Energy Minimization89.989.089.589.689.190.590.272.172.571.671.772.272.596.795.788.188.688.488.688.7
Deterministic Learned Placement87.988.088.489.188.189.088.971.671.772.372.071.272.196.396.588.688.487.587.588.0
Stochastic Placement87.587.387.787.387.587.186.770.971.071.071.570.670.396.295.487.386.887.987.087.0
Stochastic + Projection89.589.088.789.788.988.889.873.673.572.773.673.373.796.797.390.489.890.290.490.7
Standard Diffusion + Constraints90.290.089.089.789.988.989.572.972.972.971.672.472.897.797.187.188.588.488.087.6
Semantic Greedy Placement90.189.588.989.788.889.288.972.573.172.073.472.273.697.597.188.288.787.988.588.7
Timage (K=1)89.894.086.992.090.788.089.373.775.674.073.572.675.298.297.490.992.290.790.291.0
Timage (K=5)90.988.791.893.193.091.090.677.473.172.771.982.876.999.098.091.792.492.694.691.7

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.