- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents Timage, a simple yet effective way to rethink multimodal reasoning: instead of feeding text questions and images into a model separately, it renders the question directly onto the image itself.
To place the text naturally, Timage uses a Constrained Schrödinger Bridge (CSB) to solve two practical layout challenges: finding empty space that avoids covering key foreground objects, and adjusting the font style and visibility to fit the scene.
At inference time, the underlying Multimodal Large Language Model (MLLM) remains completely frozen and takes only the composite image as input. Benchmarks on VMCBench and MME show that a frozen 7B model using Timage not only beats fine-tuned baselines (LoRA and full fine-tuning), but also outperforms much larger commercial models like GPT-4o.
Critical strengths
Transforming multimodal alignment from a model-internal fusion/fine-tuning problem into an input-level image rendering formulation is conceptually innovative. Feeding only the composite image into a frozen backbone bridges the spatial-semantic gap using the model's native OCR and visual grounding capabilities.
The use of a CSB with projections offers an elegant mathematical treatment of the inherently one-to-many layout generation problem, appropriately balancing soft cross-attention semantic pull against hard foreground obstacle avoidance.
The approach is inherently human-interpretable. Users can visually inspect exactly where the model's textual cues are anchored relative to scene contents.
Critical weaknesses
Time cost: Inlaying text requires multiple pre-processing modules: foreground instance segmentation, saliency detection, a cross-attention extraction forward pass, and an iterative 20-step CSB sampling process. This poses a significant hurdle for deployments.
Upstream vision quality:The admissible mask fundamentally relies on the reliability of the external instance segmenter and saliency estimator. In cluttered scenes where off-the-shelf segmenters fail, the non-occlusion barrier could fail and occlude critical target details.
Long and multi-turn prompts:The experiments primarily focus on short VQA queries. In real-world applications requiring complex multi-turn dialogs, few-shot in-context demonstrations, or long instructions, rendering everything onto the image without causing visual clutter or massive font reduction remains unproven.
Baselines: Models tested and evaluation suits are not up to date. More like 2025 paper.
Questions for authors
Please see weaknesses.
8 comments
Q2: Robustness to Imperfect Upstream Visual Estimates
We evaluate Timage's robustness to upstream visual estimation errors through three complementary experiments, covering instance segmentation, saliency estimation, and their joint effects under different scene complexities. All robustness experiments use K=5.
(1) Instance Segmentation Robustness.
We perturb Mask R-CNN masks through erosion, dilation (5% and 10% area changes), and random instance dropping (10%, 20%, and 30%). As shown in Table 1, accuracy decreases from 87.7% to 86.1% under 30% instance dropping, indicating robustness to moderate segmentation errors.
(2) Saliency Estimation Robustness.
We perturb U²-Net saliency maps using Gaussian noise (σ = 0.03, 0.06, 0.10), Gaussian blur (σ = 1.0 and 2.0 pixels), and threshold shifts (±0.05 and ±0.10). As shown in Table 2, accuracy remains at 86.3% under the strongest noise and 86.6% under the largest threshold shift.
(3) Joint Robustness across Scene Complexities.
We group images by foreground object count: simple (1–2), moderate (3–10), and crowded (more than 10), using the original Mask R-CNN predictions. As shown in Table 3, joint perturbations reduce accuracy by 0.7, 1.1, and 1.5 percentage points, respectively, suggesting greater sensitivity to upstream errors in crowded scenes.
Additionally, Fig. 4 provides qualitative examples under complex visual conditions and incomplete query-relevance information, complementing these quantitative results.
Table 1. Robustness to Instance Segmentation Perturbations (%) (K=5)
| Setting | Perturbation Strength | Avg. Acc. (%) |
|---|---|---|
| Original Mask | 0% | 87.7 |
| Mask Erosion | 5% | 87.4 |
| Mask Erosion | 10% | 86.9 |
| Mask Dilation | 5% | 87.6 |
| Mask Dilation | 10% | 87.1 |
| Instance Drop | 10% | 87.3 |
| Instance Drop | 20% | 86.7 |
| Instance Drop | 30% | 86.1 |
Table 2. Robustness to Saliency Estimation Perturbations (%) (K=5)
| Setting | Perturbation Strength | Avg. Acc. (%) |
|---|---|---|
| Original Saliency | 0% | 87.7 |
| Gaussian Noise | σ = 0.03 | 87.4 |
| Gaussian Noise | σ = 0.06 | 86.9 |
| Gaussian Noise | σ = 0.10 | 86.3 |
| Gaussian Blur | σ = 1.0 px | 87.5 |
| Gaussian Blur | σ = 2.0 px | 86.8 |
| Threshold Shift | ±0.05 | 87.2 |
| Threshold Shift | ±0.10 | 86.6 |
Table 3. Joint Robustness across Scene Complexities (%) (K=5)
| Scene | Foreground Objects | Auxiliary Maps | Avg. Acc. (%) |
|---|---|---|---|
| Simple | 1–2 | Original | 90.4 |
| Simple | 1–2 | Jointly Perturbed | 89.7 |
| Moderate | 3–10 | Original | 87.6 |
| Moderate | 3–10 | Jointly Perturbed | 86.5 |
| Crowded | >10 | Original | 85.2 |
| Crowded | >10 | Jointly Perturbed | 83.7 |
Q3: Applicability to Long Instructions and Multi-turn Visual Interactions
We appreciate the reviewer’s suggestion to further discuss the applicability of Timage to longer instructions and more complex visual interaction scenarios.
(1) Handling Long Instructions. Timage incorporates a query-adaptive Ink Budget that adjusts text rendering according to instruction length and available visual space. As described in Section 3.2 and illustrated in Fig. 5, this mechanism supports adaptive font scaling and multi-line rendering for longer instructions. We also evaluate its performance under different query lengths in Table 9 of Appendix D.3. Timage achieves 87.7% accuracy on short, keyword-based queries and 87.3% on longer textual descriptions, showing that the rendering mechanism can accommodate longer instructions without a substantial drop in performance.
(2) Multi-turn Visual Interactions. Timage introduces a generative visual input reformulation paradigm, in which textual instructions are transformed into visual content and rendered directly onto the original image. The resulting image is then provided to the frozen MLLM, rather than supplying the instruction through a separate text input. This changes how task instructions are presented to the model, rather than simply adding visual markers to guide its attention.
Extending this paradigm to multi-turn interactions introduces a different challenge: the model needs to account for dialogue context and semantic dependencies across turns, rather than treating each instruction independently. In particular, relevant information from previous turns needs to be considered when reformulating the visual input for a subsequent query. We consider this an important direction for extending the proposed input paradigm and will discuss it in the future work section.
(3) Few-shot Scenarios. Timage may also be extended to few-shot settings. In this case, the input reformulation process would need to account for not only the current instruction but also task-relevant demonstrations. Selecting representative examples and organizing their information within the available visual space, while maintaining readability, are important considerations for this extension. We will discuss this direction in the future work section.
We will incorporate these discussions into the limitations and future work section to clarify the current scope of Timage and its potential extensions to more complex visual understanding scenarios.
Q1: Time cost
We thank the reviewer for raising concerns regarding the computational efficiency and practical deployment of Timage. We evaluate its efficiency from two aspects.
(1) Inference Time Breakdown.
We profile the complete Timage inference pipeline, including instance segmentation, saliency estimation, query-relevance extraction, cSB sampling, text rendering, and downstream MLLM inference. As shown in Table 1, query-relevance extraction and downstream MLLM inference account for the majority of the total latency. Importantly, auxiliary visual information is computed only once for each image–question pair and reused across subsequent layout sampling steps, avoiding redundant preprocessing overhead.
(2) Accuracy–Latency Trade-off.
We further evaluate the effects of different cSB refinement steps on inference efficiency and task performance under the K=1 setting. As shown in Table 2, reducing the sampling steps from 20 to 10 and 5 lowers the total latency to 1,750 ms and 1,705 ms, respectively, while maintaining accuracies of 85.9% and 84.2%. The 20-step configuration achieves the highest accuracy of 87.3% among the evaluated settings, while fewer steps provide practical alternatives for latency-sensitive applications.
Moreover, Timage uses the relatively compact Qwen2.5-VL-7B backbone, avoiding the computational requirements of substantially larger MLLMs. Combined with one-time auxiliary feature extraction and flexible cSB sampling, this design provides a practical balance between model size, inference efficiency, and task performance. The sampling budget can be adjusted according to deployment requirements without changing the underlying model architecture or training procedure.
Table 1. Inference Time Breakdown of Timage (K=1)
Table 2. Accuracy–Latency Trade-off under Different cSB Sampling Steps (K=1)