acceptodds
Under review as a conference paper at ICLR 2027

VisionFoundry: Teaching VLMs Visual Perception with Synthetic Images

Abstract

Vision-language models (VLMs) still struggle with visual perception tasks such as spatial understanding and viewpoint recognition, largely because natural image datasets provide limited supervision for low-level visual skills. Can targeted synthetic supervision address these weaknesses without reference images or manual annotation? To investigate this, we introduce VisionFoundry, an automated pipeline that takes only a task name as input, uses LLMs to synthesize paired questions, answers, and text-to-image (T2I) prompts, generates images with T2I models, and filters samples via multimodal verification. With VisionFoundry, we construct VisionFoundry-10k, a synthetic VQA dataset spanning 10 perception tasks. Finetuning on VisionFoundry-10k consistently improves perception benchmarks across three open-source backbones (e.g., +6.7% on MMVP-pair and +10.5% on CV-Bench-3D for Qwen2.5-VL-3B-Instruct) while preserving broader capabilities and showing positive data scaling. The same synthetic supervision also yields consistent gains under reinforcement learning (RL) across all three backbones, and the framework remains effective under open-source synthesis and self-verification. Our findings demonstrate that automated synthetic supervision offers an effective and scalable path toward systematic VLM training. Code: https://anonymous.4open.science/r/VisionFoundry_Anonymous/README.md.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Synthetic Data, VLM, Visual Perception

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_wDD8OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents VisionFoundary, a three-stage synthetic data generation pipeline for VLM training. Based on this pipeline, the authors construct VisionFoundary-10K and conduct experiments to validate its effectiveness. The results show performance improvements across several benchmarks, particularly on perception-centric tasks.

Critical strengths
  1. The paper is well-written and clearly presents the proposed data generation pipeline with sufficient implementation details.

  2. The experiments are reasonably comprehensive, demonstrating the effectiveness of the proposed dataset and pipeline while investigating the impact of synthetic data on model performance.

Critical weaknesses
  1. The novelty of this work is somewhat limited. Synthetic data generation for VLM training has been widely explored, but the paper does not clearly distinguish its contributions from prior methods or provide sufficient comparisons to demonstrate its advantages.

  2. The proposed pipeline has several limitations. First, the generated images may inherit the distributional biases of the underlying image generation models, potentially limiting data diversity and introducing gaps in resolution, realism, and complexity compared to real-world images. However, the paper lacks an analysis of these distributional characteristics and their impact on model performance. Second, the pipeline appears primarily applicable to spatial understanding and perception-centric tasks, while its applicability to other tasks, such as Math, STEM, OCR, and chart understanding, remains unclear.

  3. The experiments are mainly conducted on Qwen2.5-VL, which limits the generalizability of the conclusions. The observed gains may partly come from addressing the model's existing weaknesses in visual perception. Additional evaluations on stronger and more recent VLMs, such as Qwen3-VL and Qwen3.5, would provide more convincing evidence of the pipeline's effectiveness.

Questions for authors
  1. What are the key novel contributions of VisionFoundary compared to existing synthetic data generation methods?

  2. Have the authors analyzed the distributional characteristics of the generated images, such as diversity, resolution, and realism? How do these characteristics affect downstream model performance?

  3. Does VisionFoundary-10K provide similar improvements on stronger VLMs, such as Qwen3-VL and Qwen3.5?

8 comments

1. The novelty of this work is somewhat limited. Synthetic data generation for VLM training has been widely explored, but the paper does not clearly distinguish its contributions from prior methods or provide sufficient comparisons to demonstrate its advantages.

Thank you for your comment. VisionFoundry contributes a task-driven approach to constructing complete visual supervision. Starting from target perceptual tasks and structured configurations, it jointly generates QA and answer-conditioned image prompts, synthesizes images, and verifies that they support the answers. No reference images or existing QA are required. We also study the resulting data in both SFT and RL. The table below summarizes the differences from prior work.

Method\small \text{Method}Starting point for dataconstruction\small \begin{array}{l}\text{Starting point for data}\\\\ \text{construction}\end{array}Image source\small \text{Image source}Supervision construction andquality control\small \begin{array}{l}\text{Supervision construction and}\\\\ \text{quality control}\end{array}Main focus\small \text{Main focus}
SVIT\small \textbf{SVIT}Existing images and annotationssuch as descriptions and boundingboxes\small \begin{array}{l}\text{Existing images and annotations}\\\\ \text{such as descriptions and bounding}\\\\ \text{boxes}\end{array}Reuses existing images\small \text{Reuses existing images}Generates multiple types of visualinstructions from imageannotations\small \begin{array}{l}\text{Generates multiple types of visual}\\\\ \text{instructions from image}\\\\ \text{annotations}\end{array}Scaling up visual instruction data\small \text{Scaling up visual instruction data}
MMEvol\small \textbf{MMEvol}Existing images and seedinstructions\small \begin{array}{l}\text{Existing images and seed}\\\\ \text{instructions}\end{array}Reuses existing images\small \text{Reuses existing images}Evolves and filters perceptual,reasoning, and interactioninstructions\small \begin{array}{l}\text{Evolves and filters perceptual,}\\\\ \text{reasoning, and interaction}\\\\ \text{instructions}\end{array}Increasing instruction complexityand diversity\small \begin{array}{l}\text{Increasing instruction complexity}\\\\ \text{and diversity}\end{array}
ProVision\small \textbf{ProVision}Existing images and scene graphs\small \text{Existing images and scene graphs}Reuses existing images\small \text{Reuses existing images}Programmatically generates QA aboutobjects, attributes, relations,depth, etc., from scene graphs\small \begin{array}{l}\text{Programmatically generates QA about}\\\\ \text{objects, attributes, relations,}\\\\ \text{depth, etc., from scene graphs}\end{array}Constructing visual perceptionsupervision\small \begin{array}{l}\text{Constructing visual perception}\\\\ \text{supervision}\end{array}
SpatialVLM\small \textbf{SpatialVLM}Existing natural images\small \text{Existing natural images}Reuses existing images\small \text{Reuses existing images}Uses detection, segmentation, anddepth estimation to constructspatial QA and filters ambiguoussamples\small \begin{array}{l}\text{Uses detection, segmentation, and}\\\\ \text{depth estimation to construct}\\\\ \text{spatial QA and filters ambiguous}\\\\ \text{samples}\end{array}3D spatial and metric geometricreasoning\small \begin{array}{l}\text{3D spatial and metric geometric}\\\\ \text{reasoning}\end{array}
Img-Diff\small \textbf{Img-Diff}Existing image descriptions andtheir rewritten versions\small \begin{array}{l}\text{Existing image descriptions and}\\\\ \text{their rewritten versions}\end{array}Generates image pairsfrom descriptions\small \begin{array}{l}\text{Generates image pairs}\\\\ \text{from descriptions}\end{array}Filters using image similarity,image–text matching, anddifference localization\small \begin{array}{l}\text{Filters using image similarity,}\\\\ \text{image–text matching, and}\\\\ \text{difference localization}\end{array}Learning fine-grained differencesfrom image pairs\small \begin{array}{l}\text{Learning fine-grained differences}\\\\ \text{from image pairs}\end{array}
VisionFoundry\small \textbf{VisionFoundry}Target perceptual tasks andconfigurations\small \begin{array}{l}\textbf{Target perceptual tasks and}\\\\ \textbf{configurations}\end{array}Synthesizes target sceneswith T2I\small \begin{array}{l}\textbf{Synthesizes target scenes}\\\\ \textbf{with T2I}\end{array}Jointly generates QA andanswer-conditioned image prompts,then verifies image–answerconsistency\small \begin{array}{l}\textbf{Jointly generates QA and}\\\\ \textbf{answer-conditioned image prompts,}\\\\ \textbf{then verifies image–answer}\\\\ \textbf{consistency}\end{array}Constructing complete perceptualVQA data from task objectives\small \begin{array}{l}\textbf{Constructing complete perceptual}\\\\ \textbf{VQA data from task objectives}\end{array}

We additionally compare VisionFoundry with ProVision, a closely related method for constructing visual perception supervision, using 10K samples and the same training configuration. VisionFoundry achieves higher scores on all five reported metrics:

Training data\small \text{Training data}\small \text{MMVP single (\\%)}\small \text{MMVP pair (\\%)}\small \text{CV-Bench-2D (\\%)}\small \text{CV-Bench-3D (\\%)}\small \text{RealWorldQA (\\%)}
ProVision 10K\small \text{ProVision 10K}67.3\small \text{67.3}39.3\small \text{39.3}71.1\small \text{71.1}74.7\small \text{74.7}65.5\small \text{65.5}
VisionFoundry 10K\small \text{VisionFoundry 10K}68.3\small \textbf{68.3}42.0\small \textbf{42.0}72.4\small \textbf{72.4}76.5\small \textbf{76.5}66.9\small \textbf{66.9}

2. The proposed pipeline has several limitations. First, the generated images may inherit the distributional biases of the underlying image generation models, potentially limiting data diversity and introducing gaps in resolution, realism, and complexity compared to real-world images. However, the paper lacks an analysis of these distributional characteristics and their impact on model performance. Second, the pipeline appears primarily applicable to spatial understanding and perception-centric tasks, while its applicability to other tasks, such as Math, STEM, OCR, and chart understanding, remains unclear.

We have added controlled experiments on data diversity, resolution, and realism.

Data diversity. We compare training on a randomly sampled set of 1K examples ten times with training on the full set of 10K examples once.

Training data\small \text{Training data}\small \text{MMVP single (\\%)}\small \text{MMVP pair (\\%)}\small \text{CV-Bench-2D (\\%)}\small \text{CV-Bench-3D (900-example test split, \\%)}\small \text{RealWorldQA (\\%)}
Random 1K × 10\small \text{Random 1K }\times \text{ 10}67.3\small \text{67.3}38.7\small \text{38.7}71.2\small \text{71.2}77.7\small \textbf{77.7}64.4\small \text{64.4}
VisionFoundry 10K × 1\small \text{VisionFoundry 10K }\times \text{ 1}68.3\small \textbf{68.3}42.0\small \textbf{42.0}72.4\small \textbf{72.4}77.7\small \textbf{77.7}66.9\small \textbf{66.9}

Using more distinct examples achieves higher scores on MMVP, CV-Bench-2D, and RealWorldQA, and matches the score on the CV-Bench-3D test set, indicating that increasing the diversity of the training data helps improve visual perception capabilities.

Resolution. We use a 1K dataset and downscale the images so that their longest side is 512 pixels, while keeping the QA and training settings unchanged.

Training images\small \text{Training images}\small \text{MMVP single (\\%)}\small \text{MMVP pair (\\%)}\small \text{CV-Bench-2D (\\%)}\small \text{CV-Bench-3D (\\%)}\small \text{RealWorldQA (\\%)}
Longest side of 512 pixels\small \text{Longest side of 512 pixels}64.3\small \text{64.3}36.0\small \text{36.0}68.0\small \text{68.0}73.8\small \text{73.8}65.5\small \text{65.5}
Original resolution\small \text{Original resolution}65.0\small \textbf{65.0}36.7\small \textbf{36.7}69.2\small \textbf{69.2}74.3\small \textbf{74.3}65.8\small \textbf{65.8}

In this setting, training with lower-resolution images leads to a small decrease, but generated images can still provide effective visual perception supervision at a slightly reduced resolution.

Realism. With the same QA, we compare default settings with images regenerated after adding a photorealistic style requirement. Each group contains 1K paired samples.

Training images\small \text{Training images}\small \text{MMVP single (\\%)}\small \text{MMVP pair (\\%)}\small \text{CV-Bench-2D (\\%)}\small \text{CV-Bench-3D (\\%)}\small \text{RealWorldQA (\\%)}
Original generated images\small \text{Original generated images}64.7\small \textbf{64.7}35.3\small \textbf{35.3}68.0\small \text{68.0}74.9\small \textbf{74.9}66.5\small \textbf{66.5}
Images generated in a photorealistic style\small \text{Images generated in a photorealistic style}64.3\small \text{64.3}35.3\small \textbf{35.3}68.5\small \textbf{68.5}73.8\small \text{73.8}65.8\small \text{65.8}

Images generated in a photorealistic style don't not bring consistent gains on all metrics. This indicates that, in our experiments, a stronger photorealistic style does not necessarily lead to better visual perception performance, which may indirectly reflect the role of data diversity.

Regarding applicability to other tasks, the goal of VisionFoundry is to construct supervision for visual perception, focusing on capabilities such as spatial relations, orientation, viewpoint, and object attributes. We have not claimed that the current experiments have established the effectiveness of this pipeline for constructing data for Math, STEM, OCR, or chart tasks. Some samples in these tasks require the precise rendering of text, numbers, formulas, or structured charts. For such content, programmatic rendering can provide more direct control over the content and can also complement T2I generation. Might directly using T2I actually be putting the cart before the horse? Combining task configurations and verification mechanisms with generation methods suited to these domains is a direction worth exploring and we leave it to future work, as already noted in the Limitations paragraph of the paper. The improvements achieved by VisionFoundry on benchmarks in these domains in the paper primarily reflect improvements in core visual perception driven by the generalization ability of our method.

3. The experiments are mainly conducted on Qwen2.5-VL, which limits the generalizability of the conclusions. The observed gains may partly come from addressing the model's existing weaknesses in visual perception. Additional evaluations on stronger and more recent VLMs, such as Qwen3-VL and Qwen3.5, would provide more convincing evidence of the pipeline's effectiveness.

Thank you for your suggestion. Our main experiments have already evaluated VisionFoundry on three model families: Qwen2.5-VL, MiMo-VL, and Llama-Vision. To further examine its effectiveness on newer models, we have added before-and-after training comparisons for Qwen3-VL-2B and Qwen3.5-2B.

Model\small \text{Model}\small \text{MMVP single (\\%)}\small \text{MMVP pair (\\%)}\small \text{CV-Bench-2D (\\%)}\small \text{CV-Bench-3D (\\%)}\small \text{RealWorldQA (\\%)}
Qwen3-VL-2B baseline\small \text{Qwen3-VL-2B baseline}72.3\small \text{72.3}46.7\small \text{46.7}72.7\small \text{72.7}87.3\small \text{87.3}65.1\small \text{65.1}
Qwen3-VL-2B VisionFoundry\small \text{Qwen3-VL-2B VisionFoundry}73.3\small \textbf{73.3}48.7\small \textbf{48.7}73.9\small \textbf{73.9}87.8\small \textbf{87.8}66.3\small \textbf{66.3}
Qwen3.5-2B baseline\small \text{Qwen3.5-2B baseline}75.3\small \text{75.3}51.3\small \text{51.3}78.4\small \text{78.4}87.8\small \text{87.8}71.6\small \text{71.6}
Qwen3.5-2B VisionFoundry\small \text{Qwen3.5-2B VisionFoundry}76.3\small \textbf{76.3}53.3\small \textbf{53.3}78.9\small \textbf{78.9}88.0\small \textbf{88.0}71.8\small \textbf{71.8}

As shown, both models improve on all reported visual perception metrics after training on VisionFoundry data. These results indicate that the effect of VisionFoundry is not limited to the models used in the original paper and can also extend to newer VLMs.

Reviewer_fupEOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes an automated pipeline to improve vision-language models’ visual perception through targeted synthetic supervision. Starting from a task name, an LLM generates question–answer pairs and detailed image prompts that explicitly specify the visual facts required to answer each question. A text-to-image model produces the images, while a multimodal verifier checks their consistency with the intended answers, filtering failures or triggering corrective edits without requiring reference images or manual labeling. The authors use this pipeline to construct a dataset called VisionFoundry-10K, containing 10,000 examples across 10 perception tasks, including spatial relationships, depth ordering, viewpoint, orientation, and color recognition. This paper also conducts a comprehensive experiments to verify the proposed pipeline and datasets effectiveness by showing the benefits for perception improvements.

Critical strengths
  1. The paper has conducted a very comprehensive evaluation suite. In Table 1, it spans 3 distinctive VLM families and 15 metrics over 13 benchmarks.
  2. Training on the proposed VisionFoundary10k dataset for one epoch improves all five reported perception metrics across Qwen, Mimo, and Llama. The improvement is quite noticeable as Qwen got 10.5% improvement on CV-Bench-3D and Mimo got 11.4% improvement.
  3. The quality control in the proposed method is also supported strongly by annotation auditing and downstream ablation study.
  4. The proposed dataset might have a long-term impact to the community of VLM study and development.
Critical weaknesses
  1. In Table 1 and Appendix B, it states that experiments results are averaged over four independent runs, but it didn't report standard deviations or confidence interval.
  2. In Table 7, training on "Color and Appearance" data achieves 73.7 on CV-Bench-3D, compared with 72.4 points from "Depth Order" data and 73.9 points from "Orientation and Direction" data. These results are compatible with useful cross-task transfer, but do not establish that targeting a particular skill preferentially repairs that skill.
  3. In section 4.4, the comparison was based on randomly sampled LLaVA-Instruct data, but VisionFoundry is delibrately constructed from perceptual skills relevant to the evaluations. The advantage of the construction reflects better task selection, but has little weak support to demonstrate that synthetic image is a better source of supervision. It would be very helpful if the authors could provide an ablation study to separate the advantage of synthesis from task targeting.
Questions for authors

A few technical detail questions:

  1. How does 10K training compare with repeated 1K training at the same number of updates, with checkpoints selected on a separate validation set?
  2. Which tasks are represented in the 140-candidate audit, and what is the accuracy of the full self-verifier dataset under independent human review?
  3. In the shared-QA image-source experiment, are both branches checked for answer correctness and retained using identical rules? How often does a natural caption or synthetic reconstruction fail to preserve the QA fact?
3 comments

1. In Table 1 and Appendix B, it states that experiments results are averaged over four independent runs, but it didn't report standard deviations or confidence interval.

Thank you for your suggestion. We have additionally reported the standard deviations of the Table 1 experiments across four random seeds. The table below reports the mean ± standard deviation of Qwen, MiMo, and Llama on the benchmarks.

Model\small \text{Model}\small \text{MMVP single (\\%)}\small \text{MMVP pair (\\%)}\small \text{CV-Bench-2D (\\%)}\small \text{CV-Bench-3D (\\%)}\small \text{RealWorldQA (\\%)}
Qwen2.5-VL-3B\small \text{Qwen2.5-VL-3B}68.3 ± 0.6\small \text{68.3 }\pm \text{ 0.6}42.0 ± 0.6\small \text{42.0 }\pm \text{ 0.6}72.4 ± 0.2\small \text{72.4 }\pm \text{ 0.2}76.5 ± 0.6\small \text{76.5 }\pm \text{ 0.6}66.9 ± 0.4\small \text{66.9 }\pm \text{ 0.4}
MiMo-VL-7B\small \text{MiMo-VL-7B}77.7 ± 0.4\small \text{77.7 }\pm \text{ 0.4}57.3 ± 0.6\small \text{57.3 }\pm \text{ 0.6}79.0 ± 0.5\small \text{79.0 }\pm \text{ 0.5}83.7 ± 0.5\small \text{83.7 }\pm \text{ 0.5}67.5 ± 0.4\small \text{67.5 }\pm \text{ 0.4}
Llama-3.2-11B-Vision\small \text{Llama-3.2-11B-Vision}71.7 ± 0.4\small \text{71.7 }\pm \text{ 0.4}46.7 ± 0.5\small \text{46.7 }\pm \text{ 0.5}71.7 ± 0.1\small \text{71.7 }\pm \text{ 0.1}75.3 ± 0.2\small \text{75.3 }\pm \text{ 0.2}64.6 ± 0.3\small \text{64.6 }\pm \text{ 0.3}

The standard deviations of these visual perception metrics across the four training runs are small, indicating that the results are relatively stable across different training seeds.

2. In Table 7, training on "Color and Appearance" data achieves 73.7 on CV-Bench-3D, compared with 72.4 points from "Depth Order" data and 73.9 points from "Orientation and Direction" data. These results are compatible with useful cross-task transfer, but do not establish that targeting a particular skill preferentially repairs that skill.

We conducted an additional task-specific training experiment and evaluated the resulting checkpoints on the Depth and Distance subsets of CV-Bench-3D. The table below shows the improvements from each type of training data on the corresponding question categories:

Training data\small \text{Training data}\small \text{Depth (\\%)}\small \text{Distance (\\%)}
Baseline\small \text{Baseline}73.0\small \text{73.0}59.0\small \text{59.0}
Orientation and Direction (1K)\small \text{Orientation and Direction (1K)}77.0\small \text{77.0}67.7\small \text{67.7}
Depth Order (1K)\small \text{Depth Order (1K)}78.8\small \textbf{78.8}69.0\small \text{69.0}
Color and Appearance (1K)\small \text{Color and Appearance (1K)}78.0\small \text{78.0}67.7\small \text{67.7}
Relative Distance (1K)\small \text{Relative Distance (1K)}78.3\small \text{78.3}69.7\small \textbf{69.7}

Training on Depth Order data achieves the highest score on depth-related tasks, while training on Relative Distance data achieves the highest score on distance-related tasks. This demonstrates that beneficial cross-task transfer exists while also showing effective improvements targeted at specific tasks.

3. In section 4.4, the comparison was based on randomly sampled LLaVA-Instruct data, but VisionFoundry is delibrately constructed from perceptual skills relevant to the evaluations. The advantage of the construction reflects better task selection, but has little weak support to demonstrate that synthetic image is a better source of supervision. It would be very helpful if the authors could provide an ablation study to separate the advantage of synthesis from task targeting.

Thank you for your suggestion. First, the statement that “synthetic images themselves are a better source of supervision” you mentioned is based on the fact that, within the VisionFoundry framework, the T2I prompt used to generate a synthetic image is generated jointly with the question and answer. Compared with natural images, which require a captioning step before questions can be constructed, this offers a potential advantage in image–question alignment from the outset. For example, complex or counterfactual visual scenarios that do not exist in natural data can be constructed using synthetic data. Therefore, “synthetic images themselves are a better source of supervision” does not mean that we claim synthetic data are superior regardless of the content or controlled conditions.

Thus, we do not claim that synthetic images are superior to natural images under all content and control conditions. Instead, we aim to investigate whether the complete image and QA synthesis process can provide additional benefits when constructing supervision for target visual perception tasks.

Following your suggestion, we have already provided two related ablation experiments in Appendix H.

The first compares natural and synthetic images for the same task. As shown in Figure 10, we sample 1K natural images from LLaVA-Instruct, generate image captions, and construct Orientation and Direction QA pairs from these captions. We then compare this dataset with 1K synthetic image–QA examples for the same task from VisionFoundry. Both groups use the same model and training settings.

Training data\small \text{Training data}\small \text{CV-Bench-2D (\\%)}\small \text{CV-Bench-3D (\\%)}\small \text{RealWorldQA (\\%)}\small \text{BLINK (\\%)}\small \text{MMMU-Val (\\%)}
Baseline\small \text{Baseline}67.3\small \text{67.3}66.0\small \text{66.0}65.0\small \text{65.0}48.1\small \text{48.1}48.0\small \textbf{48.0}
Natural images + synthetic QA\small \text{Natural images + synthetic QA}67.6\small \text{67.6}68.7\small \text{68.7}65.5\small \text{65.5}48.5\small \text{48.5}47.2\small \text{47.2}
VisionFoundry synthetic images + synthetic QA\small \text{VisionFoundry synthetic images + synthetic QA}68.4\small \textbf{68.4}73.9\small \textbf{73.9}66.4\small \textbf{66.4}49.1\small \textbf{49.1}48.0\small \textbf{48.0}

These results show that, for the same task and dataset size, VisionFoundry's complete synthesis process achieves higher scores. However, the specific images and QA pairs differ between these two groups.

The second compares natural images and synthetic reconstructions under identical QA supervision. To further examine the role of image source, Figure 11 uses natural images to generate captions and targeted QA, and then reconstructs synthetic images from the same captions. The two groups use exactly the same QA pairs, paired with natural images and synthetic reconstructions, respectively, for training.

Training data\small \text{Training data}\small \text{CV-Bench-2D (\\%)}\small \text{CV-Bench-3D (\\%)}\small \text{RealWorldQA (\\%)}\small \text{BLINK (\\%)}\small \text{ScreenSpot-Pro (\\%)}
Baseline\small \text{Baseline}67.3\small \text{67.3}66.0\small \text{66.0}65.0\small \text{65.0}48.1\small \text{48.1}19.4\small \text{19.4}
Natural images + identical QA\small \text{Natural images + identical QA}67.6\small \text{67.6}68.7\small \text{68.7}65.5\small \text{65.5}47.2\small \text{47.2}20.2\small \text{20.2}
Synthetic reconstructions + identical QA\small \text{Synthetic reconstructions + identical QA}67.9\small \textbf{67.9}71.3\small \textbf{71.3}66.3\small \textbf{66.3}49.3\small \textbf{49.3}20.9\small \textbf{20.9}

With QA held fixed, the synthetic reconstruction group still achieves higher scores on the reported metrics, with CV-Bench-3D increasing from 68.7 to 71.3. This indicates that, beyond task-targeted QA construction, the image synthesis process may also provide additional training value in this setting.

1. How does 10K training compare with repeated 1K training at the same number of updates, with checkpoints selected on a separate validation set?

We have added the following experiment: we compare 10 passes over a randomly sampled set of 1K examples with one pass over the full 10K examples.

We split CV-Bench-3D at the image level into a validation set of 300 and a test set of 900, use the validation set to select checkpoints. All results use the same scoring protocol.

Training data\small \text{Training data}Checkpoint\small \text{Checkpoint}\small \text{MMVP single (\\%)}\small \text{MMVP pair (\\%)}\small \text{CV-Bench-2D (\\%)}\small \begin{array}{l}\text{CV-Bench-3D (test}\\\\ \text{split, \\%)}\end{array}\small \text{RealWorldQA (\\%)}
Randomly sampled 1K × 10\small \text{Randomly sampled 1K }\times \text{ 10}Best on validation set\small \text{Best on validation set}67.3\small \text{67.3}38.7\small \text{38.7}71.9\small \text{71.9}77.0\small \text{77.0}65.2\small \text{65.2}
Randomly sampled 1K × 10\small \text{Randomly sampled 1K }\times \text{ 10}Final checkpoint\small \text{Final checkpoint}67.3\small \text{67.3}38.7\small \text{38.7}71.2\small \text{71.2}77.7\small \textbf{77.7}64.4\small \text{64.4}
10K\small \textbf{10K}Final checkpoint (also best onvalidation set)\small \begin{array}{l}\text{Final checkpoint (also best on}\\\\ \text{validation set)}\end{array}68.3\small \textbf{68.3}42.0\small \textbf{42.0}72.4\small \textbf{72.4}77.7\small \textbf{77.7}66.9\small \textbf{66.9}

At the same number of updates, the 10K dataset outperforms repeated training on 1K examples on most visual perception benchmarks. This shows that data diversity can affect VisionFoundry's performance: under the same budget, more diverse data yield better results.

2. Which tasks are represented in the 140-candidate audit, and what is the accuracy of the full self-verifier dataset under independent human review?

The audit of 140 candidate samples covers all 10 tasks in VisionFoundry, with 14 candidate samples per task, as detailed below:

Task\small \text{Task}Number of audited candidates\small \text{Number of audited candidates}
Orientation and Direction\small \text{Orientation and Direction}14\small \text{14}
Viewpoint and Perspective\small \text{Viewpoint and Perspective}14\small \text{14}
Positional and Relational Context\small \text{Positional and Relational Context}14\small \text{14}
Spatial Relationship\small \text{Spatial Relationship}14\small \text{14}
State and Condition\small \text{State and Condition}14\small \text{14}
Structural and Physical Characteristics\small \text{Structural and Physical Characteristics}14\small \text{14}
Color and Appearance\small \text{Color and Appearance}14\small \text{14}
Depth Order\small \text{Depth Order}14\small \text{14}
Relative Distance\small \text{Relative Distance}14\small \text{14}
Real-World Spatial Understanding\small \text{Real-World Spatial Understanding}14\small \text{14}
Total\small \textbf{Total}140\small \textbf{140}

In addition, we conducted an independent human review of the self-verifier dataset, selecting 200 training examples to check the alignment between the images and QA pairs. Of these, 186 examples were correctly aligned, yielding an image–text alignment accuracy of 93.0%, consistent with our existing verification expectations.

3. In the shared-QA image-source experiment, are both branches checked for answer correctness and retained using identical rules? How often does a natural caption or synthetic reconstruction fail to preserve the QA fact?

The two branches use the same questions and answers, and both branches are checked for answer correctness and retained using identical rules. As requested, we checked answer correctness for the natural images and synthetic reconstructions separately, on an example-by-example basis. We randomly sampled 100 pairs from the paired data and manually examined whether the natural images, image captions, and synthetic reconstructions preserved the visual facts needed to answer the shared questions. The results are as follows:

Check\small \text{Check}Number checked\small \text{Number checked}Correct\small \text{Correct}Percentage\small \text{Percentage}
The natural image supports the answer to the shared QA\small \text{The natural image supports the answer to the shared QA}100\small \text{100}98\small \text{98}\small \text{98.0\\%}
The caption preserves the visual facts needed to answer the question\small \text{The caption preserves the visual facts needed to answer the question}100\small \text{100}98\small \text{98}\small \text{98.0\\%}
The synthetic reconstruction supports the answer to the shared QA\small \text{The synthetic reconstruction supports the answer to the shared QA}100\small \text{100}97\small \text{97}\small \text{97.0\\%}
Both images support the same answer\small \text{Both images support the same answer}100\small \text{100}97\small \text{97}\small \text{97.0\\%}

This provides further support for the reliability of the shared-QA comparison in Figure 11.

Reviewer_UTeCOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

VisionFoundry generates synthetic VQA data from task names using an LLM, a text-to-image model, and a VLM verifier. Training on its 10K samples improves perception across three backbones under both SFT and GRPO, while general benchmark scores mostly stay unchanged.

Critical strengths
  1. Matched-QA comparisons between synthetic and natural images, along with self-verification experiments, help address whether the gains simply come from Gemini distillation.

  2. Perception gains are consistent across three backbones and four-run averages, with improvements under both SFT and RL. Performance also scales with data size.

  3. The verifier audit covers all 140 samples in the generation trajectory, including rejected ones. It reports 99% precision, 9% false rejects, and a false-positive example.

Critical weaknesses
  1. Some baselines look unreliable: MiMo’s MMBench jumps from 50.5 to 81.6 and MathVista from 39.1 to 52.4, while Llama scores 0.0 on ScreenSpot-Pro. These suggest baseline formatting or inference-mode issues, weakening the general-capability claim and confidence in MiMo’s perception gains.

  2. Most appendix ablations use only one task, 1K samples, and Qwen-3B, without variance estimates. Several differences are below one point and may reflect run-to-run noise. The pipeline also follows a familiar LLM → T2I → VLM-filter design, so the contribution is mainly empirical.

Questions for authors

See the weaknesses section.

3 comments

1. Some baselines look unreliable: MiMo’s MMBench jumps from 50.5 to 81.6 and MathVista from 39.1 to 52.4, while Llama scores 0.0 on ScreenSpot-Pro. These suggest baseline formatting or inference-mode issues, weakening the general-capability claim and confidence in MiMo’s perception gains.

Thank you for pointing this out! We checked MiMo’s scores on MMBench, as well as all other benchmarks, and found that, under our matched comparison conditions, the MiMo model itself exhibits certain biases, such as a particularly strong tendency to choose option C in multiple-choice questions. This is a limitation of the baseline model itself, rather than an issue with our evaluation suite. Below are several concrete examples; only the text is shown here.

ID\small \text{ID}Question\small \text{Question}Ground-truth answer\small \text{Ground-truth answer}Baseline\small \text{Baseline}SFT\small \text{SFT}
1500\small \text{1500}How many bananas are there in the image?\small \text{How many bananas are there in the image?}B. 2\small \text{B. 2}C. 4\small \text{C. 4}B. 2\small \text{B. 2}
1959\small \text{1959}What’s the relationship between the two creatures?\small \text{What's the relationship between the two creatures?}D. Symbiotic relationship\small \text{D. Symbiotic relationship}C. Parasitic relationships\small \text{C. Parasitic relationships}D. Symbiotic relationship\small \text{D. Symbiotic relationship}
1075\small \text{1075}Which season was the picture taken in?\small \text{Which season was the picture taken in?}D. winter\small \text{D. winter}C. fall\small \text{C. fall}D. winter\small \text{D. winter}
1097\small \text{1097}Are the two pens the same size in the picture?\small \text{Are the two pens the same size in the picture?}B. Not the same\small \text{B. Not the same}A. same\small \text{A. same}B. Not the same\small \text{B. Not the same}
904\small \text{904}Which object relationship is correct?\small \text{Which object relationship is correct?}B. The cat is inside the vase.\small \text{B. The cat is inside the vase.}D. The cat is in front of the vase.\small \text{D. The cat is in front of the vase.}B. The cat is inside the vase.\small \text{B. The cat is inside the vase.}

Regarding the increase on MathVistaMini from 39.1 to 52.4, we checked the results and found an issue in the MathVista score extraction code in the version of VLMEvalKit we used, which affected only the MiMo evaluation. The corrected VisionFoundry results are an increase from a baseline score of 62.0 to 70.0. We have corrected this in the revised manuscript. Thank you for bringing this to our attention.

We also checked Llama’s score of 0.0 on ScreenSpot-Pro. The actual score was 0.01%, but it was reported as 0.0 after rounding. This is related to the instruction-following capability of the Llama baseline itself, rather than an issue with the evaluation suite.

These are our clarifications regarding the results in question. We hope they address your concerns!

2.1 Most appendix ablations use only one task, 1K samples, and Qwen-3B, without variance estimates. Several differences are below one point and may reflect run-to-run noise.

Thank you for your suggestion. We have added standard deviations across four random seeds for the experiments in Table 1. The table below reports the mean ± standard deviation of Qwen, MiMo, and Llama on the benchmarks. For the remaining experiments, including the appendix ablations, all standard deviations are below 0.6 percentage points. We omit the detailed statistics for brevity.

Model\small \text{Model}\small \text{MMVP single (\\%)}\small \text{MMVP pair (\\%)}\small \text{CV-Bench-2D (\\%)}\small \text{CV-Bench-3D (\\%)}\small \text{RealWorldQA (\\%)}
Qwen2.5-VL-3B\small \text{Qwen2.5-VL-3B}68.3 ± 0.6\small \text{68.3 }\pm \text{ 0.6}42.0 ± 0.6\small \text{42.0 }\pm \text{ 0.6}72.4 ± 0.2\small \text{72.4 }\pm \text{ 0.2}76.5 ± 0.6\small \text{76.5 }\pm \text{ 0.6}66.9 ± 0.4\small \text{66.9 }\pm \text{ 0.4}
MiMo-VL-7B\small \text{MiMo-VL-7B}77.7 ± 0.4\small \text{77.7 }\pm \text{ 0.4}57.3 ± 0.6\small \text{57.3 }\pm \text{ 0.6}79.0 ± 0.5\small \text{79.0 }\pm \text{ 0.5}83.7 ± 0.5\small \text{83.7 }\pm \text{ 0.5}67.5 ± 0.4\small \text{67.5 }\pm \text{ 0.4}
Llama-3.2-11B-Vision\small \text{Llama-3.2-11B-Vision}71.7 ± 0.4\small \text{71.7 }\pm \text{ 0.4}46.7 ± 0.5\small \text{46.7 }\pm \text{ 0.5}71.7 ± 0.1\small \text{71.7 }\pm \text{ 0.1}75.3 ± 0.2\small \text{75.3 }\pm \text{ 0.2}64.6 ± 0.3\small \text{64.6 }\pm \text{ 0.3}

As shown, the standard deviations of these visual perception metrics across the four training runs are small, and the results are relatively stable across training seeds.

2.2 The pipeline also follows a familiar LLM → T2I → VLM-filter design, so the contribution is mainly empirical.

VisionFoundry contributes a task-driven approach to constructing complete visual supervision. Starting from target perceptual tasks and structured configurations, it jointly generates QA and answer-conditioned image prompts, synthesizes images, and verifies that they support the answers. No reference images or existing QA are required. We also study the resulting data in both SFT and RL. The table below summarizes the differences from prior work.

Method\small \text{Method}Starting point for dataconstruction\small \begin{array}{l}\text{Starting point for data}\\\\ \text{construction}\end{array}Image source\small \text{Image source}Supervision construction andquality control\small \begin{array}{l}\text{Supervision construction and}\\\\ \text{quality control}\end{array}Main focus\small \text{Main focus}
SVIT\small \textbf{SVIT}Existing images and annotationssuch as descriptions and boundingboxes\small \begin{array}{l}\text{Existing images and annotations}\\\\ \text{such as descriptions and bounding}\\\\ \text{boxes}\end{array}Reuses existing images\small \text{Reuses existing images}Generates multiple types of visualinstructions from imageannotations\small \begin{array}{l}\text{Generates multiple types of visual}\\\\ \text{instructions from image}\\\\ \text{annotations}\end{array}Scaling up visual instruction data\small \text{Scaling up visual instruction data}
MMEvol\small \textbf{MMEvol}Existing images and seedinstructions\small \begin{array}{l}\text{Existing images and seed}\\\\ \text{instructions}\end{array}Reuses existing images\small \text{Reuses existing images}Evolves and filters perceptual,reasoning, and interactioninstructions\small \begin{array}{l}\text{Evolves and filters perceptual,}\\\\ \text{reasoning, and interaction}\\\\ \text{instructions}\end{array}Increasing instruction complexityand diversity\small \begin{array}{l}\text{Increasing instruction complexity}\\\\ \text{and diversity}\end{array}
ProVision\small \textbf{ProVision}Existing images and scene graphs\small \text{Existing images and scene graphs}Reuses existing images\small \text{Reuses existing images}Programmatically generates QA aboutobjects, attributes, relations,depth, etc., from scene graphs\small \begin{array}{l}\text{Programmatically generates QA about}\\\\ \text{objects, attributes, relations,}\\\\ \text{depth, etc., from scene graphs}\end{array}Constructing visual perceptionsupervision\small \begin{array}{l}\text{Constructing visual perception}\\\\ \text{supervision}\end{array}
SpatialVLM\small \textbf{SpatialVLM}Existing natural images\small \text{Existing natural images}Reuses existing images\small \text{Reuses existing images}Uses detection, segmentation, anddepth estimation to constructspatial QA and filters ambiguoussamples\small \begin{array}{l}\text{Uses detection, segmentation, and}\\\\ \text{depth estimation to construct}\\\\ \text{spatial QA and filters ambiguous}\\\\ \text{samples}\end{array}3D spatial and metric geometricreasoning\small \begin{array}{l}\text{3D spatial and metric geometric}\\\\ \text{reasoning}\end{array}
Img-Diff\small \textbf{Img-Diff}Existing image descriptions andtheir rewritten versions\small \begin{array}{l}\text{Existing image descriptions and}\\\\ \text{their rewritten versions}\end{array}Generates image pairsfrom descriptions\small \begin{array}{l}\text{Generates image pairs}\\\\ \text{from descriptions}\end{array}Filters using image similarity,image–text matching, anddifference localization\small \begin{array}{l}\text{Filters using image similarity,}\\\\ \text{image–text matching, and}\\\\ \text{difference localization}\end{array}Learning fine-grained differencesfrom image pairs\small \begin{array}{l}\text{Learning fine-grained differences}\\\\ \text{from image pairs}\end{array}
VisionFoundry\small \textbf{VisionFoundry}Target perceptual tasks andconfigurations\small \begin{array}{l}\textbf{Target perceptual tasks and}\\\\ \textbf{configurations}\end{array}Synthesizes target sceneswith T2I\small \begin{array}{l}\textbf{Synthesizes target scenes}\\\\ \textbf{with T2I}\end{array}Jointly generates QA andanswer-conditioned image prompts,then verifies image–answerconsistency\small \begin{array}{l}\textbf{Jointly generates QA and}\\\\ \textbf{answer-conditioned image prompts,}\\\\ \textbf{then verifies image–answer}\\\\ \textbf{consistency}\end{array}Constructing complete perceptualVQA data from task objectives\small \begin{array}{l}\textbf{Constructing complete perceptual}\\\\ \textbf{VQA data from task objectives}\end{array}

We additionally compare VisionFoundry with ProVision using 10K samples and the same training configuration. VisionFoundry achieves higher scores on all five reported metrics:

Training data\small \text{Training data}\small \text{MMVP single (\\%)}\small \text{MMVP pair (\\%)}\small \text{CV-Bench-2D (\\%)}\small \text{CV-Bench-3D (\\%)}\small \text{RealWorldQA (\\%)}
ProVision 10K\small \text{ProVision 10K}67.3\small \text{67.3}39.3\small \text{39.3}71.1\small \text{71.1}74.7\small \text{74.7}65.5\small \text{65.5}
VisionFoundry 10K\small \text{VisionFoundry 10K}68.3\small \textbf{68.3}42.0\small \textbf{42.0}72.4\small \textbf{72.4}76.5\small \textbf{76.5}66.9\small \textbf{66.9}

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.