- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents VisionFoundary, a three-stage synthetic data generation pipeline for VLM training. Based on this pipeline, the authors construct VisionFoundary-10K and conduct experiments to validate its effectiveness. The results show performance improvements across several benchmarks, particularly on perception-centric tasks.
Critical strengths
-
The paper is well-written and clearly presents the proposed data generation pipeline with sufficient implementation details.
-
The experiments are reasonably comprehensive, demonstrating the effectiveness of the proposed dataset and pipeline while investigating the impact of synthetic data on model performance.
Critical weaknesses
-
The novelty of this work is somewhat limited. Synthetic data generation for VLM training has been widely explored, but the paper does not clearly distinguish its contributions from prior methods or provide sufficient comparisons to demonstrate its advantages.
-
The proposed pipeline has several limitations. First, the generated images may inherit the distributional biases of the underlying image generation models, potentially limiting data diversity and introducing gaps in resolution, realism, and complexity compared to real-world images. However, the paper lacks an analysis of these distributional characteristics and their impact on model performance. Second, the pipeline appears primarily applicable to spatial understanding and perception-centric tasks, while its applicability to other tasks, such as Math, STEM, OCR, and chart understanding, remains unclear.
-
The experiments are mainly conducted on Qwen2.5-VL, which limits the generalizability of the conclusions. The observed gains may partly come from addressing the model's existing weaknesses in visual perception. Additional evaluations on stronger and more recent VLMs, such as Qwen3-VL and Qwen3.5, would provide more convincing evidence of the pipeline's effectiveness.
Questions for authors
-
What are the key novel contributions of VisionFoundary compared to existing synthetic data generation methods?
-
Have the authors analyzed the distributional characteristics of the generated images, such as diversity, resolution, and realism? How do these characteristics affect downstream model performance?
-
Does VisionFoundary-10K provide similar improvements on stronger VLMs, such as Qwen3-VL and Qwen3.5?
8 comments
2. The proposed pipeline has several limitations. First, the generated images may inherit the distributional biases of the underlying image generation models, potentially limiting data diversity and introducing gaps in resolution, realism, and complexity compared to real-world images. However, the paper lacks an analysis of these distributional characteristics and their impact on model performance. Second, the pipeline appears primarily applicable to spatial understanding and perception-centric tasks, while its applicability to other tasks, such as Math, STEM, OCR, and chart understanding, remains unclear.
We have added controlled experiments on data diversity, resolution, and realism.
Data diversity. We compare training on a randomly sampled set of 1K examples ten times with training on the full set of 10K examples once.
| \small \text{MMVP single (\\%)} | \small \text{MMVP pair (\\%)} | \small \text{CV-Bench-2D (\\%)} | \small \text{CV-Bench-3D (900-example test split, \\%)} | \small \text{RealWorldQA (\\%)} | |
|---|---|---|---|---|---|
Using more distinct examples achieves higher scores on MMVP, CV-Bench-2D, and RealWorldQA, and matches the score on the CV-Bench-3D test set, indicating that increasing the diversity of the training data helps improve visual perception capabilities.
Resolution. We use a 1K dataset and downscale the images so that their longest side is 512 pixels, while keeping the QA and training settings unchanged.
| \small \text{MMVP single (\\%)} | \small \text{MMVP pair (\\%)} | \small \text{CV-Bench-2D (\\%)} | \small \text{CV-Bench-3D (\\%)} | \small \text{RealWorldQA (\\%)} | |
|---|---|---|---|---|---|
In this setting, training with lower-resolution images leads to a small decrease, but generated images can still provide effective visual perception supervision at a slightly reduced resolution.
Realism. With the same QA, we compare default settings with images regenerated after adding a photorealistic style requirement. Each group contains 1K paired samples.
| \small \text{MMVP single (\\%)} | \small \text{MMVP pair (\\%)} | \small \text{CV-Bench-2D (\\%)} | \small \text{CV-Bench-3D (\\%)} | \small \text{RealWorldQA (\\%)} | |
|---|---|---|---|---|---|
Images generated in a photorealistic style don't not bring consistent gains on all metrics. This indicates that, in our experiments, a stronger photorealistic style does not necessarily lead to better visual perception performance, which may indirectly reflect the role of data diversity.
Regarding applicability to other tasks, the goal of VisionFoundry is to construct supervision for visual perception, focusing on capabilities such as spatial relations, orientation, viewpoint, and object attributes. We have not claimed that the current experiments have established the effectiveness of this pipeline for constructing data for Math, STEM, OCR, or chart tasks. Some samples in these tasks require the precise rendering of text, numbers, formulas, or structured charts. For such content, programmatic rendering can provide more direct control over the content and can also complement T2I generation. Might directly using T2I actually be putting the cart before the horse? Combining task configurations and verification mechanisms with generation methods suited to these domains is a direction worth exploring and we leave it to future work, as already noted in the Limitations paragraph of the paper. The improvements achieved by VisionFoundry on benchmarks in these domains in the paper primarily reflect improvements in core visual perception driven by the generalization ability of our method.
3. The experiments are mainly conducted on Qwen2.5-VL, which limits the generalizability of the conclusions. The observed gains may partly come from addressing the model's existing weaknesses in visual perception. Additional evaluations on stronger and more recent VLMs, such as Qwen3-VL and Qwen3.5, would provide more convincing evidence of the pipeline's effectiveness.
Thank you for your suggestion. Our main experiments have already evaluated VisionFoundry on three model families: Qwen2.5-VL, MiMo-VL, and Llama-Vision. To further examine its effectiveness on newer models, we have added before-and-after training comparisons for Qwen3-VL-2B and Qwen3.5-2B.
| \small \text{MMVP single (\\%)} | \small \text{MMVP pair (\\%)} | \small \text{CV-Bench-2D (\\%)} | \small \text{CV-Bench-3D (\\%)} | \small \text{RealWorldQA (\\%)} | |
|---|---|---|---|---|---|
1. The novelty of this work is somewhat limited. Synthetic data generation for VLM training has been widely explored, but the paper does not clearly distinguish its contributions from prior methods or provide sufficient comparisons to demonstrate its advantages.
Thank you for your comment. VisionFoundry contributes a task-driven approach to constructing complete visual supervision. Starting from target perceptual tasks and structured configurations, it jointly generates QA and answer-conditioned image prompts, synthesizes images, and verifies that they support the answers. No reference images or existing QA are required. We also study the resulting data in both SFT and RL. The table below summarizes the differences from prior work.
We additionally compare VisionFoundry with ProVision, a closely related method for constructing visual perception supervision, using 10K samples and the same training configuration. VisionFoundry achieves higher scores on all five reported metrics: