- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper study self-evolving vision-language models, which generate questions from unlabeled images and train on them. The authors argue that prior methods label these questions with unreliable proxies such as majority voting or a model judge (a human evaluation in the paper finds many of such labels to be wrong). The proposed method, VQS, use more reliable programming system to compute answers. The model parses each image into a structured record (scene graph, chart table, diagram graph, or infographic entries), and hand-written templates generate questions and compute their answers from this record. Questions are filtered by claim-level self-verification, a blind check, and a difficulty band, and the solver is trained on the remaining questions with GRPO. The parser is also improved without labels, by selecting its best self-verified parse as an SFT target. On Qwen3-VL with different scales, VQS improves the average over ten benchmarks by up to 3.18 points and outperforms the strongest prior self-evolving method at each scale. Human raters find 94% of VQS answers correct, versus 76% for majority voting.
Critical strengths
-
The problem is well-motivated. The paper clearly points out the main bottleneck in current self-evolving VLMs (using human evaluation to show the insufficient label accuracy of majority-vote labels & model-judge labels) and propose a direct fix. Their fix follows naturally from this diagnosis: compute the answer with a program, and only ask the model to verify small facts, which can be more robust in creating useful answers. Then they further verify with human judgers.
-
Their results show consistent gains across scales and benchmarks. VQS improves every one of the ten benchmarks at three Qwen3-VL scales, and its average gain is larger than that of the strongest prior method at each scale.
Critical weaknesses
-
Insufficient isolation of the benefit of program-verified supervision: A central motivation of VQS is that previous methods like majority voting and model-based judging produce unreliable pseudo-labels, whereas program-verified answers provide more reliable supervision for self-evolution. While Table 2 shows improved pseudo-label accuracy, the downstream comparisons in Table 1 simultaneously change question generation, data selection, and reward construction. Therefore, it remains unclear how much of the improvement can be attributed specifically to the more reliable answer supervision. A more direct comparison would train models on the "same generated image-question pairs", using majority-vote, model-judge, and program-computed labels, respectively, while keeping the training data size and GRPO settings fixed. This would help determine whether the improved label reliability itself drives the performance gains, beyond the benefits of programmatic question generation.
-
Lack of a controlled comparison for iterative self-evolution: The paper reports continued improvements over three self-evolution cycles, but each cycle also adds 96 GRPO training steps. It is unclear whether the gains come from regenerating QA data or simply from longer training. A compute-matched baseline that continues training on the first-round QA data, without regenerating new questions, would help isolate the benefit of iterative self-evolution.
Questions for authors
-
The proposed verification checks atomic facts extracted from the structured parse, but how does it handle missing information? For example, an incomplete parse could lead to incorrect answers for counting, max or min, or object-absence questions, even if the extracted facts are correct. Are there explicit checks to ensure completeness for such questions? Or this is simply not a common mistake?
-
While the evaluation is quite completed, I would like to ask, why do the authors use a different benchmark set from VISE, excluding VQAv2, AI2D, ChartQA, CaptionQA, and RealWorldQA while including LogicVista, MMStar, and SEEDBench? In particular, ChartQA and AI2D seem quite relevant to the proposed method. It would be nice to see these additional results.
-
Could the authors provide more details about the training image sources? Section 4 mentions using images from VISE, which reports training on 4,000 COCO images, whereas VQS mentions using 12,000 images for parser training. It would be helpful to clarify this difference and whether the two methods use the same image pool.
-
Since the generated training questions are constrained by predefined schemas and certain template families, could the authors discuss the potential limitations of this design? In particular, does the observed improvement extend to questions or reasoning skills that are not directly covered by these templates, or are the gains primarily concentrated on similar question types?
3 comments
Q3: Training Images
VISE trains on 4,000 COCO photos. VQS uses the same 4,000 photos plus 8,000 chart, diagram and infographic images, because our parser needs these other domains. The 8,000 images come from the ChartQA training split, from AI2D outside its test split and from the real infographics of InfoDet outside its test list, with all annotations removed. We corrected Section 4, which said only that the images come from VISE, to name every source and split. We also checked by image hashing that none of the 8,000 images appear in the ChartQA, AI2D or InfoVQA test splits.
To isolate the extra images, we also trained VQS on the 4K COCO photos alone and VISE on our 12K images with its released code. The table gives the 2B gain over Base, averaged over the ten benchmarks.
| Training images | VISE | VQS |
|---|---|---|
| 4K COCO photos | +1.01 | +2.61 |
| 12K images (4K COCO + 8K charts, diagrams, infographics) | +1.12 | +3.18 |
VQS trained on the 4K photos alone (+2.61) still beats VISE trained on all 12K images (+1.12), so the extra images do not explain the gap. We added this table to Appendix A.1.
Q4: Template Limitations
The templates limit what VQS can ask. They cover four visual domains and a fixed set of operations such as counting, comparing, ranking, arithmetic and following relations, so they cannot ask questions that need outside knowledge, commonsense or temporal reasoning. We keep them fixed because every answer key must come from a program we trust. When the model writes its own questions and programs instead, the solver drops from 62.4 to 61.4 (Table 6), so evolving the templates first needs a way to check new programs without labels. We leave this to future work and state it in the limitations paragraph.
Still, the gains reach beyond the templates. We hand-tagged 1,000 random questions per benchmark (all questions for MMMU and LogicVista) as expressible or not by our templates and measured the 2B gain on each part.
| Benchmark | Expressible, n (%) | Rest, n | 2B gain (expressible) | 2B gain (rest) |
|---|---|---|---|---|
| GQA | 410 (41%) | 590 | +1.95 | +0.85 |
| InfoVQA | 270 (27%) | 730 | +4.07 | +2.47 |
| ScienceQA | 90 (9%) | 910 | +8.89 | +7.36 |
| MMMU | 54 (6%) | 846 | +9.26 | +7.80 |
| All ten | 1,683 (18%) | 7,665 | +3.42 | +3.10 |
VQS gains +3.10 on questions the templates cannot express, close to +3.42 on those they can. On MMMU and ScienceQA, where only 6% and 9% of questions are expressible, it still gains +7.80 and +7.36 on the rest. Many of these questions fall outside the templates only because they also need subject knowledge, yet answering them still requires reading a chart, table or figure and combining several values, which the multi-step templates train. This is not a format effect, because the same training questions with majority-vote labels gain only 0.72 points, against 3.18 with VQS labels (W1).
Dear Reviewer hEMf,
Thank you again for your detailed review and your positive comments about the paper's motivation and gains. In our response, we
- labeled the same training questions in three ways and found that most of the gain comes from more accurate labels,
- showed that new questions in each cycle help, while training longer on the old ones does not,
- added the five other VISE benchmarks and listed every training image source.
As the discussion period is closing, we would appreciate knowing whether this resolves your concerns. We're glad to clarify anything else. If you're satisfied with our responses, we would be grateful if you’d consider updating your score.
Thank you again for all your time and support!
We thank the reviewer for the detailed and constructive feedback. We answer each point below. Changes in the revised PDF are marked in blue.
W1: Answer Verification
When the number of training questions is held fixed, VQS labels give 62.40, against 60.31 with model-judge labels and 59.94 with majority-vote labels. VQS gains 3.18 points over the base model (59.22), but the same questions with majority-vote or model-judge labels gain only 0.72 or 1.09 points. Hence, 66% to 77% of the gains come from the more accurate labels rather than from the questions.
For this experiment, we took the 8,000 training questions of the main VQS run and labeled each question in three ways, by majority vote over five solver answers, by the model judge, and by our templates and per-claim fact checks. Because Table 2 measured each method only on the questions it kept, we also checked the labels on 300 of these shared questions. The table below reports their accuracy next to the downstream results.
Solver accuracy rises with label accuracy, so the gain does not come only from how we generate questions. We added this experiment to the appendix.
W2: Compute-Matched Cycles
We find that training longer on the old questions adds +0.03 points after 96 more steps and −0.60 after 192, whereas new questions add 0.23 and then 0.43 points per cycle (Figure 8). The drop after 192 steps comes from LogicVista, which collapses when training continues on the same questions. For this experiment, we started from the cycle-1 model and trained two more rounds of 96 GRPO steps on the cycle-1 questions only. Each round used the same reweighting of questions, resampled 8K training items and merged the LoRA weights from the previous round, exactly as in the actual cycles. The table below compares the two settings.
The gains in Figure 8 therefore come from regenerating the questions rather than from longer training. We added this run to the appendix and refer to it in Section 5.
Q1: Incomplete Parses
For counting and absence questions, the checker is asked about the answer itself. For “How many mugs are shown?” it checks “Is there only 1 mug shown?” (Figure 2), so a missed second mug makes it answer no and the question is rejected. Absence questions are checked the same way, by asking whether the object is present (Appendix A.5.1). We do the same for max and min questions, where we ask the claim “Is X the largest (smallest) value shown?”.
Q2: Benchmark Choice
We chose the ten benchmarks to cover the three skill groups of Appendix A.2 (perception, knowledge and reasoning, and general). Seven are shared with VISE, and we added LogicVista, MMStar and SEEDBench because they test multi-step reasoning and image grounding, where we expected computed answers to help most. VISE falls below Base only on these three, which we evaluated ourselves, but every other baseline also loses points on at least two of them at every scale, while VQS gains on all three.
Following your suggestion, we evaluated all five VISE benchmarks at 2B, taking Base and VISE from the VISE paper and running VQS under our protocol. Each gain is over Base.
VQS improves over Base on all five, most on AI2D and RealWorldQA. Against VISE, it is higher on AI2D, ChartQA and RealWorldQA, ties on VQAv2 and is lower on CaptionQA, so the five-benchmark averages are close (75.88 against 75.76). Unlike VISE, it trains on ChartQA and AI2D training images, but never on their questions or labels, and none of its training images appears in their test splits (Q3 below). We added these results to the appendix.