acceptodds
Under review as a conference paper at ICLR 2027

Program-Verified Self-Evolution for Vision-Language Models

23 views

Abstract

Self-evolving vision-language models train on questions they generate from unlabeled images. Since these questions have no gold answers, prior methods label them by majority vote over sampled answers or by a model judge. In a human evaluation, we find that 24% of majority-vote labels and 18% of model-judge labels produced during self-evolution are wrong. To address this problem, we present Verifiable QA Generation for Self-Evolving Models (VQS), which changes how the model judges answers. Instead of voting on an answer, the model parses each image into a structured record, such as a scene graph, a chart table, or a diagram graph. Fixed programs then write a question from the record and compute its answer. The model still acts as a visual checker, but it only confirms the individual facts the program reads, one short claim at a time. These claim-level checks select the parser's training targets, so the parser also improves without labels. Human raters find 94% of VQS answers correct, against 76% for majority voting. Across ten benchmarks, VQS improves Qwen3-VL by up to 3.18 points at the 2B, 4B, and 8B scales and outperforms the strongest self-evolving baseline at each. Gains keep growing over three training rounds, reaching 3.84 points at 2B. Our code and models will be released.

Primary area:
foundation or frontier models, including LLMs
Keywords:
vision-language models, self-evolving training, reinforcement learning with verifiable rewards, GRPO, reward hacking, label-free training, synthetic question generation, structured scene parsing, multimodal reasoning, difficulty filtering, evaluation protocol

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_hEMfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper study self-evolving vision-language models, which generate questions from unlabeled images and train on them. The authors argue that prior methods label these questions with unreliable proxies such as majority voting or a model judge (a human evaluation in the paper finds many of such labels to be wrong). The proposed method, VQS, use more reliable programming system to compute answers. The model parses each image into a structured record (scene graph, chart table, diagram graph, or infographic entries), and hand-written templates generate questions and compute their answers from this record. Questions are filtered by claim-level self-verification, a blind check, and a difficulty band, and the solver is trained on the remaining questions with GRPO. The parser is also improved without labels, by selecting its best self-verified parse as an SFT target. On Qwen3-VL with different scales, VQS improves the average over ten benchmarks by up to 3.18 points and outperforms the strongest prior self-evolving method at each scale. Human raters find 94% of VQS answers correct, versus 76% for majority voting.

Critical strengths
  1. The problem is well-motivated. The paper clearly points out the main bottleneck in current self-evolving VLMs (using human evaluation to show the insufficient label accuracy of majority-vote labels & model-judge labels) and propose a direct fix. Their fix follows naturally from this diagnosis: compute the answer with a program, and only ask the model to verify small facts, which can be more robust in creating useful answers. Then they further verify with human judgers.

  2. Their results show consistent gains across scales and benchmarks. VQS improves every one of the ten benchmarks at three Qwen3-VL scales, and its average gain is larger than that of the strongest prior method at each scale.

Critical weaknesses
  1. Insufficient isolation of the benefit of program-verified supervision: A central motivation of VQS is that previous methods like majority voting and model-based judging produce unreliable pseudo-labels, whereas program-verified answers provide more reliable supervision for self-evolution. While Table 2 shows improved pseudo-label accuracy, the downstream comparisons in Table 1 simultaneously change question generation, data selection, and reward construction. Therefore, it remains unclear how much of the improvement can be attributed specifically to the more reliable answer supervision. A more direct comparison would train models on the "same generated image-question pairs", using majority-vote, model-judge, and program-computed labels, respectively, while keeping the training data size and GRPO settings fixed. This would help determine whether the improved label reliability itself drives the performance gains, beyond the benefits of programmatic question generation.

  2. Lack of a controlled comparison for iterative self-evolution: The paper reports continued improvements over three self-evolution cycles, but each cycle also adds 96 GRPO training steps. It is unclear whether the gains come from regenerating QA data or simply from longer training. A compute-matched baseline that continues training on the first-round QA data, without regenerating new questions, would help isolate the benefit of iterative self-evolution.

Questions for authors
  1. The proposed verification checks atomic facts extracted from the structured parse, but how does it handle missing information? For example, an incomplete parse could lead to incorrect answers for counting, max or min, or object-absence questions, even if the extracted facts are correct. Are there explicit checks to ensure completeness for such questions? Or this is simply not a common mistake?

  2. While the evaluation is quite completed, I would like to ask, why do the authors use a different benchmark set from VISE, excluding VQAv2, AI2D, ChartQA, CaptionQA, and RealWorldQA while including LogicVista, MMStar, and SEEDBench? In particular, ChartQA and AI2D seem quite relevant to the proposed method. It would be nice to see these additional results.

  3. Could the authors provide more details about the training image sources? Section 4 mentions using images from VISE, which reports training on 4,000 COCO images, whereas VQS mentions using 12,000 images for parser training. It would be helpful to clarify this difference and whether the two methods use the same image pool.

  4. Since the generated training questions are constrained by predefined schemas and certain template families, could the authors discuss the potential limitations of this design? In particular, does the observed improvement extend to questions or reasoning skills that are not directly covered by these templates, or are the gains primarily concentrated on similar question types?

3 comments

We thank the reviewer for the detailed and constructive feedback. We answer each point below. Changes in the revised PDF are marked in blue.

W1: Answer Verification

When the number of training questions is held fixed, VQS labels give 62.40, against 60.31 with model-judge labels and 59.94 with majority-vote labels. VQS gains 3.18 points over the base model (59.22), but the same questions with majority-vote or model-judge labels gain only 0.72 or 1.09 points. Hence, 66% to 77% of the gains come from the more accurate labels rather than from the questions.

For this experiment, we took the 8,000 training questions of the main VQS run and labeled each question in three ways, by majority vote over five solver answers, by the model judge, and by our templates and per-claim fact checks. Because Table 2 measured each method only on the questions it kept, we also checked the labels on 300 of these shared questions. The table below reports their accuracy next to the downstream results.

Label sourceLabel accuracy (shared questions)2B avg.Gain over base
None (base)–59.22–
Majority vote75.859.94+0.72
Model judge81.760.31+1.09
VQS94.062.40+3.18

Solver accuracy rises with label accuracy, so the gain does not come only from how we generate questions. We added this experiment to the appendix.

W2: Compute-Matched Cycles

We find that training longer on the old questions adds +0.03 points after 96 more steps and −0.60 after 192, whereas new questions add 0.23 and then 0.43 points per cycle (Figure 8). The drop after 192 steps comes from LogicVista, which collapses when training continues on the same questions. For this experiment, we started from the cycle-1 model and trained two more rounds of 96 GRPO steps on the cycle-1 questions only. Each round used the same reweighting of questions, resampled 8K training items and merged the LoRA weights from the previous round, exactly as in the actual cycles. The table below compares the two settings.

2B gain over baseAfter cycle 1+96 steps+192 steps
New questions each cycle (Fig. 8)3.183.413.84
Cycle-1 questions only3.183.212.58

The gains in Figure 8 therefore come from regenerating the questions rather than from longer training. We added this run to the appendix and refer to it in Section 5.

Q1: Incomplete Parses

For counting and absence questions, the checker is asked about the answer itself. For “How many mugs are shown?” it checks “Is there only 1 mug shown?” (Figure 2), so a missed second mug makes it answer no and the question is rejected. Absence questions are checked the same way, by asking whether the object is present (Appendix A.5.1). We do the same for max and min questions, where we ask the claim “Is X the largest (smallest) value shown?”.

Q2: Benchmark Choice

We chose the ten benchmarks to cover the three skill groups of Appendix A.2 (perception, knowledge and reasoning, and general). Seven are shared with VISE, and we added LogicVista, MMStar and SEEDBench because they test multi-step reasoning and image grounding, where we expected computed answers to help most. VISE falls below Base only on these three, which we evaluated ourselves, but every other baseline also loses points on at least two of them at every scale, while VQS gains on all three.

Following your suggestion, we evaluated all five VISE benchmarks at 2B, taking Base and VISE from the VISE paper and running VQS under our protocol. Each gain is over Base.

BenchmarkBaseVISEVQS
VQAv278.3778.5478.53 (+0.16)
AI2D73.6776.4276.55 (+2.88)
ChartQA79.1680.0880.36 (+1.20)
CaptionQA77.0479.1677.96 (+0.92)
RealWorldQA63.4164.5866.01 (+2.60)

VQS improves over Base on all five, most on AI2D and RealWorldQA. Against VISE, it is higher on AI2D, ChartQA and RealWorldQA, ties on VQAv2 and is lower on CaptionQA, so the five-benchmark averages are close (75.88 against 75.76). Unlike VISE, it trains on ChartQA and AI2D training images, but never on their questions or labels, and none of its training images appears in their test splits (Q3 below). We added these results to the appendix.

Q3: Training Images

VISE trains on 4,000 COCO photos. VQS uses the same 4,000 photos plus 8,000 chart, diagram and infographic images, because our parser needs these other domains. The 8,000 images come from the ChartQA training split, from AI2D outside its test split and from the real infographics of InfoDet outside its test list, with all annotations removed. We corrected Section 4, which said only that the images come from VISE, to name every source and split. We also checked by image hashing that none of the 8,000 images appear in the ChartQA, AI2D or InfoVQA test splits.

To isolate the extra images, we also trained VQS on the 4K COCO photos alone and VISE on our 12K images with its released code. The table gives the 2B gain over Base, averaged over the ten benchmarks.

Training imagesVISEVQS
4K COCO photos+1.01+2.61
12K images (4K COCO + 8K charts, diagrams, infographics)+1.12+3.18

VQS trained on the 4K photos alone (+2.61) still beats VISE trained on all 12K images (+1.12), so the extra images do not explain the gap. We added this table to Appendix A.1.

Q4: Template Limitations

The templates limit what VQS can ask. They cover four visual domains and a fixed set of operations such as counting, comparing, ranking, arithmetic and following relations, so they cannot ask questions that need outside knowledge, commonsense or temporal reasoning. We keep them fixed because every answer key must come from a program we trust. When the model writes its own questions and programs instead, the solver drops from 62.4 to 61.4 (Table 6), so evolving the templates first needs a way to check new programs without labels. We leave this to future work and state it in the limitations paragraph.

Still, the gains reach beyond the templates. We hand-tagged 1,000 random questions per benchmark (all questions for MMMU and LogicVista) as expressible or not by our templates and measured the 2B gain on each part.

BenchmarkExpressible, n (%)Rest, n2B gain (expressible)2B gain (rest)
GQA410 (41%)590+1.95+0.85
InfoVQA270 (27%)730+4.07+2.47
ScienceQA90 (9%)910+8.89+7.36
MMMU54 (6%)846+9.26+7.80
All ten1,683 (18%)7,665+3.42+3.10

VQS gains +3.10 on questions the templates cannot express, close to +3.42 on those they can. On MMMU and ScienceQA, where only 6% and 9% of questions are expressible, it still gains +7.80 and +7.36 on the rest. Many of these questions fall outside the templates only because they also need subject knowledge, yet answering them still requires reading a chart, table or figure and combining several values, which the multi-step templates train. This is not a format effect, because the same training questions with majority-vote labels gain only 0.72 points, against 3.18 with VQS labels (W1).

Dear Reviewer hEMf,

Thank you again for your detailed review and your positive comments about the paper's motivation and gains. In our response, we

  • labeled the same training questions in three ways and found that most of the gain comes from more accurate labels,
  • showed that new questions in each cycle help, while training longer on the old ones does not,
  • added the five other VISE benchmarks and listed every training image source.

As the discussion period is closing, we would appreciate knowing whether this resolves your concerns. We're glad to clarify anything else. If you're satisfied with our responses, we would be grateful if you’d consider updating your score.

Thank you again for all your time and support!

Reviewer_GxjROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses answer verification in self-evolving vision-language models, where a model writes its own questions from unlabeled images and must therefore also supply its own labels; the authors first show, by human rating, that the usual proxies such as majority voting and model judges are often wrong. They instead compute the answer rather than estimate it: a parser turns each image into a structured record, fixed templates read that record to write questions and compute their reference answers with deterministic programs, and a per-claim check plus two filters decide which pairs train the solver. One model serves as parser, checker and solver and is trained with GRPO, and the method is evaluated on ten benchmarks at three scales and on further model families.

Critical strengths
  • The central idea is genuinely new: instead of estimating an answer from sampled outputs, the paper shows that a large class of questions can have their answer computed from a structured reading of the image.
  • The individual pieces are validated rather than only the final numbers, so a reader can see why the method works and where it can fail.
  • The empirical support is broad and consistent: many benchmarks, several model scales and families, and gains that continue over several rounds.
Critical weaknesses
  • The comparison cannot be checked. Only one sentence in the Reproducibility Statement says where the baseline numbers come from, and the appendix gives settings for this method alone, so it is unclear which rows the authors ran.
  • The task space is too narrow for a framework called self-evolving. Questions can only come from a hand-written template library covering four domains, and the gain still rises with the number of templates, so the loop is not what improves the model.
  • The contribution is overstated. The paper presents per-claim verification as the load-bearing idea, but its own ablation shows a single whole-parse judgement comes close and that training the parser accounts for most of the gain.
Questions for authors
  • Which rows of Table 1 did you run yourselves, and how far is a locally reproduced baseline from the published number you cite?
  • How much of your benchmarks falls inside the set of questions your templates can express?
  • Does the gain come from per-claim verification or from training the parser at all? Would a single judgement over the whole parse recover most of it?
3 comments

We sincerely thank the reviewer for their thoughtful feedback. Changes in the revised PDF are marked in blue.

W1/Q1: Baseline Sources

Of the ten benchmarks in Table 1, seven also appear in the VISE paper (GQA, OK-VQA, InfoVQA, ScienceQA, MMMU, MMBench, EmbSpatial). On these seven, every baseline row, including Base, is copied from the VISE paper. On the other three (LogicVista, MMStar and SEEDBench), we evaluated every row ourselves with the same lmms-eval setup as VQS. For VisPlay, Vision-Zero, EvoLMM and iReasoner, we used the Qwen3-VL checkpoints that the VISE authors trained for their comparison.

To check the copied numbers, we reran Base and VISE (from its released checkpoints) on all ten benchmarks.

ScaleBase (Table 1)Base (our rerun)VISE (Table 1)VISE (our rerun)VQS
2B59.2259.0260.2359.9662.40
4B66.7066.7367.7467.5269.01
8B68.8168.6969.4769.3171.24

The largest difference between a rerun and Table 1 is 0.27 points, far below the margin of VQS over the rerun VISE (1.49 to 2.44 points). If VQS is compared against the rerun Base, it gains +3.38, +2.28, and +2.55 points at 2B, 4B, and 8B, all within 0.20 points of the gains in Table 1.

In the revision, we mark the source of every cell in Table 1 and add the full baseline settings and the reruns above to Appendix A.1.

W2-1: Narrow Task Space

We agree that the template families may limit what VQS can ask to four visual domains (photos, charts, diagrams and infographics). However, these domains still cover the image types of the benchmarks commonly used for self-evolving VLMs, from natural photos in GQA and OK-VQA to charts, diagrams and infographics in InfoVQA, ScienceQA and MMMU. Also, adding a new family takes only about 10 lines of code (Appendix A.5.2), so extending VQS to new domains is cheap. Additionally, we find that the benchmark questions that are not expressible by our templates gain +3.10 from our training, while the expressible questions gain +3.42 (Q2 below). We leave templates that evolve with the model to future work and state this in the limitations.

W2-2: Gains from the Loop

More template families do help (Figure 5c), but the training cycles also improve the model while the families stay fixed. Each cycle retrains the parser and parses the same images again, so the same families produce new and more accurate questions. Cycles 2 and 3 add 0.23 and 0.43 points (Figure 8), while training for the same extra steps on the cycle-1 questions gives +0.03 points after 96 steps and −0.60 after 192 (see our reply to hEMf W2). The gain therefore comes from generating more questions for each cycle.

W3/Q3: Per-Claim vs Whole-Parse Verification

The gain comes from both training the parser and checking the solver's data claim by claim. Fine-tuning the parser on any of its parses, even a random one, lifts the 2B solver by 3.5 points. Choosing that parse by per-claim scores adds 0.9 points over a random parse and 0.4 over one whole-parse judgment. For the solver's data, per-claim checks add 1.4 points over one whole-answer judgment and 1.5 over no fact-check. Hence, per-claim checks are needed in both stages (parser training data selection and solver training data filtering) to achieve our total gains.

To measure the filter's effect, we replaced the claim-level fact-check on the solver's data with a single yes/no judgment of the whole answer, made by the solver itself. Out of 10,500 candidate questions, the per-claim check keeps 8,000 (76%), the whole-answer judgment keeps 9,240 (88%), and no check keeps all of them. Every solver trains for the same 96 GRPO steps on 8,000 questions sampled from its kept set, so the training size is matched. The first four rows vary the parser's target and the last three vary the solver-data filter.

Parser targetSolver-data filterParse prec.Answer acc. (150 samples)2B solver avg.
Untrained parserPer-claim80.886.858.0
Random parsePer-claim82.191.061.5
Whole-parse judgmentPer-claim85.094.062.0
Per-claim (VQS)Per-claim87.696.162.4
Per-claimWhole-answer judgment87.690.761.0
Per-claimNone (Table 4)87.688.760.9

We find that the gap is small for the parser's target, although per-claim selection still gives more precise parses (87.6 against 85.0) and more correct answer keys (96.1 against 94.0). For the solver's data, a whole-answer judgment does not recover the gain and scores close to no fact-checking at all (61.0 against 60.9).

Q2: Template Coverage

Only 18% of benchmark questions can be written by our templates, yet VQS gains about as much on the questions they cannot express (+3.10) as on those they can (+3.42). To measure this, we hand-tagged a random sample of 1,000 questions from each benchmark, or all of its questions when it has fewer than 1,000 (900 MMMU and 448 LogicVista). A question is expressible when one of our templates can write it. The table below gives the 2B gain on each part.

BenchmarkExpressible, n (%)Rest, n2B gain (expressible)2B gain (rest)
GQA410 (41%)590+1.95+0.85
InfoVQA270 (27%)730+4.07+2.47
ScienceQA90 (9%)910+8.89+7.36
MMMU54 (6%)846+9.26+7.80
All ten1,683 (18%)7,665+3.42+3.10

The new Appendix A.6 gives this table.

Dear Reviewer GxjR,

Thank you again for your thoughtful review and your positive comments about the paper's novelty and broad, consistent results. In our response, we

  • reran base and VISE on all ten benchmarks to confirm the copied baselines,
  • showed that the gains across cycles come from new questions, not from longer training,
  • showed that per-claim checks matter in both stages

As the discussion period ends soon, we would love to hear whether this addresses your concerns. We're happy to answer any further questions. If you're satisfied with our responses, we would be grateful if you’d consider updating your score.

Thank you again for all your time and support!

Comments on the paper

2 comments

We sincerely thank the reviewer for the careful and constructive feedback. Changes in the revised PDF are marked in blue.

W1/Q1: Baseline Control and Cost

Verification helps beyond the extra training and curation. With the number of training questions held fixed, changing only the labels gives 62.40 with VQS labels against 60.31 with the model judge and 59.94 with majority vote (see our reply to hEMf W1). Adding more training images doesn’t explain the gap either. When we train VQS on VISE's 4,000 COCO photos alone, VQS still reaches 61.83 (+2.61) against 60.23 (+1.01) for VISE (see our reply to hEMf Q3).

On the seven benchmarks shared with VISE, all baseline rows in Table 1, including Base, are copied from the VISE paper. We ran all the baselines on LogicVista, MMStar and SEEDBench using the same setup as VQS. To check the copied numbers, we reran the Base model and VISE (released checkpoints) on all ten benchmarks under our protocol.

ScaleBase (Table 1)Base (our rerun)VISE (Table 1)VISE (our rerun)VQS
2B59.2259.0260.2359.9662.40
4B66.7066.7367.7467.5269.01
8B68.8168.6969.4769.3171.24

The largest difference from Table 1 is 0.27 points, far below the margin of VQS over the rerun VISE (1.49 to 2.44 points).

Regarding the budget, VQS is the cheapest of the three methods. We timed one full 2B run of each method on 8 MI210 GPUs, using the released code of VISE and VisPlay and counting every VQS step from parser sampling to data curation.

One 2B runVQSVISEVisPlay
Parser sampling and claim checks (GPU-hours)14––
Parser fine-tuning (GPU-hours)3––
Data curation with filters (GPU-hours)9––
RL training (GPU-hours)1513148
Total (GPU-hours)4113148
Parser fine-tuning steps284––
RL training steps964,000120
10-benchmark avg.62.4060.2359.48

VQS reaches the highest accuracy at the lowest cost and with the fewest RL steps. We added this table to Appendix A.1.

Q2: Matched-Retention Comparison

At matched retention, VQS answers are correct 9.1 to 13.6 points more often than those of the model judge and majority vote. A 2B solver trained on the VQS answers scores 1.78 points higher than one trained on the model judge's answers and 2.30 points higher than one trained on majority-vote answers.

For correctness, we used the same 500 candidate questions as Table 2. VQS keeps 380 of them (76%), so the other two methods also keep their 380 most confident answers. We check each kept answer along with the chart values and diagram relations relevant to it. For downstream accuracy, VQS keeps 8,000 of the 10,500 questions of the main run (76%), so the other two methods also keep their 8,000 most confident answers. We trained a 2B solver on each set with the same settings.

MethodCorrect, 380 kept (%)Chart-value errorsDiagram-relation errors2B avg., 8,000 kept
VQS94.43.6%7.9%62.40
Majority vote80.818.7%24.2%60.10
Model judge85.314.1%19.5%60.62

Diagram relations are the hardest case for all three methods. Human checks find errors in 7.9% of the diagram relations behind kept VQS answers. Appendix A.3 reports that at least two other checkers reject 26.7% of diagram arrows, so disagreement between checkers is much higher than the measured error rate. We added this table to the appendix.

Dear Reviewer QuFG,

Thank you again for your careful review and your positive comments about the paper's impact, broad evaluation, and results. In our response, we

  • showed that verification helps beyond extra training,
  • added GPU-hours for our experiments, where VQS is the cheapest method,
  • compared the methods at matched retention.

As the discussion period ends soon, we'd be grateful to hear whether this addresses your concerns. We're happy to answer any further questions. If you're satisfied with our responses, we would be grateful if you’d consider updating your score.

Thank you again for all your time and support!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.