acceptodds
Under review as a conference paper at ICLR 2027

PixelProof: Visual Question Generation through Forward–Inverse Agreement

27 views

Abstract

From pre-training on human-written tokens to reinforcement learning on verifiable rewards (RLVR), obtaining cheaply verifiable data is the fuel for current large language models. RLVR often works in mathematics and similar domains because the labels are verifiable by construction and establish a strong self-supervision loop that bootstraps model capabilities. Constructing this loop outside math domains, especially in vision, is an interesting open challenge. We introduce PixelProof, a method that takes human-input prompts and automatically synthesizes novel visual-question images via Python where labels are verifiable by the generation loop itself (without human supervision). That is, PixelProof parameterizes a set of questions by 3 Python programs: (1) a Forward program that translates scene specifications into an expected answer; (2) a Renderer that renders an image; and (3) an Inverse program that progressively transforms the input image using image processing tools to derive its own verification answer independently. Comparing the answers between the Forward (what is designed) vs. Inverse programs (what is observed) self-supervises the generation. Generation can be steered toward (a) a human-defined topic e.g., music; (b) where in the image the answer's evidence lies; and (c) harder questions. As an initial demonstration of PixelProof, five coding agents working across nine visual reasoning directions produced over 2,400 verified questions. Fine-tuning three open-weight models on the newly generated questions improves their performance on held-out questions by an average of +10.7 points, and by +1.9 points on average across 16 external benchmarks. PixelProof shows a promising loop for automatically synthesizing visual questions for evaluating and training vision-language models.

Primary area:
foundation or frontier models, including LLMs
Keywords:
visual reasoning, benchmark generation, automated evaluation, executable verification

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.60 over 5 reviews, above 70% of the others.
Reviewer_APYAOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces an agent-based framework for generating synthetic puzzle-like visual reasoning questions. Each question world includes a renderer, a forward program that computes the answer from scene parameters, and an inverse program that recovers the answer from pixels. Forward-inverse agreement is used to verify the generated data. The authors generate over 2,400 verified question worlds and demonstrate improvements through VLM SFT.

Critical strengths
  1. Useful verification mechanism. Pixel-based inverse verification addresses potential inconsistencies between generated images and their intended answers. interesting and useful idea.

  2. Extensive generation and evaluation. The framework produces diverse visual reasoning tasks, with substantial experiments on generation, verification, human validation, and eval on model performance.

  3. Automatically constructing new question worlds and their verification programs is potentially a scalable data generation method.

Critical weaknesses
  1. Limited methodological novelty and missing essential baselines. The authors compare model performance before and after fine-tuning on PixelProof, which demonstrates that their data can be useful. but LLM-driven visual data generation and programmatically verified puzzles have been explored in Multimodal Self-Instruct, CoSyn, AlgoPuzzleVQA, and related work. The main novelty is forward-inverse verification, but comparisons against matched-budget forward-only generation and VLM-based verification are missing. Its incremental value is unclear.

  2. Limited generalization beyond synthetic visual reasoning. Fine-tuning improves accuracy by 10.7 on held-out generated worlds, but only +1.9 on average across 16 external benchmarks. Improvements on general visual understanding tasks are modest, while two benchmarks with substantial gains contributed example questions to the generation profiles.

  3. unfair frontier versus open-weight model comparison in section 4.5. The reported performance gap is difficult to interpret because the models are not matched for scale or reasoning configuration.

  4. Insufficient evidence of novel reasoning challenges. Agents generate new questions using predefined profiles, implementation demonstrations, and previously accepted worlds. I prefer to regard this paper as a solution of a specific benchmark: VLMsAreBlind.

Questions for authors

What is the primary contribution of PixelProof? Is the main goal to generate challenging, verifiable visual reasoning data for evaluation, or to produce training data that improves VLM capabilities? Although both are discussed, the experimental emphasis seems to favor the former, while the abstract primarily motivates the work through RLVR and model improvement. Could the authors clarify this discrepancy in the paper's framing? I you emphasize the model improvement, you really need to include other puzzel-like data syn baselines.

4 comments
AuthorsOpenReview ↗

Response to Reviewer APYA (Part 1 of 2)

Thank you for your review.

You raised four concerns: what the primary contribution is, whether the forward–inverse check adds anything over prior generators and over simpler baselines, how far the training gains generalize, whether the frontier and open-weight models are compared fairly, and whether the questions offer new reasoning challenges beyond VLMsAreBlind. We addressed each of them and give the new evidence below.

Our main new result is an ablation that removes or replaces the inverse check. We used three discovery profiles, two hours each, comparing four ways of deciding whether a world is kept. It shows that adding the inverse does not degrade generation: the agents still produce questions with it, at a somewhat lower throughput. In return, every label is checked against the pixels by a fixed program. None of the worlds it kept was defective, it costs nothing per question, and it rarely rejects a correct world, where a VLM judge rejected many.

The primary contribution

Is the main goal to generate challenging, verifiable visual reasoning data for evaluation, or to produce training data that improves VLM capabilities?

The core contribution is the idea that a visual question can be verified the way a program can: by running a second program against it, one that sees only the pixels. In PixelProof, coding agents write reusable question worlds, and in each world an inverse program must recover the answer from the pixels alone, so every label is checked without a model and without a person. The paper shows that this works at scale: current coding agents can write both the questions and the inverse programs that answer them from the pixels. The paper then tests the data in both uses, evaluation and training. RLVR appears in the abstract only as the general motivation for wanting cheaply verifiable data. However, for the rebuttal we also ran a first RL test on one model: GRPO on Qwen2.5-VL-7B, with a match against the recorded answer as the only reward, raised held-out accuracy from 44.5% to 53.4%.

Forward-only generation and VLM-based verification

The main novelty is forward-inverse verification, but comparisons against matched-budget forward-only generation and VLM-based verification are missing.

Thank you for bringing up this point! Based on your feedback, we have rerun the generation process with four different setups:

  1. Inverse program (PixelProof): agent writes an inverse program that can recover the answer from the image alone.
  2. No check: forward-only generation with no check of the answers.
  3. Agent's own tests: agent can run any code and can inspect its own images.
  4. VLM judge: a separate VLM judge verifies the image instead of the inverse program.

In each setup, the generation process has a fixed 2 hours of wall clock time to output as many valid worlds as possible. We repeated this setup across three different generating agents (GPT-5.6-Sol, GPT-5.6-Luna and DeepSeek-V4-Flash). In the VLM judge setup, the judge is a separate instance of the agent model (for DeepSeek-V4-Flash, the judge is GPT-5.6-Sol).

Then to measure the data quality of each setup, we first flagged potentially problematic data by evaluating GPT-6.1-Terra on the questions and then Claude Opus 5.5 on the questions that GPT-6.1-Terra answered incorrectly. If both Terra and Opus answered incorrectly, an author then manually checked those samples to determine if the generated question contained any errors.

Table R1: The inverse program is a cheap and effective way to ensure data quality.

Generation setup# Worlds kept ↑# Worlds with flawed questions ↓Total API cost ↓
Inverse program (PixelProof)1350$53
No check1593$38
Agent’s own tests1394$52
VLM judge1130$69

The worlds kept with the agent's own tests or with no check contained errors: wrong labels, broken renders and ambiguous questions. In contrast, the worlds kept by PixelProof or the VLM judge contained no errors.

The VLM judge had no false positives (flawed worlds accepted), but many false negatives (correct worlds rejected). On the 119 questions where the judge disagreed with the recorded answer, Terra and Opus agreed with the recorded answer on 115 and never with the judge. Additionally, the VLM judge process cost 30% more than PixelProof, and a judge has to be paid again for every new question sampled from a world: checking 100,000 questions costs 70 to 1,250 US dollars with a judge, against about 83 minutes of CPU time and no model call with the inverse program.

In summary, only PixelProof and the VLM judge kept no flawed world. Of the two, PixelProof cost less, kept more worlds, and almost never rejected a correct world, while the judge rejected many. The full study is in Appendix L. The step-by-step animations on the project page show what the inverse programs compute from the pixels: https://projectpixelproof.github.io/animations.html

AuthorsOpenReview ↗

Response to Reviewer APYA (Part 2 of 2)

(continued)

What the inverse adds, and what it does not

Its incremental value is unclear.

The ablation above shows what the inverse adds over the simpler alternatives. Against forward-only generation, none of the worlds it kept was defective. Against a VLM judge, it is a fixed CPU program with negligible cost per question, where a judge costs 70 to 1,250 US dollars per 100,000 questions, and it rarely rejects a correct world, where the judge rejected many. And requiring it costs nothing in training value: we fine-tuned Gemma 4 12B on the data from each of the four conditions (16,082 examples each, one run each), and held-out accuracy rose from 57.8% to 67.9%, 69.1%, 67.2% and 68.2%, about ten points in every case (Appendix K.7).

Multimodal Self-Instruct, CoSyn and AlgoPuzzleVQA

LLM-driven visual data generation and programmatically verified puzzles have been explored in Multimodal Self-Instruct, CoSyn, AlgoPuzzleVQA, and related work.

These methods differ from PixelProof in what the answer is checked against. Our related-work sentence calling them "PixelProof without the inverse program" was imprecise; we have corrected it, and Appendix L.6 of the revised PDF now has a table of what each related method checks. In the question-answer data of Multimodal Self-Instruct and CoSyn, a language model writes each answer from the drawing code, and no program checks it against the image. AlgoPuzzleVQA computes answers with solvers written by people for 18 puzzle types, and nothing checks that the image shows the state the solver used. In PixelProof the agents write the generators themselves, and a program that sees only the pixels must recover every answer.

We also trained on the data the two papers released (the chart and table questions of Multimodal Self-Instruct and all figure types of CoSyn): the same number of examples as ours (27,207), the same recipe, Gemma 4 12B, one run each. On our held-out questions the models reach 59.4% (Multimodal Self-Instruct), 60.8% (CoSyn) and 71.0% (PixelProof), from 57.8% for the base model. Appendix K.8 discusses the comparison further.

On the 15 external benchmarks the averages are 59.4% (base model), 59.3% (PixelProof) and 56.2% (each of the other two). We do not read this as a ranking of the methods.

Generalization beyond the generated questions

Improvements on general visual understanding tasks are modest, while two benchmarks with substantial gains contributed example questions to the generation profiles.

We found that a couple of external benchmarks had been evaluated with an outdated script, and corrected it. The average change is now +1.3 points over 15 external benchmarks. The largest correction is on VLMsAreBlind, one of the two benchmarks that contributed example questions: its gain is +1.1 points and not +14.0. The largest gain is on PuzzleVQA (+6.4), which contributed no examples. So the external gains are positive on average, modest, and concentrated on puzzle-style benchmarks, as expected from the kind of questions we train on. We do not claim gains on general visual understanding. The revised paper says this and now discloses the overlap in example questions in the main text. The forward–inverse construction itself is not limited to 2D puzzles: we show this on photographs and 3D scenes (see below). The table no longer includes RAVEN, because of a documented defect in its answer choices (Hu et al., 2021).

Frontier and open-weight models

The reported performance gap is difficult to interpret because the models are not matched for scale or reasoning configuration.

The two groups are not matched, and the revised Section 4.5 now says that the comparison is descriptive and that the gap may reflect both scale and reasoning setting.

New reasoning challenges, and VLMsAreBlind

I prefer to regard this paper as a solution of a specific benchmark: VLMsAreBlind.

The paper does not target one benchmark. It uses nine profiles to separate questions by the kind of reasoning they require, and the collection covers all nine. Ten of the 45 profile example questions were adapted from existing benchmarks, five from VLMsAreBlind and five from VLMsAreBiased; the other 35 were not. Fine-tuning on the collection moves VLMsAreBlind itself by only +1.1 points.

The worlds in the paper are simple 2D images drawn with Python code, and we chose them to establish the idea first. For the rebuttal we gave the agent two other ways to make images, an image-generation model (GPT Image 2.6 Flare) for photographs and Blender for 3D scenes, and kept the forward–inverse check unchanged. In first runs, 20 of 40 photograph worlds and 20 of 23 Blender worlds passed the check. So the loop goes beyond puzzle-like drawings. Examples, and short questions that a frontier model answers incorrectly, are on the project page: https://projectpixelproof.github.io/3d-scenes-and-photographs.html and https://projectpixelproof.github.io/hard-questions.html

First, I appreciate the authors' effort in running additional experiments. However, the new ablations raise further concerns about the practical value of inverse verification.

"we fine-tuned Gemma 4 12B on the data from each of the four conditions (16,082 examples each, one run each), and held-out accuracy rose from 57.8% to 67.9%, 69.1%, 67.2% and 68.2%, about ten points in every case (Appendix K.7)." any idea on why no-check and VLM judge is better than PixelProof?

More concerningly, PixelProof provides no improvement over the base Gemma 4 12B on external benchmarks (59.3% vs. 59.4%). Why wasn't this investigated further across all 3 base models? Without evidence of generalization, what is the practical value of PixelProof as training data?

AuthorsOpenReview ↗

A clarification on the ablation and the external benchmarks

Thank you for the follow-up. We would like to clarify both points.

Why are no check and the VLM judge higher than PixelProof? For this particular experiment, the differences are within noise. Each of the four numbers (67.2% to 69.1%) comes from a single training run, scored on one held-out set of 1,691 questions, and at that sample size the sampling error alone is about 1.1 points (Appendix K.7). The main takeaway is that all four setups behave the same, with a gain of about ten points each (Most of the worlds in all four training sets also come from GPT-5.6 Sol, the strongest of the three agents). The inverse program adds a constraint to generation, and one of the main question in the paper and in this experiment to answer is whether that constraint hinders the data. It clearly does not. (Please note that this noise level applies to this experiment only. The +1.3 points on the external benchmarks below is a different measurement: an average over four training runs, three models and 15 benchmarks).

What does the inverse program add, then? Correct answers. Among the worlds kept without an inverse program, 7 of 298 were defective, and among those kept with it, 0 of 135 (Table A35). The gap grows as the coding agent gets weaker: without a check the defect rate goes from 0.9% for GPT-5.6 Sol to 10% for DeepSeek-V4-Flash. The inverse program also offers another advantage that we should emphasize: a step-by-step program that reads the image and solves the problem, which could later be converted into reasoning traces for training.

External benchmarks on all three base models. We did run this: Table A27 reports all three models on the 15 external benchmarks, with four training runs each, and the average gain is +1.3 points. The 59.3% against 59.4% you quote is Gemma 4 12B, the smallest of the three. Gemma 4 31B gains 0.6 points, and Qwen3.8-27B gains 3.4 points and improves on all 15 benchmarks.

To close: we see PixelProof as a general loop for generating questions. It is not limited to one kind of image: in the paper it generates puzzles, and in this rebuttal we showed that the same loop also works on photographs and on 3D scenes rendered with Blender. It is not limited to one model either. We ran it with several coding agents, and we also had one model write the inverse program for worlds that a different model built, and it works. In every case the idea is the same: the answer is verified by running a program on the pixels.

Reviewer_kMH7OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

PixelProof generates synthetic VQA data by checking that answers computed from scene parameters match those recovered from rendered images. It produces about 2,400 verified worlds across nine topics. LoRA SFT improves accuracy by 10.7 points on held-out worlds and 1.9 points across 16 external benchmarks.

Critical strengths
  1. The inverse check catches rendering errors that would otherwise leave incorrect labels. Figure 4 and the 29% repair rate show why this matters.

  2. Data quality is checked through scene replay, image replacement, human review, and answer-position balancing. Evaluation holds out entire worlds.

  3. Generation follows spatial requests with 95.5% compliance, while error feedback produces questions that push open models near chance.

Critical weaknesses
  1. There is no forward-only baseline with the same agents and budget. The training gains could come from data scale and diversity rather than inverse verification.

  2. Transfer gains mainly come from VLMsAreBlind and PuzzleVQA, which resemble the training data. Other benchmarks show little improvement or declines, including CLEVR and MathVista. The RLVR use proposed in the abstract is not tested.

Questions for authors

See the weakness section.

2 comments
AuthorsOpenReview ↗

Response to Reviewer kMH7 (Part 1 of 2)

Thank you for your review.

You raised three concerns: the paper has no forward-only baseline run with the same agents and budget, the transfer to external benchmarks comes mostly from benchmarks that resemble our training data, and the RLVR use is not tested. We addressed each of them and give the new evidence below.

For the first, we ran an ablation. We used three discovery profiles, two hours each, comparing four ways of deciding whether a world is kept. What it shows is that requiring an inverse program does not limit what the agents can generate. It does cost some throughput, which is the price of having a program verify every world, and in return none of the worlds it kept was defective. Compared with a VLM used as a judge, the inverse is free to run, it is a CPU program and not model inference, and it rarely rejects a correct world, where the judge rejected many. For the other two concerns, we corrected the external-benchmark table (the average gain is +1.3 points, positive but modest) and ran a first RL experiment, in which the answers work as a reward.

The missing forward-only baseline

There is no forward-only baseline with the same agents and budget.

Thank you for bringing up this point! Based on your feedback, we have rerun the generation process with four different setups:

  1. Inverse program (PixelProof): agent writes an inverse program that can recover the answer from the image alone.
  2. No check: forward-only generation with no check of the answers.
  3. Agent's own tests: agent can run any code and can inspect its own images.
  4. VLM judge: a separate VLM judge verifies the image instead of the inverse program.

In each setup, the generation process has a fixed 2 hours of wall clock time to output as many valid worlds as possible. We repeated this setup across three different generating agents (GPT-5.6-Sol, GPT-5.6-Luna and DeepSeek-V4-Flash). In the VLM judge setup, the judge is a separate instance of the agent model (for DeepSeek-V4-Flash, the judge is GPT-5.6-Sol).

Then to measure the data quality of each setup, we first flagged potentially problematic data by evaluating GPT-6.1-Terra on the questions and then Claude Opus 5.5 on the questions that GPT-6.1-Terra answered incorrectly. If both Terra and Opus answered incorrectly, an author then manually checked those samples to determine if the generated question contained any errors.

Table R1: The inverse program is a cheap and effective way to ensure data quality.

Generation setup# Worlds kept ↑# Worlds with flawed questions ↓Total API cost ↓
Inverse program (PixelProof)1350$53
No check1593$38
Agent’s own tests1394$52
VLM judge1130$69

The worlds kept with the agent's own tests or with no check contained errors: wrong labels, broken renders and ambiguous questions. In contrast, the worlds kept by PixelProof or the VLM judge contained no errors.

The VLM judge had no false positives (flawed worlds accepted), but many false negatives (correct worlds rejected). On the 119 questions where the judge disagreed with the recorded answer, Terra and Opus agreed with the recorded answer on 115 and never with the judge. Additionally, the VLM judge process cost 30% more than PixelProof, and a judge has to be paid again for every new question sampled from a world: checking 100,000 questions costs 70 to 1,250 US dollars with a judge, against about 83 minutes of CPU time and no model call with the inverse program.

In summary, only PixelProof and the VLM judge kept no flawed world. Of the two, PixelProof cost less, kept more worlds (135 against 113), and almost never rejected a correct world, while the judge rejected many. The full study is in Appendix L of the revised PDF. The step-by-step animations on the project page show what the inverse programs compute from the pixels: https://projectpixelproof.github.io/animations.html

AuthorsOpenReview ↗

Response to Reviewer kMH7 (Part 2 of 2)

(continued)

Whether the training gains come from the data or from the inverse

The training gains could come from data scale and diversity rather than inverse verification.

We tested this. We fine-tuned Gemma 4 12B on the worlds from each of the four conditions (16,082 examples each, one training run each). The four training sets come from the same agents and the same profiles, so they contain the same kinds of question; they differ only in how a world was accepted. Held-out accuracy went from 57.8% for the base model to 67.9% with the inverse, 69.1% with no check, 67.2% with the agent's own tests and 68.2% with the judge. The gain is about ten points in every case, and the differences among the four are within what a single run can resolve. One likely reason is that few worlds in any condition were defective, at most 4 of 139, so the four training sets differ in only a small share of their examples. More generally, fine-tuning accuracy follows what the training data contains, not how its labels were checked. So requiring the inverse costs nothing in training value, and the gains should not be attributed to it; the revised paper says so (Appendix K.7). What the inverse contributes is something fine-tuning accuracy does not measure: every label, on every instance, is checked against the pixels by a fixed program with no model in it.

Transfer to external benchmarks

Transfer gains mainly come from VLMsAreBlind and PuzzleVQA, which resemble the training data. Other benchmarks show little improvement or declines, including CLEVR and MathVista.

We went back over the external results, found that a couple of benchmarks had been evaluated with an outdated script, and corrected it. The largest correction was on VLMsAreBlind, where the gain is +1.1 points on average and not +14.0. With the corrected table, the average change is +1.3 points over 15 external benchmarks, and it stays at +1.3 if VLMsAreBlind is left out. The largest gains are on PuzzleVQA (+6.4), VLMsAreBiased (+4.6) and AlgoPuzzleVQA (+3.1), the benchmarks closest to our training data.

Two other rows changed in the corrected table, MathVista and RAVEN. MathVista was regraded and its average is now +0.3, though two of the three models still decline on it. RAVEN is no longer in the table, because of a documented defect in how its answer choices are built (Hu et al., 2021); with RAVEN included, the corrected average is +1.2 over 16. A-OKVQA, DocVQA and CLEVR still decline. So the external gains are positive on average, modest, and concentrated on puzzle-style benchmarks. The revised paper says this directly, and it now states in the main text that 10 of the 45 profile example questions were adapted from VLMsAreBlind and VLMsAreBiased.

RLVR

The RLVR use proposed in the abstract is not tested.

We trained Qwen2.5-VL-7B with GRPO on PixelProof questions with a match against the recorded answer as the only reward. Accuracy on the 1,691 held-out questions rose from 44.5% to 53.4%, about the level that supervised fine-tuning reaches on the same data (52.4%). On the 15 external benchmarks the average moved from 60.6 to 61.8. RL can take a large amount of time and compute, so we show it for this one model.

Beyond 2D drawings

The worlds in the paper are drawn with 2D drawing code. For this rebuttal we gave the agent two other ways to make images, an image-generation model (GPT Image 2.6 Flare) for photographs and Blender for 3D scenes, and kept the forward–inverse check unchanged. In first two-hour runs, 20 of 40 photograph worlds and 20 of 23 Blender worlds passed the check. These are early tests, but the loop still produces valid questions on both kinds of image. Examples are at https://projectpixelproof.github.io/3d-scenes-and-photographs.html.

Reviewer_HnhjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposed PixelProof, an agentic framework to generate synthetic VQA data mainly focused on puzzle-like tasks. The framework generates question worlds, where each world is a reusable set of programs that can generate multiple image-question-answer samples for the same type of question.

The cross-validation idea underlying the framework is straightforward and useful. Given a task description, a coding agent builds a question world by writing a sampler to introduce randomness for scene description, a renderer to generate the image according to the scene description, and forward and inverse programs. The forward program computes the expected answer from the scene specification, while the inverse program tries to recover the answer from the rendered image. If their answers mismatch, the agent can revise the programs and reject them if final verification fails.

The authors obtain over 2,400 verified question worlds and report about 852K generated image-question pairs in the main generation experiments. They sample 10,860 pairs to benchmark existing frontier and open-weight VLMs, indicate that open-weight models perform much worse than frontier models on these questions.. They also fine-tune three open-weight VLMs on about 28K samples and report improvements on held-out synthetic questions and some external benchmarks.

Critical strengths

S1. The cross-validation idea is straightforward and useful. The proposed forward–inverse agreement can check whether the expected answers can be recovered from the rendered images to secure the effcitveness of accuracy metrics. The paper also provides examples where inverse programs detect rendering errors and help agents fix them.

S2. Extensive experiments and analysis have been conducted in this work. The authors evaluate five coding-agent configurations across nine visual reasoning profiles. They also study generation diversity, verification reliability, question difficulty, human judgments, and downstream fine-tuning.

Critical weaknesses

W1. The benefit of inverse verification is not clearly isolated. While the paper shows that inverse programs can detect and fix errors, it does not directly compare against generation without inverse verification.

W2. The added value of PixelProof data over existing visual QA training data is unclear. As stated in Appendix E.2, existing benchmarks VLMsAreBlind and VLMsAreBiased are used as inspiration for question generation, then what about the SFT performance if only these datasets are used as training set and test on the same held-out test sets of Table A28 and A29?

W3. The paper does not study if the SFT performance will scale up along with the generated data scale, despite scalability being a main motivation of synthetic data.

W4. The improvements on real-world visual tasks are limited. The average gain across 16 external benchmarks is only 1.9 points, with most gains coming from related synthetic reasoning tasks.

More details and suggestions are provided in the below Questions For Authors.

Questions for authors

Q1 (W1). Can the authors compare PixelProof with a version without inverse verification? I think it would be interesting to understand the trade-off between generating more data and spending additional computation on inverse verification. If inverse verification does not bring enough improvement to justify its cost, would it be better to use the same budget to generate more synthetic data? A comparison under the same total computational budget, including data quality and downstream SFT performance, would help answer this question.

Q2 (W2). Can the authors try SFT with the same number of training instances from existing visual QA datasets, and evaluate on the same held-out test sets as Tables A28 and A29? It would also be interesting to see whether combining existing data with PixelProof provides additional gains. This could help show what new value PixelProof data brings beyond existing training data.

Q3 (W3). Can the authors report a data scaling experiment, for example using 1×, 3×, and 10× the current 28K training size, or even smaller subsets? Since the paper already reports about 852K generated image-question pairs, I think this would be worth exploring. It would also be interesting to compare more examples from the same worlds against more unique worlds with a fixed training size. Do the gains on synthetic and real-world benchmarks continue to increase?

Q4 (W4). Can the authors provide a more detailed analysis of transfer to real-image tasks that require similar skills, such as counting, spatial relations, and object correspondence? This could help us better understand whether PixelProof improves general visual reasoning or mainly helps with related synthetic tasks.

Q5 (W1). Would it be possible to directly compare PixelProof with existing code-based generation methods, such as Multimodal Self-Instruct[1] and CoSyn[2] cited in the paper, under a similar total computational budget? Following Q1, I think this comparison could be useful to show whether the additional cost of inverse verification leads to better data quality or downstream performance. A comparison on overlapping task types would be especially helpful.

References:

[1] Wenqi Zhang, Zhenglin Cheng, Yuanyu He, Mengna Wang, Yongliang Shen, Zeqi Tan, Guiyang Hou, Mingqian He, Yanna Ma, Weiming Lu, and Yueting Zhuang. Multimodal Self-Instruct: Synthetic abstract image and visual reasoning instruction using language model. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 2024.

[2] Yue Yang, Ajay Patel, Matt Deitke, Tanmay Gupta, Luca Weihs, Andrew Head, Mark Yatskar, Chris Callison-Burch, Ranjay Krishna, Aniruddha Kembhavi, and Christopher Clark. Scaling text-rich image understanding via code-guided synthetic multimodal data generation. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2025.

6 comments
AuthorsOpenReview ↗

Response to Reviewer Hnhj (Part 1 of 4)

Thank you for your review.

You raised four concerns: the benefit of the inverse is not isolated, the value of PixelProof data over existing data is unclear, scaling with training size is not studied, and the gains on real-image tasks are limited. We addressed each of them and give the new evidence below.

Our main new result is an ablation that removes or replaces the inverse check. We used three discovery profiles, two hours each, comparing four ways of deciding whether a world is kept. It shows that adding the inverse does not degrade generation: the agents still produce questions with it, at a somewhat lower throughput. In return, every label is checked against the pixels by a fixed program. None of the worlds it kept was defective, it costs nothing per question, and it rarely rejects a correct world, where a VLM judge rejected many. Fine-tuning on the data from each of the four conditions gives about the same gain, so requiring the inverse costs nothing in training value.

A version without inverse verification

Can the authors compare PixelProof with a version without inverse verification?

Thank you for bringing up this point! Based on your feedback, we have rerun the generation process with four different setups:

  1. Inverse program (PixelProof): agent writes an inverse program that can recover the answer from the image alone.
  2. No check: forward-only generation with no check of the answers.
  3. Agent's own tests: agent can run any code and can inspect its own images.
  4. VLM judge: a separate VLM judge verifies the image instead of the inverse program.

In each setup, the generation process has a fixed 2 hours of wall clock time to output as many valid worlds as possible. We repeated this setup across three different generating agents (GPT-5.6-Sol, GPT-5.6-Luna and DeepSeek-V4-Flash). In the VLM judge setup, the judge is a separate instance of the agent model (for DeepSeek-V4-Flash, the judge is GPT-5.6-Sol).

Then to measure the data quality of each setup, we first flagged potentially problematic data by evaluating GPT-6.1-Terra on the questions and then Claude Opus 5.5 on the questions that GPT-6.1-Terra answered incorrectly. If both Terra and Opus answered incorrectly, an author then manually checked those samples to determine if the generated question contained any errors.

Table R1: The inverse program is a cheap and effective way to ensure data quality.

Generation setup# Worlds kept ↑# Worlds with flawed questions ↓Total API cost ↓
Inverse program (PixelProof)1350$53
No check1593$38
Agent’s own tests1394$52
VLM judge1130$69

The worlds kept with the agent's own tests or with no check contained errors: wrong labels, broken renders and ambiguous questions. In contrast, the worlds kept by PixelProof or the VLM judge contained no errors.

The VLM judge had no false positives (flawed worlds accepted), but many false negatives (correct worlds rejected). On the 119 questions where the judge disagreed with the recorded answer, Terra and Opus agreed with the recorded answer on 115 and never with the judge. Additionally, the VLM judge process cost 30% more than PixelProof, and a judge has to be paid again for every new question sampled from a world: checking 100,000 questions costs 70 to 1,250 US dollars with a judge, against about 83 minutes of CPU time and no model call with the inverse program.

In summary, only PixelProof and the VLM judge kept no flawed world. Of the two, PixelProof cost less, kept more worlds (135 against 113), and almost never rejected a correct world, while the judge rejected many. The full study is in Appendix L of the revised PDF. The step-by-step animations on the project page show what the inverse programs compute from the pixels: https://projectpixelproof.github.io/animations.html

AuthorsOpenReview ↗

Response to Reviewer Hnhj (Part 2 of 4)

(continued)

Verification, or more data for the same budget?

If inverse verification does not bring enough improvement to justify its cost, would it be better to use the same budget to generate more synthetic data?

We measured both sides of this trade-off in the ablation. The cost of the inverse is in worlds, not in data: within the same two hours it kept somewhat fewer worlds than generating with no check (101 against 111 with Sol, 31 against 37 with Luna, 6 against 9 with DeepSeek). The amount of data is not limited by it, because a world is not one question. It is a program with parameters, from which any number of new instances can be sampled, each with its own image and answer. That is also where the check pays off: nobody can inspect every instance of a world by hand, and a VLM judge has to be paid for each one (between 70 to 1,250 US dollars for 100,000 questions, and many hours of model inference, depending on the model used as judge). The inverse is a fixed program, so every newly sampled instance can be run against it, and it checks the same 100,000 in about 83 minutes on a CPU, with no model call.

How much the check is worth depends on the coding agent. Without it, 2 of 200 worlds from Sol, the strongest agent, were defective, against 3 of 77 from Luna and 2 of 21 from DeepSeek; with the inverse, none was defective for any of the three. These counts are small, but they point one way: with a strong agent, more unchecked data is a reasonable use of the budget, and with a weaker or cheaper agent the check removes a larger share of faulty worlds.

Downstream fine-tuning under the same budget

A comparison under the same total computational budget, including data quality and downstream SFT performance, would help answer this question.

We tested this. We fine-tuned Gemma 4 12B on the worlds from each of the four conditions (16,082 examples each, one training run each). The four training sets come from the same agents and the same profiles, so they contain the same kinds of question; they differ only in how a world was accepted. Held-out accuracy went from 57.8% for the base model to 67.9% with the inverse, 69.1% with no check, 67.2% with the agent's own tests and 68.2% with the judge. The gain is about ten points in every case, and the differences among the four are within what a single run can resolve. One likely reason is that few worlds in any condition were defective, at most 4 of 139, so the four training sets differ in only a small share of their examples. So requiring the inverse costs nothing in training value, and the gains should not be attributed to it; the revised paper says so (Appendix K.7). What it buys is on the data-quality side: a check on every label, on every instance.

Multimodal Self-Instruct and CoSyn

Would it be possible to directly compare PixelProof with existing code-based generation methods, such as Multimodal Self-Instruct[1] and CoSyn[2] cited in the paper, under a similar total computational budget?

We compared with their data directly. Both methods are fixed pipelines of single model calls, built before coding agents that work for hours on one task, so there is no common unit of budget in which to match them with ours. We therefore took the training data the two papers released (the chart and table questions of Multimodal Self-Instruct and all figure types of CoSyn), used the same number of examples as for ours (27,207), and fine-tuned Gemma 4 12B with the same recipe, one run each. On our held-out questions the models reach 59.4% (Multimodal Self-Instruct), 60.8% (CoSyn) and 71.0% (PixelProof). The test questions are from PixelProof. Appendix K.8 discusses the comparison further. The two methods also differ from ours in what the answer is checked against: for their question-answer data, a language model writes each answer from the drawing code, and no program checks it against the image (Appendix L.6).

On the 15 external benchmarks the averages are 59.4% (base model), 59.3% (PixelProof) and 56.2% (each of the other two). We do not read this as a ranking of the methods.

AuthorsOpenReview ↗

Response to Reviewer Hnhj (Part 3 of 4)

(continued)

Existing visual QA data of the same size, and combined

Can the authors try SFT with the same number of training instances from existing visual QA datasets, and evaluate on the same held-out test sets as Tables A28 and A29?

Yes, With Gemma 4 12B and the same recipe, we trained on VLMsAreBlind and VLMsAreBiased, on the same number of PixelProof examples, and on the two together, and evaluated on the 1,691 held-out questions of Table A28. We did not run the smaller test set of Table A29.

Training dataExamplesHeld-out accuracy (%)
None (base model)057.8
VLMsAreBlind + VLMsAreBiased10,58463.9
PixelProof, same size10,58470.3
VLMsAreBlind + VLMsAreBiased + PixelProof27,20770.7

At the same size, PixelProof data reaches 70.3% and the two benchmarks 63.9%. Adding the benchmarks to PixelProof data changes little. Each row is a single run, and the test questions are ours, as you asked.

Scaling with training size

Can the authors report a data scaling experiment, for example using 1×, 3×, and 10× the current 28K training size, or even smaller subsets?

We ran the smaller subsets and the full size. At 3× and 10× the training collapsed with the recipe used for the other sizes, so we have no valid result to report for them. We fine-tuned Gemma 4 12B on five nested subsets of the training data, one run each:

Training examplesHeld-out accuracy (%)Mean of eight external benchmarks (%)
0 (base model)57.852.2
2,64665.553.6
5,29265.952.9
10,58470.352.7
18,90071.153.0
27,19673.753.2

Held-out accuracy rises with training size, from 65.5% to 73.7%. The external mean stays within about a point of the base model at every size.

We also ran the comparison of more examples per world against more worlds. At a fixed size of about 27,700 examples, drawing them from 136, 272 or 1,256 worlds gives 70.0%, 71.3% and 70.2% on the held-out questions. These are within the variation between runs, so we see no measurable effect of the number of worlds at this size. Both experiments are in Appendix K.9 and K.10.

Reviewer_1KD7OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Paper and contributions

PixelProof is an automated pipeline for generating synthetic visual questions whose intended answers are checked against what is actually recoverable from the rendered pixels. A coding agent creates reusable "question worlds": a scene sampler, renderer, forward program that answers from scene specifications, and inverse program that answers from the image alone. Disagreement guides code repair; passing worlds generate many labeled instances.

The central contribution is pixel-based verification beyond trusting the generation code, without relying on a VLM judge. Additional contributions are an iterative generation-and-repair loop and steering by question family, spatial distribution of evidence, and evaluator-model errors. Five coding-agent configurations produced over 2,400 replay-verified worlds, not merely 2,400 individual questions.

Validation

  • Verification usefulness: 29% of accepted worlds underwent repairs following inverse failures; about half of repair events involved renderer/sampler edits. A controlled substitution test across 100 worlds detected every answer-changing image replacement in 300 trials.
  • Robustness and clarity: A separate 200-scene unseen replay recorded failures in approximately 1.6% of initially accepted worlds, with missing records reported separately. Two human reviewers flagged no concern on 91.6% of 800 instances.
  • Steerability: 95.5% of measured, replay-verified spatially steered worlds matched the requested pattern. Feedback-guided generation increased the share classified as hard for frontier models from 4.5% to 8.3%, although frontier accuracy remained high overall.
  • Training utility: LoRA supervised fine-tuning of three open-weight VLMs yielded average gains of +10.7 percentage points on held-out worlds, +8.5 on excluded question families, +5.3 on the harder generated collection, and +1.9 across 16 external benchmarks.
Critical strengths
  • Verification checks the rendered evidence as well as the generator's intended answer. The inverse program must recover the answer from pixels alone, addressing a concrete weakness of procedural data generation: correct scene metadata can still produce an incorrect or unreadable image. The paper substantiates this with rendering-error examples, repairs following inverse failures in 29% of accepted worlds, and successful detection of answer-changing image substitutions. This makes the approach a consequential improvement in synthetic-data quality control rather than simply another question generator.

  • The generated data demonstrate learning value beyond individual generators. Fine-tuning three open-weight VLMs improves accuracy by an average of 10.7 percentage points on held-out worlds across four splits and 8.5 points when an entire question family is excluded from training. The modest, mixed external-benchmark gains provide additional evidence of transfer. Holding out whole generators strengthens the conclusion that the data teach reusable capabilities rather than merely variations of training instances, supporting the framework's practical value.

Critical weaknesses
  • The experiments do not isolate the benefit of the central contribution. There is no matched comparison against generation without an inverse program, or against model-based verification, controlling for generation budget and training-data size. Repair examples show that the inverse can catch errors, but downstream improvements could largely reflect additional synthetic training data rather than inverse verification specifically. This leaves unresolved whether the proposed mechanism materially improves data quality and learning relative to simpler alternatives—a key consideration for acceptance.

  • Program agreement provides a narrower validity check than the "verifiable visual questions" framing suggests. The same agent authors both answer programs, so agreement can preserve shared misunderstandings of the question; fresh-scene replay tests robustness within the generator rather than independently establishing semantic correctness. Human review flagged clarity or answer concerns in 8.4% of sampled instances despite program verification. Although the paper acknowledges this limitation, it lacks an independent check that systematically addresses it, weakening confidence in the framework as a source of automatically reliable evaluation labels or training rewards.

Questions for authors
  • What does human review establish about label correctness? Of the 67 instances flagged by at least one reviewer, how many involved incorrect recorded answers versus wording or interpretational ambiguity? Were these concerns adjudicated, and were any cases identified where both programs agreed on an answer inconsistent with the question? This distinction materially affects confidence in the generated labels, beyond agreement and replay robustness.
  • What improvement is attributable specifically to inverse verification? Do you have a controlled comparison against forward-only generation, with comparable generation budgets, training-data sizes, and evaluation splits? Human-assessed validity and downstream accuracy would be particularly informative. Such evidence would help distinguish the value of the paper's central verification mechanism from the benefit of synthetic-data generation generally; this is a request supporting the current claims, not a suggestion for extending the framework to new domains.
2 comments
AuthorsOpenReview ↗

Response to Reviewer 1KD7 (Part 1 of 2)

Thank you for your review.

You raised two concerns and two questions that go with them: the experiments do not isolate what the inverse program contributes, and agreement between two programs written by one agent is a narrower check than "verifiable" suggests, which also raises the question of what the human review says about label correctness. We addressed each of them and give the new evidence below.

For the first, we ran the controlled comparison as an ablation. We used three discovery profiles, two hours each, comparing four ways of deciding whether a world is kept. It shows that adding the inverse does not degrade generation: the agents still produce questions with it, at a somewhat lower throughput. In return, every label is checked against the pixels by a fixed program. None of the worlds it kept was defective, it costs nothing per question, and it rarely rejects a correct world, where a VLM judge rejected many. Fine-tuning on the data from each of the four gives about the same gain, so the downstream gains are not what the inverse contributes, and the revised paper says so. For the second, we give the breakdown of the 67 flagged instances, the cases we found where both programs agreed on a wrong reading, and a run in which a different model writes the inverse.

A controlled comparison against forward-only generation and a VLM judge

There is no matched comparison against generation without an inverse program, or against model-based verification, controlling for generation budget and training-data size.

Thank you for bringing up this point! Based on your feedback, we have rerun the generation process with four different setups:

  1. Inverse program (PixelProof): agent writes an inverse program that can recover the answer from the image alone.
  2. No check: forward-only generation with no check of the answers.
  3. Agent's own tests: agent can run any code and can inspect its own images.
  4. VLM judge: a separate VLM judge verifies the image instead of the inverse program.

In each setup, the generation process has a fixed 2 hours of wall clock time to output as many valid worlds as possible. We repeated this setup across three different generating agents (GPT-5.6-Sol, GPT-5.6-Luna and DeepSeek-V4-Flash). In the VLM judge setup, the judge is a separate instance of the agent model (for DeepSeek-V4-Flash, the judge is GPT-5.6-Sol).

Then to measure the data quality of each setup, we first flagged potentially problematic data by evaluating GPT-6.1-Terra on the questions and then Claude Opus 5.5 on the questions that GPT-6.1-Terra answered incorrectly. If both Terra and Opus answered incorrectly, an author then manually checked those samples to determine if the generated question contained any errors.

Table R1: The inverse program is a cheap and effective way to ensure data quality.

Generation setup# Worlds kept ↑# Worlds with flawed questions ↓Total API cost ↓
Inverse program (PixelProof)1350$53
No check1593$38
Agent’s own tests1394$52
VLM judge1130$69

The worlds kept with the agent's own tests or with no check contained errors: wrong labels, broken renders and ambiguous questions. In contrast, the worlds kept by PixelProof or the VLM judge contained no errors.

The VLM judge had no false positives (flawed worlds accepted), but many false negatives (correct worlds rejected). On the 119 questions where the judge disagreed with the recorded answer, Terra and Opus agreed with the recorded answer on 115 and never with the judge. Additionally, the VLM judge process cost 30% more than PixelProof, and a judge has to be paid again for every new question sampled from a world: checking 100,000 questions costs 70 to 1,250 US dollars with a judge, against about 83 minutes of CPU time and no model call with the inverse program.

In summary, only PixelProof and the VLM judge kept no flawed world. Of the two, PixelProof cost less, kept more worlds (135 against 113), and almost never rejected a correct world, while the judge rejected many. The full study is in Appendix L of the revised PDF. The step-by-step animations on the project page show what the inverse programs compute from the pixels: https://projectpixelproof.github.io/animations.html

AuthorsOpenReview ↗

Response to Reviewer 1KD7 (Part 2 of 2)

(continued)

Downstream accuracy with and without the inverse

downstream improvements could largely reflect additional synthetic training data rather than inverse verification specifically.

We tested this. We fine-tuned Gemma 4 12B on the worlds from each of the four conditions (16,082 examples each, one training run each). The four training sets come from the same agents and the same profiles, so they contain the same kinds of question; they differ only in how a world was accepted. Held-out accuracy went from 57.8% for the base model to 67.9% with the inverse, 69.1% with no check, 67.2% with the agent's own tests and 68.2% with the judge. The gain is about the same for each setup. One likely reason is that few worlds in any condition were defective, at most 4 of 139, so the four training sets differ in only a small share of their examples.

So the gains do come from the generated data and should not be attributed to the inverse. What the inverse contributes is the check itself: every label, on every instance, is tested against the pixels by a fixed program with no model in it, and requiring it costs nothing in training value.

What the human review says about label correctness

Of the 67 instances flagged by at least one reviewer, how many involved incorrect recorded answers versus wording or interpretational ambiguity?

The human review shows how often a careful reader had any concern about an instance. It was not designed to certify each label, and the 8.4% is not a rate of wrong labels. The two reviewers raised 80 concerns on the 67 flagged instances. Going by the reasons they wrote, 15 of the 80 question the recorded answer and 65 are about wording or ambiguity. So at most 15 of the 800 instances carry a doubt about the label, and at least 52 of the 67 were flagged only for clarity. One reviewer marked 19 further instances with the single word "confusing"; counting them gives 85 of 800, and all 19 concern clarity. These concerns were not adjudicated: we report what the reviewers wrote and did not assign new answers. The revised paper now says this where the 8.4% is reported.

Cases where both programs agree on a wrong reading

were any cases identified where both programs agreed on an answer inconsistent with the question?

Yes, we found such cases. Agreement cannot detect a misreading that the two programs share. We had the same concern while building the collection, and the revised Limitations paragraph now says it.

In our manual checks while running the loop, we found only two clear cases where the two programs agreed for the wrong reason. In a world written by DeepSeek, the stripe that carries the answer was drawn in white on a white background, so the inverse reads it from exact pixel values but a person cannot see it. The other was a world written by Luna that draws a clock and asks about a detail of its face, where the two programs agreed in the same way. After submission we also searched systematically: we screened 2,172 worlds for cases where all three frontier models give the same answer and it differs from the recorded one, and inspected the 12 most contradicted. Five of those were shared misreadings of the question. This was a targeted search of the most suspicious worlds, so it gives a lower bound and not a rate.

An independent check

it lacks an independent check that systematically addresses it

We tested this for the rebuttal. The method does not require one agent to write both programs: the inverse can be written by a different model that never sees the forward program, and this works. Claude Opus 5.5 wrote inverse programs for 105 worlds from the images and question text alone, and they agreed with the recorded answers on 96 of them. Five of the other nine were worlds we had already found defective (Appendix L.5).

Reviewer_3bdGOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

PixelProof generates visual questions with verifiable answers. Each question has three parts. A sampler and renderer produce a scene and its image. A forward program computes the answer from the scene specification. An inverse program sees only the image and independently computes the answer. A question is kept only if the two agree, and a fixed verifier rechecks agreement on fresh scenes. The authors run five coding-agent configurations across nine discovery profiles, giving 45 campaigns, 1,331 accepted worlds, and about 850K image-question pairs.

Critical strengths

1.A simple, practical verification idea. Forward/inverse agreement gives a label check without a VLM judge. Because the inverse program sees only pixels, it catches rendering bugs. The paper reports that 29% of worlds triggered an inverse-program error during generation, and nearly half of those were fixed by changing how the image was rendered (Fig. 4). 2.Large-scale generation evidence. The study covers 1,426 episodes, 1,331 accepted worlds, and about 852K evaluated image-question pairs. Repetition analysis shows little overlap: no two worlds share question text, and only 16 of 845,650 pairs exceed 0.90 similarity. 3. Steerability is shown empirically. Spatially steered worlds match the requested region shape 95.5% of the time, and discovery profiles raise category coverage from 5.20 to 7.09 of 9.

Critical weaknesses

1.The key control is missing. The paper never compares training with the inverse-agreement filter against no filter, forward-only filtering, VLM-judge filtering, or existing code-based generators such as Multimodal Self-Instruct and CoSyn. Section 3.1 says those methods are "PixelProof without the inverse program," but they are not run at matched budget. So the causal claim that verification drives the gains is not established. 2.External transfer is overstated. In Table A27, the 16-benchmark average of +1.9 pp is driven by VLMsAreBlind at +14.0. Without it, the average is about +1.1 pp. Five of the 16 averages are negative (A-OKVQA, DocVQA, CLEVR, MathVista, RAVEN). 3.The leave-one-profile-out statistics are weak. Each cell is a single seed, and run-to-run variance is very large. Measurement scores 0.0% in one 8-step run and 75.6% in a repeat. Prior conflict scores 40.6% and 41.3% at 8 steps but 84.3% at 16 steps. Gemma 4 12B's recipe was changed to 16 accumulation steps after the 8-step runs collapsed. The paper discloses this, but the change was made post hoc. The "26 of 27 combinations improve" claim cannot be separated from this noise without multiple seeds. 4.The "harder questions" claim lacks support. In Section 4.6, frontier accuracy drops only about 1.7 pp under model-feedback steering. Open-weight models fall to 30.7%, close to 28% chance. Near-chance accuracy could reflect ambiguity or label noise rather than difficulty. Section 4.4 also finds that the profile with the most reviewer concerns is the one the authors attribute part of its difficulty to ambiguity. The paper needs human solve rates on the hard subset to separate difficulty from error. The comparison with earlier experiments is also confounded by different discovery-profile settings. 5.The verification guarantee is narrower than the framing. Both programs are written by the same agent from the same question text. Agreement checks that answers are recoverable from pixels and catches rendering bugs, but it cannot detect a shared misreading of the question.

Questions for authors

as in weakness

6 comments
AuthorsOpenReview ↗

Response to Reviewer 3bdG (Part 1 of 4)

Thank you for your review.

You raised five concerns: the paper has no control that removes or replaces the inverse check, the external transfer is overstated, the leave-one-profile-out results rest on single runs, low accuracy on the steered questions could be noise and not difficulty, and one agent writes both programs. We addressed each of them and give the new evidence below.

Our main new result is an ablation that removes or replaces the inverse check. We used three discovery profiles, two hours each, comparing four ways of deciding whether a world is kept. It shows that adding the inverse does not degrade generation. The agents still produce questions with it, at a somewhat lower throughput. In return, every label is checked against the pixels by a fixed program: none of the worlds it kept was defective, it costs nothing per question, and it rarely rejects a correct world, where a VLM judge rejected many.

On fine-tuning, the paper does not claim that the inverse is the cause of the gains. Its fine-tuning experiments test whether the generated data is useful for training, and the ablation confirms that data generated with and without the inverse trains equally well at this scale. We have clarified this in the text. On transfer, we corrected the external table: the average gain is +1.3 points, modest and concentrated on puzzle-style benchmarks.

The missing control

The key control is missing. The paper never compares training with the inverse-agreement filter against no filter, forward-only filtering, VLM-judge filtering, or existing code-based generators

Thank you for bringing up this point! Based on your feedback, we have rerun the generation process with four different setups:

  1. Inverse program (PixelProof): agent writes an inverse program that can recover the answer from the image alone.
  2. No check: forward-only generation with no check of the answers.
  3. Agent's own tests: agent can run any code and can inspect its own images.
  4. VLM judge: a separate VLM judge verifies the image instead of the inverse program.

In each setup, the generation process has a fixed 2 hours of wall clock time to output as many valid worlds as possible. We repeated this setup across three different generating agents (GPT-5.6-Sol, GPT-5.6-Luna and DeepSeek-V4-Flash). In the VLM judge setup, the judge is a separate instance of the agent model (for DeepSeek-V4-Flash, the judge is GPT-5.6-Sol).

Then to measure the data quality of each setup, we first flagged potentially problematic data by evaluating GPT-6.1-Terra on the questions and then Claude Opus 5.5 on the questions that GPT-6.1-Terra answered incorrectly. If both Terra and Opus answered incorrectly, an author then manually checked those samples to determine if the generated question contained any errors.

Table R1: The inverse program is a cheap and effective way to ensure data quality.

Generation setup# Worlds kept ↑# Worlds with flawed questions ↓Total API cost ↓
Inverse program (PixelProof)1350$53
No check1593$38
Agent’s own tests1394$52
VLM judge1130$69

The worlds kept with the agent's own tests or with no check contained errors: wrong labels, broken renders and ambiguous questions. In contrast, the worlds kept by PixelProof or the VLM judge contained no errors.

The VLM judge had no false positives (flawed worlds accepted), but many false negatives (correct worlds rejected). On the 119 questions where the judge disagreed with the recorded answer, Terra and Opus agreed with the recorded answer on 115 and never with the judge. Additionally, the VLM judge process cost 30% more than PixelProof, and a judge has to be paid again for every new question sampled from a world: checking 100,000 questions costs 70 to 1,250 US dollars with a judge, against about 83 minutes of CPU time and no model call with the inverse program.

In summary, only PixelProof and the VLM judge kept no flawed world. Of the two, PixelProof cost less, kept more worlds (135 against 113), and almost never rejected a correct world, while the judge rejected many. The full study is in Appendix L of the revised PDF. The step-by-step animations on the project page show what the inverse programs compute from the pixels: https://projectpixelproof.github.io/animations.html

AuthorsOpenReview ↗

Response to Reviewer 3bdG (Part 2 of 4)

(continued)

What the paper claims for verification, and what it does not

So the causal claim that verification drives the gains is not established.

The paper does not claim that the inverse check is what makes fine-tuning work. Our claim for the inverse is about how the data is produced: it lets us generate questions at scale with every answer verified against the image. This verification needs no model, so it adds no token cost and no inference time per question. It rarely rejects a correct question, where a VLM judge rejected many. And, as the ablation shows, it does not limit what can be generated. The fine-tuning experiments ask a different question: is the generated data useful for training?

To test whether the inverse check affects the training gains, we fine-tuned Gemma 4 12B on the worlds from each of the four conditions (16,082 examples each, one training run each). The four training sets come from the same agents and the same profiles, so they contain the same kinds of question; they differ only in how a world was accepted. Held-out accuracy went from 57.8% for the base model to 67.9% with the inverse, 69.1% with no check, 67.2% with the agent's own tests and 68.2% with the judge. The gain is about ten points in every case, and the differences among the four are within what a single run can resolve. One likely reason is that few worlds in any condition were defective, at most 4 of 139, so the four training sets differ in only a small share of their examples. More generally, fine-tuning accuracy follows what the training data contains, not how its labels were checked. So requiring the inverse costs nothing in training value, and the gains should not be attributed to it. The revised paper reports the experiment in Appendix K.7.

Multimodal Self-Instruct and CoSyn

Section 3.1 says those methods are "PixelProof without the inverse program," but they are not run at matched budget.

The two methods have a different purpose: they produce chart, table and document data at scale, and for that data a language model writes each answer from the drawing code. PixelProof differs in one respect: a program that sees only the image must recover every answer. Appendix L.6 now has a table of what each related method checks its answers against.

We also trained on the data the two papers released (the chart and table questions of Multimodal Self-Instruct and all figure types of CoSyn): the same number of examples as ours (27,207) with Gemma 4 12B. On our held-out questions the models reach 59.4% (Multimodal Self-Instruct), 60.8% (CoSyn) and 71.0% (PixelProof), from 57.8% for the base model.

On the 15 external benchmarks the averages are 59.4% (base model), 59.3% (PixelProof), 56.2% (Multimodal Self-Instruct), and 56.2% (CoSyn).

External transfer

External transfer is overstated. In Table A27, the 16-benchmark average of +1.9 pp is driven by VLMsAreBlind at +14.0.

We went back over the external results, found that a couple of benchmarks had been evaluated with an outdated script, and corrected it. The largest correction was on VLMsAreBlind, where the gain is +1.1 points on average and not +14.0. With the corrected table, the average change is +1.3 points over 15 external benchmarks, and it stays at +1.3 if VLMsAreBlind is left out. The largest gains are on PuzzleVQA (+6.4), VLMsAreBiased (+4.6) and AlgoPuzzleVQA (+3.1), the benchmarks closest to our training data.

Two other rows changed in the corrected table, MathVista and RAVEN. MathVista was regraded and its average is now +0.3, though two of the three models still decline on it. RAVEN is no longer in the table, because of a documented defect in how its answer choices are built (Hu et al., 2021). So the external gains are positive on average, modest, and concentrated on puzzle-style benchmarks.

AuthorsOpenReview ↗

Response to Reviewer 3bdG (Part 3 of 4)

(continued)

Leave-one-profile-out

The leave-one-profile-out statistics are weak. Each cell is a single seed, and run-to-run variance is very large.

We added training seeds. For Gemma 4 12B we repeated the study three times for each of the nine excluded profiles, changing only the training seed (27 runs in total). 24 of the 27 runs score above the base model, and the average gain is 5.6 points.

In seven of the nine profiles, all three runs improve, and they differ from each other by at most 3.3 points. In the other two profiles, measurement and topology, one of the three runs failed in training: its loss rose midway and never recovered, and it ended below the base model (36.9% against 68.2% on measurement; 59.2% against 63.2% on topology). To check that these were training failures and not a property of the profiles, we ran three more seeds on them, and all three improved over the base model, by 6.5 to 8.8 points.

The per-run results are in Appendix N.

Are the steered questions hard, or noisy?

Near-chance accuracy could reflect ambiguity or label noise rather than difficulty.

We checked in four ways whether these questions are noise.

First, more reasoning solves them. We evaluated two of the open-weight models on the same 1,260 questions with reasoning turned on, which lets them spend many more tokens per answer. Accuracy rose from 29.8% to 80.6% for Qwen3.8-27B and from 30.7% to 63.3% for Gemma 4 31B. A wrong label caps accuracy whatever the model does, so this rise is not what noise would give. Appendix M of the revised PDF has the results for all three open-weight models on all questions.

Second, the frontier models agree with the labels. On 171 of the 252 model-feedback worlds, all three frontier models answer every question correctly, while the open-weight models score 31.8% on those same worlds (chance is 27.8%).

Third, two authors worked through 25 of the questions that the frontier models did not solve. They are very hard and slow even for a person, but they are answerable, and 24 of the 25 recorded answers were correct. Fourth, in the human review, reviewers questioned the recorded answer in 3 of their 400 judgments on these questions.

So the low accuracy comes from difficulty and not from noise. Where the difficulty comes from is a separate matter, and measuring hardness is itself hard. Part of it is the question text: some of these questions are long, which is common in the worlds written by Opus. Length is something the loop can control. To show that it can also produce questions that are short and still hard, we ran it again with that request. The project page has the results: questions of 8 to 16 words, in plain wording, that a frontier model answers incorrectly, each with the program that fixes its answer: https://projectpixelproof.github.io/hard-questions.html

In the revised paper, the claim for this experiment is now specific. Given a model's mistakes as feedback, the loop generates new questions that this model also fails. The checks above show that these questions are hard, not noisy. The comparison with the earlier experiments is not a controlled one, and the revised text says so.

Comments on the paper

1 comment
AuthorsOpenReview ↗

summary for all reviewers and ACs

We sincerely thank all five reviewers for their thoughtful reviews, which have made the paper stronger! We have replied to each of you individually and uploaded a revised PDF; below is a summary for your consideration.

What the inverse program adds (Appendix L)

  • Following your suggestion, we ran the same coding agents, prompts and time budget, changing only the checker.
  • Without the inverse program: 7 of the 298 kept worlds were defective (vs. 0 out of 135 defective worlds with the inverse program).
  • Alternatively, a VLM judge also kept no defective worlds, but it rejected many correct ones and incur extra cost for every new question. The key advantage of our method is leveraging the generated inverse program that runs cheaply on a CPU.
  • Qualitatively, these animations show step-by-step how the Renderer and Inverse program works and convinced us that these programs are legitimately doing their jobs as expected: https://projectpixelproof.github.io/animations.html

Other additions requested by reviewers (details in the individual replies)

  • Comparison of inverse programs written by a different model
  • A test showing the hard questions are answerable with more reasoning
  • A same-size comparison with existing synthetic data
  • Results with more training data and more seeds
  • A first reinforcement-learning experiment showing that PixelProof generated data can be used to improve some models

Easier to see than to read

When incorporating 3D Blender or an image generator model in the loop, our PixelProof pipeline can also 3D scenes and realistic AI-generated images. We are excited to invite you to take a look:

Thank you again for your interest and consideration!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.