- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces an agent-based framework for generating synthetic puzzle-like visual reasoning questions. Each question world includes a renderer, a forward program that computes the answer from scene parameters, and an inverse program that recovers the answer from pixels. Forward-inverse agreement is used to verify the generated data. The authors generate over 2,400 verified question worlds and demonstrate improvements through VLM SFT.
Critical strengths
-
Useful verification mechanism. Pixel-based inverse verification addresses potential inconsistencies between generated images and their intended answers. interesting and useful idea.
-
Extensive generation and evaluation. The framework produces diverse visual reasoning tasks, with substantial experiments on generation, verification, human validation, and eval on model performance.
-
Automatically constructing new question worlds and their verification programs is potentially a scalable data generation method.
Critical weaknesses
-
Limited methodological novelty and missing essential baselines. The authors compare model performance before and after fine-tuning on PixelProof, which demonstrates that their data can be useful. but LLM-driven visual data generation and programmatically verified puzzles have been explored in Multimodal Self-Instruct, CoSyn, AlgoPuzzleVQA, and related work. The main novelty is forward-inverse verification, but comparisons against matched-budget forward-only generation and VLM-based verification are missing. Its incremental value is unclear.
-
Limited generalization beyond synthetic visual reasoning. Fine-tuning improves accuracy by 10.7 on held-out generated worlds, but only +1.9 on average across 16 external benchmarks. Improvements on general visual understanding tasks are modest, while two benchmarks with substantial gains contributed example questions to the generation profiles.
-
unfair frontier versus open-weight model comparison in section 4.5. The reported performance gap is difficult to interpret because the models are not matched for scale or reasoning configuration.
-
Insufficient evidence of novel reasoning challenges. Agents generate new questions using predefined profiles, implementation demonstrations, and previously accepted worlds. I prefer to regard this paper as a solution of a specific benchmark: VLMsAreBlind.
Questions for authors
What is the primary contribution of PixelProof? Is the main goal to generate challenging, verifiable visual reasoning data for evaluation, or to produce training data that improves VLM capabilities? Although both are discussed, the experimental emphasis seems to favor the former, while the abstract primarily motivates the work through RLVR and model improvement. Could the authors clarify this discrepancy in the paper's framing? I you emphasize the model improvement, you really need to include other puzzel-like data syn baselines.
4 comments
Response to Reviewer APYA (Part 2 of 2)
(continued)
What the inverse adds, and what it does not
Its incremental value is unclear.
The ablation above shows what the inverse adds over the simpler alternatives. Against forward-only generation, none of the worlds it kept was defective. Against a VLM judge, it is a fixed CPU program with negligible cost per question, where a judge costs 70 to 1,250 US dollars per 100,000 questions, and it rarely rejects a correct world, where the judge rejected many. And requiring it costs nothing in training value: we fine-tuned Gemma 4 12B on the data from each of the four conditions (16,082 examples each, one run each), and held-out accuracy rose from 57.8% to 67.9%, 69.1%, 67.2% and 68.2%, about ten points in every case (Appendix K.7).
Multimodal Self-Instruct, CoSyn and AlgoPuzzleVQA
LLM-driven visual data generation and programmatically verified puzzles have been explored in Multimodal Self-Instruct, CoSyn, AlgoPuzzleVQA, and related work.
These methods differ from PixelProof in what the answer is checked against. Our related-work sentence calling them "PixelProof without the inverse program" was imprecise; we have corrected it, and Appendix L.6 of the revised PDF now has a table of what each related method checks. In the question-answer data of Multimodal Self-Instruct and CoSyn, a language model writes each answer from the drawing code, and no program checks it against the image. AlgoPuzzleVQA computes answers with solvers written by people for 18 puzzle types, and nothing checks that the image shows the state the solver used. In PixelProof the agents write the generators themselves, and a program that sees only the pixels must recover every answer.
We also trained on the data the two papers released (the chart and table questions of Multimodal Self-Instruct and all figure types of CoSyn): the same number of examples as ours (27,207), the same recipe, Gemma 4 12B, one run each. On our held-out questions the models reach 59.4% (Multimodal Self-Instruct), 60.8% (CoSyn) and 71.0% (PixelProof), from 57.8% for the base model. Appendix K.8 discusses the comparison further.
On the 15 external benchmarks the averages are 59.4% (base model), 59.3% (PixelProof) and 56.2% (each of the other two). We do not read this as a ranking of the methods.
Generalization beyond the generated questions
Improvements on general visual understanding tasks are modest, while two benchmarks with substantial gains contributed example questions to the generation profiles.
We found that a couple of external benchmarks had been evaluated with an outdated script, and corrected it. The average change is now +1.3 points over 15 external benchmarks. The largest correction is on VLMsAreBlind, one of the two benchmarks that contributed example questions: its gain is +1.1 points and not +14.0. The largest gain is on PuzzleVQA (+6.4), which contributed no examples. So the external gains are positive on average, modest, and concentrated on puzzle-style benchmarks, as expected from the kind of questions we train on. We do not claim gains on general visual understanding. The revised paper says this and now discloses the overlap in example questions in the main text. The forward–inverse construction itself is not limited to 2D puzzles: we show this on photographs and 3D scenes (see below). The table no longer includes RAVEN, because of a documented defect in its answer choices (Hu et al., 2021).
Frontier and open-weight models
The reported performance gap is difficult to interpret because the models are not matched for scale or reasoning configuration.
The two groups are not matched, and the revised Section 4.5 now says that the comparison is descriptive and that the gap may reflect both scale and reasoning setting.
New reasoning challenges, and VLMsAreBlind
I prefer to regard this paper as a solution of a specific benchmark: VLMsAreBlind.
The paper does not target one benchmark. It uses nine profiles to separate questions by the kind of reasoning they require, and the collection covers all nine. Ten of the 45 profile example questions were adapted from existing benchmarks, five from VLMsAreBlind and five from VLMsAreBiased; the other 35 were not. Fine-tuning on the collection moves VLMsAreBlind itself by only +1.1 points.
The worlds in the paper are simple 2D images drawn with Python code, and we chose them to establish the idea first. For the rebuttal we gave the agent two other ways to make images, an image-generation model (GPT Image 2.6 Flare) for photographs and Blender for 3D scenes, and kept the forward–inverse check unchanged. In first runs, 20 of 40 photograph worlds and 20 of 23 Blender worlds passed the check. So the loop goes beyond puzzle-like drawings. Examples, and short questions that a frontier model answers incorrectly, are on the project page: https://projectpixelproof.github.io/3d-scenes-and-photographs.html and https://projectpixelproof.github.io/hard-questions.html
First, I appreciate the authors' effort in running additional experiments. However, the new ablations raise further concerns about the practical value of inverse verification.
"we fine-tuned Gemma 4 12B on the data from each of the four conditions (16,082 examples each, one run each), and held-out accuracy rose from 57.8% to 67.9%, 69.1%, 67.2% and 68.2%, about ten points in every case (Appendix K.7)." any idea on why no-check and VLM judge is better than PixelProof?
More concerningly, PixelProof provides no improvement over the base Gemma 4 12B on external benchmarks (59.3% vs. 59.4%). Why wasn't this investigated further across all 3 base models? Without evidence of generalization, what is the practical value of PixelProof as training data?
A clarification on the ablation and the external benchmarks
Thank you for the follow-up. We would like to clarify both points.
Why are no check and the VLM judge higher than PixelProof? For this particular experiment, the differences are within noise. Each of the four numbers (67.2% to 69.1%) comes from a single training run, scored on one held-out set of 1,691 questions, and at that sample size the sampling error alone is about 1.1 points (Appendix K.7). The main takeaway is that all four setups behave the same, with a gain of about ten points each (Most of the worlds in all four training sets also come from GPT-5.6 Sol, the strongest of the three agents). The inverse program adds a constraint to generation, and one of the main question in the paper and in this experiment to answer is whether that constraint hinders the data. It clearly does not. (Please note that this noise level applies to this experiment only. The +1.3 points on the external benchmarks below is a different measurement: an average over four training runs, three models and 15 benchmarks).
What does the inverse program add, then? Correct answers. Among the worlds kept without an inverse program, 7 of 298 were defective, and among those kept with it, 0 of 135 (Table A35). The gap grows as the coding agent gets weaker: without a check the defect rate goes from 0.9% for GPT-5.6 Sol to 10% for DeepSeek-V4-Flash. The inverse program also offers another advantage that we should emphasize: a step-by-step program that reads the image and solves the problem, which could later be converted into reasoning traces for training.
External benchmarks on all three base models. We did run this: Table A27 reports all three models on the 15 external benchmarks, with four training runs each, and the average gain is +1.3 points. The 59.3% against 59.4% you quote is Gemma 4 12B, the smallest of the three. Gemma 4 31B gains 0.6 points, and Qwen3.8-27B gains 3.4 points and improves on all 15 benchmarks.
To close: we see PixelProof as a general loop for generating questions. It is not limited to one kind of image: in the paper it generates puzzles, and in this rebuttal we showed that the same loop also works on photographs and on 3D scenes rendered with Blender. It is not limited to one model either. We ran it with several coding agents, and we also had one model write the inverse program for worlds that a different model built, and it works. In every case the idea is the same: the answer is verified by running a program on the pixels.
Response to Reviewer APYA (Part 1 of 2)
Thank you for your review.
You raised four concerns: what the primary contribution is, whether the forward–inverse check adds anything over prior generators and over simpler baselines, how far the training gains generalize, whether the frontier and open-weight models are compared fairly, and whether the questions offer new reasoning challenges beyond VLMsAreBlind. We addressed each of them and give the new evidence below.
Our main new result is an ablation that removes or replaces the inverse check. We used three discovery profiles, two hours each, comparing four ways of deciding whether a world is kept. It shows that adding the inverse does not degrade generation: the agents still produce questions with it, at a somewhat lower throughput. In return, every label is checked against the pixels by a fixed program. None of the worlds it kept was defective, it costs nothing per question, and it rarely rejects a correct world, where a VLM judge rejected many.
The primary contribution
The core contribution is the idea that a visual question can be verified the way a program can: by running a second program against it, one that sees only the pixels. In PixelProof, coding agents write reusable question worlds, and in each world an inverse program must recover the answer from the pixels alone, so every label is checked without a model and without a person. The paper shows that this works at scale: current coding agents can write both the questions and the inverse programs that answer them from the pixels. The paper then tests the data in both uses, evaluation and training. RLVR appears in the abstract only as the general motivation for wanting cheaply verifiable data. However, for the rebuttal we also ran a first RL test on one model: GRPO on Qwen2.5-VL-7B, with a match against the recorded answer as the only reward, raised held-out accuracy from 44.5% to 53.4%.
Forward-only generation and VLM-based verification
Thank you for bringing up this point! Based on your feedback, we have rerun the generation process with four different setups:
In each setup, the generation process has a fixed 2 hours of wall clock time to output as many valid worlds as possible. We repeated this setup across three different generating agents (GPT-5.6-Sol, GPT-5.6-Luna and DeepSeek-V4-Flash). In the VLM judge setup, the judge is a separate instance of the agent model (for DeepSeek-V4-Flash, the judge is GPT-5.6-Sol).
Then to measure the data quality of each setup, we first flagged potentially problematic data by evaluating GPT-6.1-Terra on the questions and then Claude Opus 5.5 on the questions that GPT-6.1-Terra answered incorrectly. If both Terra and Opus answered incorrectly, an author then manually checked those samples to determine if the generated question contained any errors.
Table R1: The inverse program is a cheap and effective way to ensure data quality.The worlds kept with the agent's own tests or with no check contained errors: wrong labels, broken renders and ambiguous questions. In contrast, the worlds kept by PixelProof or the VLM judge contained no errors.
The VLM judge had no false positives (flawed worlds accepted), but many false negatives (correct worlds rejected). On the 119 questions where the judge disagreed with the recorded answer, Terra and Opus agreed with the recorded answer on 115 and never with the judge. Additionally, the VLM judge process cost 30% more than PixelProof, and a judge has to be paid again for every new question sampled from a world: checking 100,000 questions costs 70 to 1,250 US dollars with a judge, against about 83 minutes of CPU time and no model call with the inverse program.
In summary, only PixelProof and the VLM judge kept no flawed world. Of the two, PixelProof cost less, kept more worlds, and almost never rejected a correct world, while the judge rejected many. The full study is in Appendix L. The step-by-step animations on the project page show what the inverse programs compute from the pixels: https://projectpixelproof.github.io/animations.html