acceptodds
Withdrawn from ICLR 2027

Drawing Answers: Benchmarking Visual Reasoning in Image Generative Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Can image generative models solve a problem by drawing its answer? Tracing a route, completing a grid, and selecting a winning move require images that satisfy explicit rules and spatial constraints. We introduce Drawing Answers, a dataset and benchmark spanning 22 tasks in seven reasoning families. It contains 152,366 problem–answer image pairs across 26 configurations, including four paired representations of the same underlying problems. Task-specific generators, solvers, and structured annotations support supervised learning and evaluation of generated solutions. We evaluate unadapted FLUX.1 Kontext and GPT Image 2, and train task-specific FLUX LoRA adapters, evaluating six training checkpoints and reporting the highest test outcome per configuration. Gomoku peaks at 30,000 steps, where success rises from 0.0% for unadapted FLUX to 92.0%, compared with 10.0% for GPT Image 2. Performance varies substantially across tasks: accurate local edits coexist with persistent difficulty in globally constrained solutions. Paired encoding experiments further reveal different representation preferences across models. Drawing Answers provides a controlled resource for studying the capabilities, learnability, and remaining challenges of visual reasoning through image generation.

Primary area:
datasets and benchmarks
Keywords:
visual reasoning, image generation, image editing, datasets and benchmarks, synthetic data

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Jo6BOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Drawing Answers is a benchmark for solving visually specified problems by generating answer images. It contains 152,366 problem-answer pairs across 22 tasks and 26 configurations, with structured annotations, task-specific generators and evaluators, and four aligned representation comparisons. The paper evaluates unadapted FLUX.1 Kontext and GPT Image 2 and trains 26 task-specific FLUX LoRA adapters, reporting test-set peaks over six checkpoints as well as full training trajectories. Results show substantial supervised gains on several routing and board tasks, persistent failures on globally constrained problems, and component-level differences between producing a well-formed image and solving the underlying task.

Critical strengths
  1. The benchmark is a substantial resource design: 152,366 paired examples cover diverse visual answer types, explicit constraints, and task-specific split units (Section 3 and Appendix B). Structured annotations and evaluators that accept alternative legal solutions support task-level assessment beyond generic image quality.

  2. The four aligned representation comparisons preserve underlying problem identities while changing visual encoding. Together with split rules based on source images, layouts, symmetries, or game families, this enables more controlled analysis of representation sensitivity and reduces obvious train-test overlap.

  3. The component analyses distinguish well-formed output from correct reasoning, such as correctly highlighted objects with incorrect counts or high cell coverage with invalid checkpoint order (Section 4.4). Full training trajectories also substantiate large learning gains without relying solely on peak selection: fixed-30k Gomoku, ColorConnect, and TautReach scores remain 92%, 87%, and 99% (Table 7).

Critical weaknesses
  1. The main cross-model conclusions rely on image parsers whose reliability on actual generated outputs is not adequately established. Reference answers, unchanged inputs, and controlled corruptions are useful calibration, but they do not quantify false positives or false negatives for each model's output style. The supplementary README explicitly states that the base output-domain semantic validity audit remains incomplete. Since many base/GPT scores are near zero while adapted FLUX can approach 100%, a representative human audit is needed to distinguish reasoning failures from extraction or preservation-gate failures. I appreciate that the paper separately labels Billiards, Jigsaw, and Bongard as diagnostics; the concern also applies to validation of the principal binary comparisons.

  2. The submitted artifact does not yet establish the benchmark's practical reproducibility, which is central to its contribution. The supplementary README and docs/REPRODUCING.md state that the package omits dataset images, frozen dataset rows, reference/prediction images, and certain construction inputs, while the anonymous artifact URL is only a placeholder. The included code and per-sample scores are valuable, but they permit score aggregation rather than independent image-level rescoring or verified reconstruction of the exact benchmark. This limits the practical utility of the resource and, together with the evaluator-validation gap, is central to my weak-rejection recommendation.

  3. Test-set checkpoint selection complicates interpretation of the reported comparisons. Appendix C explicitly selects the highest test score over six checkpoints and describes early monitoring on test questions, without a separate validation split. This is transparently disclosed, and the fixed-30k results still support the large learning effects. However, peak-based estimates and smaller differences between representations need fixed-checkpoint or validation-selected results with uncertainty to support fair comparison.

Questions for authors
  1. Can the authors provide a stratified human audit of generated outputs from all three model settings, including per-task parser false-positive/false-negative rates and the effect of any corrections on the main comparisons? This would directly address my principal validity concern.

  2. Is there an accessible anonymized release containing the exact frozen benchmark inputs, references, split identifiers, and a representative set of generated outputs, or a complete reconstruction procedure with all required frozen construction inputs? This would materially affect my assessment of the resource contribution.

  3. Appendix C transparently identifies the results as test-set peaks and describes early monitoring on test questions. Can the authors distinguish exploratory peak analyses from a fixed-checkpoint or validation-selected evaluation, and provide uncertainty for the paired representation comparisons? The fixed-30k results already support the large learning gains, but smaller differences need a cleaner estimate.

Reviewer_xBwdOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

Drawing Answers benchmarks whether image generative models can solve visually specified problems by drawing their answers. It comprises 22 tasks across seven reasoning families, with 152,366 problem–answer image pairs across 26 configurations. Task-specific generators, solvers, and structured annotations support training and evaluation, while four paired encoding studies vary visual representation without changing the underlying problems.

The authors compare unadapted FLUX.1 Kontext, GPT Image 2, and 26 separately trained FLUX LoRA adapters. Task-specific evaluation checks answer correctness and, where required, scene preservation. The results show substantial gains from supervised adaptation on several routing and board-game tasks, alongside persistent difficulties in globally constrained tasks. Component analyses distinguish correct object selection, numerical answers, visual completeness, and full rule satisfaction. The main adapted-model results report the highest test outcome across six checkpoints per configuration.

Critical strengths

The benchmark provides a substantial and structured resource for visual problem solving through image generation. The common image-and-instruction-to-image interface covers diverse tasks, and the paired training data allow researchers to study both existing capabilities and task-specific learnability. Structured references and task-dependent validity checks are well suited to evaluating solutions beyond visual similarity. Accepting alternative legal solutions where appropriate is also a useful design choice.

Aligned encoding variants and component-level analyses provide useful diagnostic insights. The four pairs of encoding variants preserve the underlying problems and train/test splits, helping examine how visual representations affect model performance. The component-level analyses further distinguish requirements such as quantity prediction, target highlighting, and overall rule satisfaction, revealing failure modes that aggregate success rates can obscure.

Critical weaknesses

The benchmark evaluates generated answers using structured annotations and task-specific rules, and calibration with reference answers and negative controls provides useful initial evidence. However, the paper does not report a systematic human audit of actual generated outputs, leaving the evaluator’s false acceptance and false rejection rates under different rendering styles, spatial deviations, and borderline cases unclear. Paperfold’s success rate drops from 15% to 5% when the positional tolerance is tightened, illustrating that threshold choices can substantially affect reported results. Although this sensitivity does not itself imply that the metric is inappropriate, the paper should better justify the tolerances and assess whether the main model comparisons remain stable under reasonable threshold variations.

The incremental value over existing benchmarks needs to be more clearly established. The aligned encoding variants and component-level analyses provide useful diagnostic information, but comparisons with the closest prior work, including Reason2Gen and EAR/AMAZE, remain high-level. Given that prior work already encompasses diverse visual problem-solving tasks, training resources, and logical evaluation, the paper should clarify which gaps in capability coverage its additional tasks address and what additional analytical capabilities its structured evaluation and representation comparisons offer. This would help establish the benchmark’s distinctive contribution.

Questions for authors

Have evaluator decisions been manually audited on generated outputs? Please report the audit sample size and selection, false-positive and false-negative rates, and the main failure modes. How robust are the important conclusions to extraction thresholds and harmless rendering variations?

What are the main incremental advantages of Drawing Answers over the closest prior work, particularly Reason2Gen and EAR/AMAZE? Could the authors clarify which gaps in capability coverage the additional tasks address?

The paired encodings preserve problem identity, which is a useful control, but each encoding is trained separately and compared at its own test-selected peak. Do the representation preferences persist at matched checkpoints and across repeated training runs? This would help distinguish encoding effects from checkpoint selection and run variability.

Reviewer_t23tOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a dataset and benchmark including 22 tasks and 152k data points for validating if image editing model can perform visual reasoning by drawing answers. Using task-specific lora, sft can significantly improve the base model's performance on some tasks. However, a gap remains between "local correctness" and "satisfying all rubrics."

Critical strengths
  • This benchmark differentiates visual quality and task correctness, which is important. The proposed dataset covers 22 tasks in 7 reasoning families, including training dataset, structured annotation, and evaluation pipeline.
Critical weaknesses
  • This paper doesn't include a separate validation set but directly uses the test set to choose the best checkpoint. Besides, the authors mention that some training runs use test examples for monitoring performance, which might further introduce the selection bias.

  • The experiments can't reveal if the learned reasoning capability is transferable. This paper trains a specific lora for each task which is impossible in the real-world setting. It would be better to show the results from joint multi-task training or cross-task transfer experiments.

Questions for authors
  • Recent LLMs can easily solve most proposed tasks and they are good at drawing the outputs with tools and coding. It would be better to provide comparisons against LLM-based approaches or agentic pipelines.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.