- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Drawing Answers is a benchmark for solving visually specified problems by generating answer images. It contains 152,366 problem-answer pairs across 22 tasks and 26 configurations, with structured annotations, task-specific generators and evaluators, and four aligned representation comparisons. The paper evaluates unadapted FLUX.1 Kontext and GPT Image 2 and trains 26 task-specific FLUX LoRA adapters, reporting test-set peaks over six checkpoints as well as full training trajectories. Results show substantial supervised gains on several routing and board tasks, persistent failures on globally constrained problems, and component-level differences between producing a well-formed image and solving the underlying task.
Critical strengths
-
The benchmark is a substantial resource design: 152,366 paired examples cover diverse visual answer types, explicit constraints, and task-specific split units (Section 3 and Appendix B). Structured annotations and evaluators that accept alternative legal solutions support task-level assessment beyond generic image quality.
-
The four aligned representation comparisons preserve underlying problem identities while changing visual encoding. Together with split rules based on source images, layouts, symmetries, or game families, this enables more controlled analysis of representation sensitivity and reduces obvious train-test overlap.
-
The component analyses distinguish well-formed output from correct reasoning, such as correctly highlighted objects with incorrect counts or high cell coverage with invalid checkpoint order (Section 4.4). Full training trajectories also substantiate large learning gains without relying solely on peak selection: fixed-30k Gomoku, ColorConnect, and TautReach scores remain 92%, 87%, and 99% (Table 7).
Critical weaknesses
-
The main cross-model conclusions rely on image parsers whose reliability on actual generated outputs is not adequately established. Reference answers, unchanged inputs, and controlled corruptions are useful calibration, but they do not quantify false positives or false negatives for each model's output style. The supplementary README explicitly states that the base output-domain semantic validity audit remains incomplete. Since many base/GPT scores are near zero while adapted FLUX can approach 100%, a representative human audit is needed to distinguish reasoning failures from extraction or preservation-gate failures. I appreciate that the paper separately labels Billiards, Jigsaw, and Bongard as diagnostics; the concern also applies to validation of the principal binary comparisons.
-
The submitted artifact does not yet establish the benchmark's practical reproducibility, which is central to its contribution. The supplementary README and docs/REPRODUCING.md state that the package omits dataset images, frozen dataset rows, reference/prediction images, and certain construction inputs, while the anonymous artifact URL is only a placeholder. The included code and per-sample scores are valuable, but they permit score aggregation rather than independent image-level rescoring or verified reconstruction of the exact benchmark. This limits the practical utility of the resource and, together with the evaluator-validation gap, is central to my weak-rejection recommendation.
-
Test-set checkpoint selection complicates interpretation of the reported comparisons. Appendix C explicitly selects the highest test score over six checkpoints and describes early monitoring on test questions, without a separate validation split. This is transparently disclosed, and the fixed-30k results still support the large learning effects. However, peak-based estimates and smaller differences between representations need fixed-checkpoint or validation-selected results with uncertainty to support fair comparison.
Questions for authors
-
Can the authors provide a stratified human audit of generated outputs from all three model settings, including per-task parser false-positive/false-negative rates and the effect of any corrections on the main comparisons? This would directly address my principal validity concern.
-
Is there an accessible anonymized release containing the exact frozen benchmark inputs, references, split identifiers, and a representative set of generated outputs, or a complete reconstruction procedure with all required frozen construction inputs? This would materially affect my assessment of the resource contribution.
-
Appendix C transparently identifies the results as test-set peaks and describes early monitoring on test questions. Can the authors distinguish exploratory peak analyses from a fixed-checkpoint or validation-selected evaluation, and provide uncertainty for the paired representation comparisons? The fixed-30k results already support the large learning gains, but smaller differences need a cleaner estimate.