acceptodds
Under review as a conference paper at ICLR 2027

OrigamiWorldBench: Benchmarking Visual Instruction-Conditioned Next-State Generation

Abstract

Recent advances in image editing models have largely centered on natural-language instructions. In real-world diagrams and manuals, however, actions are often specified visually through arrows, lines, and symbols and may induce state transitions of the target object. Generating the resulting state from such visual instructions therefore requires models both to interpret visually specified actions and to faithfully realize their geometric and structural consequences. To study this capability, we introduce OrigamiWorldBench, a benchmark for next-state generation conditioned on visually specified actions. Origami is well suited to this setting because origami operations can be visually specified and their outcomes obey clear geometric and structural constraints. Given a current-state image overlaid with a visual prompt and an auxiliary text instruction describing the operation, models directly generate the corresponding one-step-ahead state image. To evaluate geometric and structural fidelity, which is difficult to capture with conventional image-similarity metrics, we introduce an aspect-wise VLM-as-a-Judge protocol. Using OrigamiWorldBench, we evaluate a set of recent image generation and editing models. To quantify the gap between current models and human performance under the same visual instructions, we additionally compare model predictions with origami states physically folded by human participants. We further validate the proposed aspect-wise VLM evaluation against human judgments and conduct an input-cue analysis. We plan to release the benchmark dataset and code upon acceptance.

Primary area:
datasets and benchmarks
Keywords:
image generation, geometry-aware generation, visual prompt, vision-language models, benchmark

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_wacpOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

OrigamiWorldBench evaluates whether models can generate the next-state image from an origami image, operation markings, and text instructions. It contains 2,160 samples across seven operation types, evaluates 11 models, and validates a five-criterion VLM evaluation protocol against human judgments.

Critical strengths

1.The task is clearly defined. Origami provides a controlled setting for evaluating visual instruction understanding and geometric structure preservation.

2.Human validation supports the evaluation protocol: 35 participants evaluated 360 images, with correlations of Pearson r=0.802 and Spearman ρ=0.773 between average human and VLM scores.

Critical weaknesses

1.The incremental value is insufficiently established. VIBE already studies visual instruction-driven editing and the generation of visual consequences of actions. Although cited, the paper does not sufficiently explain which capability differences the origami setting reveals that existing benchmarks cannot capture, limiting its contribution as a standalone benchmark.

2.Dataset construction is insufficiently specified. Section 3.2 reports the overall process and counts, but does not fully specify the constituent states, transitions, configurations, marking variants, or their mappings. The transition deduplication criteria are also unclear. These omissions limit assessment of structural coverage, sample redundancy, and reproducibility.

3.Human comparisons and ablations require qualified interpretation. Humans physically manipulate paper, whereas models generate images; their performance gap cannot directly establish a reasoning gap under matched conditions. Likewise, removing arrows without reducing average performance does not uniquely support “limited use of directional information.” Cue redundancy and accompanying text-prompt changes remain alternative explanations (Sections 4.3 and 4.5; Appendix A.2).

Questions for authors

1.What distinct capabilities does this benchmark assess beyond VIBE, and which results demonstrate this added value?

2.Can the authors provide state–transition–configuration mappings, transition deduplication criteria, and all six marking variants for the same instance?

3.Can the authors clarify the role and comparability of the physical human reference, and distinguish observed ablation results from explanations of model behavior?

5 comments
AuthorsOpenReview ↗

Response to Reviewer wacp (1/2)

Thank you for your thoughtful and constructive review. We appreciate your concerns regarding the distinction from VIBE and our benchmark's added value (W1 & Q1), dataset construction and reproducibility (W2 & Q2), and the interpretation of human-folded references and ablation results (W3 & Q3). We have clarified our evaluation focus and dataset construction, added experimental analyses, and revised our interpretation of the results.

Note on the evaluation model

Because Gemini-3.7-Flash is no longer available, we use Gemini-3.8-Flash for additional evaluations. On the 360-sample analysis subset, the two judges produce identical rankings across 11 models (Spearman ρ = 1.000), with 95.2% aspect-level agreement (Cohen's κ = 0.837; Appendix E). The revised manuscript retains the original Gemini-3.7-Flash results for the main benchmark evaluation and specifies the judge used for each analysis. For the camera-ready version, we will re-evaluate the main benchmark outputs using Gemini-3.8-Flash and update the related analyses.

W1 & Q1. Distinction from VIBE and the added value of our benchmark

Thank you for this important comment. We acknowledge that VIBE evaluates a broad range of visual, instruction-driven image-editing tasks, including morphological transformations and the generation of physical or causal consequences. Our benchmark complements VIBE by focusing on the geometric and structural consequences of instructed state transitions in a controlled origami setting.

Specifically, our benchmark requires models to (1) ground fold lines and arrows in the current origami state, (2) infer the resulting state transition, and (3) generate the next state while preserving unaffected geometry and visible structure. We evaluate these requirements together within each task.

To examine the resulting errors in detail, we compare generated images with the ground-truth next state along five geometric and structural aspects. For example, in the full-benchmark evaluation using Gemini-3.7-Flash, GPT-Image-2 achieved an average Orientation score of 0.669, compared with 0.188 for Surface Structure and 0.290 for Visible Layering. These differences reveal variation across geometric and structural aspects that a single overall score does not capture.

We have clarified the distinction from VIBE in Section 2.1 and added a discussion of the capabilities and results highlighted by our benchmark in Section 4.2.

W2 & Q2. Dataset construction details, deduplication criteria, and reproducibility

Thank you for highlighting the need for greater clarity. We have revised Section 3.2 to clarify the dataset construction process and added canonical state illustrations, a transition table, and operation-mark style examples to Appendix A.

The dataset is organized as follows:

  1. Canonical states (63) (Section 3.2; Appendix A.1, Figure 4): We first created 63 canonical origami states. We considered two states equivalent if one could be transformed into the other by 90° rotations, horizontal reflection, and scaling. The 63 canonical states are distinct under these transformations.

  2. Unique transitions (78) (Section 3.2; Appendix A.1, Table 7): We count unique transitions using the ordered pair of canonical current- and next-state IDs. Entries with the same pair are grouped as one transition, even if their operations or orientations differ. For example, 90°, 180°, and 270° in-plane rotations of the same state retain the same canonical state ID and are counted as one transition. Table 7 lists the 78 unique state-ID pairs with their corresponding operation labels.

  3. Base configurations (360) (Section 3.2): We combined state images with operation marks and applied controlled rotations, reflections, and paper-color variations. We recorded the positions and orientations of the marks and kept them geometrically consistent with the corresponding state transition.

  4. Visual prompt variants (2,160 samples) (Section 3.2; Appendix A.2, Figure 5): Each base configuration was rendered with six visual prompt variants. These variants share the same current state, next state, action text, and underlying operation geometry, but differ in the visual rendering of the marks. Figure 5 illustrates the six operation-mark styles for each operation type, including differences in arrowhead shape, stroke appearance, and color.

The state IDs in Figure 4 correspond to the current- and next-state IDs in Table 7, making the relationships between canonical states and transitions explicit. Section 3.2 further explains how these transitions are expanded into 360 base configurations and then into 2,160 samples through the six marking variants.

These additions clarify the transition deduplication criteria and dataset construction process, making the dataset's structural coverage and sample redundancy easier to assess.

AuthorsOpenReview ↗

Response to Reviewer wacp (2/2)

W3 & Q3. Human reference and ablation results

1. Human reference: comparability and dataset validity

We agree that physical manipulation could give participants access to structures not visible in the model input. Thus, the human–model score gap cannot be interpreted as a reasoning gap under matched conditions. The experiment primarily examined whether humans could understand and execute the intended transitions using the visual prompts, action text, and physical paper, providing evidence about instruction clarity and dataset validity under these conditions.

“Human reference upper bound” was intended as a reference for a hypothetical model that could accurately infer the current state and perform physically faithful transitions. However, differences in available information, human folding errors, and evaluation variability prevent the human score from being a strict upper bound. Accordingly, we removed “human reference upper bound” and “Human–model performance comparison” from Section 4.3, separated human and model results into different tables, and clarified their different conditions. We also clarified the role of the human-folded results in the Abstract and Conclusion.

Additional human-drawn next-state study. To assess next-state inference without physical manipulation, four participants drew the expected next state for 30 samples—10 each for valley folds, mountain folds, and unfold operations—using the current-state image, visual prompt, and action text. An author visually compared the drawings with the ground truth and excluded three unanswered items.

OperationMatch rate with ground truth
Valley fold97.4%
Mountain fold73.7%
Unfold87.5%
Overall86.3%

Overall, 86.3% of the judged drawings matched the ground truth without physical manipulation. The lower match rate for mountain folds was mainly because drawings depicted valley folds instead. We added the study procedure and results to Section 4.3, with further details in Appendix B.6.

2. Ablation results: visual and textual cue changes

We agree that the limited effect of removing arrows does not necessarily mean that models ignore directional information. In the original ablation, we also changed the action text, making it difficult to separate the effects of visual and textual changes.

Additional cue ablation. We evaluated both models on the same 360-sample subset under two conditions: Default Text, which retains the original action text, and Matched Text, which changes the text to match the modified visual instruction. The action text specifies the operation but does not explicitly describe the arrow direction. We evaluated the generated images using Gemini-3.8-Flash.

ArrowFold lineAction textGPT-Image-2Gemini-3.1-Flash-Image
✓✓Default0.3260.228
✗✓Default0.3160.229
✓✗Default0.2580.169
✗✗Default0.2480.149
✗✓Matched0.2940.234
✓✗Matched0.2510.175
✗✗Matched0.2190.176

With Default Text, removing only the arrow had a small effect on the average score (GPT-Image-2: Δ −0.010; Gemini-3.1-Flash-Image: Δ +0.001), while removing only the fold line caused larger decreases (Δ −0.068 and Δ −0.059, respectively). For the same arrow-removed input, changing from Default to Matched Text reduced GPT-Image-2’s score by 0.022, but changed Gemini-3.1-Flash-Image’s score by +0.005. Thus, the text change affected the models differently.

Additional arrow-reversal experiment. To further examine sensitivity to arrow direction, we selected 229 samples where reversing the arrow direction changes the correct next state, including valley folds, mountain folds, pleat folds, and 90° or 270° rotations. For each sample, we reversed only the arrow direction while keeping the current-state image, fold line, and action text fixed. Using Gemini-3.8-Flash, scores against the original ground-truth images changed by Δ +0.003 for GPT-Image-2 and Δ −0.016 for Gemini-3.1-Flash-Image relative to the original-arrow condition.

In this ablation, fold-line removal had a larger average effect than arrow removal. The limited effect of arrow removal may reflect redundancy with the remaining fold line, which, together with the current state and action text, may allow the intended operation to be inferred in some cases. We added the detailed results of both the cue-removal and arrow-reversal experiments to Section 4.5 of the revised manuscript and revised the interpretation to distinguish the observed results from possible explanations of model behavior.

AuthorsOpenReview ↗

Brief Follow-up on Discussion

Dear Reviewer wacp,

Thank you very much for taking the time to review our submission. As the discussion phase approaches its deadline, we would like to send a brief reminder. If there are any points you would like us to elaborate on or any remaining questions we can address, we would be glad to respond. Thank you again for your helpful feedback and careful consideration.

Reviewer_6NCgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces OrigamiWorldBench, a benchmark for generating the next origami state from a current-state image with overlaid visual marks and an auxiliary text instruction. The authors propose an aspect-wise VLM-as-a-Judge protocol to evaluate geometric and structural fidelity, and use it to evaluate recent image generation and editing models. They also compare model outputs with human-folded results and validate the VLM judge against human ratings.

Critical strengths
  1. The benchmark is carefully constructed, with controlled transformations and mark variants that allow fine-grained analysis.

  2. The human-folded reference is a valuable addition. It verifies that the task is solvable and gives a meaningful upper bound.

  3. The evaluation protocol is validated against human judgments and compared with conventional metrics, which makes the results more credible.

Critical weaknesses
  1. The benchmark only covers origami, so it is hard to call it a "World" benchmark, and it is unclear whether the findings would generalize to other scenarios. In addition, the sample size is relatively small.

  2. The paper does not explore any way to improve performance. Only showing that current models perform poorly makes the contribution feel limited.

  3. The unedited input already scores higher than almost all models. Future methods might hack this metric to get high scores without actually performing the correct edit.

Questions for authors
  1. Whether using Gemini-3.7-Flash as the judge favors Gemini-3.1-Flash-Image, since they come from the same family?

  2. Why is Gemini-3.7-Flash cited as OpenAI?

  3. The paper only reports the unedited-input score overall (Table 3). Could the authors also report it for each operation type in Table 2?

8 comments
AuthorsOpenReview ↗

Response to Reviewer 6NCg (1/3)

Thank you for your thoughtful and constructive review. We appreciate your concerns regarding the benchmark scope and dataset size (W1), directions for improving model performance (W2), the unedited-input baseline (W3 and Q3), and the choice of VLM judge (Q1). We address these points through additional analyses, clarify the benchmark's contributions and limitations, and correct the citation error identified in Q2.

Note on the evaluation model

Because Gemini-3.7-Flash is no longer available, we use Gemini-3.8-Flash for additional evaluations. On the 360-sample analysis subset, the two judges produce identical rankings across 11 models (Spearman ρ = 1.000), with 95.2% aspect-level agreement (Cohen's κ = 0.837; Appendix E). The revised manuscript retains the original Gemini-3.7-Flash results for the main benchmark evaluation and specifies the judge used for each analysis. For the camera-ready version, we will re-evaluate the main benchmark outputs using Gemini-3.8-Flash and update the related analyses.

W1. Benchmark scope and dataset size

Thank you for this comment. We acknowledge that the term "World" may suggest broader domain coverage than our origami-focused benchmark provides. We have therefore renamed the benchmark "OrigamiGenBench" and revised the title to "OrigamiGenBench: Benchmarking Visual Instruction-Conditioned Next-State Generation."

Our goal is not to cover arbitrary real-world state transitions, but to use origami as a controlled testbed for evaluating whether models can interpret visually specified actions and faithfully generate their geometric and structural consequences. Extending this evaluation to other visually instructed transformations would let us examine these capabilities across a broader range of scenarios.

Regarding dataset size, our benchmark contains 2,160 samples, comparable in scale to related image generation and editing benchmarks, including GenPuzzle (2,005), GGBench (1,411), and KRIS-Bench (1,267). These 2,160 samples comprise six visual prompt variants for each of 360 base configurations derived from 78 unique transitions. This design enables systematic analysis of sensitivity to the appearance of visual instructions while keeping the underlying state transition and action semantics fixed.

W2. Performance bottlenecks and a potential improvement framework

Thank you for this comment. Although our primary contribution is an evaluation benchmark, our analyses reveal three key challenges that suggest concrete directions for improving current models.

  1. Visual instruction grounding: In response to another reviewer’s suggestion, we expanded our input-cue analysis and found that removing fold lines reduced performance for both evaluated models under Gemini-3.8-Flash, highlighting the importance of grounding visual cues in the current state.

  2. Geometric state-transition reasoning: In the original full-benchmark evaluation using Gemini-3.7-Flash, GPT-Image-2 performed substantially better on Rotation (0.735) and Crease (0.931) than on Valley Fold (0.256) and Pleat Fold (0.303), revealing substantial performance differences across operation types.

  3. Structure-preserving generation: In the same evaluation, despite achieving an Orientation score of 0.669, GPT-Image-2 scored much lower on Surface Structure (0.188) and Visible Layering (0.290), indicating difficulty generating structurally correct next states.

These results suggest that jointly grounding visual instructions, reasoning about geometric state transitions, and generating structurally faithful images remains challenging for current end-to-end models. To address these challenges, we discuss a possible agentic framework for future work that decomposes next-state generation into four stages:

  1. Instruction Grounding: Convert visual marks and action text into a structured representation of the target region, fold axis, and direction.

  2. Geometric Reasoning: Predict the resulting geometry and layer configuration.

  3. Structure-Preserving Generation: Generate the next-state image while maintaining unaffected regions.

  4. Verification and Refinement: Check consistency with the original instructions and predicted geometry, and refine the generated result without accessing the ground-truth image.

This decomposition provides a concrete, testable direction for improving next-state generation. We added a discussion to Section 4.6 of the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer 6NCg (2/3)

W3 & Q3. Re-evaluation and operation-wise analysis of unedited-input baselines

Thank you for this important comment. In the original manuscript, we evaluated the current-state image without visual annotations (Input State) as the unedited-input baseline. To directly assess the strategy of reproducing the model's input, we additionally evaluated the annotated input image, including arrows and fold lines (Visual Prompt), as the predicted output.

The following table reports the aspect-wise and overall scores of both baselines and the two highest-scoring models, all evaluated on the same 360-sample analysis subset using Gemini-3.8-Flash.

Model / BaselineSilhouetteInternal LinesOrientationSurface StructureVisible LayeringOverall
GPT-Image-20.3860.2250.6310.1440.2440.326
Gemini-3.1-Flash-Image0.2940.1280.4560.0940.1690.228
Visual Prompt0.2500.2330.4140.1190.1580.235
Input State0.3640.1500.3640.0750.1000.211

To address Q3, we also report the operation-wise scores of both baselines below, evaluated on the same subset using the same judge. The operation macro average is the unweighted mean across the seven operation types.

BaselineValley FoldUnfoldMountain FoldFlipRotationCreasePleat FoldMacro Avg.
Visual Prompt0.1160.1600.1870.1770.7710.6670.0500.304
Input State0.1300.1930.1790.3490.4240.3500.1330.251

The Visual Prompt baseline obtains relatively high scores for Rotation and Crease. We agree that, as the reviewer points out, a model may receive substantial credit for some operations by returning the input unchanged without performing the instructed edit. The scores measure agreement with the target across individual aspects, so an unchanged input can receive credit for aspects that already match. Therefore, a relatively high score alone does not establish successful execution of the instructed edit.

Under the same evaluation conditions, GPT-Image-2 achieves an overall score of 0.326, exceeding the Visual Prompt baseline of 0.235. Reporting this baseline lets us assess whether future models improve on the simple input-copying strategy, although it does not eliminate the possibility of exploiting the metric.

Our aspect-wise evaluation also helps identify model strengths and weaknesses when the overall score is below the baseline. For example, Gemini-3.1-Flash-Image scores below Visual Prompt overall (0.228 vs. 0.235), but exceeds it on Orientation (0.456 vs. 0.414). Thus, the evaluation provides diagnostic information about which aspects each model reproduces successfully and which remain challenging.

In Section 4.2 (Benchmark Results) of the revised manuscript, we use Visual Prompt as the primary unedited-input baseline and clarify that overall scores should be interpreted alongside the baseline and aspect-wise results. We also report the operation-wise and step-wise scores for both Visual Prompt and Input State in Appendix D.

AuthorsOpenReview ↗

Response to Reviewer 6NCg (3/3)

Q1. Judge model selection and cross-model consistency

We selected Gemini-3.7-Flash as a representative VLM judge in the original manuscript. However, our evaluation protocol is model-agnostic and can be applied to other VLMs. To assess consistency across judges, we also evaluated the outputs using Gemini-3.8-Flash, GPT-5.5, and Claude Sonnet 5.5. We applied the same five-aspect evaluation criteria to outputs from 11 image generation models on 360 samples and compared their rankings and inter-judge agreement.

Image Generation ModelGemini 3.7Gemini 3.8GPT-5.5Claude Sonnet 5.5
GPT-Image-21111
Gemini-3.1-Flash-Image2232
FLUX.1-Kontext-dev3323
GPT-Image-1.54468
OmniGen25555
FLUX.2-klein-4B6644
Qwen-Image-Edit-25117787
FLUX.2-klein-9B8876
JoyAI-Image-Edit9999
FLUX.2-dev10101110
LongCat-Image-Edit11111011

GPT-Image-2 ranked first under all four judges. Gemini-3.1-Flash-Image ranked second not only under Gemini-family judges but also under Claude Sonnet 5.5, and third under GPT-5.5, suggesting that its strong relative performance is not specific to Gemini-family judges. Furthermore, Gemini-3.7-Flash showed high model-level agreement with GPT-5.5 (Pearson r = 0.956; Spearman ρ = 0.936) and Claude Sonnet 5.5 (r = 0.961; ρ = 0.891). The agreement rate across 19,800 binary aspect-level judgments between Gemini-3.7-Flash and Claude Sonnet 5.5 was 90.8%.

These results support broad consistency in the leading model rankings and performance trends across different VLM families, although individual model positions and absolute scores vary across judges. We have added the detailed cross-judge comparisons to Appendix E, with a reference in Section 4.1 of the revised manuscript.

Q2. Correction of the Gemini-3.7-Flash citation

Thank you for pointing this out. This was a citation error. We have corrected the Gemini-3.7-Flash citation in Section 4.1 (Experimental Setup) and the reference list of the revised manuscript.

Reviewer_5ARpOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

OrigamiWorldBench evaluates whether image generation and editing models can interpret visually specified origami operations and generate the corresponding next-state image. It contains 2,160 samples derived from 78 unique transitions, covering seven operations with controlled variations in orientation, paper color, and instruction-mark appearance. The paper introduces a five-aspect VLM evaluation covering silhouette, internal lines, orientation, surface structure, and layering. Experiments evaluate 11 models, compare their outputs with physically folded human references, validate automatic scores against human judgments, and analyze the effects of removing input cues. Results reveal substantial difficulties in preserving geometric and structural fidelity.

Critical strengths

The benchmark targets a useful combination of capabilities: interpreting diagrammatic actions and generating their geometric consequences, rather than merely selecting a correct answer. Its controlled construction separates underlying transitions from instruction-mark appearance, enabling targeted robustness analysis. The evaluation is also supported by substantial validation: human judgments on 360 outputs, comparisons with conventional image-similarity metrics, and an aspect-wise versus holistic scoring comparison. The unedited-input baseline and operation-level breakdowns provide useful context for interpreting model failures.

Critical weaknesses

The main concerns are the evaluator’s ability to distinguish models and the interpretation of the visual-grounding evidence. The reported human–VLM correlation pools outputs from seven generative models with substantially higher-scoring human-folded references; it may therefore partly reflect broad differences between sources rather than reliable discrimination among model outputs. Model-only correlations and agreement on model rankings are needed to establish the metric’s suitability for benchmarking. In addition, the aggregate arrow-removal experiment does not distinguish failure to interpret arrows from cases where direction is redundant or geometrically equivalent. The human-folding comparison also provides participants with physical folded states, offering access to structure and manipulation feedback unavailable from the models’ single images.

Questions for authors
  1. What are the human–VLM correlations after excluding human-folded references, and how closely do human-derived model rankings agree with the automatic rankings?

  2. For cases where arrow direction changes the correct target, how does performance change when the arrow is reversed while the current state, fold line, and action text remain fixed?

  3. Were folding participants permitted to inspect or manipulate the physical current state before committing to an interpretation?

  4. Is there an image-only human assessment of whether the intended next state is uniquely inferable, particularly for hidden layers and unfolding operations?

  5. Since the 2,160 samples derive from 78 unique transitions, do model rankings and performance differences remain stable under confidence intervals clustered by underlying transition?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 5ARp (1/3)

Thank you for your thoughtful and constructive review. We appreciate your concerns regarding the reliability of our VLM-based evaluation (Q1), the interpretation of directional cues (Q2), the role of human-folded references and image-only inferability (Q3–Q4), and the stability of model rankings under transition-level clustering (Q5). We address these concerns through additional analyses and experiments, and clarify the interpretation of our findings below.

Note on the evaluation model

Because Gemini-3.7-Flash is no longer available, we use Gemini-3.8-Flash for additional evaluations. On the 360-sample analysis subset, the two judges produce identical rankings across 11 models (Spearman ρ = 1.000), with 95.2% aspect-level agreement (Cohen's κ = 0.837; Appendix E). The revised manuscript retains the original Gemini-3.7-Flash results for the main benchmark evaluation and specifies the judge used for each analysis. For the camera-ready version, we will re-evaluate the main benchmark outputs using Gemini-3.8-Flash and update the related analyses.

Q1. Human–VLM correlation and model ranking agreement

Thank you for raising this concern. The following analyses are based on the Gemini-3.7-Flash evaluation results used in the original manuscript.

After excluding the 45 human-folded images from the 360-image human-evaluation set, the Pearson and Spearman correlations for the remaining 315 model-generated images were 0.684 and 0.673, respectively, compared with 0.802 and 0.773 when including the human-folded references. This decrease suggests that the higher-scoring human-folded references contributed to the overall correlation. Nevertheless, the correlations remain moderately strong, indicating meaningful agreement with human judgments even when restricted to model-generated outputs.

The Spearman rank correlation between human-derived and VLM-derived rankings across the seven models was 0.714. GPT-Image-2 ranked first under both evaluations, while the remaining models differed by up to two positions.

We have added these model-only correlations and the ranking comparison to Section 4.4 of the revised manuscript.

Q2. Effect of reversing arrow directions

Thank you for this suggestion. To examine the role of directional cues, we selected 229 samples from the 360-sample analysis subset for which reversing the arrow direction changes the correct next state. These samples involve Valley Fold, Mountain Fold, Pleat Fold, and Rotation (90° and 270°). We reversed only the arrow directions while keeping the current-state image, fold lines, and action text fixed, and evaluated the generated outputs against the original ground-truth images using Gemini-3.8-Flash. This analysis measures changes in agreement with the original targets; it does not directly measure correctness under the reversed instructions.

ModelOriginalReversedΔ
GPT-Image-20.2690.272+0.003
Gemini-3.1-Flash-Image0.2260.210−0.016

The average scores against the original ground truth changed only slightly after arrow reversal. Although score changes varied across operation types, we observed no consistent decrease.

As a complementary analysis, we evaluated cue removal on the 360-sample analysis subset while retaining the original action text, which specifies the operation but not the arrow direction. Using the same judge, we measured score changes relative to the original-input condition.

ModelArrow removed (Δ)Fold line removed (Δ)
GPT-Image-2−0.010−0.068
Gemini-3.1-Flash-Image+0.001−0.059

Fold-line removal had a larger average effect than arrow removal under the tested conditions. The limited effect of arrow removal may reflect redundancy with the remaining fold line, current state, and action text.

We have added the cue-removal and arrow-reversal results to Section 4.5 of the revised manuscript and revised their interpretation accordingly.

AuthorsOpenReview ↗

Response to Reviewer 5ARp (2/3)

Q3. Human-folding protocol and experimental purpose

Participants were not explicitly instructed to inspect the physical current state before folding, but inspection and manipulation, including turning the paper over, were not prohibited. Participants could therefore access structural information and manipulation feedback unavailable from the models’ single input images. Thus, under matched conditions, the human–model score gap cannot be interpreted as a reasoning gap.

However, the primary purpose of the human-folded experiment was not to compare human and model performance under strictly matched conditions, but to examine whether participants could understand and execute the intended state transitions from the visual prompts and action text. We intended this experiment to provide evidence about dataset validity by assessing whether the instructions were understandable to humans.

We have therefore replaced the “human reference upper bound” framing with a “human-folded reference” and removed “Human–model performance comparison” from Section 4.3. We now present the human-folded and model results in separate tables and explicitly clarify their different experimental conditions. We also revised the Abstract and Conclusion to reflect this interpretation. Appendix B.4 provides details on physical manipulation.

We additionally conducted a human-drawn next-state study without physical manipulation, as described in our response to Q4.

Q4. Human-drawn next-state study

Thank you for this suggestion. To test whether participants can infer the intended next state without physical manipulation, we conducted an additional human-drawn next-state study.

Four participants were asked to draw the expected next state for 30 samples—10 each for Valley Fold, Mountain Fold, and Unfold— given the current-state image, visual prompt, and action text, without physically folding paper. An author visually compared each drawing with the ground-truth image and assigned a binary match judgment. Three unanswered items were excluded.

OperationMatch rate with ground truth
Valley fold97.4%
Mountain fold73.7%
Unfold87.5%
Overall86.3%

Overall, 86.3% of the judged drawings matched the ground truth without physical manipulation. The lower match rate for mountain folds was mainly because drawings depicted valley folds instead. These results provide evidence of image-only inferability in many of the tested cases. Our assessment focuses on agreement with the intended next state, rather than an exhaustive evaluation of alternative valid interpretations.

We have added the study procedure and results to Section 4.3, with further details in Appendix B.6.

AuthorsOpenReview ↗

Response to Reviewer 5ARp (3/3)

Q5. Stability of model rankings and performance differences under transition-level clustering

Thank you for this suggestion. We conducted two complementary analyses using the full 2,160-sample benchmark results evaluated with Gemini-3.7-Flash to assess the effect of dependencies among samples sharing the same underlying transition.

First, we computed the mean score for each of the 78 unique transitions and calculated the Transition Mean ± SD, weighting all transitions equally. We also ranked the 11 models within each transition and calculated the Transition Rank ± SD across transitions. These standard deviations describe variation across transitions. We compared these results with the original Overall Mean and Overall Rank, calculated across all 2,160 samples.

Second, we performed a transition-clustered bootstrap with 5,000 resamples and seed 0. In each iteration, we sampled 78 transitions with replacement, using the same sampled transitions for all models, retained their constituent samples, and recomputed the sample-count-weighted overall scores. We obtained Clustered 95% CIs from the 2.5th and 97.5th percentiles and additionally computed confidence intervals for pairwise score differences and bootstrap ranking distributions.

ModelOverall MeanOverall RankTransition Mean ± SDTransition Rank ± SDClustered 95% CI
GPT-Image-20.37110.411 ± 0.2702.33 ± 1.66[0.312, 0.437]
Gemini-3.1-Flash-Image0.25020.284 ± 0.1914.31 ± 2.90[0.212, 0.291]
FLUX.1-Kontext-dev0.22830.254 ± 0.2074.66 ± 2.02[0.184, 0.277]
GPT-Image-1.50.21040.230 ± 0.1725.33 ± 3.12[0.178, 0.246]
OmniGen20.19050.213 ± 0.1975.97 ± 2.72[0.149, 0.237]
FLUX.2-klein-4B0.18960.206 ± 0.2315.81 ± 2.64[0.140, 0.243]
FLUX.2-klein-9B0.18370.203 ± 0.2056.04 ± 2.73[0.136, 0.238]
Qwen-Image-Edit-25110.17680.205 ± 0.2426.94 ± 2.81[0.130, 0.231]
JoyAI-Image-Edit0.13290.169 ± 0.2087.38 ± 2.72[0.102, 0.166]
FLUX.2-dev0.099100.127 ± 0.1888.33 ± 2.93[0.067, 0.137]
LongCat-Image-Edit0.091110.093 ± 0.1158.88 ± 1.69[0.066, 0.120]

Ranking models by their equally weighted Transition Mean produced nearly the same ordering as the original overall ranking, with only the seventh- and eighth-ranked models exchanging positions. The top three remained GPT-Image-2, Gemini-3.1-Flash-Image, and FLUX.1-Kontext-dev.

The cluster bootstrap further supported the leading model’s stability: GPT-Image-2 ranked first in 100% of resamples, and its pairwise score differences against all ten other models had 95% CIs excluding zero. Its advantage over Gemini-3.1-Flash-Image was 0.120 (95% CI: [0.073, 0.173]). Gemini-3.1-Flash-Image ranked second in 84.1% of resamples, while FLUX.1-Kontext-dev ranked third in 64.5%.

These results support the leading model's stability and the main performance trends, even after accounting for dependencies among samples sharing the same underlying transition.

We have added the transition-level statistics, cluster-bootstrap confidence intervals, and summaries of the paired comparisons and ranking stability to Appendix F of the revised manuscript.

AuthorsOpenReview ↗

Brief Follow-up on Discussion

Dear Reviewer 5ARp,

Thank you very much for taking the time to review our submission. As the discussion phase approaches its deadline, we would like to send a brief reminder. If there are any points you would like us to elaborate on or any remaining questions we can address, we would be glad to respond. Thank you again for your helpful feedback and careful consideration.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Discussion

Dear ACs and Reviewers,

We sincerely thank all reviewers for their careful consideration and valuable feedback throughout the review and discussion. Reviewers recognized the benchmark’s controlled construction and its value for evaluating visual instruction understanding and preserving geometric structure. They also highlighted validation against human judgments and the usefulness of the unedited-input baseline and operation-wise analyses for interpreting model failures. Two of the three reviewers gave weak acceptance ratings. We have carefully addressed all reviewer comments through revisions, additional experiments, and clarifications, as summarized below.

Main points addressed

  1. Scope and dataset construction (6NCg W1; wacp W1–W2/Q1–Q2). We renamed the benchmark OrigamiGenBench, clarified its focus on geometric and structural state transitions and its distinction from prior benchmarks, and detailed the construction process and deduplication criteria.

  2. Evaluator validity (5ARp Q1; 6NCg Q1). Human–VLM correlations restricted to model-generated images were r = 0.684 and ρ = 0.673. The leading model ranked first under both human and automatic evaluations and across all four VLM judges. The second-ranked model also placed second and third under judges from other model families, supporting its strong relative performance beyond same-family evaluation.

  3. Transition-level robustness (5ARp Q5). Equal transition weighting preserved the model ordering except for a seventh/eighth-place swap. The leading model ranked first in all 5,000 transition-clustered bootstrap resamples, and its paired score differences against every other model had 95% confidence intervals excluding zero.

  4. Human references and image-only assessment (5ARp Q3–Q4; wacp W3/Q3). We clarified the role of human-folded results as a reference for instruction understandability and execution, acknowledging differences in available information. In an additional image-only drawing study, 86.3% of judged drawings matched the ground truth, supporting image-only inferability in many tested cases.

  5. Cue contributions and interpretation (5ARp Q2; wacp W3/Q3). We added cue-removal experiments with action text fixed and arrow-reversal experiments. Fold-line removal affected performance more than arrow removal, while arrow reversal produced no consistent decrease in agreement with the original targets. We distinguished observations from interpretations, including the possibility of inferring the operation from the remaining cues.

  6. Unedited inputs and action consistency (6NCg W3/Q3). We introduced an Action Consistency gate assessing whether the intended operation was attempted, enabling penalties for unedited outputs and unrelated changes. This criterion complements the original five-aspect evaluation, whose findings remain unchanged. Human–VLM correlations for five-aspect scores on annotated unedited inputs were r = 0.660 and ρ = 0.649, close to those for model-generated images.

  7. Directions for improvement (6NCg W2). We discussed an agentic framework separating instruction grounding, geometric reasoning, generation, and verification/refinement as a testable direction for future improvement.

Revisions (in red) and planned additions

  • Title, Sections 2–3, and Appendix A: clarified scope, contribution, dataset construction, and deduplication.

  • Abstract, Section 4, Conclusion, and Appendices B, D–F: clarified human references; added evaluator validation, baseline analyses, the image-only study, controlled cue experiments, and transition-level robustness analyses; discussed improvement directions.

  • Planned additions: report five-aspect scores, Action Consistency rates, and Gated scores; clarify their complementary roles; add unedited-input human evaluation and state-preservation results. As the original judge is unavailable, we will re-evaluate the main outputs with the updated judge used for additional evaluations.

We sincerely thank the reviewers for helping us strengthen the evaluation and clarify the benchmark’s contribution and findings.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.