- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
OrigamiWorldBench evaluates whether models can generate the next-state image from an origami image, operation markings, and text instructions. It contains 2,160 samples across seven operation types, evaluates 11 models, and validates a five-criterion VLM evaluation protocol against human judgments.
Critical strengths
1.The task is clearly defined. Origami provides a controlled setting for evaluating visual instruction understanding and geometric structure preservation.
2.Human validation supports the evaluation protocol: 35 participants evaluated 360 images, with correlations of Pearson r=0.802 and Spearman ρ=0.773 between average human and VLM scores.
Critical weaknesses
1.The incremental value is insufficiently established. VIBE already studies visual instruction-driven editing and the generation of visual consequences of actions. Although cited, the paper does not sufficiently explain which capability differences the origami setting reveals that existing benchmarks cannot capture, limiting its contribution as a standalone benchmark.
2.Dataset construction is insufficiently specified. Section 3.2 reports the overall process and counts, but does not fully specify the constituent states, transitions, configurations, marking variants, or their mappings. The transition deduplication criteria are also unclear. These omissions limit assessment of structural coverage, sample redundancy, and reproducibility.
3.Human comparisons and ablations require qualified interpretation. Humans physically manipulate paper, whereas models generate images; their performance gap cannot directly establish a reasoning gap under matched conditions. Likewise, removing arrows without reducing average performance does not uniquely support “limited use of directional information.” Cue redundancy and accompanying text-prompt changes remain alternative explanations (Sections 4.3 and 4.5; Appendix A.2).
Questions for authors
1.What distinct capabilities does this benchmark assess beyond VIBE, and which results demonstrate this added value?
2.Can the authors provide state–transition–configuration mappings, transition deduplication criteria, and all six marking variants for the same instance?
3.Can the authors clarify the role and comparability of the physical human reference, and distinguish observed ablation results from explanations of model behavior?
5 comments
Response to Reviewer wacp (2/2)
W3 & Q3. Human reference and ablation results
1. Human reference: comparability and dataset validity
We agree that physical manipulation could give participants access to structures not visible in the model input. Thus, the human–model score gap cannot be interpreted as a reasoning gap under matched conditions. The experiment primarily examined whether humans could understand and execute the intended transitions using the visual prompts, action text, and physical paper, providing evidence about instruction clarity and dataset validity under these conditions.
“Human reference upper bound” was intended as a reference for a hypothetical model that could accurately infer the current state and perform physically faithful transitions. However, differences in available information, human folding errors, and evaluation variability prevent the human score from being a strict upper bound. Accordingly, we removed “human reference upper bound” and “Human–model performance comparison” from Section 4.3, separated human and model results into different tables, and clarified their different conditions. We also clarified the role of the human-folded results in the Abstract and Conclusion.
Additional human-drawn next-state study. To assess next-state inference without physical manipulation, four participants drew the expected next state for 30 samples—10 each for valley folds, mountain folds, and unfold operations—using the current-state image, visual prompt, and action text. An author visually compared the drawings with the ground truth and excluded three unanswered items.
| Operation | Match rate with ground truth |
|---|---|
| Valley fold | 97.4% |
| Mountain fold | 73.7% |
| Unfold | 87.5% |
| Overall | 86.3% |
Overall, 86.3% of the judged drawings matched the ground truth without physical manipulation. The lower match rate for mountain folds was mainly because drawings depicted valley folds instead. We added the study procedure and results to Section 4.3, with further details in Appendix B.6.
2. Ablation results: visual and textual cue changes
We agree that the limited effect of removing arrows does not necessarily mean that models ignore directional information. In the original ablation, we also changed the action text, making it difficult to separate the effects of visual and textual changes.
Additional cue ablation. We evaluated both models on the same 360-sample subset under two conditions: Default Text, which retains the original action text, and Matched Text, which changes the text to match the modified visual instruction. The action text specifies the operation but does not explicitly describe the arrow direction. We evaluated the generated images using Gemini-3.8-Flash.
| Arrow | Fold line | Action text | GPT-Image-2 | Gemini-3.1-Flash-Image |
|---|---|---|---|---|
| ✓ | ✓ | Default | 0.326 | 0.228 |
| ✗ | ✓ | Default | 0.316 | 0.229 |
| ✓ | ✗ | Default | 0.258 | 0.169 |
| ✗ | ✗ | Default | 0.248 | 0.149 |
| ✗ | ✓ | Matched | 0.294 | 0.234 |
| ✓ | ✗ | Matched | 0.251 | 0.175 |
| ✗ | ✗ | Matched | 0.219 | 0.176 |
With Default Text, removing only the arrow had a small effect on the average score (GPT-Image-2: Δ −0.010; Gemini-3.1-Flash-Image: Δ +0.001), while removing only the fold line caused larger decreases (Δ −0.068 and Δ −0.059, respectively). For the same arrow-removed input, changing from Default to Matched Text reduced GPT-Image-2’s score by 0.022, but changed Gemini-3.1-Flash-Image’s score by +0.005. Thus, the text change affected the models differently.
Additional arrow-reversal experiment. To further examine sensitivity to arrow direction, we selected 229 samples where reversing the arrow direction changes the correct next state, including valley folds, mountain folds, pleat folds, and 90° or 270° rotations. For each sample, we reversed only the arrow direction while keeping the current-state image, fold line, and action text fixed. Using Gemini-3.8-Flash, scores against the original ground-truth images changed by Δ +0.003 for GPT-Image-2 and Δ −0.016 for Gemini-3.1-Flash-Image relative to the original-arrow condition.
In this ablation, fold-line removal had a larger average effect than arrow removal. The limited effect of arrow removal may reflect redundancy with the remaining fold line, which, together with the current state and action text, may allow the intended operation to be inferred in some cases. We added the detailed results of both the cue-removal and arrow-reversal experiments to Section 4.5 of the revised manuscript and revised the interpretation to distinguish the observed results from possible explanations of model behavior.
Brief Follow-up on Discussion
Dear Reviewer wacp,
Thank you very much for taking the time to review our submission. As the discussion phase approaches its deadline, we would like to send a brief reminder. If there are any points you would like us to elaborate on or any remaining questions we can address, we would be glad to respond. Thank you again for your helpful feedback and careful consideration.
Response to Reviewer wacp (1/2)
Thank you for your thoughtful and constructive review. We appreciate your concerns regarding the distinction from VIBE and our benchmark's added value (W1 & Q1), dataset construction and reproducibility (W2 & Q2), and the interpretation of human-folded references and ablation results (W3 & Q3). We have clarified our evaluation focus and dataset construction, added experimental analyses, and revised our interpretation of the results.
Note on the evaluation model
Because Gemini-3.7-Flash is no longer available, we use Gemini-3.8-Flash for additional evaluations. On the 360-sample analysis subset, the two judges produce identical rankings across 11 models (Spearman ρ = 1.000), with 95.2% aspect-level agreement (Cohen's κ = 0.837; Appendix E). The revised manuscript retains the original Gemini-3.7-Flash results for the main benchmark evaluation and specifies the judge used for each analysis. For the camera-ready version, we will re-evaluate the main benchmark outputs using Gemini-3.8-Flash and update the related analyses.
W1 & Q1. Distinction from VIBE and the added value of our benchmark
Thank you for this important comment. We acknowledge that VIBE evaluates a broad range of visual, instruction-driven image-editing tasks, including morphological transformations and the generation of physical or causal consequences. Our benchmark complements VIBE by focusing on the geometric and structural consequences of instructed state transitions in a controlled origami setting.
Specifically, our benchmark requires models to (1) ground fold lines and arrows in the current origami state, (2) infer the resulting state transition, and (3) generate the next state while preserving unaffected geometry and visible structure. We evaluate these requirements together within each task.
To examine the resulting errors in detail, we compare generated images with the ground-truth next state along five geometric and structural aspects. For example, in the full-benchmark evaluation using Gemini-3.7-Flash, GPT-Image-2 achieved an average Orientation score of 0.669, compared with 0.188 for Surface Structure and 0.290 for Visible Layering. These differences reveal variation across geometric and structural aspects that a single overall score does not capture.
We have clarified the distinction from VIBE in Section 2.1 and added a discussion of the capabilities and results highlighted by our benchmark in Section 4.2.
W2 & Q2. Dataset construction details, deduplication criteria, and reproducibility
Thank you for highlighting the need for greater clarity. We have revised Section 3.2 to clarify the dataset construction process and added canonical state illustrations, a transition table, and operation-mark style examples to Appendix A.
The dataset is organized as follows:
Canonical states (63) (Section 3.2; Appendix A.1, Figure 4): We first created 63 canonical origami states. We considered two states equivalent if one could be transformed into the other by 90° rotations, horizontal reflection, and scaling. The 63 canonical states are distinct under these transformations.
Unique transitions (78) (Section 3.2; Appendix A.1, Table 7): We count unique transitions using the ordered pair of canonical current- and next-state IDs. Entries with the same pair are grouped as one transition, even if their operations or orientations differ. For example, 90°, 180°, and 270° in-plane rotations of the same state retain the same canonical state ID and are counted as one transition. Table 7 lists the 78 unique state-ID pairs with their corresponding operation labels.
Base configurations (360) (Section 3.2): We combined state images with operation marks and applied controlled rotations, reflections, and paper-color variations. We recorded the positions and orientations of the marks and kept them geometrically consistent with the corresponding state transition.
Visual prompt variants (2,160 samples) (Section 3.2; Appendix A.2, Figure 5): Each base configuration was rendered with six visual prompt variants. These variants share the same current state, next state, action text, and underlying operation geometry, but differ in the visual rendering of the marks. Figure 5 illustrates the six operation-mark styles for each operation type, including differences in arrowhead shape, stroke appearance, and color.
The state IDs in Figure 4 correspond to the current- and next-state IDs in Table 7, making the relationships between canonical states and transitions explicit. Section 3.2 further explains how these transitions are expanded into 360 base configurations and then into 2,160 samples through the six marking variants.
These additions clarify the transition deduplication criteria and dataset construction process, making the dataset's structural coverage and sample redundancy easier to assess.