acceptodds
Withdrawn from ICLR 2027

Can AI Speak the Language of Origami?

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Building AI systems that can plan, act, and create in the physical world requires more than pattern recognition. Such systems must reason about the generative mechanisms and constraints that govern physical processes, using structured representations that connect observations, actions, and their effects. Yet existing benchmarks often study these capabilities in isolation, focusing either on visual recognition or on abstract symbolic and programmatic reasoning. Origami offers a natural testbed for integrating these abilities: constructing shapes through sequences of folds requires visual perception, reasoning about geometric and physical constraints, and multi-step planning, while remaining sufficiently structured for systematic evaluation. We introduce OrigamiBench, a benchmark for evaluating programmatic understanding of the mechanisms underlying origami synthesis and the ability to induce such mechanisms in a high-level language of physically grounded fold actions. Experiments with modern vision-language models show that increasing model scale alone does not reliably improve reasoning about physical transformations. Moreover, models struggle to ground programmatic representations in visual observations, indicating that their visual and linguistic representations remain only weakly integrated.

Primary area:
datasets and benchmarks
Keywords:
programmatic representations, program synthesis, computational origami, temporal reasoning, planning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_LK6XOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes OrigamiBench, a new benchmark for testing whether vision-language models can understand origami as a sequence of physically grounded actions that transform a sheet of paper into a target shape. To support this, the authors introduce a high-level executable language that represents origami as sequences of formal folding operations. The benchmark contains manually annotated, randomly generated, and LLM-generated folding trajectories, with all visual observations rendered synthetically. Models are evaluated on tasks ranging from predicting or inferring individual folds and resulting states to reasoning over multi-step folding sequences and generating a complete sequence of folds that transforms an initial state into a target state. The results show that while current models can often reason about individual folding steps or infer plausible local transformations, they lack a robust causal understanding of the folding process and struggle to plan or execute sequences of folds over multiple steps.

Critical strengths
  1. The problem is interesting, and the paper introduces a distinctive evaluation of leading VLMs on executable sequential origami reasoning, including grounding visual states in fold-action programs and inducing folding procedures. While prior benchmarks have already evaluated VLMs on origami-based spatial reasoning, this work appears novel in its emphasis on executable action-sequence reasoning and synthesis.

  2. The benchmark contains a diverse set of trajectories, including manually annotated human-designed origami procedures as well as randomly and LLM-generated procedures, although all visual observations are synthetically rendered and limited to 2D.

  3. The evaluation is broad, testing models across increasingly demanding levels of reasoning, from single-step state/action understanding to multi-step execution and ordering, plan completion and steering, and finally generation of a complete folding program from initial and target visual states.

Critical weaknesses
  1. The scope of the benchmark is relatively narrow, focusing on sequential action reasoning within 2D flat-foldable origami.

  2. The work focuses exclusively on evaluating existing VLMs and does not investigate methods for training models on the task. Hence, while the paper identifies limitations of current models, it does not propose methods for improving performance on the tasks it introduces.

  3. While Origami is the art of turning 2D sheets to 3D structure, the benchmark is restricted to flat-foldable, essentially 2D origami and does not evaluate genuinely 3D folded structures. Although the authors mention 3D origami as future work, it remains unclear whether the proposed language and tests and results naturally extend to non-flat 3D configurations.

  4. All images presented to the VLMs are synthetically rendered. This raises questions about how well the results transfer to real photographs or videos. Moreover, the renderings appear relatively schematic rather than photorealistic, so the benchmark may emphasize reasoning over an abstract visual representation that does not translate to real world visual conditions.

  5. The paper does not include a human baseline. Human evaluation would help calibrate task difficulty and clarify whether low model performance reflects limitations of current VLMs or intrinsic difficulty or ambiguity in the benchmark itself.

Questions for authors

See weaknesses section

Reviewer_5xHoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces OrigamiBench, a benchmark that connects visual origami states to executable programs through a high-level language of atomic folds, compound actions, and reusable folding primitives. It provides an execution environment and nine tasks spanning local action understanding, sequence ordering, and complete program generation. Experiments on manually annotated, randomly generated, and LLM-generated trajectories evaluate multiple VLMs, reporting difficulties in visual–programmatic grounding and multi-step reasoning, as well as inconsistent improvements with model scale. The main contributions are the action representation, execution environment, and structured evaluation suite.

Critical strengths
  1. The language explicitly represents fold geometry, direction, layer selection, and compound actions, while the environment connects compilation, foldability checking, and rendering. This supports studying how actions transform states rather than only recognizing static configurations, which is the paper’s strongest contribution.
  2. Corresponding variants of Tasks 1 and 2 share states and candidate options, while Tasks 2c and 3 share execution-validated candidate actions with distinct rendered outcomes. These comparisons help investigate the value of action information and the asymmetry between program-to-vision grounding and vision-to-program induction.
Critical weaknesses
  1. Appendix H.3.1 fixes the scrambling fraction to f=1.0f=1.0 and accepts permutations whose inversion count is within one of the maximum. Consequently, inputs are always almost completely reversed. Under this stated construction, simply reversing the displayed order without inspecting any image leaves at most one incorrect pair; for n=9n=9, this guarantees Kendall’s τ≥0.944\tau \geq 0.944. Task 6, in contrast, uses uniformly random permutations. Therefore, Task 5 scores and the performance gap between Tasks 5 and 6 cannot be straightforwardly interpreted as evidence of stronger visual-ordering ability. A reversal-only baseline and evaluation with uniformly shuffled inputs are necessary.
  2. The prompt explicitly requires recovering the recorded action order even when another ordering could produce a physically equivalent final state, although the model only receives the endpoint images and shuffled actions. Removing text-identical actions does not establish that the remaining instances exclude commuting actions or alternative valid execution orders. The evaluation may therefore penalize valid solutions and conflate reasoning errors with ambiguity about the original history. An ambiguity audit or execution-equivalence-aware scoring is needed.
Questions for authors
  1. What scores does a reversal-only, image-free baseline achieve on Task 5? Does the Task 5–Task 6 performance gap persist when Task 5 uses uniformly random permutations?
  2. What fraction of Task 6 instances admit multiple valid action orders? How do model rankings and the main conclusions change when execution-equivalent solutions are accepted?
  3. For Tasks 7 and 9, does “coverage” mean parseable output, executable programs, or successful goal attainment? Since CPD is reported only on scoreable outputs, what are the execution-success and target-state-matching rates when all instances, including failures, are included in the denominator?
Reviewer_EGR6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces OrigamiBench, built around a Lisp-like language representing atomic folds, compound actions, paper turns, and reusable folding primitives. An environment compiles actions into FOLD graphs, checks flat-foldability, and renders intermediate states. The authors construct manually annotated, randomly generated, and LLM-generated trajectories and define nine tasks spanning local transitions, action understanding, state/action ordering, plan completion, and full plan generation. Experiments compare proprietary and open-weight vision-language models and identify difficulties with harder distractors, action grounding, and multi-step reasoning.

Critical strengths
  1. A potentially useful unified action representation. Explicit layer arguments, compound combine operations, and expandable primitives could support reasoning about folding procedures beyond crease-pattern or terminal-state representations.

  2. A systematic task decomposition. The nine tasks connect local transitions, global trajectory reasoning, and goal-conditioned program generation. Distractors constructed through AST interventions offer a promising diagnostic mechanism.

  3. Multiple data sources and model families. Human-designed and synthetic trajectories can provide complementary distributions, while GPT, Qwen, and InternVL results provide a basis for studying visual understanding of the action language.

Critical weaknesses
  1. The dataset size is materially inconsistent. Table 1 (p.5, L216–225) reports 100 manually annotated, 1,000 LLM-generated, and 10,000 randomly generated trajectories, totaling 11,100. The accompanying text instead states that 80 random trajectories were selected and 80 valid LLM trajectories were obtained in total (L246–260). If these are evaluation subsets, their relationship to the full corpus must be stated. As written, it is unclear which dataset version underlies the statistics and evaluations. Please provide a reconciled manifest with full and evaluated counts by source, distinct trajectories and questions per task, and separation between LLM-generation demonstrations and evaluation instances. Uncertainty should account for multiple questions derived from the same trajectory.

  2. The connection between the central generative metrics and executable planning success is insufficiently specified. Table 2 (p.6) assigns CPD, Crease Pattern Dissimilarity, to Tasks 7/9, while §4.3 (p.9) primarily discusses coverage and CPD rankings on scoreable subsets. The available material does not establish the implemented CPD definition, normalization, treatment of mountain/valley assignments and layers, or whether every action and the final target state are also checked. Crease similarity alone does not establish that a program executes correctly and reaches the goal; conversely, a program different from the reference may still be successful. Please report, over all evaluation instances, the fraction of programs that execute legally and reach an equivalent target state. CPD can remain a complementary diagnostic. Parsing success, execution success, goal attainment, and conditional quality should be separated. Stronger conditional results on Qwen's covered subset are not directly comparable with results from models covering more instances.

  3. Physical validity requires separating feasible terminal configurations from legal state transitions. Section 2.2 (p.4, L197–200) describes local flat-foldability checks and a search for a globally consistent folded configuration of the candidate FOLD graph. This establishes configuration feasibility, but the description alone does not establish reachability from the current state under the requested action, particularly for simultaneous combine semantics, selected layers, and connectivity. The compiler may provide additional guarantees, but the relevant invariants or validation evidence are not explicit in the main text. Please define the transition semantics and test negative examples where a flat-folded endpoint exists but the requested transition is illegal. This asks for precise guarantees under the symbolic action model, not a full material-mechanics or robotics simulator.

  4. The novelty discussion understates a direct task overlap with ORIGAMISPACE. Section 3 (p.7, L349–353) already cites ORIGAMISPACE, but characterizing prior benchmarks as primarily reasoning over crease patterns or folded states obscures its explicit process-level task. ORIGAMISPACE §4.2 (p.5) requires recovering the chronological order of shuffled folding-process images, directly overlapping the task family of Task 5 here. The differences concern multiple-choice versus generated orderings, image construction, and metrics. ORIGAMISPACE also includes target-conditioned CP generation and iterative crease editing using compiler feedback (§3.2, §4.4, §5.2). The contribution should therefore be positioned around the explicit action language, standardized executable trajectories, and additional action-reasoning/program-generation tasks. Multi-step reasoning and generative interaction alone are not the distinguishing advances. A task-level table separating inherited, extended, and new capabilities would clarify this, especially for Tasks 5 and 9.

Questions for authors
  1. What is the actual dataset size, and what do the two counts of 80 represent relative to Table 1's 11,100 trajectories? Which trajectories support each task?

  2. How is CPD implemented for Tasks 7/9, and which failures are excluded by coverage? Can you report unconditional legal-execution and target-matching success, accepting equivalent alternative programs?

  3. How does the compiler guarantee transition consistency for combine, layer selection, and consecutive states? What tests distinguish feasible endpoints from illegal requested transitions?

  4. Which tasks extend ORIGAMISPACE's temporal image ordering and interactive CP generation, and which introduce additional capability requirements?

  5. Do correct/removed/incorrect-action comparisons with matched questions and distractors support the grounding interpretation? Can the cited appendices and Tables 3–9 be provided?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.