acceptodds
Under review as a conference paper at ICLR 2027

Origami as a Spatial and Geometric Reasoning Benchmark

Abstract

Multimodal language models perform well on visual recognition and on tool use, but these abilities do not transfer directly to spatial and geometric reasoning. Geometric structure does not reduce to text or to a single image: exact angles and incidences, which surface lies in front of which, and the order in which operations were applied are not recoverable from a semantic description of a scene. We introduce CP2Seq, a benchmark for evaluating geometric reasoning and sequential planning in multimodal language models through origami. The benchmark contains 600 procedurally generated samples, each requiring a model to produce a valid sequence of folds from a crease pattern and a target folded state. Samples are produced by a deterministic folding engine. Ground truth is recorded during generation. We release the generation and evaluation framework to support additional samples and experiments across reference depths and action models. During evaluation, the engine executes proposed folds or rejects invalid actions with explicit feedback, while the model remains responsible for selecting actions and searching for a solution. We evaluate multimodal language models under progressively greater tool assistance, and against a deterministic breadth-first search over the same action set. Candidate sequences are assessed by executing them and comparing the resulting folded state with the target, allowing for planar translations, rotations, and reflections. Performance falls off sharply with sequence depth: in the Luna ablations no configuration solves a medium or hard sample (11 or more folds), while some-layers solves rise from 6/30 to 18/30 with higher reasoning effort. Failures are dominated by search rather than by illegal moves: in the broader evaluation, 48% of episodes end by revisiting earlier states. A breadth-first search over the same actions, under its own time budget, solves 54.7% of easy-group attempts, so short sequences are within reach of systematic search.

Primary area:
datasets and benchmarks
Keywords:
spatial reasoning ·benchmark, multimodal LLMs, interactive environment, verifiable rewards, tool use, verifier, search and planning, synthetic data, computational origami

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_xGpNOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces CP2Seq, a benchmark for evaluating multimodal language models on geometric planning though origami folding tasks. Given a crease pattern and target folded state, the evaluated models are required to generate a valid sequence of folding actions to reproduce the target end state. The benchmark contains 600 procedurally generated instances across different sequence lengths and folding constraints, and provides an interactive environment with fold verification, geometric observations, and varying levels of tool assistance for evaluation. The authors evaluated GPT-5.6 Luna and GPT-6 Luna on the benchmark, showing that performance deteriorates substantially as folding sequences become longer. Although tool assistance and increased reasoning improve success on easier instances, the higher difficulty instances remain challenging.

Critical strengths
  • the evaluation of models' geometric planning and executing a sequence of folds is novel compared to prior origami benchmarks
  • the procedurally generated instances and the interactive evaluation environment provide flexible but also programmatically verifiable success criteria
  • the experiments on varying levels of tool access and reasoning effort is interesting
Critical weaknesses
  • the model evaluations in this paper are very limited. The authors only evaluated the GPT Luna family models through the Codex CLI, which is insufficient. At the very least, the authors should evaluate several open-source large language models at varying scales, as well as more capable closed-source models
  • the authors cited repeated-state cycling as a dominant failure model, but it is not immediately indicative of the underlying failure cause of the evaluated models. For example, the same failure can be caused by inadequate geometric understanding or insufficient backtracking in reasoning, thus limiting the insightfulness of the error analysis
Questions for authors
  • "the evaluated models are closed and served through the Codex CLI, so their weights and sampling cannot be pinned" -- could the authors instead evaluate through public APIs, which could allow for more reproducible experiments, for example, for the sampling settings?
  • could the authors also provide a more extensive set of evaluated models, or explain why the GPT Luna family is sufficient?
  • could the authors also provide a more fine-grained analysis of the evaluated model's failure modes in their reasoning process, supported by quantitative evidence?
2 comments

Thank you for the review. Answers in order.

Q1. Could the models be evaluated through public APIs for reproducible sampling settings?

We agree that would be better, and we have not done it: the Luna runs use Codex CLI and the added Sonnet runs use Claude Code CLI, which we now document rather than imply direct-API reproducibility. An endpoint with explicit sampling settings would still not pin closed weights or guarantee deterministic serving, so we also made the evidence replayable instead of only re-runnable: every accepted state in the released artifacts re-executes offline, and the supplement now ships the comparator, the audit scripts, 118 runs and 1,906 episodes. Direct-API replication remains listed as needed work.

Q2. Could more models be evaluated, or is the Luna family sufficient?

We do not claim it is sufficient, and the revision no longer makes any claim that would need it. Luna was the strongest family we had a working tool loop for when the sweep started, and holding one family fixed let the budget cover the tool, effort and modality ablations, which are 360 episodes of up to 80 turns with large per-turn state; we then added Claude Sonnet 5.5 on 20 shared instances, a second family from a second vendor, where it solves 14/20 against a random expectation of 15.3/20 and a Luna single-trial median of 9/20. The multi-scale open-weight evaluation you ask for is the right experiment and is not done: that is a budget limit we are seeking support to lift, not a judgement that it is unnecessary.

We would add that the result the revision now leads with does not rest on which models were evaluated. On the same 40 instances and the same legal action set, random legal folding solves 57% of easy and 96% of some-layers episodes against 50% and 80% for the best Luna condition, and BFS solves 5/10 medium instances (Section 5.1, Table 4, Figure 4). A model given exact layer geometry, per-action legality checks, the full legal-action list and automatic target comparison does worse than undirected sampling in the same space, and Sonnet narrows that gap without closing it. That is a statement about this class of agent setup rather than a ranking, which is why a wider model sweep would strengthen it but is not what it depends on.

Q3. Could the failure analysis be more fine-grained and quantitative?

Agreed, and cycling is no longer offered as a cause: it terminates 191 of 305 unsuccessful completed episodes, which we now present as a protocol stopping label. In its place the revision reports observable trajectory measures. Replaying every accepted state in 143 logged episodes and stopping at the first match raises solves only from 25 to 27, so failures rarely reach the target without recognising it. Distinct-state coverage is 0.11 to 0.25 of random folding over comparable budgets. Bounded repair search finds a fix within three removed folds for 71 of 104 full restarts and 161 of 280 shorter backtracks, so discarded positions were often recoverable. After the first backtrack, 340 of 931 accepted Luna folds revisit abandoned states against 26 of 243 for Sonnet (Section 5.3, Appendix B.14, Appendix D).

What these do not do is identify the cause. We have no access to internal reasoning through this interface and no completed controlled intervention, so we do not attribute the behaviour to geometric understanding, memory or backtracking policy. Controlled interventions on memory, action semantics and geometric state are stated as the next measurement, and the recorded, replayable trajectories are what make them runnable.

What changed in the revision: identity-aware scoring with an audit of all 536 accepted episodes, correcting the ablation total from 55/360 to 50/360; matched random and BFS baselines with stated budgets, placed on one axis with the models in Figure 4; tools and input modality in one grid including text-only arms (Table 3); the Sonnet evaluation and five repeated Luna trials (Table 6); instance-clustered uncertainty for reasoning effort; trajectory measures in place of termination attribution; and an abstract, conclusion and title rewritten to state what the experiments establish.

Reviewer_xGpNOpenReview ↗

Thank you for the response

I thank the authors for their response. However, they seem to have mostly acknowledged my concerns rather than addressing them directly. The only new evidence the authors seemed to provide is benchmarking Claude Sonnet 5.5, ran through Claude Code CLI rather than more reproducible API setup.

My main concerns remains: the benchmark, in its current form, lack extensive benchmarking on current open and closed source LLM models, under reproducible setups. The authors also provided very limited insights on how models fail on this benchmark, and how it could guide future model development. The authors mentioned that the reason for not providing more fine-grained error analysis was because they have "no access to internal reasoning" of the models -- I believe this could be resolved by evaluating open-source models, which would allow the authors to audit their reasoning traces. I thus still lean towards rejection and will maintain my score.

Reviewer_Fff9OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces CP2Seq, a benchmark requiring a model to construct an executable folding sequence from a crease pattern and a target folded state, starting from an unfolded square. Its 600 procedurally generated instances cover all-layers and some-layers simple folds. An interactive environment supports execution, legality checking, observation, and backtracking. Evaluation accepts alternative solutions that reproduce both the required creases and terminal state. The experiments vary tool assistance and reasoning effort and include a BFS baseline. Assistance improves some short-sequence results, while none of the controlled model configurations solves a medium or hard instance.

Critical strengths
  1. A meaningful extension of the task formulation. Recovering executable fold actions from a supplied crease pattern and target differs from generating crease-pattern code. Actions in current folded coordinates require layer selection, connectivity reasoning, and sequential planning.

  2. A substantial environment and evaluation design. Recorded generation trajectories establish solutions under the implemented action model. Execution-based scoring accepts alternative sequences and accounts for global translation, rotation, and turnover. The appendices document interfaces, tolerances, and protocol details.

  3. Useful transparency. The manuscript acknowledges strong assistance from target feedback, incompatible historical run budgets, five protocol failures included in the denominator, and a comparison-tool defect. These disclosures help delimit the results.

Critical weaknesses
  1. Terminal verification does not fully implement the paper-region semantics emphasized in the task description. Appendix A.1 (p.12, L643–647) states that original-sheet identities matter when comparing completed constructions, whereas §6 and Appendix A.6 (pp.9, 14, L725–728) acknowledge that the comparator checks current-plane polygons, parity, and rank without explicitly matching original-sheet region identities. Geometrically identical regions originating at different sheet locations may therefore require a stronger correspondence check. The paper does not establish whether separate crease-pattern matching already rules out this ambiguity. This is not evidence that a reported success is necessarily wrong, but it limits the claim of exact verification. Please specify the intended equivalence relation, add identity-sensitive adversarial tests, and report an audit of accepted solutions. If these identity differences are intentionally considered equivalent, the task definition should say so and justify that choice.

  2. The feedback defect affects the controlled ablation, and the post-hoc checks do not fully bound its impact. Appendix B.13 (p.30, L1596–1604) reports that equal layer counts with unalignable bottom layers could produce matches despite zero matched layers. Five incorrect submissions followed this feedback, including two automatic-feedback ablation episodes. The independent final scorer was reportedly unaffected by this particular bug, which is reassuring. However, trajectories were generated under incorrect feedback: changing feedback can change subsequent search. Inspecting final states of incorrect submissions and confirming that 536 solved episodes still receive matches does not establish that only the five identified failures were affected, or that the correction can only improve performance. Please audit all relevant feedback calls, rerun at least the known affected cases, and distinguish original from corrected results.

  3. Controlled coverage and baselines are insufficient for broader capability conclusions. The 360 planned evaluations reuse only 40 distinct instances, taking the first ten per group; the some-layers subset contains only five-fold examples (§5.1, p.8). The main controlled comparison is restricted to the Luna family. BFS's 725 attempts contain repeated instances and varying budgets and are not matched to these ablations (Appendix B.2, p.15). Although this limitation is disclosed, a matched comparison is still needed to interpret model performance. A bounded addition would evaluate random legal-action selection and BFS on the same instances with explicit resource budgets, alongside a text-only geometry condition to establish the contribution of images. Statistical inference for reasoning effort should also account for instances reused across tool conditions, using instance-clustered intervals or an appropriate paired analysis.

  4. Termination categories do not establish the claimed capability attribution. Illegal actions consume budget but cannot directly terminate an episode; repeating the same fold from the same position three times can trigger cycling termination (§5.2, pp.8–9). Thus, 215/384 earlier failures ending in cycling describes the protocol's stopping labels, rather than establishing the relative contributions of illegal actions, state misunderstanding, and feedback errors. Rejection rates, trajectory-level error analysis, and a small sensitivity check on the cycling threshold would strengthen the interpretation. Otherwise, conclusions should remain descriptive. Similarly, reference depth is confounded with geometric complexity, and the extrapolated hard-group BFS cost curve rests on only two instances; it should not be interpreted as intrinsic task complexity (Figure 4, Appendix B.2).

Questions for authors
  1. Among sequences matching the required CP, can the current comparator accept a terminal state with different original-sheet region correspondence? Can you provide a proof, adversarial tests, and an audit of accepted solutions?

  2. What happens to the two known affected ablation episodes after fixing matches? Were all intermediate feedback calls audited, rather than only the final states of incorrect submissions?

  3. Can you provide matched random-legal/BFS baselines with explicit budgets, and a text-only input condition?

  4. How uncertain is the reasoning-effort improvement when clustering by instance? Does relaxing the cycling threshold change the conclusions?

2 comments

Thank you for the review. Answers in order.

Q1. Yes. An audit of 8,000 reference-depth rollouts finds eight states that match the crease pattern and the plane relation yet carry different sheet regions, so crease-pattern matching does not remove the ambiguity. We now define E_id, which requires each layer to contain the corresponding original-sheet region under one global assignment-preserving symmetry of the crease pattern; identity-sensitive tests use congruent layers with permuted source regions and shifted reference replays, and re-scoring all 536 accepted episodes reverses 22, correcting the ablation total from 55/360 to 50/360 (Section 4.2.1, Table 2, Appendices A.6 to A.7). We claim no formal proof: verification is numerical under stated tolerances and the revision says so.

Q2. They have not been rerun under corrected feedback, so we cannot say what the corrected trajectories would do, and their outcomes stay labelled as measurements taken under defective feedback. Intermediate calls were audited wherever per-turn logs survive: all 1,025 calls across 65 episodes, of which one differs under corrected alignment logic and two report a plane-only match that fails E_id. The image-ablation episodes retained no per-turn logs, so those trajectories cannot be cleared, and terminal E_id rescoring is reported separately rather than as a corrected-feedback result (Section 6, Appendix B.13).

Q3. Both are in. Random folding and BFS use the same 40 instances, the same legal action set and E_id: random is 100 episodes of 80 folds per instance, BFS reports 80 to 100,000 expansions under a reference-depth-plus-four limit and a 600 second cap (Table 4, Figure 4). Removing images while keeping exact numeric geometry gives 3/40, 5/40 and 11/40 against 2/40, 5/40 and 9/40 with images, with images alone at 1/40 (Table 3).

Q4. Resampling instances with all their conditions together gives a difference of 0.40, a 95% interval of roughly [0.23, 0.53] and an exact sign-flip p = 0.0078, from 10,000 resamples with seed 1, so we no longer treat the 30 outcomes as independent. Relaxing the threshold requires new continuations, since terminated trajectories cannot establish their own counterfactuals, and those runs have not been done. The revision claims no robustness to that change.

W3. The instance count is unchanged at 40 and we no longer generalise beyond it. Sonnet 5.5 solves 14/20 on 20 shared instances against a random expectation of 15.3/20, and five Luna repeats solve 7, 10, 6, 10 and 9, so the ordering holds for a second family without becoming a ranking (Table 6). A multi-scale open-weight evaluation is not done: the ablation grid alone is 360 episodes of up to 80 turns, and that is a budget limit we are seeking support to lift.

W4. Cycling terminates 191 of 305 unsuccessful completed episodes under E_id, and we present that as a stopping label, not an attribution. In its place we report observable measures: an oracle stop at the first matching state adds two solves across 143 logged episodes, distinct-state coverage is 0.11 to 0.25 of random folding, and 71 of 104 full restarts discard a position repairable within three removed folds (Section 5.3, Appendix B.14). We removed the depth-wall reading, since reference depth covaries with layer count and the hard-group extrapolation rests on two instances.

What changed in the revision: identity-aware scoring with tests and the 536-episode audit; matched random and BFS baselines on one axis with the models (Figure 4); tools and modality in one grid with the text-only arms (Table 3); Sonnet and repeated trials (Table 6); instance-clustered uncertainty; trajectory measures in place of termination attribution; abstract, conclusion and title rewritten, which also shortens the main text; and a rebuilt supplement with the comparator, audit scripts, 118 runs and 1,906 episodes.

The corrected verifier and the new baselines on the same instances address my main concerns, so I now recommend a score of 3 (Weak Acceptance), up from 2 (Weak Rejection).

Reviewer_xZkyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes the CP2Seq origami reasoning benchmark, which requires multimodal models to generate executable folding sequences from a flat sheet of paper based on a crease diagram and a target folded state, in order to jointly evaluate geometric reasoning, spatial state tracking, and multi-step planning capabilities. The main contribution is the programmatic construction of 600 samples covering both full-layer and partial-layer folds, along with the provision of an interactive environment and an automatic verification mechanism that checks creases and the final folded state via playback and accepts different valid folding paths. In terms of validation, the authors conducted ablation experiments on the Luna model with varying levels of tool support and reasoning intensity, and established a breadth-first search baseline; the results show that tool assistance and increased reasoning effort can improve performance on short-sequence tasks.

Critical strengths

This paper integrates geometric transformations, paper-layer relationship tracking, and multi-step planning into an executable origami task, enabling an examination of the model’s ability to maintain and update spatial states during continuous operations, and offering clear evaluation value. The programmable generator records valid reference sequences for each sample, and the evaluation determines success based on the creases and layered states resulting from actual execution, allowing for multiple valid paths. This design, which combines interactive tools with an extensible data generation workflow, provides a verifiable and continuously expandable experimental environment for research on spatial planning.

Critical weaknesses

First, the controlled ablation covered only 40 distinct samples, primarily from the Luna series, and the BFS control group did not standardize the samples or computational budget; therefore, the model’s coverage, the adequacy of the comparison, and the generalizability of the conclusions remain limited. Second, there are critical gaps in the determination of target states that require clarification: Appendix A.1 indicates that the identity of the source paper region is significant for target comparison, but the final-state comparator in A.6 only checks the current geometry, orientation, and stacking order, without explicitly verifying the correspondence with the source paper region. The paper has not sufficiently explained whether this simplification guarantees target equivalence or whether it affects the success rate.

Questions for authors

Since the final-state comparator does not explicitly verify the identity of the source paper region, is it possible that configurations with the same shape, orientation, and sequence—but corresponding to different source paper regions—might be classified as successful? Can the current verification process rule out this scenario, and would it affect the success rate reported? For controlled ablation, the first 10 samples in each group are used—are their generation and ordering independent of model performance, and are these samples representative in terms of geometric complexity and search difficulty? The appendix discloses that the target feedback tool has erroneously reported successful matches. Why can its impact be limited to the listed cases of erroneous submissions? Has the impact of erroneous feedback on other search trajectories and termination behaviors been examined?

1 comment

改成每问三句,全文 618 词。文件在 reply-reviewer-1.txt。


Thank you for the review. Answers in order.

Q1. Can same shape, orientation and stacking order with different source regions be scored as successful, and does it affect the reported rate?

Yes, and 22 of the 536 previously accepted episodes did exactly that; the submitted comparator checked only current-plane geometry and could not rule it out. We added E_id, which requires each layer to carry the corresponding original-sheet region under one global assignment-preserving symmetry, with missing region data failing rather than falling back. The controlled ablation total is corrected from 55/360 to 50/360, with the affected cells marked in Table 2 (Section 4.2.1, Appendices A.6 to A.7).

Q2. Are the first ten samples per group independent of model performance, and representative?

Independent, yes: IDs follow generation order and consecutive seeds, and the corpus manifest was fixed before the ablations ran. Representative, no: the subset matches on reference depth, easy-group 6.2 against 6.8, but is lighter in layers, 26.4 against 33.0, and we have no measurement of representativeness in search difficulty. We therefore do not extrapolate the 40-instance estimates to all 600 samples (Appendix B.15).

Q3. Why can the defective feedback be limited to the listed incorrect submissions, and were other trajectories and terminations examined?

It cannot be, and the revision no longer claims it. In the 65 episodes whose per-turn logs survive we replayed all 1,025 comparison calls: one differs under corrected alignment logic and two report a plane-only match that fails E_id, while the image-ablation logs were not retained, so those trajectories cannot be cleared. The two known affected episodes have not been rerun under corrected feedback, so the trajectory-level sign is unknown, and a stopping audit over 143 logged episodes raises solves only from 25 to 27 (Section 6, Appendices B.13 and B.14).

W1. BFS did not standardise samples or budget.

Random legal folding and BFS now use the same 40 instances, the same legal action set and E_id, with stated budgets: random is 100 episodes of 80 folds per instance, BFS reports 80 to 100,000 expansions under a reference-depth-plus-four limit and a 600 second cap (Table 4, Figure 4). We do not claim equal compute and say so explicitly. Random solves 57% of easy and 96% of some-layers episodes against 50% and 80% for the best Luna condition, so low model success does not indicate unreachable targets.

W2. Coverage is limited to 40 samples and the Luna series.

Sonnet 5.5 solves 14/20 on the 20 shared instances against a random expectation of 15.3/20, and five Luna repeats on those instances solve 7, 10, 6, 10 and 9, so the ordering holds for a second family and a second vendor. The instance count is unchanged and we no longer generalise beyond it. The multi-scale open-weight evaluation is not done: the ablation grid alone is 360 episodes of up to 80 turns, and that is a budget limit we are seeking support to lift.

What changed in the revision:

  • Identity-aware scoring E_id, an audit of all 536 accepted episodes, and the corrected total of 50/360 (Section 4.2.1, Table 2, Appendices A.6 to A.7).
  • Matched random and BFS baselines on the same instances and action set, with budgets stated (Section 5.1, Table 4, Figure 4).
  • Tool support and input modality in one grid, including text-only arms (Table 3), and a Sonnet evaluation with five repeated Luna trials (Table 6).
  • Trajectory analyses reported as observations rather than causes: stopping audit, state coverage, and repair distance at restarts (Section 5.3).
  • Abstract, conclusion and title rewritten to state what the experiments establish; supplement rebuilt with the identity-aware comparator, the audit scripts, 118 runs and 1,906 episodes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.