- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces CP2Seq, a benchmark for evaluating multimodal language models on geometric planning though origami folding tasks. Given a crease pattern and target folded state, the evaluated models are required to generate a valid sequence of folding actions to reproduce the target end state. The benchmark contains 600 procedurally generated instances across different sequence lengths and folding constraints, and provides an interactive environment with fold verification, geometric observations, and varying levels of tool assistance for evaluation. The authors evaluated GPT-5.6 Luna and GPT-6 Luna on the benchmark, showing that performance deteriorates substantially as folding sequences become longer. Although tool assistance and increased reasoning improve success on easier instances, the higher difficulty instances remain challenging.
Critical strengths
- the evaluation of models' geometric planning and executing a sequence of folds is novel compared to prior origami benchmarks
- the procedurally generated instances and the interactive evaluation environment provide flexible but also programmatically verifiable success criteria
- the experiments on varying levels of tool access and reasoning effort is interesting
Critical weaknesses
- the model evaluations in this paper are very limited. The authors only evaluated the GPT Luna family models through the Codex CLI, which is insufficient. At the very least, the authors should evaluate several open-source large language models at varying scales, as well as more capable closed-source models
- the authors cited repeated-state cycling as a dominant failure model, but it is not immediately indicative of the underlying failure cause of the evaluated models. For example, the same failure can be caused by inadequate geometric understanding or insufficient backtracking in reasoning, thus limiting the insightfulness of the error analysis
Questions for authors
- "the evaluated models are closed and served through the Codex CLI, so their weights and sampling cannot be pinned" -- could the authors instead evaluate through public APIs, which could allow for more reproducible experiments, for example, for the sampling settings?
- could the authors also provide a more extensive set of evaluated models, or explain why the GPT Luna family is sufficient?
- could the authors also provide a more fine-grained analysis of the evaluated model's failure modes in their reasoning process, supported by quantitative evidence?
2 comments
Thank you for the response
I thank the authors for their response. However, they seem to have mostly acknowledged my concerns rather than addressing them directly. The only new evidence the authors seemed to provide is benchmarking Claude Sonnet 5.5, ran through Claude Code CLI rather than more reproducible API setup.
My main concerns remains: the benchmark, in its current form, lack extensive benchmarking on current open and closed source LLM models, under reproducible setups. The authors also provided very limited insights on how models fail on this benchmark, and how it could guide future model development. The authors mentioned that the reason for not providing more fine-grained error analysis was because they have "no access to internal reasoning" of the models -- I believe this could be resolved by evaluating open-source models, which would allow the authors to audit their reasoning traces. I thus still lean towards rejection and will maintain my score.
Thank you for the review. Answers in order.
Q1. Could the models be evaluated through public APIs for reproducible sampling settings?
We agree that would be better, and we have not done it: the Luna runs use Codex CLI and the added Sonnet runs use Claude Code CLI, which we now document rather than imply direct-API reproducibility. An endpoint with explicit sampling settings would still not pin closed weights or guarantee deterministic serving, so we also made the evidence replayable instead of only re-runnable: every accepted state in the released artifacts re-executes offline, and the supplement now ships the comparator, the audit scripts, 118 runs and 1,906 episodes. Direct-API replication remains listed as needed work.
Q2. Could more models be evaluated, or is the Luna family sufficient?
We do not claim it is sufficient, and the revision no longer makes any claim that would need it. Luna was the strongest family we had a working tool loop for when the sweep started, and holding one family fixed let the budget cover the tool, effort and modality ablations, which are 360 episodes of up to 80 turns with large per-turn state; we then added Claude Sonnet 5.5 on 20 shared instances, a second family from a second vendor, where it solves 14/20 against a random expectation of 15.3/20 and a Luna single-trial median of 9/20. The multi-scale open-weight evaluation you ask for is the right experiment and is not done: that is a budget limit we are seeking support to lift, not a judgement that it is unnecessary.
We would add that the result the revision now leads with does not rest on which models were evaluated. On the same 40 instances and the same legal action set, random legal folding solves 57% of easy and 96% of some-layers episodes against 50% and 80% for the best Luna condition, and BFS solves 5/10 medium instances (Section 5.1, Table 4, Figure 4). A model given exact layer geometry, per-action legality checks, the full legal-action list and automatic target comparison does worse than undirected sampling in the same space, and Sonnet narrows that gap without closing it. That is a statement about this class of agent setup rather than a ranking, which is why a wider model sweep would strengthen it but is not what it depends on.
Q3. Could the failure analysis be more fine-grained and quantitative?
Agreed, and cycling is no longer offered as a cause: it terminates 191 of 305 unsuccessful completed episodes, which we now present as a protocol stopping label. In its place the revision reports observable trajectory measures. Replaying every accepted state in 143 logged episodes and stopping at the first match raises solves only from 25 to 27, so failures rarely reach the target without recognising it. Distinct-state coverage is 0.11 to 0.25 of random folding over comparable budgets. Bounded repair search finds a fix within three removed folds for 71 of 104 full restarts and 161 of 280 shorter backtracks, so discarded positions were often recoverable. After the first backtrack, 340 of 931 accepted Luna folds revisit abandoned states against 26 of 243 for Sonnet (Section 5.3, Appendix B.14, Appendix D).
What these do not do is identify the cause. We have no access to internal reasoning through this interface and no completed controlled intervention, so we do not attribute the behaviour to geometric understanding, memory or backtracking policy. Controlled interventions on memory, action semantics and geometric state are stated as the next measurement, and the recorded, replayable trajectories are what make them runnable.
What changed in the revision: identity-aware scoring with an audit of all 536 accepted episodes, correcting the ablation total from 55/360 to 50/360; matched random and BFS baselines with stated budgets, placed on one axis with the models in Figure 4; tools and input modality in one grid including text-only arms (Table 3); the Sonnet evaluation and five repeated Luna trials (Table 6); instance-clustered uncertainty for reasoning effort; trajectory measures in place of termination attribution; and an abstract, conclusion and title rewritten to state what the experiments establish.