acceptodds
Withdrawn from ICLR 2027

TenonBench: Evaluating Spatiotemporal Capabilities with Mortise-and-Tenon Assemblies

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Evaluating whether vision-language models (VLMs) can reason about feasible motions and operation order is essential for their use in embodied planning. However, existing benchmarks provide limited coverage of changing geometric constraints, do not fully distinguish necessary from optional ordering, and often assess perception and planning separately. To address these gaps, we introduce TenonBench, a spatiotemporal reasoning benchmark that combines interlocking assembly tasks, multi-solution evaluation, and paired perception–reasoning analysis. We construct 178 parametric mortise-and-tenon assemblies with 1,200 parts and use a geometric solver to provide ground truth for 4,065 questions spanning part identification, structural understanding, and reasoning about how movement direction and preceding removals determine motion feasibility. Our partial-order evaluation distinguishes necessary dependencies from optional ordering that varies across valid solutions, while complete sequences are verified by replaying each removal against the remaining assembly. Within multi-turn shared conversations, we further pair reasoning questions with perception probes about the same parts and jointly analyze their responses to examine whether reasoning errors persist when the corresponding perception questions are answered correctly. Evaluation of 23 VLMs yields a best overall accuracy of 77.2% and a mean of 53.7%, highlighting TenonBench as a challenging testbed for geometry-grounded spatiotemporal reasoning.

Primary area:
datasets and benchmarks
Keywords:
Vision-Language Models, Spatiotemporal Reasoning, Disassembly Planning, Benchmark, Geometric Constraints

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_TmRuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduce a new benchmark called TenonBench, which evaluates VLMs on geometric reasoning about mortise-and-tenon assemblies, covering contact, directional mobility, blocking relations, necessary precedence, and sequence validity. It provides solver-generated labels and evaluates 23 models, with additional analysis pairing structural questions with operation-reasoning questions.

Critical strengths

Distinguishing necessary dependencies from optional ordering is useful: producing a feasible sequence does not establish an understanding of which constraints must hold. Solver-based labels and stepwise sequence validation accommodate multiple valid solutions. Reporting invented and missed dependencies separately also reveals errors that overall accuracy would hide.

Critical weaknesses

The main concern is whether the benchmark isolates the capabilities it claims to measure. Correctly counting contacts does not establish correct geometric perception: the appendix itself shows models giving the right count while identifying the wrong contact relationships. The paired analysis therefore cannot reliably separate perception failures from reasoning failures.

Geometric ground truth also does not guarantee that the answer is recoverable from the supplied fixed-view images. Without answerability checks and stronger shortcut controls, errors may reflect missing visual information, while correct answers may exploit directional or language priors. The proposed image-removal and ID-permutation controls were not run.

All questions about an assembly share a conversation, so earlier answers may influence later judgments. Without independent-context or question-order comparisons, these effects remain mixed with spatial reasoning performance. Finally, the connection to embodied planning is unvalidated: the robot demonstrations execute solver-generated plans, and the downstream correlation study remains a proposal.

Questions for authors

Can humans reliably answer the questions from exactly the images provided to the models, particularly when relevant geometry is occluded?

Does reasoning improve when models receive correct contact identities and directional blocking relations, rather than merely answering contact-count probes?

How do results change under independent conversations, reordered questions, and image-removal or ID-permutation controls?

Do benchmark scores predict models’ ability to generate or execute feasible plans beyond selecting among candidate answers?

Reviewer_Sdr3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper uses mortise-and-tenon assemblies to test spatial understanding and disassembly reasoning. It evaluates 23 models on 4k+ questions.

Critical strengths
  1. Distinguishing required from optional removal orders makes the evaluation fairer to multiple valid solutions.
  2. Testing contact, movement, and removal order across 23 models gives a useful picture of their weaknesses.
Critical weaknesses
  1. The reported checks do not fully establish that a single finite-displacement test guarantees collision-free removal along the entire path.
  2. Some key connections may be hidden. Without human evaluation or extra-view controls, missing visual information remains a possible explanation for errors.
Questions for authors

Under what geometric conditions does the finite-displacement test guarantee complete collision-free removal? Could human testing or extra-view controls establish whether the provided images contain enough information to answer the questions?

Reviewer_nNB1OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

TenonBench evaluates vision-language models on geometric constraints and disassembly dependencies using rendered mortise-and-tenon assemblies. It contains 4,065 multiple-choice questions across 178 assemblies, covering part identification, contact relations, directional mobility, blocker attribution, necessary precedence, and sequence validity. The paper evaluates 23 models and analyzes performance across task categories, assembly complexity, and paired contact-count and operation-reasoning questions.

Critical strengths
  1. Interlocking assemblies offer a useful setting for studying how local geometric relations constrain operations on an entire structure. The tasks connect contact and blocking relations to motion feasibility and disassembly order.
  2. Distinguishing necessary precedence from an arbitrary valid ordering is a worthwhile evaluation objective. Evaluating both pairwise dependencies and complete-sequence validity can reveal errors that comparison with a single reference sequence would miss.
Critical weaknesses
  1. Severe presentation problems substantially obstruct understanding. The paper introduces several overlapping taxonomies and repeatedly summarizes its components without establishing a clear narrative from the research question to the design choices and findings. Sections 4.2 and 4.3 are particularly difficult to follow: task codes, definitions, qualifications, and numerical comparisons are densely interleaved, with frequent parentheses, dashes, and semicolons further interrupting the exposition. For example, Section 4.2 refers to whether a part can move along “−X,” but the main text does not clearly establish the coordinate convention or explain how these directions correspond to the rendered inputs. Since motion direction is central to the benchmark, this is a task-specification issue as well as a readability issue.
  2. The paper does not establish a sufficiently focused contribution or sufficiently informative analysis. Although the related-work section discusses existing benchmarks, the paper does not clearly demonstrate which specific shortcomings motivate this benchmark and how its design addresses them. The experimental narrative largely catalogs performance differences across models and task categories, without sufficiently explaining what these differences reveal about the capabilities models lack. The appendices add useful observations about directional priors, confounding factors, and incorrect contact relations, but provide limited systematic evidence explaining the observed failures. In particular, the authors acknowledge that a correct contact count does not establish correct understanding of the geometry relevant to an operation. This limitation substantially restricts the diagnostic value of the paired-probe design: it remains unclear what the analysis establishes beyond an association between performance on two related tasks, or how it helps distinguish perceptual errors from reasoning failures. Overall, the paper leaves the reader to reconstruct the connection between the benchmark’s motivation, its evaluation design, and the scientific insights supported by its results.
Questions for authors
  1. Compared with the assembly and spatial benchmarks discussed in Section 2, what specific evaluation gap is uniquely or substantially better addressed by TenonBench? Please connect that gap to the benchmark’s design and to a concrete finding that existing evaluations would not reveal.
  2. What evidence distinguishes failures of operation reasoning from failures to identify the relevant parts, contacts, or blockers? In particular, how should the paired-probe results be interpreted given that a correct contact count can coexist with incorrect contact identities?
  3. What are the central findings that readers should take away from the experiments, and how does each follow from the reported evidence? The experimental sections require substantial restructuring around these findings. In its current form, the presentation is not sufficiently clear for publication.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.