- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduce a new benchmark called TenonBench, which evaluates VLMs on geometric reasoning about mortise-and-tenon assemblies, covering contact, directional mobility, blocking relations, necessary precedence, and sequence validity. It provides solver-generated labels and evaluates 23 models, with additional analysis pairing structural questions with operation-reasoning questions.
Critical strengths
Distinguishing necessary dependencies from optional ordering is useful: producing a feasible sequence does not establish an understanding of which constraints must hold. Solver-based labels and stepwise sequence validation accommodate multiple valid solutions. Reporting invented and missed dependencies separately also reveals errors that overall accuracy would hide.
Critical weaknesses
The main concern is whether the benchmark isolates the capabilities it claims to measure. Correctly counting contacts does not establish correct geometric perception: the appendix itself shows models giving the right count while identifying the wrong contact relationships. The paired analysis therefore cannot reliably separate perception failures from reasoning failures.
Geometric ground truth also does not guarantee that the answer is recoverable from the supplied fixed-view images. Without answerability checks and stronger shortcut controls, errors may reflect missing visual information, while correct answers may exploit directional or language priors. The proposed image-removal and ID-permutation controls were not run.
All questions about an assembly share a conversation, so earlier answers may influence later judgments. Without independent-context or question-order comparisons, these effects remain mixed with spatial reasoning performance. Finally, the connection to embodied planning is unvalidated: the robot demonstrations execute solver-generated plans, and the downstream correlation study remains a proposal.
Questions for authors
Can humans reliably answer the questions from exactly the images provided to the models, particularly when relevant geometry is occluded?
Does reasoning improve when models receive correct contact identities and directional blocking relations, rather than merely answering contact-count probes?
How do results change under independent conversations, reordered questions, and image-removal or ID-permutation controls?
Do benchmark scores predict models’ ability to generate or execute feasible plans beyond selecting among candidate answers?