- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces MechBench, a 2541-question benchmark evaluating LLM's knowledge, analysis, and design capabilities in parallel mechanisms. It comprises 200 development questions, 1791 standard questions spanning six capability categories, and 550 challenge questions covering nine tasks requiring structurally and geometrically consistent reasoning. Its main contributions are the benchmark, MechForge- a construction pipeline combining domain evidence, LLM-assisted and programmatic generation, and two-stage expert review; and task-aware evaluation using exact matching, expert-defined rubrics, and executable mechanism checks. The authors also develop MechGPT, a domain-adapted Qwen3-8B model, to investigate the benefits of specialized training.
Critical strengths
The given work has the following strengths: 1- The benchmark checks whether answers are mechanically valid. It checks whether proposed mechanisms satisfy requirements such as closed connections, allowed motion, and target trajectories. This is a meaningful strength because an answer can sound convincing while describing a mechanism that does not work. 2- The results reveal a clear gap between knowledge and problem-solving. Even the strongest model scores 96.12 on standard questions but only 59.68 on challenge tasks. The detailed results show where models struggle, making the benchmark useful for identifying weaknesses that ordinary knowledge tests may miss.
Critical weaknesses
1- The reliability of the scores is not established well enough. The expert check compares average scores but does not report agreement on individual answers, where scoring errors could cancel out. Execution failures are also excluded from averages without clearly reporting their frequency for each model. Since the benchmark’s main contribution depends on trustworthy scores, these gaps make the model comparisons harder to judge. 2- The paper does not sufficiently explain what causes poor challenge performance. Models may fail because they misunderstand mechanics, cannot express their answers in the required format, or struggle with the verifier interface. The reported analysis does not clearly separate these possibilities. This matters because the paper interprets low scores as evidence of weak mechanical reasoning, while the evaluation may also measure difficulties in formatting and submitting solutions.
Questions for authors
I have the following questions for the authors: 1- On the 150 questions re-scored by experts, how closely did automated and expert scores agree on individual answers? Did disagreements change model rankings or any main conclusions? 2- How many questions were excluded from each model’s results because of execution failures, and what caused those failures? Would the model comparisons change if these cases received zero? 3- What evidence separates errors in mechanical reasoning from answer-format or extraction failures? How often did a model provide a substantively correct answer that could not be processed by the verifier? 4- For program-generated questions, were training and test sets separated by generator or mechanism family, or only by individual instances? This distinction matters for judging whether MechGPT generalizes to new problems or benefits from closely related training examples.
2 comments
Thank you for your detailed response. I feel my questions have been answered and thus I have increased the score to 3.
Scoring reliability, verification failures, and evaluation scope
Question 1:
Response: Thank you for requesting item-level evidence. Appendix B, Table 6 compares 150 identical items per model (50 single-choice, 50 fill-in, and 50 open-ended), totaling 1,050 paired scores. It reports identical, higher, and lower expert scores, exact agreement (76.67%–90.00%), MAE (3.54–9.37 points), and counts with absolute differences above 20 points. These statistics supplement mean-score comparisons by showing both disagreement frequency and magnitude; errors on different items cannot simply cancel out in these measures.
Location: Section 5.2; Appendix B, Table 6.
Question 2:
Response: Thank you for asking us to clarify this. We corrected component-label input-protocol and local runtime compatibility issues, then re-verified the same original responses without new model answers. Missing required information receives zero and remains in the mean denominator. Verifiable responses receive weighted partial credit; where designated hard constraints apply, any failure caps the final score at 39. Verifier errors are reviewed rather than automatically scored as model zeros. Current comparisons use finalized scores after these corrections. A hard-constraint failure caps the raw score rather than automatically making it zero. Partial credit records satisfied criteria, not a valid design capable of continuously realizing the required motion or trajectory. Timeouts and internal errors are verifier-side exceptions, distinct from missing answer content.
Location: Section 4.4.
Question 3:
Response: Thank you for this question. A closed-service LLM faithfully converts original answers into the required structure, including answers not already in interface format. It cannot add missing information or correct mechanism content. We reviewed all responses, fixed protocol/runtime issues, and rescored the original answers; some former zeros became nonzero. Component-label normalization was strictly one-to-one: it resolved input-protocol rejections without altering the proposed mechanism. Runtime fixes allowed verification to complete. No new answers or design corrections were introduced. Category means and the overall mean across the 550 challenge questions were updated from this rescoring. After review, currently unsuccessful answers lack necessary information or contain incorrect information, rather than failing solely through interface or format mismatch. Complete candidates are assessed by task constraints and screw-subspace equivalence.
Location: Sections 4.4 and 5.3; Figure 5.
Question 4:
Response: Thank you for highlighting this distinction. Training and test items are split proportionally within question types, without generator or mechanism-family isolation. Question type specifies the response format, rather than a single mechanism structure or fixed solution procedure. A shared response format can cover different topology, joints, geometry, constraints, and solution procedures. Six capability categories and nine executable tasks provide varied evaluation conditions, but this split does not establish generalization to unseen generators or mechanism families.
Location: Sections 3.2 and 5.2.
Other concern 1: score reliability
Concern: Mean-score checks and unclear failure handling weaken confidence in comparisons.
Response: Thank you for raising this concern. Questions 1–3 above clarify the added item-level statistics, rescoring, and failure treatment. Partial credit is not evidence of a valid mechanism realizing the required motion or trajectory.
Location: Section 4.4; Appendix B, Table 6.
Other concern 2: causes of low challenge scores
Concern: Low scores may mix reasoning errors with formatting or interface difficulties.
Response: Thank you for this suggestion. Figure 5 separates mobility, motion-signature, screw-space, trajectory-geometry, and sampled-trajectory checks. On 214 mobility items, scalar properties are easier than complete screw-space construction; the 35 paired endpoint-trajectory items distinguish instantaneous motion from finite-trajectory conditions. For example, GPT-5.6-Sol passes all scalar mobility and motion-signature checks on the 214-item subset, but its strict output screw-space pass rate is 23.8%; MechGPT reaches 4.2% on output screw space. Thus, recognizing a motion class differs from constructing a geometry-consistent six-dimensional basis. Together with the faithful conversion and review described in Question 3, these diagnostics support a more specific interpretation of answer-content limitations.
Location: Sections 4.4 and 5.3; Figure 5.