acceptodds
Under review as a conference paper at ICLR 2027

MechBench: A Domain-Specific LLM Benchmark for Parallel Mechanisms

Abstract

Parallel mechanisms pose distinctive challenges to large language models (LLMs), as valid reasoning depends jointly on mechanism topology, loop closure, geometry, joint types, and modeling assumptions. Yet existing LLM benchmarks provide little systematic coverage of such domain-specific knowledge, constraint-sensitive analysis, and constructive design. To fill this gap, we introduce MECHBENCH, a 2,541-question benchmark for evaluating LLMs in parallel mechanisms across professional knowledge, mechanism analysis, and design. MECHBENCH contains standard and challenge tiers spanning six capability categories and nine executable analysis and design tasks. It is constructed through MECHFORGE, a domain-grounded framework that combines expert-organized knowledge, LLM-assisted question generation, programmatic instance generation, two-stage expert review, and task-aware scoring with exact matching, rubric-based evaluation, and program verification. We further develop MECHGPT as a diagnostic domain-adapted model. Experiments show that strong knowledge performance does not directly translate into executable mechanism reasoning: leading closed-service models achieve over 95 on the standard tier but only 47–60 on the challenge tier, while domain adaptation substantially improves MechGPT over its Qwen3-8B baselines but leaves a large gap in constrained analysis and design. These results establish MECHBENCH as a systematic testbed for measuring domain-specific LLM capabilities in mechanism science.

Primary area:
datasets and benchmarks
Keywords:
Large language models, Domain-specific language models, Benchmarking and evaluation, Parallel mechanisms

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_gLpCOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces MechBench, a 2541-question benchmark evaluating LLM's knowledge, analysis, and design capabilities in parallel mechanisms. It comprises 200 development questions, 1791 standard questions spanning six capability categories, and 550 challenge questions covering nine tasks requiring structurally and geometrically consistent reasoning. Its main contributions are the benchmark, MechForge- a construction pipeline combining domain evidence, LLM-assisted and programmatic generation, and two-stage expert review; and task-aware evaluation using exact matching, expert-defined rubrics, and executable mechanism checks. The authors also develop MechGPT, a domain-adapted Qwen3-8B model, to investigate the benefits of specialized training.

Critical strengths

The given work has the following strengths: 1- The benchmark checks whether answers are mechanically valid. It checks whether proposed mechanisms satisfy requirements such as closed connections, allowed motion, and target trajectories. This is a meaningful strength because an answer can sound convincing while describing a mechanism that does not work. 2- The results reveal a clear gap between knowledge and problem-solving. Even the strongest model scores 96.12 on standard questions but only 59.68 on challenge tasks. The detailed results show where models struggle, making the benchmark useful for identifying weaknesses that ordinary knowledge tests may miss.

Critical weaknesses

1- The reliability of the scores is not established well enough. The expert check compares average scores but does not report agreement on individual answers, where scoring errors could cancel out. Execution failures are also excluded from averages without clearly reporting their frequency for each model. Since the benchmark’s main contribution depends on trustworthy scores, these gaps make the model comparisons harder to judge. 2- The paper does not sufficiently explain what causes poor challenge performance. Models may fail because they misunderstand mechanics, cannot express their answers in the required format, or struggle with the verifier interface. The reported analysis does not clearly separate these possibilities. This matters because the paper interprets low scores as evidence of weak mechanical reasoning, while the evaluation may also measure difficulties in formatting and submitting solutions.

Questions for authors

I have the following questions for the authors: 1- On the 150 questions re-scored by experts, how closely did automated and expert scores agree on individual answers? Did disagreements change model rankings or any main conclusions? 2- How many questions were excluded from each model’s results because of execution failures, and what caused those failures? Would the model comparisons change if these cases received zero? 3- What evidence separates errors in mechanical reasoning from answer-format or extraction failures? How often did a model provide a substantively correct answer that could not be processed by the verifier? 4- For program-generated questions, were training and test sets separated by generator or mechanism family, or only by individual instances? This distinction matters for judging whether MechGPT generalizes to new problems or benefits from closely related training examples.

2 comments
AuthorsOpenReview ↗

Scoring reliability, verification failures, and evaluation scope

Question 1:

Response: Thank you for requesting item-level evidence. Appendix B, Table 6 compares 150 identical items per model (50 single-choice, 50 fill-in, and 50 open-ended), totaling 1,050 paired scores. It reports identical, higher, and lower expert scores, exact agreement (76.67%–90.00%), MAE (3.54–9.37 points), and counts with absolute differences above 20 points. These statistics supplement mean-score comparisons by showing both disagreement frequency and magnitude; errors on different items cannot simply cancel out in these measures.

Location: Section 5.2; Appendix B, Table 6.

Question 2:

Response: Thank you for asking us to clarify this. We corrected component-label input-protocol and local runtime compatibility issues, then re-verified the same original responses without new model answers. Missing required information receives zero and remains in the mean denominator. Verifiable responses receive weighted partial credit; where designated hard constraints apply, any failure caps the final score at 39. Verifier errors are reviewed rather than automatically scored as model zeros. Current comparisons use finalized scores after these corrections. A hard-constraint failure caps the raw score rather than automatically making it zero. Partial credit records satisfied criteria, not a valid design capable of continuously realizing the required motion or trajectory. Timeouts and internal errors are verifier-side exceptions, distinct from missing answer content.

Location: Section 4.4.

Question 3:

Response: Thank you for this question. A closed-service LLM faithfully converts original answers into the required structure, including answers not already in interface format. It cannot add missing information or correct mechanism content. We reviewed all responses, fixed protocol/runtime issues, and rescored the original answers; some former zeros became nonzero. Component-label normalization was strictly one-to-one: it resolved input-protocol rejections without altering the proposed mechanism. Runtime fixes allowed verification to complete. No new answers or design corrections were introduced. Category means and the overall mean across the 550 challenge questions were updated from this rescoring. After review, currently unsuccessful answers lack necessary information or contain incorrect information, rather than failing solely through interface or format mismatch. Complete candidates are assessed by task constraints and screw-subspace equivalence.

Location: Sections 4.4 and 5.3; Figure 5.

Question 4:

Response: Thank you for highlighting this distinction. Training and test items are split proportionally within question types, without generator or mechanism-family isolation. Question type specifies the response format, rather than a single mechanism structure or fixed solution procedure. A shared response format can cover different topology, joints, geometry, constraints, and solution procedures. Six capability categories and nine executable tasks provide varied evaluation conditions, but this split does not establish generalization to unseen generators or mechanism families.

Location: Sections 3.2 and 5.2.

Other concern 1: score reliability

Concern: Mean-score checks and unclear failure handling weaken confidence in comparisons.

Response: Thank you for raising this concern. Questions 1–3 above clarify the added item-level statistics, rescoring, and failure treatment. Partial credit is not evidence of a valid mechanism realizing the required motion or trajectory.

Location: Section 4.4; Appendix B, Table 6.

Other concern 2: causes of low challenge scores

Concern: Low scores may mix reasoning errors with formatting or interface difficulties.

Response: Thank you for this suggestion. Figure 5 separates mobility, motion-signature, screw-space, trajectory-geometry, and sampled-trajectory checks. On 214 mobility items, scalar properties are easier than complete screw-space construction; the 35 paired endpoint-trajectory items distinguish instantaneous motion from finite-trajectory conditions. For example, GPT-5.6-Sol passes all scalar mobility and motion-signature checks on the 214-item subset, but its strict output screw-space pass rate is 23.8%; MechGPT reaches 4.2% on output screw space. Thus, recognizing a motion class differs from constructing a geometry-consistent six-dimensional basis. Together with the faithful conversion and review described in Question 3, these diagnostics support a more specific interpretation of answer-content limitations.

Location: Sections 4.4 and 5.3; Figure 5.

Thank you for your detailed response. I feel my questions have been answered and thus I have increased the score to 3.

Reviewer_WNqUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses a specific task domain benchmark - mechanics, kinetics, and mechanism design. The paper asked two research questions: 1. Do LLMs understand domain knowledge about parallel mechanism? 3. Can the domain-specific performance improve with CPT or SFT.

The resulting artifacts are 1. a benchmark focusing on mechanics. 2. a problem construction and evaluation framework. 3. a domain specifically adapted model.

How to construct the benchmark questions: 1. expert domain knowledge sourcing. 2. conclude domain topics and extend. 3. evidence-based and program-based question generation. 4. LLM check and human expert check.

Critical strengths
  • To my knowledge, Mechanism design and analysis are largely verifiable tasks, yet remain challenging for current LLMs. So this brings a new perspective to current LLM development.

  • Human annotated benchmark is always contribution to the community.

Critical weaknesses
  • I think this paper should emphasize more on the verifiable part of the mechanics problem, and a solid verifier would be valuable.

  • There are already benchmark focusing on similar engineering domain, such as EngDesign, and some of them provide verifier. Need discussion about the difference on required mechanisms.

  • The authors have not released nor promised to release the benchmark and data construction framework.

Questions for authors

do you plan to release the benchmark?

1 comment
AuthorsOpenReview ↗

Benchmark release; Executable verification; Comparison with EngDesign; Release plan

Question 1: Benchmark release

Question 1: do you plan to release the benchmark?

Response: Thank you for your question. We plan to release the benchmark and its data-construction framework. The Ethics and Reproducibility Statements in the revised manuscript explain that release will follow source licenses and external-service terms; reference answers, item-level scoring resources, and verifier assets remain under organizer control until these conditions are met.

Location in the revised manuscript: Reproducibility Statement and Ethics Statement.

Executable verification

Reviewer comment (summary): The paper should explain the executable verification component more clearly.

Response: Thank you for this suggestion. Sections 4.4 and 5.3 clarify the executable checks. A candidate mechanism specifies links, a fixed base, an output platform, and R/P/U/S joints. The verifier recomputes topology, loop closure, geometry, mobility, output motion, and internal self-motion, and checks these properties against the task constraints. Figure 5 further reports pass rates for individual verification criteria, providing a more specific diagnosis of low aggregate scores.

Location in the revised manuscript: Sections 4.4 and 5.3; Figure 5.

Comparison with EngDesign

Reviewer comment (summary): How MechBench differs from existing engineering benchmarks such as EngDesign.

Response: Thank you for pointing out this relevant work. We have carefully compared the scope of EngDesign with that of MechBench. EngDesign is a broad engineering-design benchmark with 101 design tasks and 473 gradable items across nine engineering domains. It emphasizes open-ended designs, objective trade-offs, and functional verification through domain simulation or performance scripts. MechBench offers complementary coverage: it focuses on parallel mechanisms, a specialized domain with strong structural constraints, and contains 2,541 questions spanning six professional-knowledge capabilities and nine mechanism analysis, modification, completion, and constrained-design tasks.

MechBench explicitly encodes topological connectivity, loop closure, joint types and axes, whole-mechanism mobility, output motion, and internal motion, screw spaces, and continuous trajectories as structured inputs and executable checks. Candidates undergo deterministic payload-parsing, topology, closure, motion, and geometry checks; equivalent motion representations are assessed through the corresponding motion and constraint subspaces. MechForge combines domain-evidence organization, programmatic instance generation, and two-stage expert review to connect professional knowledge with executable mechanism reasoning. EngDesign therefore provides breadth across engineering domains, while MechBench provides depth and diagnostic detail for constraint-sensitive parallel-mechanism reasoning.

Related manuscript content: Section 4.4.

Release plan

Reviewer comment (summary): Whether the benchmark and data-construction framework will be released.

Response: Thank you for your comment. We plan to release the benchmark and its data-construction framework. The Ethics and Reproducibility Statements in the revised manuscript explain that release will follow source licenses and external-service terms; reference answers, item-level scoring resources, and verifier assets remain under organizer control until these conditions are met.

Location in the revised manuscript: Reproducibility Statement and Ethics Statement.

Reviewer_fUdeOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The authors introduces MechBench, a domain-specific benchmark containing 2,541 questions designed to evaluate large language models on parallel-mechanism knowledge, constraint-sensitive analysis, and constructive mechanical design. To construct the benchmark, the authors develop MechForge, which integrates a domain-specific knowledge hierarchy, LLM-assisted question generation, programmatic instance generation, and two-stage expert review. Evaluation combines exact answer matching, rubric-based scoring, and program-based verification of topology, kinematic constraints, loop closure, and motion feasibility.

The paper also introduces MechGPT, a domain-adapted Qwen3-8B model trained through continued pretraining and supervised fine-tuning. Experiments evaluate seven models, including closed-service and open-weight LLMs. The strongest model achieves 96.12 on the standard benchmark but only 59.68 on the challenge tier.

Critical strengths
  1. MechBench is trying to move beyond conventional knowledge-based evaluation by introducing challenge tasks that require models to satisfy structural, geometric, and kinematic constraints. In particular, its executable evaluation of mechanism mobility, joint replacement, completion, and constrained design addresses limitations of benchmarks that rely primarily on answer matching or LLM-based judgments. This is quite important because a mechanically plausible explanation does not necessarily correspond to a physically or mathematically valid mechanism. The separation of standard and challenge tiers provides a meaningful way to investigate this discrepancy.

  2. MechForge combines expert-organized domain knowledge, programmatic question generation, LLM-assisted generation, and two-stage researcher review. The resulting benchmark spans six knowledge capabilities and nine analysis/design tasks, with different scoring approaches depending on response type. The authors also conduct expert rechecks of scoring, attempt to separate training and evaluation sources, and validate the feasibility of difficult trajectory-design references. These measures strengthen the benchmark's credibility and back up the contribution more than a straightforward collection of automatically generated questions.

Critical weaknesses
  1. A central contribution of MechBench is its ability to evaluate constraint-sensitive engineering reasoning through program verification. However, the experimental results reveal that all evaluated models receive zero scores on the 35 specified-trajectory design questions, while several other constructive-design categories also have near-zero scores. Although the authors verify that the reference mechanisms are feasible and provide independently constructed solutions for a subset, these checks do not fully establish that the evaluation pipeline reliably recognizes alternative valid solutions produced by language models.

I believe this is important because engineering design problems can admit multiple mechanically equivalent solutions involving different joint arrangements, geometric parameters, or motion representations. Without broader validation of the verifier's acceptance of equivalent constructions, and an analysis separating genuinely invalid designs from representation.

Since executable verification is one of the paper's primary contributions, this uncertainty directly affects confidence in the benchmark's validity and the interpretation of its central findings.

Questions for authors

Q1 The benchmark reports zero scores for all models on specified-trajectory design tasks and near-zero scores in some other constructive-design categories. Could the authors please clarify how the verifier handles alternative mechanically valid solutions with different topologies, joint configurations, or equivalent motion representations?

Q2 Table 3 reports differences between historical scores and expert-reviewed scores of up to approximately 3.7 points. Could the authors provide item-level or criterion-level agreement statistics, including the frequency and magnitude of scoring disagreements? This would help assess whether the conclusions, particularly comparisons between similarly performing models, are robust to evaluator variation.

Q3 MechGPT is trained using both domain continued pretraining and supervised fine-tuning. Could the authors provide separate CPT-only and SFT-only ablations to quantify the contribution of each stage? It would be useful to clarify how improvements depend on genuine domain knowledge versus familiarity with benchmark-like question formats.

1 comment
AuthorsOpenReview ↗

Clarifications on verifier validity, scoring agreement, and model adaptation

Question 1: The benchmark reports zero scores for all models on specified-trajectory design tasks and near-zero scores in some other constructive-design categories. Could the authors please clarify how the verifier handles alternative mechanically valid solutions with different topologies, joint configurations, or equivalent motion representations?

Response: Thank you for this question. We reviewed all original responses and the verification pipeline. We corrected input-protocol compatibility issues through one-to-one component-label normalization and fixed local runtime compatibility issues, then rescored the same responses. Models did not answer again, and no mechanism-design content was added or corrected. Some former zeros became nonzero; category means and the overall mean across 550 challenge questions were updated.

For complete mechanism descriptions, the verifier recomputes structural, geometric, and motion properties and checks task constraints. Screw-subspace equivalence accepts different labels, vector orders, and bases. Different topologies or joint configurations are evaluated from their own properties. Partial credit does not establish design validity: for tasks with designated hard constraints, any such failure caps the final score at 39.

Location: Sections 4.4 and 5.3.

Question 2: Table 3 reports differences between historical scores and expert-reviewed scores of up to approximately 3.7 points. Could the authors provide item-level or criterion-level agreement statistics, including the frequency and magnitude of scoring disagreements? This would help assess whether the conclusions, particularly comparisons between similarly performing models, are robust to evaluator variation.

Response: Thank you for suggesting item-level statistics. Appendix B, Table 6 compares the same 150 items per model (50 single-choice, 50 fill-in, and 50 open-ended), yielding 1,050 paired scores. It reports identical, increased, and decreased expert scores, exact agreement, mean absolute error (MAE), and counts with absolute differences above 20 points. Exact agreement ranges from 76.67% to 90.00%, and MAE from 3.54 to 9.37 points. These quantify disagreement frequency and magnitude beyond mean-score comparisons.

Location: Section 5.2; Appendix B, Table 6.

Question 3: MechGPT is trained using both domain continued pretraining and supervised fine-tuning. Could the authors provide separate CPT-only and SFT-only ablations to quantify the contribution of each stage? It would be useful to clarify how improvements depend on genuine domain knowledge versus familiarity with benchmark-like question formats.

Response: Thank you for this suggestion. CPT is intended to inject mechanism-domain knowledge; SFT trains its use in question answering, analysis, and structured outputs. We have not conducted CPT-only or SFT-only ablations, so their independent contributions cannot be quantified. We interpret the reported gains as the effect of joint adaptation.

Location: Sections 4 and 7.

Other concern: verifier validity

Concern: Whether feasible references alone establish acceptance of alternative valid designs.

Response: Thank you for highlighting this distinction. Reference feasibility and candidate acceptance serve different purposes. Table 5 reports checks for all 35 specified-trajectory reference mechanisms and independent-construction checks for 10 tasks. Candidate acceptance follows the task constraints and screw-subspace comparisons described above, rather than exact matching to a reference representation. The rescoring correction and the distinction between partial credit and valid design are explained in our answer to Question 1.

Location: Sections 4.4 and 5.3; Table 5.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.