- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
BIMArena is a benchmark of 409 expert-built tasks for computer-use agents working in two commercial BIM authoring tools, Revit 2027 and Archicad 29. It has three tracks: Atomic (123 tasks), Long-Sequence (143), and Reasoning (143). Tasks run in the real environment and are scored from the final model state via the native API. Three metrics, partial completion (PCS), collateral-free rate (CFR), and success rate (SR) are designed for evaluation. The authors test 11 models and agents with and without retrieval support (official documentation plus 46 expert-refined skills). Performance drops sharply with task depth, and the benefit of the support shrinks as tasks get deeper.
Critical strengths
-
Realistic professional software with deterministic, state-based scoring that also catches collateral damage. BIMArena drives the real Revit and Archicad interfaces inside isolated VMs. It reads the final model through native APIs and checks typed checkpoints against it, with no LLM judge involved. Reporting CFR separately is particularly useful, as it captures if the agent makes changes that are not intended.
-
The paired (Long-Sequence, Reasoning) design gives a controlled breakdown of where agents fail and is interesting.
Critical weaknesses
-
Apart from the change of domain, the benchmark adds few new evaluation ideas or findings. Every vertical domain may deserve its own benchmark, but a new domain is not by itself a contribution at the level ICLR expects. A new benchmark should measure a capability, or use a measurement, that existing benchmarks do not. Set the BIM domain aside and compare what is left with prior work:
- Main finding: agents that are strong on general benchmarks fail in professional software. Earlier professional-software benchmarks already report this. ProSoftArena and DeskCraft show it across professional applications; ScienceBoard shows it in scientific software.
- Performance falls as tasks get deeper or longer. This is expected, and benchmarks with difficulty tiers or long-horizon splits already show it, e.g., OSWorld 2.0.
- External operational knowledge helps only partly, and noisy documentation can hurt. Augmenting agents with demonstrations or tutorials has been studied before (e.g., the memory module in MMINA [1]), and the harm from irrelevant retrieved context is well known from the RAG literature.
- Realistic environment with state-based scoring is the standard design of OSWorld, WindowsAgentArena, ProSoftArena and DeskCraft.
- Metrics. PCS is the same idea as checkpoint-based partial credit (TheAgentCompany). Building models that satisfy geometric and code-compliance constraints is already the target of BIM-Edit, ComCADBench, and BMEval. The difference here is that the agent works through the GUI rather than generating code or actions.
-
The paper's distinctive claims are not yet well supported.
- Reasoning-track scoring may reject correct solutions. Each Reasoning task is scored against the same expert target as its Long-Sequence pair. It does not work when the input is a minimum requirement, because many models satisfy such requirements. In Fig. 17, for example, undersized bedrooms can be enlarged by moving different walls by different amounts. Yet checkpoints like those in Listing 1 require positions within 100 mm and exact element counts, so a compliant alternative would fail.
- Each model was run only once, and several claims rest on one or two tasks. All results come from a single run at temperature 1.0 with no confidence intervals.
[1] MMInA: Benchmarking Multihop Multimodal Internet Agents
Questions for authors
-
What does BIMArena measure that existing benchmarks do not? Please state, apart from the domain, which capability or measurement is new.
-
Have you audited a sample of failed Reasoning runs manually to estimate how often the evaluator rejects a correct model?
-
How did failed runs end? For failed Long-Sequence and Reasoning runs, what fraction hit the 300-round or 60-minute cap, what fraction self-reported completion, and what fraction abandoned the task?
-
How often were the retrieval tools called? Please give calls per task for each tool, by track and model.
5 comments
Response to Reviewer zHev – Part II
(Continued, Part 2/3)
W2.1 & Q2: Evaluator Validity and Manual Audit of Failed Reasoning Tasks
A: We thank the reviewer for this important concern. For tasks allowing multiple valid solutions, we use constraints derived from architectural handbooks, standards, and expert-defined. For example, for the task mentioned by the reviewer, the evaluator verifies whether the child's bedroom provides at least 2600 x 3100 mm of clear space and the parents' bedroom at least 3600 x 3500 mm, without requiring the walls to be moved to fixed positions.
To further validate our evaluator, an independent BIM expert reviewed 300 BIM models generated by different agents during our repeated-run experiments, covering 100 models from each track. For each model, the expert assessed task success, partial completion, and unintended modifications based on the task description and a comparison of the initial and agent-generated final BIM models, without access to automated scores. We also examined sensitivity to geometric tolerances (0.5×, 1.0×, and 2.0× the original thresholds).
As shown in Table 1, the original tolerance yields the highest overall agreement with expert judgments. On Reasoning tasks, the evaluator achieves 100% SR agreement with expert judgments at the original tolerance. This provides an evidence that our constraint-based evaluation can accommodate different valid modeling solutions without requiring an exact match to a single reference model.
Table 1: Evaluator–human agreement and tolerance sensitivity on 300 expert-assessed BIM models (100 per track). SR Agreement and CFR Agreement: percentage of models on which the automated and expert binary verdicts coincide. Cohen's κ: chance-corrected agreement on binary task success. PCS Agreement: percentage of goal checkpoints, pooled over all models, on which the automated and expert verdicts coincide. PCS MAE: mean absolute difference between automated and expert-assessed per-model partial completion scores.
| Subset | Tolerance | SR Agreement | Cohen's κ | PCS Agreement | PCS MAE | CFR Agreement |
|---|---|---|---|---|---|---|
| All 300 models | 0.5× | 94.67% | 0.892 | 85.58% | 10.46% | 80.00% |
| All 300 models | 1.0× | 96.67% | 0.932 | 89.85% | 7.32% | 85.33% |
| All 300 models | 2.0× | 94.67% | 0.892 | 86.11% | 9.54% | 82.67% |
| Reasoning only | 0.5× | 100.00% | 1.000 | 87.06% | 8.78% | 88.00% |
| Reasoning only | 1.0× | 100.00% | 1.000 | 91.06% | 6.56% | 92.00% |
| Reasoning only | 2.0× | 98.00% | 0.947 | 83.89% | 10.11% | 88.00% |
We have updated the Appendix C.5 Validity of the Evaluator with these results and the corresponding analysis.
W2.2: Experimental Robustness Through Repeated Evaluations
A: We thank the reviewer for this suggestion. We conducted three independent runs on 60 tasks (14.7% of BIMArena) under both support settings including operational skills and official documentation (w/ and w/o). Tasks were selected using stratified random sampling with a fixed seed, covering 2 applications × 3 tracks (10 tasks per stratum), while preserving Long-Sequence and Reasoning task pairs. Table 2 reports the mean success rates and two-sided 95% Student's t confidence intervals across the three runs (n=3, df=2). We used temperature = 1.0 following the default model settings, as some models do not support temperature adjustments.
The repeated-run results are consistent with our main experiments: performance drops sharply with increasing task depth, substantial performance gaps persist across models, and the benefits of external support become more limited on Reasoning tasks. Weaker models consistently fail to complete Long-Sequence and Reasoning tasks, providing additional support for our two-stage evaluation strategy.
Table 2: Repeated-run robustness on the stratified 60-task subset. Values report mean SR (%) and two-sided 95% Student's t confidence intervals across three independent runs.
| Model | Support | Atomic SR | Long-Seq. SR | Reasoning SR |
|---|---|---|---|---|
| Claude Opus 5 | w/o | 95.0 [95.0, 95.0] | 53.3 [46.2, 60.5] | 35.0 [35.0, 35.0] |
| w/ | 95.0 [95.0, 95.0] | 56.7 [49.5, 63.8] | 36.7 [29.5, 43.8] | |
| Gemini 3.7 Flash | w/o | 80.0 [80.0, 80.0] | 36.7 [29.5, 43.8] | 18.3 [11.2, 25.5] |
| w/ | 88.3 [81.2, 95.5] | 40.0 [40.0, 40.0] | 20.0 [20.0, 20.0] | |
| GPT-5.6 Sol | w/o | 73.3 [66.2, 80.5] | 33.3 [26.2, 40.5] | 11.7 [4.5, 18.8] |
| w/ | 80.0 [80.0, 80.0] | 41.7 [34.5, 48.8] | 16.7 [9.5, 23.8] | |
| Qwen3.7-Plus | w/o | 0.0 [0.0, 0.0] | 0.0 [0.0, 0.0] | 0.0 [0.0, 0.0] |
| w/ | 5.0 [5.0, 5.0] | 0.0 [0.0, 0.0] | 0.0 [0.0, 0.0] | |
| EvoCUA-32B | w/o | 5.0 [5.0, 5.0] | 0.0 [0.0, 0.0] | 0.0 [0.0, 0.0] |
We have added the Appendix D.4 Run-to-Run Robustness with these results and the corresponding analysis in our revised manuscript.
Response to Reviewer zHev – Part III
(Continued, Part 3/3)
Q3: Failure Termination Analysis for Long-Sequence and Reasoning Tasks
A: We thank the reviewer for this question. We analyzed the termination patterns of failed Long-Sequence and Reasoning runs across both support settings, as shown in Table 3.
Table 3: Termination patterns of failed runs.
| Track | Support | Round Limit | Time Limit | Reported Complete | Abandoned |
|---|---|---|---|---|---|
| Long-Seq. | w/o | 8.9% | 2.2% | 78.4% | 10.5% |
| Long-Seq. | w/ | 5.6% | 2.5% | 76.3% | 15.6% |
| Reasoning | w/o | 3.9% | 0.6% | 87.6% | 8.0% |
| Reasoning | w/ | 3.5% | 1.6% | 83.7% | 11.2% |
Most failed runs self-reported completion, accounting for 76.3–78.4% of Long-Sequence failures and 83.7–87.6% of Reasoning failures. Relatively few runs reached the round or time limits. Interestingly, despite their higher complexity, Reasoning tasks reached these round and time limits less frequently than Long-Sequence tasks, with combined rates 6.6 percentage points lower without support and 3.0 percentage points lower with support. Instead, agents more often terminated on their own by claiming task completion. This suggests that failures are primarily related to agents' inability to recognize unmet requirements and accurately assess task completion, rather than insufficient execution budgets. We have added a detailed analysis in Appendix E.2 (Failure Termination Analysis) of the revised manuscript.
Q4: Retrieval Tool Usage Analysis Across Models and Task Levels
A: We thank the reviewer for this suggestion. We analyzed the average number of documentation and operational skill retrieval calls per task across models and evaluation tracks, as shown in Table 4.
Table 4: Average documentation (Doc) and operational skill (Skill) retrieval calls per task under operational support.
| Model | Atomic Doc/Skill | Long-Seq. Doc/Skill | Reasoning Doc/Skill |
|---|---|---|---|
| Claude Opus 5 | 0.07 / 1.42 | 0.12 / 1.97 | 0.04 / 2.00 |
| Gemini 3.7 Flash | 0.46 / 1.15 | 0.79 / 3.52 | 0.57 / 3.16 |
| GPT-5.6 Sol | 0.26 / 1.67 | 1.43 / 4.38 | 1.27 / 4.11 |
| GPT-5.6 Terra | 0.51 / 1.72 | 0.82 / 4.31 | 0.63 / 3.89 |
| GPT-5.6 Luna | 0.50 / 1.67 | 0.58 / 4.29 | 1.10 / 3.94 |
| Claude Sonnet 5 | 0.12 / 1.43 | 0.24 / 2.59 | 0.11 / 2.42 |
| Qwen3.8-Max | 0.04 / 1.09 | – | – |
| Qwen3.7-Plus | 0.05 / 0.33 | – | – |
| Muse Spark 1.1 | 1.33 / 1.43 | – | – |
The results show that agents consistently invoke operational skills more frequently than documentation, with retrieval usage increasing substantially on complex tasks. For example, GPT-5.6 Sol increases its skill retrieval calls from 1.67 on Atomic tasks to 4.38 on Long-Sequence and 4.11 on Reasoning tasks. However, the corresponding SR improvements from operational support decrease from +10.6 to +7.7 and +4.2 percentage points, respectively. Similarly, Gemini 3.7 Flash increases its skill retrieval calls from 1.15 to 3.52 and 3.16, while its SR improvements decrease from +7.3 to +4.2 and 0.0 points. Claude Opus 5 even exhibits a 1.4-point decrease in Reasoning SR despite invoking operational skills 2.00 times per task. These findings suggest that increased retrieval usage does not translate into proportional performance improvements, highlighting the diminishing benefits of operational support as task depth increases. We have added these statistics and further analysis to Appendix D.5 (Operational Support Usage Analysis) of the revised manuscript.
Response to Reviewer zHev – Part I
We sincerely thank the reviewer for the insightful comments and constructive suggestions. We address each concern and question as follows.
A: We thank the reviewer for this detailed assessment. BIMArena's main contribution lies in providing a systematic and progressive evaluation of the depth of agent competence within a single knowledge-intensive professional domain, enabling finer-grained diagnosis of capabilities that are difficult to distinguish in broad benchmarks.
1. Breadth vs. Depth (W1.1). OSWorld 2.0 [1] covers 108 tasks across 7 domains (15.4 tasks/domain); ScienceBoard [2] contains 169 tasks across 6 domains and 6 applications (28.2 tasks/domain, 28.2 tasks/application); ProSoftArena [3] includes 456 tasks across 6 domains and 13 applications (76.0 tasks/domain, 35.1 tasks/application); and DeskCraft [4] comprises 538 tasks across 5 domains and 11 applications (107.6 tasks/domain, 48.9 tasks/application). While these benchmarks provide valuable cross-domain evaluation, their broad coverage limits the number of tasks dedicated to each domain. In contrast, BIMArena provides 409 tasks within a single domain and 204.5 tasks per application, covering 23 fundamental capabilities and their combinations in complex professional workflows.
2. Beyond Long-Horizon Difficulty (W1.2). While OSWorld 2.0 explicitly focuses on long-horizon tasks, ProSoftArena and DeskCraft also include some long-sequence tasks. However, longer workflows amplify error propagation and make failure diagnosis increasingly challenging. BIMArena addresses this through its interconnected three-track design: Long-Sequence and Reasoning tasks are composed of fundamental capabilities evaluated in the Atomic track, while Long-Sequence and Reasoning tasks are paired with the same modeling objective. This structure enables more systematic diagnosis of complex workflow failures. By progressively evaluating capabilities from basic operations to complex professional workflows, BIMArena offers a transferable benchmark design for other specialized domains and provides a more interpretable diagnostic framework to guide the development of domain-specific agents.
3. Operational Knowledge (W1.3). We acknowledge that prior work has explored external operational knowledge (MMInA [5]). Rather than proposing a new retrieval method, BIMArena investigates how the effectiveness of operational support changes with professional task depth. Our results show that support improves Atomic SR by 7.5 percentage points on average, but Reasoning SR by only 0.8 points. Beyond the known negative effects of noisy documentation, our findings reveal diminishing benefits of external operational knowledge as workflows become more complex and knowledge intensive.
4. State-Based Evaluation and Metrics (W1.4–W1.5). We acknowledge that state-based evaluation and partial completion scoring are not methodological innovations of BIMArena. Rather, we adopt these established approaches in BIM authoring environments, where mature native APIs enable reliable verification of geometric and semantic model states, as demonstrated by existing BIM/CAD benchmarks. Our contribution is to extend these evaluation principles to GUI-based computer use in professional BIM software. Our evaluation additionally differs by introducing the CFR, a BIM-specific metric that measures whether existing model elements remain unaffected during task execution.
Broader Significance. The lack of in-depth domain-specialized agent benchmarks is not unique to BIM. We select BIM as a representative testbed because its complex workflows and domain knowledge requirements, together with mature software infrastructure and native APIs, enable rigorous state-based evaluation. We hope this depth-oriented approach can inspire similar benchmarks in other professional domains and advance the development of domain-specialized agents.
We have expanded Appendix A.6: Comparison with Existing Benchmarks, and Section 1 Introduction, to clarify these contributions.
References
[1] Yuan, Mengqi, et al. "OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks." arXiv:2606.29537 (2026).
[2] Sun, Qiushi, et al. "ScienceBoard: Evaluating Multimodal Autonomous Agents in Realistic Scientific Workflows." ICLR (2026).
[3] Ai, Jiaxin, et al. "ProSoftArena: Benchmarking Hierarchical Capabilities of Multi-modal Agents in Professional Software Environments." CVPR (2026).
[4] Wang, Wenkai, et al. "DeskCraft: Benchmarking Desktop Agents on Professional Workflows and Human-in-the-Loop Collaboration." arXiv:2606.03103 (2026).
[5] Tian, Shulin, et al. "MMInA: Benchmarking Multihop Multimodal Internet Agents." arXiv:2404.09992 (2024).