acceptodds
Under review as a conference paper at ICLR 2027

BIMArena: Benchmarking Domain-Specialized Computer-Use in Professional Building Modeling

11 views

Abstract

Computer-use agents are advancing rapidly; yet, existing benchmarks focus on general-purpose applications and do not fully reflect the complexity of real-world workflows. Moreover, they tend to prioritize breadth across applications over depth within professional workflows that span diverse functionalities and tightly coupled operations. In this work, we introduce BIMArena, a domain-specialized benchmark for evaluating the depth of computer-use agent capabilities in Building Information Modeling (BIM) authoring environments, a complex, knowledge-intensive professional domain that remains underexplored in current benchmarks. BIMArena comprises 409 tasks that progressively evaluate fundamental professional software operations, long-sequence workflows, and domain reasoning tasks. We further investigate whether external operational support through software documentation and expert-curated operational skills improves agent performance. Experiments are conducted in isolated real-computer environments using a programmatic evaluation framework. They reveal a pronounced capability gap: the strongest agent achieves a 98.4% success rate on fundamental software operations, yet only 26.6% on realistic, professional tasks. Our analysis shows that models strong on general computer-use benchmarks can still struggle with professional tasks beyond basic user interface operations. Operational support also becomes less effective with increasing task depth and can even be detrimental.

Primary area:
datasets and benchmarks
Keywords:
Computer-Use Agents, GUI Agents, Building Information Modeling, Professional Software, Benchmarking, Long-Horizon Task Execution, Domain-Specific Reasoning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_zHevOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

BIMArena is a benchmark of 409 expert-built tasks for computer-use agents working in two commercial BIM authoring tools, Revit 2027 and Archicad 29. It has three tracks: Atomic (123 tasks), Long-Sequence (143), and Reasoning (143). Tasks run in the real environment and are scored from the final model state via the native API. Three metrics, partial completion (PCS), collateral-free rate (CFR), and success rate (SR) are designed for evaluation. The authors test 11 models and agents with and without retrieval support (official documentation plus 46 expert-refined skills). Performance drops sharply with task depth, and the benefit of the support shrinks as tasks get deeper.

Critical strengths
  • Realistic professional software with deterministic, state-based scoring that also catches collateral damage. BIMArena drives the real Revit and Archicad interfaces inside isolated VMs. It reads the final model through native APIs and checks typed checkpoints against it, with no LLM judge involved. Reporting CFR separately is particularly useful, as it captures if the agent makes changes that are not intended.

  • The paired (Long-Sequence, Reasoning) design gives a controlled breakdown of where agents fail and is interesting.

Critical weaknesses
  • Apart from the change of domain, the benchmark adds few new evaluation ideas or findings. Every vertical domain may deserve its own benchmark, but a new domain is not by itself a contribution at the level ICLR expects. A new benchmark should measure a capability, or use a measurement, that existing benchmarks do not. Set the BIM domain aside and compare what is left with prior work:

    • Main finding: agents that are strong on general benchmarks fail in professional software. Earlier professional-software benchmarks already report this. ProSoftArena and DeskCraft show it across professional applications; ScienceBoard shows it in scientific software.
    • Performance falls as tasks get deeper or longer. This is expected, and benchmarks with difficulty tiers or long-horizon splits already show it, e.g., OSWorld 2.0.
    • External operational knowledge helps only partly, and noisy documentation can hurt. Augmenting agents with demonstrations or tutorials has been studied before (e.g., the memory module in MMINA [1]), and the harm from irrelevant retrieved context is well known from the RAG literature.
    • Realistic environment with state-based scoring is the standard design of OSWorld, WindowsAgentArena, ProSoftArena and DeskCraft.
    • Metrics. PCS is the same idea as checkpoint-based partial credit (TheAgentCompany). Building models that satisfy geometric and code-compliance constraints is already the target of BIM-Edit, ComCADBench, and BMEval. The difference here is that the agent works through the GUI rather than generating code or actions.
  • The paper's distinctive claims are not yet well supported.

    • Reasoning-track scoring may reject correct solutions. Each Reasoning task is scored against the same expert target as its Long-Sequence pair. It does not work when the input is a minimum requirement, because many models satisfy such requirements. In Fig. 17, for example, undersized bedrooms can be enlarged by moving different walls by different amounts. Yet checkpoints like those in Listing 1 require positions within 100 mm and exact element counts, so a compliant alternative would fail.
    • Each model was run only once, and several claims rest on one or two tasks. All results come from a single run at temperature 1.0 with no confidence intervals.

[1] MMInA: Benchmarking Multihop Multimodal Internet Agents

Questions for authors
  • What does BIMArena measure that existing benchmarks do not? Please state, apart from the domain, which capability or measurement is new.

  • Have you audited a sample of failed Reasoning runs manually to estimate how often the evaluator rejects a correct model?

  • How did failed runs end? For failed Long-Sequence and Reasoning runs, what fraction hit the 300-round or 60-minute cap, what fraction self-reported completion, and what fraction abandoned the task?

  • How often were the retrieval tools called? Please give calls per task for each tool, by track and model.

5 comments
AuthorsOpenReview ↗

Response to Reviewer zHev – Part I

We sincerely thank the reviewer for the insightful comments and constructive suggestions. We address each concern and question as follows.


W1 & Q1: Core Contributions and Distinctive Capabilities of BIMArena Compared with Existing Benchmarks

A: We thank the reviewer for this detailed assessment. BIMArena's main contribution lies in providing a systematic and progressive evaluation of the depth of agent competence within a single knowledge-intensive professional domain, enabling finer-grained diagnosis of capabilities that are difficult to distinguish in broad benchmarks.

1. Breadth vs. Depth (W1.1). OSWorld 2.0 [1] covers 108 tasks across 7 domains (15.4 tasks/domain); ScienceBoard [2] contains 169 tasks across 6 domains and 6 applications (28.2 tasks/domain, 28.2 tasks/application); ProSoftArena [3] includes 456 tasks across 6 domains and 13 applications (76.0 tasks/domain, 35.1 tasks/application); and DeskCraft [4] comprises 538 tasks across 5 domains and 11 applications (107.6 tasks/domain, 48.9 tasks/application). While these benchmarks provide valuable cross-domain evaluation, their broad coverage limits the number of tasks dedicated to each domain. In contrast, BIMArena provides 409 tasks within a single domain and 204.5 tasks per application, covering 23 fundamental capabilities and their combinations in complex professional workflows.

2. Beyond Long-Horizon Difficulty (W1.2). While OSWorld 2.0 explicitly focuses on long-horizon tasks, ProSoftArena and DeskCraft also include some long-sequence tasks. However, longer workflows amplify error propagation and make failure diagnosis increasingly challenging. BIMArena addresses this through its interconnected three-track design: Long-Sequence and Reasoning tasks are composed of fundamental capabilities evaluated in the Atomic track, while Long-Sequence and Reasoning tasks are paired with the same modeling objective. This structure enables more systematic diagnosis of complex workflow failures. By progressively evaluating capabilities from basic operations to complex professional workflows, BIMArena offers a transferable benchmark design for other specialized domains and provides a more interpretable diagnostic framework to guide the development of domain-specific agents.

3. Operational Knowledge (W1.3). We acknowledge that prior work has explored external operational knowledge (MMInA [5]). Rather than proposing a new retrieval method, BIMArena investigates how the effectiveness of operational support changes with professional task depth. Our results show that support improves Atomic SR by 7.5 percentage points on average, but Reasoning SR by only 0.8 points. Beyond the known negative effects of noisy documentation, our findings reveal diminishing benefits of external operational knowledge as workflows become more complex and knowledge intensive.

4. State-Based Evaluation and Metrics (W1.4–W1.5). We acknowledge that state-based evaluation and partial completion scoring are not methodological innovations of BIMArena. Rather, we adopt these established approaches in BIM authoring environments, where mature native APIs enable reliable verification of geometric and semantic model states, as demonstrated by existing BIM/CAD benchmarks. Our contribution is to extend these evaluation principles to GUI-based computer use in professional BIM software. Our evaluation additionally differs by introducing the CFR, a BIM-specific metric that measures whether existing model elements remain unaffected during task execution.

Broader Significance. The lack of in-depth domain-specialized agent benchmarks is not unique to BIM. We select BIM as a representative testbed because its complex workflows and domain knowledge requirements, together with mature software infrastructure and native APIs, enable rigorous state-based evaluation. We hope this depth-oriented approach can inspire similar benchmarks in other professional domains and advance the development of domain-specialized agents.

We have expanded Appendix A.6: Comparison with Existing Benchmarks, and Section 1 Introduction, to clarify these contributions.

References

[1] Yuan, Mengqi, et al. "OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks." arXiv:2606.29537 (2026).

[2] Sun, Qiushi, et al. "ScienceBoard: Evaluating Multimodal Autonomous Agents in Realistic Scientific Workflows." ICLR (2026).

[3] Ai, Jiaxin, et al. "ProSoftArena: Benchmarking Hierarchical Capabilities of Multi-modal Agents in Professional Software Environments." CVPR (2026).

[4] Wang, Wenkai, et al. "DeskCraft: Benchmarking Desktop Agents on Professional Workflows and Human-in-the-Loop Collaboration." arXiv:2606.03103 (2026).

[5] Tian, Shulin, et al. "MMInA: Benchmarking Multihop Multimodal Internet Agents." arXiv:2404.09992 (2024).

AuthorsOpenReview ↗

Response to Reviewer zHev – Part II

(Continued, Part 2/3)


W2.1 & Q2: Evaluator Validity and Manual Audit of Failed Reasoning Tasks

A: We thank the reviewer for this important concern. For tasks allowing multiple valid solutions, we use constraints derived from architectural handbooks, standards, and expert-defined. For example, for the task mentioned by the reviewer, the evaluator verifies whether the child's bedroom provides at least 2600 x 3100 mm of clear space and the parents' bedroom at least 3600 x 3500 mm, without requiring the walls to be moved to fixed positions.

To further validate our evaluator, an independent BIM expert reviewed 300 BIM models generated by different agents during our repeated-run experiments, covering 100 models from each track. For each model, the expert assessed task success, partial completion, and unintended modifications based on the task description and a comparison of the initial and agent-generated final BIM models, without access to automated scores. We also examined sensitivity to geometric tolerances (0.5×, 1.0×, and 2.0× the original thresholds).

As shown in Table 1, the original tolerance yields the highest overall agreement with expert judgments. On Reasoning tasks, the evaluator achieves 100% SR agreement with expert judgments at the original tolerance. This provides an evidence that our constraint-based evaluation can accommodate different valid modeling solutions without requiring an exact match to a single reference model.

Table 1: Evaluator–human agreement and tolerance sensitivity on 300 expert-assessed BIM models (100 per track). SR Agreement and CFR Agreement: percentage of models on which the automated and expert binary verdicts coincide. Cohen's κ: chance-corrected agreement on binary task success. PCS Agreement: percentage of goal checkpoints, pooled over all models, on which the automated and expert verdicts coincide. PCS MAE: mean absolute difference between automated and expert-assessed per-model partial completion scores.

SubsetToleranceSR AgreementCohen's κPCS AgreementPCS MAECFR Agreement
All 300 models0.5×94.67%0.89285.58%10.46%80.00%
All 300 models1.0×96.67%0.93289.85%7.32%85.33%
All 300 models2.0×94.67%0.89286.11%9.54%82.67%
Reasoning only0.5×100.00%1.00087.06%8.78%88.00%
Reasoning only1.0×100.00%1.00091.06%6.56%92.00%
Reasoning only2.0×98.00%0.94783.89%10.11%88.00%

We have updated the Appendix C.5 Validity of the Evaluator with these results and the corresponding analysis.


W2.2: Experimental Robustness Through Repeated Evaluations

A: We thank the reviewer for this suggestion. We conducted three independent runs on 60 tasks (14.7% of BIMArena) under both support settings including operational skills and official documentation (w/ and w/o). Tasks were selected using stratified random sampling with a fixed seed, covering 2 applications × 3 tracks (10 tasks per stratum), while preserving Long-Sequence and Reasoning task pairs. Table 2 reports the mean success rates and two-sided 95% Student's t confidence intervals across the three runs (n=3, df=2). We used temperature = 1.0 following the default model settings, as some models do not support temperature adjustments.

The repeated-run results are consistent with our main experiments: performance drops sharply with increasing task depth, substantial performance gaps persist across models, and the benefits of external support become more limited on Reasoning tasks. Weaker models consistently fail to complete Long-Sequence and Reasoning tasks, providing additional support for our two-stage evaluation strategy.

Table 2: Repeated-run robustness on the stratified 60-task subset. Values report mean SR (%) and two-sided 95% Student's t confidence intervals across three independent runs.

ModelSupportAtomic SRLong-Seq. SRReasoning SR
Claude Opus 5w/o95.0 [95.0, 95.0]53.3 [46.2, 60.5]35.0 [35.0, 35.0]
w/95.0 [95.0, 95.0]56.7 [49.5, 63.8]36.7 [29.5, 43.8]
Gemini 3.7 Flashw/o80.0 [80.0, 80.0]36.7 [29.5, 43.8]18.3 [11.2, 25.5]
w/88.3 [81.2, 95.5]40.0 [40.0, 40.0]20.0 [20.0, 20.0]
GPT-5.6 Solw/o73.3 [66.2, 80.5]33.3 [26.2, 40.5]11.7 [4.5, 18.8]
w/80.0 [80.0, 80.0]41.7 [34.5, 48.8]16.7 [9.5, 23.8]
Qwen3.7-Plusw/o0.0 [0.0, 0.0]0.0 [0.0, 0.0]0.0 [0.0, 0.0]
w/5.0 [5.0, 5.0]0.0 [0.0, 0.0]0.0 [0.0, 0.0]
EvoCUA-32Bw/o5.0 [5.0, 5.0]0.0 [0.0, 0.0]0.0 [0.0, 0.0]

We have added the Appendix D.4 Run-to-Run Robustness with these results and the corresponding analysis in our revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer zHev – Part III

(Continued, Part 3/3)


Q3: Failure Termination Analysis for Long-Sequence and Reasoning Tasks

A: We thank the reviewer for this question. We analyzed the termination patterns of failed Long-Sequence and Reasoning runs across both support settings, as shown in Table 3.

Table 3: Termination patterns of failed runs.

TrackSupportRound LimitTime LimitReported CompleteAbandoned
Long-Seq.w/o8.9%2.2%78.4%10.5%
Long-Seq.w/5.6%2.5%76.3%15.6%
Reasoningw/o3.9%0.6%87.6%8.0%
Reasoningw/3.5%1.6%83.7%11.2%

Most failed runs self-reported completion, accounting for 76.3–78.4% of Long-Sequence failures and 83.7–87.6% of Reasoning failures. Relatively few runs reached the round or time limits. Interestingly, despite their higher complexity, Reasoning tasks reached these round and time limits less frequently than Long-Sequence tasks, with combined rates 6.6 percentage points lower without support and 3.0 percentage points lower with support. Instead, agents more often terminated on their own by claiming task completion. This suggests that failures are primarily related to agents' inability to recognize unmet requirements and accurately assess task completion, rather than insufficient execution budgets. We have added a detailed analysis in Appendix E.2 (Failure Termination Analysis) of the revised manuscript.


Q4: Retrieval Tool Usage Analysis Across Models and Task Levels

A: We thank the reviewer for this suggestion. We analyzed the average number of documentation and operational skill retrieval calls per task across models and evaluation tracks, as shown in Table 4.

Table 4: Average documentation (Doc) and operational skill (Skill) retrieval calls per task under operational support.

ModelAtomic Doc/SkillLong-Seq. Doc/SkillReasoning Doc/Skill
Claude Opus 50.07 / 1.420.12 / 1.970.04 / 2.00
Gemini 3.7 Flash0.46 / 1.150.79 / 3.520.57 / 3.16
GPT-5.6 Sol0.26 / 1.671.43 / 4.381.27 / 4.11
GPT-5.6 Terra0.51 / 1.720.82 / 4.310.63 / 3.89
GPT-5.6 Luna0.50 / 1.670.58 / 4.291.10 / 3.94
Claude Sonnet 50.12 / 1.430.24 / 2.590.11 / 2.42
Qwen3.8-Max0.04 / 1.09––
Qwen3.7-Plus0.05 / 0.33––
Muse Spark 1.11.33 / 1.43––

The results show that agents consistently invoke operational skills more frequently than documentation, with retrieval usage increasing substantially on complex tasks. For example, GPT-5.6 Sol increases its skill retrieval calls from 1.67 on Atomic tasks to 4.38 on Long-Sequence and 4.11 on Reasoning tasks. However, the corresponding SR improvements from operational support decrease from +10.6 to +7.7 and +4.2 percentage points, respectively. Similarly, Gemini 3.7 Flash increases its skill retrieval calls from 1.15 to 3.52 and 3.16, while its SR improvements decrease from +7.3 to +4.2 and 0.0 points. Claude Opus 5 even exhibits a 1.4-point decrease in Reasoning SR despite invoking operational skills 2.00 times per task. These findings suggest that increased retrieval usage does not translate into proportional performance improvements, highlighting the diminishing benefits of operational support as task depth increases. We have added these statistics and further analysis to Appendix D.5 (Operational Support Usage Analysis) of the revised manuscript.

Reviewer_ujn2OpenReview ↗
Rating
3
Confidence
3(+1)
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces BIMArena, a benchmark designed to assess computer-use agents in professional BIM environments. The benchmark consists of 409 tasks across three levels, covering fundamental software operations, long-sequence workflows, and domain-specific reasoning. The authors evaluate multiple agents in isolated computer environments using a programmatic evaluation framework. Experimental results reveal a substantial gap between basic software proficiency and complex professional capabilities. The strongest agent achieves a 98.4% success rate on atomic tasks, but only 26.6% on reasoning tasks. The study further examines external operational support, showing that its effectiveness diminishes as task complexity increases and can occasionally have negative effects.

Critical strengths
  1. The paper introduces a comprehensive benchmark covering 409 tasks across two professional BIM applications, organized into three levels of increasing task depth. The evaluation includes a diverse set of proprietary models and open-source computer-use agents. The paired design of Long-Sequence and Reasoning tasks further enables a meaningful comparison of agent performance under different levels of task specification.

  2. The authors develop an automated evaluation framework using isolated virtual environments and programmatic evaluation. The framework assesses agent performance through multiple metrics, including task success, partial completion, and unintended modifications to existing models. The paper also provides a publicly accessible code repository, which supports the reproducibility and potential reuse of the benchmark.

Critical weaknesses
  1. Limited Novelty and Insufficient Analytical Insights. The methodological novelty of BIMArena is relatively limited. As discussed in Appendix A.6, existing computer-use benchmarks, such as OSWorld and ProSoftArena, have already explored complex software interactions and state-based evaluation. BIMArena primarily extends these established evaluation paradigms to a specialized professional domain. While the paired Long-Sequence and Reasoning tasks represent a meaningful contribution, the analysis in Section 5.1 does not fully exploit this design to provide deeper insights into the distinct challenges of domain reasoning and long-horizon execution. So the paper's contributions are more incremental than fundamentally novel.

  2. Insufficient Evaluation Validation and Experimental Robustness. Although the paper develops a programmatic evaluation framework, its validation in Appendix C.5 primarily relies on unchanged initial models and expert-constructed target models. This does not fully establish whether the evaluator can reliably assess partially correct or incorrectly modified BIM models. Furthermore, each model is evaluated only once per task, leaving the robustness of the reported performance differences unclear.

Questions for authors
  1. The evaluation framework relies on programmatic metrics (SR, PCS, and CFR), but Appendix C.5 primarily validates the evaluator using initial and expert-constructed target models. Could the authors include a human evaluation of a representative subset of agent-generated BIM models, particularly partially completed and failed cases?

  2. Could the authors provide a sensitivity analysis of the geometric tolerances used in the automated evaluation framework? This would help clarify whether the reported success rates remain consistent under different evaluation thresholds.

  3. Given that each model is evaluated only once per task, how robust are the reported results to inference randomness? Could the authors provide repeated evaluations on a representative subset of tasks to assess the stability of the performance differences?

5 comments
AuthorsOpenReview ↗

Response to Reviewer ujn2 – Part I

We sincerely thank the reviewer for the insightful comments and constructive suggestions. We address each concern and question as follows.


W1.1: Novelty and Comparison with Existing Benchmarks

A: We thank the reviewer for this concern. While existing benchmarks have explored general-purpose GUI interactions in complex professional software, BIMArena's key contribution lies in evaluating the depth of agent competence within a single knowledge-intensive professional domain.

The limitations of existing benchmarks can be summarized as follows:

1. Breadth Rather Than Depth. OSWorld 2.0 [1] covers 108 tasks across 7 domains (15.4 tasks/domain); ScienceBoard [2] contains 169 tasks across 6 domains and 6 applications (28.2 tasks/domain, 28.2 tasks/application); ProSoftArena [3] includes 456 tasks across 6 domains and 13 applications (76.0 tasks/domain, 35.1 tasks/application); and DeskCraft [4] comprises 538 tasks across 5 domains and 11 applications (107.6 tasks/domain, 48.9 tasks/application). Such broad coverage inevitably limits the depth of evaluation within any single domain. In contrast, BIMArena provides 409 tasks within a single domain and 204.5 tasks per application, enabling deeper coverage of professional workflows. This depth-oriented design is further reflected in two aspects:

2. Execution Complexity. BIMArena averages 50.9 expert GUI steps per task, increasing to 69.2 for Long-Sequence and Reasoning tasks. These are approximately 3.9 and 5.4 times the reported 12.9 steps in ProSoftArena and 5.7 and 7.7 times the reported 9.0 steps in ScienceBoard.

3. Task Inputs. Unlike existing benchmarks primarily relying on explicit natural-language instructions, BIMArena incorporates architectural drawings, design requirements, and regulatory constraints, requiring agents to infer modeling objectives and plan complex workflows.

Furthermore, 4. Diagnostic Task Design is another key distinction. Error propagation is a major challenge in long-sequence tasks. Several benchmarks already organize tasks into difficulty levels (e.g., the capability hierarchy in ProSoftArena and the L1–L3 taxonomy in DeskCraft). BIMArena differs in how its levels are linked: every Long-Sequence and Reasoning task is annotated with the Atomic capabilities it is composed of, and each Reasoning task is paired with a Long-Sequence task that has the same modeling objective. Failures can therefore be analyzed in terms of their underlying causes, rather than merely attributed to task difficulty, allowing us to distinguish limitations in domain-specific reasoning from the effects of task length.

The challenges of deep professional competence are not unique to BIM but extend across knowledge-intensive domains. We select BIM as a representative testbed due to its complex workflows and substantial domain knowledge requirements. Besides, the mature software infrastructure and native APIs of BIM applications enable state-based evaluation. Our depth-oriented evaluation approach can be extended to other professional domains, providing a benchmark foundation for assessing general-purpose agents and advancing domain-specialized agents.

We have expanded Appendix A.6: Comparison with Existing Benchmarks, and Section 1 Introduction, to clarify these contributions.

References

[1] Yuan, Mengqi, et al. "OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks." arXiv:2606.29537 (2026).

[2] Sun, Qiushi, et al. "ScienceBoard: Evaluating Multimodal Autonomous Agents in Realistic Scientific Workflows." ICLR (2026).

[3] Ai, Jiaxin, et al. "ProSoftArena: Benchmarking Hierarchical Capabilities of Multi-modal Agents in Professional Software Environments." CVPR (2026).

[4] Wang, Wenkai, et al. "DeskCraft: Benchmarking Desktop Agents on Professional Workflows and Human-in-the-Loop Collaboration." arXiv:2606.03103 (2026).

AuthorsOpenReview ↗

Response to Reviewer ujn2 – Part II

(Continued, Part 2/3)


W1.2: In-Depth Analysis of Task Failures

A: We thank the reviewer for this important suggestion. To provide a more fine-grained failure analysis, we carefully reviewed all 143 paired tasks and categorized them based on their primary domain-specific requirements, assigning one or more of three labels: Visual Understanding, Geometric Understanding, and Domain-Specific Reasoning. We then compared the success rates (w/o support) of paired Long-Sequence and Reasoning tasks across these categories.

As shown in Table 1, SR decreases by an average of 12.6 percentage points from Long-Sequence to Reasoning tasks. Domain-Specific Reasoning shows the largest drop (22.7 points), compared with Visual Understanding (13.7 points) and Geometric Understanding (7.7 points). This suggests that domain-specific reasoning presents a particularly significant challenge beyond workflow execution.

Table 1: Success rate degradation from Long-Sequence to Reasoning tasks (percentage points).

ModelOverallVisualGeometricDomain
Claude Opus 5−18.9−20.0−6.4−30.6
Gemini 3.7 Flash−16.8−17.8−10.3−30.6
GPT-5.6 Sol−14.0−15.6−10.3−22.2
GPT-5.6 Terra−11.2−8.9−5.1−22.2
GPT-5.6 Luna−5.6−8.9−6.4−13.9
Claude Sonnet 5−9.1−11.1−7.7−16.7
Average−12.6−13.7−7.7−22.7

We have updated Section 5.2: Error Analysis, and Appendix E: Failure Analysis in our revised manuscript.


W2.1 & Q1 & Q2: Evaluator Reliability, Human Expert Validation and Sensitivity Analysis of Geometric Evaluation Tolerances

A: We thank the reviewer for this important concern. To further validate our evaluator beyond the initial and expert-target baselines, we designed additional analyses:

1. Human Expert Validation. An independent BIM expert reviewed 300 BIM models generated by different agents (100 from each track) on the stratified subset used in the repeated-run experiments (W2.2 & Q3). For each model, the expert assessed task success, partial completion, and unintended modifications based on the task description and a comparison of the initial and agent-generated final BIM models, without access to automated scores.

2. Geometric Tolerance Sensitivity. We examine evaluator sensitivity by varying geometric tolerances (0.5×, 1×, and 2× the original thresholds) and measuring agreement with expert judgments.

Table 2: Evaluator–human agreement and tolerance sensitivity on 300 expert-assessed BIM models (100 per track). SR Agreement and CFR Agreement: percentage of models on which the automated and expert binary verdicts coincide. Cohen's κ: chance-corrected agreement on binary task success. PCS Agreement: percentage of goal checkpoints, pooled over all models, on which the automated and expert verdicts coincide. PCS MAE: mean absolute difference between automated and expert-assessed per-model partial completion scores.

SubsetToleranceSR AgreementCohen's κPCS AgreementPCS MAECFR Agreement
All 300 models0.5×94.67%0.89285.58%10.46%80.00%
All 300 models1.0×96.67%0.93289.85%7.32%85.33%
All 300 models2.0×94.67%0.89286.11%9.54%82.67%
Reasoning only0.5×100.00%1.00087.06%8.78%88.00%
Reasoning only1.0×100.00%1.00091.06%6.56%92.00%
Reasoning only2.0×98.00%0.94783.89%10.11%88.00%

As shown in Table 2, at the original tolerance, the evaluator achieves 96.67% SR agreement with human judgments (κ = 0.932), indicating strong agreement. SR agreement remains 94.67% under both the 0.5× and 2× tolerances, and the original tolerance yields the highest overall agreement with human judgments. We also evaluated Reasoning tasks separately to assess how well our evaluator handles alternative valid modeling solutions. For such tasks, evaluation is based on task-specific constraints rather than exact reference-model matching. For instance, room dimensions and opening widths are checked against minimum or maximum requirements derived from architectural handbooks and standards. The 100% SR agreement with human judgments on Reasoning tasks provides further evidence that our evaluator can accommodate different valid modeling solutions.

We have updated the Appendix C.5 Validity of the Evaluator with these results and the corresponding analysis.

AuthorsOpenReview ↗

Response to Reviewer ujn2 – Part III

(Continued, Part 3/3)


W2.2 & Q3: Experimental Robustness Through Repeated Evaluations

A: We thank the reviewer for this suggestion. To assess statistical robustness, we conducted three independent runs on 60 tasks (14.7% of BIMArena) under both support settings (w/ and w/o). Tasks were selected using stratified random sampling with a fixed seed, covering 2 applications × 3 tracks (10 tasks per stratum), while preserving Long-Sequence and Reasoning task pairs. As shown in Table 3, we report the mean success rates and two-sided 95% Student's t confidence intervals across the three runs (n=3, df=2).

The repeated-run results are consistent with our main experiments. We observe the same key trends: performance drops sharply with increasing task depth, substantial performance gaps persist across models, and the benefits of external support become more limited on Reasoning tasks. Furthermore, weaker models consistently fail to complete Long-Sequence and Reasoning tasks, providing additional support for our two-stage evaluation strategy. These results further demonstrate the robustness of our main findings to run-to-run variability on the evaluated subset.

Table 3: Repeated-run robustness on the stratified 60-task subset. Values report mean SR (%) and two-sided 95% Student's t confidence intervals across three independent runs.

ModelSupportAtomic SRLong-Seq. SRReasoning SR
Claude Opus 5w/o95.0 [95.0, 95.0]53.3 [46.2, 60.5]35.0 [35.0, 35.0]
w/95.0 [95.0, 95.0]56.7 [49.5, 63.8]36.7 [29.5, 43.8]
Gemini 3.7 Flashw/o80.0 [80.0, 80.0]36.7 [29.5, 43.8]18.3 [11.2, 25.5]
w/88.3 [81.2, 95.5]40.0 [40.0, 40.0]20.0 [20.0, 20.0]
GPT-5.6 Solw/o73.3 [66.2, 80.5]33.3 [26.2, 40.5]11.7 [4.5, 18.8]
w/80.0 [80.0, 80.0]41.7 [34.5, 48.8]16.7 [9.5, 23.8]
Qwen3.7-Plusw/o0.0 [0.0, 0.0]0.0 [0.0, 0.0]0.0 [0.0, 0.0]
w/5.0 [5.0, 5.0]0.0 [0.0, 0.0]0.0 [0.0, 0.0]
EvoCUA-32Bw/o5.0 [5.0, 5.0]0.0 [0.0, 0.0]0.0 [0.0, 0.0]

We have added the Appendix D.4 Run-to-Run Robustness with these results and the corresponding analysis in our revised manuscript.

Reviewer_JyNWOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces BIMArena, a GUI-based computer-use benchmark for professional Building Information Modeling (BIM) software. It comprises 409 tasks across two BIM authoring applications and three progressively challenging task levels. The authors evaluate multiple multimodal models and open-source agents using programmatic model-state verification, and investigate the effects of external operational skills and software documentation. The results reveal a substantial performance gap between basic GUI operations and complex professional workflows.

Critical strengths

The research problem is practically relevant, as professional software use represents an important application scenario for computer-use agents.

The three-level task design and paired-task structure have potential research value.

The use of real GUI interactions and native API-based model-state evaluation goes beyond assessments based solely on interaction trajectories or textual judgments.

The results and failure cases offer some explanatory value, while the experiments on external operational support provide useful empirical observations.

Critical weaknesses

The incremental contribution over existing professional-software and CAD/BIM benchmarks has not been sufficiently established.

The Reasoning tasks conflate several factors, including visual understanding, geometric information extraction, and domain-specific reasoning.

Single-run evaluations lack sufficient statistical robustness analysis.

The screening-based evaluation protocol limits comprehensive comparisons across models on more challenging tasks.

The validity of the evaluation for multiple plausible modeling solutions, as well as potential confounding factors in comparisons between retrieval resources, requires further investigation.

Questions for authors

1.What is the core research contribution of BIMArena beyond existing professional-software and CAD/BIM benchmarks? Beyond focusing on a more specialized domain and longer execution workflows, can BIMArena reveal agent capability limitations that existing benchmarks fail to identify? Can the authors provide more systematic comparisons or analyses to demonstrate the distinctive research value of its task design, evaluation metrics, and paired-task structure?

2.Can the current paired-task experiments effectively disentangle domain-specific reasoning from other contributing factors? Compared with Long-Sequence tasks, Reasoning tasks require agents not only to infer modeling objectives but also to interpret architectural drawings, extract geometric information, and understand design requirements. Could the authors conduct controlled experiments or fine-grained subtask evaluations to distinguish the contributions of these factors to performance degradation, thereby providing stronger evidence for the paper's central claim regarding the limitations of domain-specific reasoning?

3.How can the robustness of the main experimental findings be established when each task is evaluated only once? For performance differences between models and the improvements attributed to external documentation and operational skills, could the authors conduct multiple independent runs on a representative subset of tasks and report confidence intervals or other appropriate uncertainty estimates? Furthermore, could the current two-stage evaluation strategy, which selects models based on an Atomic success-rate threshold, affect the comparability of results on more challenging tasks and the generalizability of the conclusions?

4.Can the current automated evaluation framework reliably assess modeling results that are reasonable but do not exactly match the expert-provided reference model? The paper validates the evaluator using the initial model and the expert-constructed target model as two endpoint cases, but this alone does not establish the validity of its judgments on intermediate results or alternative plausible solutions. Could the authors further validate the evaluation metrics through independent BIM expert assessments, agreement analysis between automated evaluation and expert judgments, and sensitivity analyses of geometric tolerances? Such analyses would also help clarify how the evaluator handles partial completion, unintended modifications, and reasonable alternative solutions.

5 comments
AuthorsOpenReview ↗

Response to Reviewer JyNW – Part I

We sincerely thank the reviewer for the insightful comments and constructive suggestions. We address each concern and question as follows.


W1 & Q1: Core Contributions and Systematic Comparison with Existing Benchmarks

A: We thank the reviewer for this important question. BIMArena's core contribution is to systematically evaluate the depth of agent competence within a single complex, knowledge-intensive professional domain, rather than breadth across diverse applications. The challenges of evaluating deep professional competence are not unique to BIM but arise across knowledge-intensive domains. The details are as follows:

To compare with existing benchmarks:

1. From the perspective of Breadth vs. Depth, while recent benchmarks increasingly incorporate professional software, their evaluation remains largely distributed across multiple domains and applications. Specifically, OSWorld 2.0 [1] covers 108 tasks across 7 domains (15.4 tasks/domain); ScienceBoard [2] contains 169 tasks across 6 domains and 6 applications (28.2 tasks/domain, 28.2 tasks/application); ProSoftArena [3] includes 456 tasks across 6 domains and 13 applications (76.0 tasks/domain, 35.1 tasks/application); and DeskCraft [4] comprises 538 tasks across 5 domains and 11 applications (107.6 tasks/domain, 48.9 tasks/application). Given the extensive functionalities and diverse workflows within each professional domain and application, distributing tasks across numerous environments inevitably limits the depth of evaluation within any single domain. In contrast, BIMArena contains 409 tasks within a single domain, averaging 204.5 tasks per application. While broad benchmarks emphasize cross-application generalizability, BIMArena prioritizes depth through concentrated coverage of diverse functionalities and interconnected workflows within a single professional domain.

2. In terms of execution complexity, BIMArena tasks require an average of 50.9 human expert GUI steps, increasing to 69.2 steps for Long-Sequence and Reasoning tasks. These are approximately 3.9 and 5.4 times the reported 12.9 steps in ProSoftArena and 5.7 and 7.7 times the reported 9.0 steps in ScienceBoard.

3. Diagnostic Task Design. Several benchmarks already organize tasks into difficulty levels (e.g., the capability hierarchy in ProSoftArena and the L1–L3 taxonomy in DeskCraft). BIMArena differs in how its levels are linked: every Long-Sequence and Reasoning task is annotated with the Atomic capabilities it is composed of, and each Reasoning task is paired with a Long-Sequence task that has the same modeling objective. Failures can therefore be analyzed in terms of their underlying causes, rather than merely attributed to task difficulty, allowing us to distinguish limitations in domain-specific reasoning from the effects of task length.

Lastly, existing BIM/CAD benchmarks [5, 6, 7] primarily focus on programmatic modeling through APIs or code-based agents, with limited evaluation of GUI-based computer-use agents in real workflow.

The lack of in-depth domain-specialized agent benchmarks is not unique to BIM but extends to other professional domains. We hope BIMArena can inspire the development of similar benchmarks in other domains and foster progress in domain-specialized agents. In the revised manuscript, we have expanded Appendix A.6 and Section 1 Introduction to provide a more comprehensive comparison with existing benchmarks and clarify BIMArena's distinctive contributions.

References

[1] Yuan, Mengqi, et al. "OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks." arXiv:2606.29537 (2026).

[2] Sun, Qiushi, et al. "ScienceBoard: Evaluating Multimodal Autonomous Agents in Realistic Scientific Workflows." ICLR (2026).

[3] Ai, Jiaxin, et al. "ProSoftArena: Benchmarking Hierarchical Capabilities of Multi-modal Agents in Professional Software Environments." CVPR (2026).

[4] Wang, Wenkai, et al. "DeskCraft: Benchmarking Desktop Agents on Professional Workflows and Human-in-the-Loop Collaboration." arXiv:2606.03103 (2026).

[5] Ai, Jiaxin, et al. "ComAct: Reframing Professional Software Manipulation via COM-as-Action Paradigm." arXiv:2606.13239 (2026).

[6] Nithyanantham, Bharathi Kannan, et al. "BIM-Edit: Benchmarking Large Language Models for IFC-Based Building Information Modeling." arXiv:2606.20146 (2026).

[7] Jiang, Yongqing, et al. "Rethinking Scientific Modeling: Toward Physically Consistent and Simulation-Executable Programmatic Generation." KDD (2026).

AuthorsOpenReview ↗

Response to Reviewer JyNW – Part II

(Continued, Part 2/3)


W2 & Q2: Disentangling Reasoning Tasks from Visual Understanding, Geometric Understanding and Domain Reasoning

A: We thank the reviewer for this important suggestion. We acknowledge that Reasoning tasks involve multiple factors, but different tasks emphasize different domain-specific capabilities. To provide a finer-grained analysis, we carefully reviewed all 143 paired tasks and categorized them based on their primary domain-specific requirements, assigning one or more of three labels: Visual Understanding, Geometric Understanding, and Domain-Specific Reasoning. We then compared the success rates (w/o support) of paired Long-Sequence and Reasoning tasks across these categories.

As shown in Table 1, SR decreases by an average of 12.6 percentage points from Long-Sequence to Reasoning tasks. Domain-Specific Reasoning shows the largest drop (22.7 points), compared with Visual Understanding (13.7 points) and Geometric Understanding (7.7 points). This suggests that domain-specific reasoning presents a particularly significant challenge beyond workflow execution.

Table 1: Success rate degradation from Long-Sequence to Reasoning tasks (percentage points).

ModelOverallVisualGeometricDomain
Claude Opus 5−18.9−20.0−6.4−30.6
Gemini 3.7 Flash−16.8−17.8−10.3−30.6
GPT-5.6 Sol−14.0−15.6−10.3−22.2
GPT-5.6 Terra−11.2−8.9−5.1−22.2
GPT-5.6 Luna−5.6−8.9−6.4−13.9
Claude Sonnet 5−9.1−11.1−7.7−16.7
Average−12.6−13.7−7.7−22.7

We have updated Section 5.2: Error Analysis, and Appendix E: Failure Analysis in our revised manuscript.


W3 & Q3.1: Statistical Robustness and Uncertainty Analysis

A: We thank the reviewer for this suggestion. To assess statistical robustness, we evaluated 60 tasks (14.7% of BIMArena) across three independent runs in total under both support settings including operational skills and official documentation (w/ and w/o). Tasks were selected via stratified random sampling with a fixed seed, covering 2 applications × 3 tracks (10 tasks per stratum), while preserving Long-Sequence and Reasoning task pairs. Table 2 reports the mean success rates and two-sided 95% Student's t confidence intervals across the three runs (n=3, df=2).

As shown in Table 2, the repeated-run results are consistent with our main experiments. We observe the same key trends: performance drops sharply with increasing task depth, substantial performance gaps persist across models, and the benefits of external support become more limited on Reasoning tasks. These results provide additional evidence that our main findings are robust to run-to-run variability on the evaluated subset.

Table 2: Repeated-run robustness on the stratified 60-task subset. Values report mean SR (%) and two-sided 95% Student's t confidence intervals across three independent runs.

ModelSupportAtomic SRLong-Seq. SRReasoning SR
Claude Opus 5w/o95.0 [95.0, 95.0]53.3 [46.2, 60.5]35.0 [35.0, 35.0]
w/95.0 [95.0, 95.0]56.7 [49.5, 63.8]36.7 [29.5, 43.8]
Gemini 3.7 Flashw/o80.0 [80.0, 80.0]36.7 [29.5, 43.8]18.3 [11.2, 25.5]
w/88.3 [81.2, 95.5]40.0 [40.0, 40.0]20.0 [20.0, 20.0]
GPT-5.6 Solw/o73.3 [66.2, 80.5]33.3 [26.2, 40.5]11.7 [4.5, 18.8]
w/80.0 [80.0, 80.0]41.7 [34.5, 48.8]16.7 [9.5, 23.8]
Qwen3.7-Plusw/o0.0 [0.0, 0.0]0.0 [0.0, 0.0]0.0 [0.0, 0.0]
w/5.0 [5.0, 5.0]0.0 [0.0, 0.0]0.0 [0.0, 0.0]
EvoCUA-32Bw/o5.0 [5.0, 5.0]0.0 [0.0, 0.0]0.0 [0.0, 0.0]

We have added the Appendix D.4 Run-to-Run Robustness with these results and the corresponding analysis in our revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer JyNW – Part III

(Continued, Part 3/3)


W4 & Q3.2: Impact of the Two-Stage Evaluation Protocol

A: We thank the reviewer for this concern. To examine potential screening bias, we extended the repeated-run evaluation to two previously excluded models (Qwen3.7-Plus and EvoCUA-32B) across all three tracks on the 60-task subset described in W3 & Q3.1. As shown in Table 2 (Part 2), both models achieved 0% SR on Long-Sequence and Reasoning tasks across three independent runs. In the revised manuscript, we have added Appendix D.4 with the analysis accordingly.


W5 & Q4: Evaluator Validity and Assessment of Alternative Modeling Solutions

A: We thank the reviewer for this important concern. To further validate our evaluator beyond the initial and expert-target baselines, we designed additional analyses:

An independent BIM expert reviewed 300 BIM models generated by different agents on the stratified subset described in Q3.1. For each model, the expert assessed task success, partial completion, and unintended modifications based on the task description and a comparison of the initial and agent-generated final BIM models, without access to automated scores. We further examined evaluator sensitivity by varying geometric tolerances (0.5×, 1×, and 2× the original thresholds) and measuring agreement with expert judgments.

Table 3: Evaluator–human agreement and tolerance sensitivity on 300 expert-assessed BIM models (100 per track). SR Agreement and CFR Agreement: percentage of models on which the automated and expert binary verdicts coincide. Cohen's κ: chance-corrected agreement on binary task success. PCS Agreement: percentage of goal checkpoints, pooled over all models, on which the automated and expert verdicts coincide. PCS MAE: mean absolute difference between automated and expert-assessed per-model partial completion scores.

SubsetToleranceSR AgreementCohen's κPCS AgreementPCS MAECFR Agreement
All 300 models0.5×94.67%0.89285.58%10.46%80.00%
All 300 models1.0×96.67%0.93289.85%7.32%85.33%
All 300 models2.0×94.67%0.89286.11%9.54%82.67%
Reasoning only0.5×100.00%1.00087.06%8.78%88.00%
Reasoning only1.0×100.00%1.00091.06%6.56%92.00%
Reasoning only2.0×98.00%0.94783.89%10.11%88.00%

At the original tolerance, the evaluator achieves 96.67% SR agreement with human judgments (κ = 0.932), reaching 100% SR agreement on Reasoning tasks. The original tolerance settings also yield the highest overall agreement with human judgments. Regarding alternative modeling solutions, many Reasoning tasks admit multiple valid configurations. Our evaluator therefore assesses task-specific constraints rather than requiring an exact match to a single reference model. For example, minimum room dimensions or required opening widths are evaluated using inequality constraints (≥ or ≤), with thresholds derived from the referenced architectural handbooks and standards. The high agreement with independent expert judgments on Reasoning tasks further supports the evaluator's ability to accommodate valid alternative solutions.

We have updated the Appendix C.5 Validity of the Evaluator with these results and the corresponding analysis.


W5: Potential Confounding Factors in Operational Support Comparisons

A: We thank the reviewer for this comment. Our main experiments evaluate the combined effect of both retrieval resources. To partially separate them, the ablation in Section 5.3 (Table 5) shows that documentation alone lowers GPT-5.6 Sol's overall SR by 4.8 points, whereas documentation plus skills raises it by 7.4 points, indicating that the gains come from the curated skills rather than from additional information. We acknowledge that this comparison is not fully controlled: the two resources differ in retrieval mechanism, context length, and relevance. We have clarified this limitation in the revised manuscript (Section 6 Conclusion).

Comments on the paper

1 comment

Dear PC, SAC, AC, and Reviewers,

We sincerely thank you for your constructive feedback and expert insights, which have substantially strengthened the manuscript. We are encouraged that the reviewers recognized BIMArena as a valuable domain-specialized, depth-oriented benchmark for evaluating computer-use agent capabilities in professional software environments.

Based on your feedback, we have made the following updates to the paper:

  • We added a run-to-run robustness analysis in Section 5.1 and Appendix D.4, confirming that our main findings are stable across repeated runs.
  • We added an expert validation of the automated evaluator in Section 4.4 and Appendix C.5, showing high agreement with independent expert judgments, and clarified in Section 3.2 that Reasoning tasks with multiple valid solutions are evaluated by constraint-based verification.
  • We expanded the error analysis in Section 5.2 and Appendix E, showing that domain-specific reasoning is the largest source of the performance drop on Reasoning tasks and that most failures stem from agents prematurely reporting completion.
  • We added an operational support usage analysis in Appendix D.5, showing that more retrieval does not translate into proportional gains on complex tasks.
  • We extended the comparison with existing benchmarks in Appendix A.6 to better position BIMArena's focus on domain depth.
  • We refined the contribution statements in Section 1 and the limitations in Section 6 to improve precision and clarity.

We provide further details in our point-by-point responses and have incorporated all additions into the revised manuscript. We are pleased that our rebuttal has successfully addressed the main concerns of reviewers JyNW, ujn2, and zHev, and we greatly appreciate all reviewers' encouragement in raising or finalizing their scores to 3 (conf. 2, reviewer JyNW), 3 (conf. 3, reviewer ujn2), and 3 (conf. 4, reviewer zHev), leading to a final overall score of 3.0. In particular, reviewer zHev raised the score from 2 to 3, and reviewer ujn2 raised confidence from 2 to 3.

Thank you again for your time and dedication in reviewing our submission.

Sincerely,

The Authors of Submission #26403

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.