acceptodds
Under review as a conference paper at ICLR 2027

AssetArena: A Benchmark for Agentic Workflows in Industrial Asset Operations

Abstract

Industrial asset operations require agents to combine condition assessment, diagnosis, evidence retrieval, and operational action within a single stateful workflow. However, evaluations remain fragmented, as deployed agents typically cover only part of this workflow, while industrial benchmarks largely assess isolated capabilities or static knowledge rather than end-to-end execution. We introduce AssetArena}, an executable benchmark for agentic workflows in industrial asset operations. AssetArena provides a unified MCP-native environment that connects heterogeneous industrial evidence, analytical tools, and operational actions, with scenarios designed so that relevant information must be acquired through interaction. Its evidence-first construction process establishes reference outcomes independently of the agent and records replayable executions for reproducible evaluation. The benchmark contains 231 human-authored, machine-verified scenarios and 85 tools across six MCP servers. We evaluate eight frontier language-model configurations under a common agent framework and further study the effects of domain skills, agent decomposition, tool-access topology, and execution harness. Models achieve 16.5–29.9% Pass@1 across the benchmark, indicating substantial headroom even for frontier models. Controlled comparisons show that agent design and execution infrastructure affect task success and interaction cost: across 280 paired runs, Stirrup agent achieves 31.1% Pass@1 versus 25% for OpenCode. AssetArena provides a common and extensible environment for evaluating model capability and agent design on challenging industrial asset-operation workflows. Reproducibility materials are available at: https://anonymous.4open.science/r/AssetArena-ICLR/

Primary area:
datasets and benchmarks
Keywords:
Agentic AI, Industrial asset operations, Industrial benchmarks, Tool-using agents, Agent evaluation, Asset management, Benchmark construction

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_fHSyOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses a limitation of previous industrial benchmarks, which mostly measure the simple question-answering abilities of LLMs. Solving real industrial problems requires LLMs to execute tasks that are more complex than simple question answering, for example, integrating various sources of data, instructions, or manuals, or clarifying or confirming the user’s intent when necessary. Motivated by this concern, the authors propose AssetArena, a benchmark consisting of 231 scenarios across seven workflow families, which mimics real industrial settings through an MCP-native environment. The benchmark provides a controlled sandbox with 85 tools across six MCP servers and heterogeneous industrial data, including work orders, telemetry, and failure-mode catalogs, together with challenging user queries that contain some degree of ambiguity. To avoid queries that seem plausible but are in fact unsolvable, the authors constructed each query only after its supporting evidence and reference answer were fixed (an evidence-first process), and eight domain experts participated in building the benchmark. The authors also provide a controlled testbed for comparing agent designs, along with a deterministic evaluator that scores trajectories against hidden references without an LLM judge. They report that even frontier models achieve only 16.5–29.9% Pass@1 across the benchmark, and their controlled experiments further show that agent-design choices mainly affect efficiency rather than accuracy.

Critical strengths
  • Expert-constructed benchmark. The benchmark was constructed by human domain experts rather than synthesized by LLMs, and it is grounded in real industrial records and physics-based simulations.
  • Vendor neutrality. The tool environment is built on the open MCP protocol rather than tied to a specific commercial platform.
  • End-to-end industrial workflows. The benchmark provides end-to-end industrial workflows that have been validated by human domain experts.
  • Reproducible evaluation. Scoring relies solely on deterministic rules rather than LLM judges, which can introduce inherent biases.
  • Principled construction methodology. The authors construct the benchmark through an evidence-first methodology that extends Agents' Last Exam (ALE), adopting its three criteria: representativeness, verifiability, and workflow completeness. Building on these criteria, the work introduces admission predicates C1–C7, which operationalize them for industrial tasks.
  • Substantial headroom. Even frontier models achieve only 16.5–29.9% Pass@1, indicating that the benchmark leaves substantial headroom before saturation.
  • Extensive experiments and analysis. The work evaluates various open- and closed-weight models, reporting Pass@1, Pass@3, Pass^3, and Bradley–Terry Elo ratings, together with a failure analysis suggesting that failures stem mainly from grounded reasoning rather than from tool calling.
  • Cost reporting. The work reports inference costs, enabling practitioners to select suitable models according to their budget.
Critical weaknesses
  • The main weakness of this paper is the inconsistency of the reported numbers, especially between the main text and the appendix. For example, Table 3 (p. 8) reports 425 passing and 1,423 failing runs, whereas Appendix G (p. 47) reports 333 successful and 1,515 failed runs, although both sum to 1,848. Similarly, Figure 14 reports 444 of 1,336 failed non-FMEA runs with output-contract mismatches, whereas the text of Appendix G (p. 46) reports 444 of 1,428. Furthermore, Table 2 lists 65 WO scenarios, but Appendix B.2 states that “the work-order scenario set contains 60 manually authored tasks.” More importantly, the per-model Pass@1 values in Figure 15 (12.6–22.1%) differ from those in Table 2 (16.5–29.9%), suggesting that results from different experimental runs may have been mixed. Since the main conclusions rely on these numbers, the authors should clarify which runs the reported results are based on, resolve these discrepancies, and verify the consistency of all reported numbers throughout the paper.
Questions for authors

Questions

Q1. Benchmark Composite Index (Figure 1). Figure 1 presents a Benchmark Composite Index, but the paper does not specify how this index is measured, including how many sub-criteria each axis contains and what they are. Could the authors clarify the criteria used to measure each axis and report the grounding and agency scores for each benchmark?

Q2. Horizontal and vertical composites (p. 3). The paper states that "Only ITBench reaches the top of the horizontal composite, and no benchmark passes half of the vertical one." However, it does not define what the horizontal and vertical composites refer to. Please clarify these terms.

Q3. Scope of the agent-design comparisons (pp. 6 and 8). In "Controlled Agent Interfaces" (p. 6), the authors evaluate skills, gateway access, and domain sub-agents. Since each of these is a design pattern with many possible implementations, the experiments compare one specific implementation of each pattern rather than the patterns themselves. However, the statement in Section 5.3 (p. 8) that "flat access dominates it outright" for sub-agents reads as a general comparison between approaches. I recommend revising such statements so that they are restricted to the implementations evaluated in this paper.

** Q4. Unsolved scenarios (p. 9).** The paper states that "126 of 231 scenarios defeat all eight" models. However, it is difficult to gauge how many of these scenarios human practitioners could actually solve. It would be helpful to have a mechanism that distinguishes whether these failures reflect the limitations of the models or issues with the scenarios themselves, such as ambiguous queries, incorrect references, or overly strict scoring. For example, a human baseline or a root-cause analysis of the unsolved scenarios would address this concern.

Q5. Fine-grained CAR metrics. The CAR family includes scenarios that cannot be answered from the available data, but the paper does not report fine-grained metrics for these cases, such as how often a model answers when it should have abstained. Could the authors include such analyses, for example, a mode confusion matrix?

Q6. Representativeness of the 35-scenario subset. The additional experiments (harness comparison, skills, sub-agents, gateway, and the 30- vs. 60-turn budget) are conducted on a subset of only 35 scenarios. Please disclose the family-level composition and difficulty distribution of this subset. The mean Pass@1 is 23.0% on the full set but 31.1% under the flat setting on the subset, which makes it difficult to regard the subset as representative of the full set's difficulty. For example, p. 50 states that "at the aggregate family level, the gateway improves HLT, TSFM, and FMEA completion while reducing CAR and FMSR completion." In Table 20, however, DeepSeek-V4's HLT result changes from 0 to 50. If the subset contains only two HLT scenarios, this corresponds to solving just one additional scenario, so the result may be over-interpreted. Please clarify this point.

Minor suggestions

  • In Table 2, the per-family results are percentages but are rounded to integers, so readers may misread them as counts. Reporting them to one decimal place may help.
7 comments
AuthorsOpenReview ↗

Response to Reviewer fHSy

We thank the reviewer for the positive assessment and careful checks. We have revised the paper and all changes are marked in blue. Please see our responses to each weakness and question below.

W1: Consistency of the reported results

W1. "The main weakness of this paper is the inconsistency of the reported numbers, especially between the main text and the appendix....."

Thank you for identifying this discrepancy. We found that Appendix G have used outdated scored aggregates, including scores preceding the CAR evaluator update. We rescored all 1,848 saved trajectories with consistent scenario-specific scoring and regenerated the failure analysis. The verified accounting is 425 passing and 1,423 failing runs, including 1,336 failed non-FMEA runs, of which 444 have missing or extra answer keys.

Appendix G and the updated Figures 15–16 now use this same cohort. Figure 16’s model-level Pass@1 values match Table 2’s 16.5–29.9%. We also corrected the WO construction count from 60 to 65 (Appendix B.3, lines 1577, 1587). These corrections are based on scoring and reporting of saved trajectories, and did not involve new model runs.

Q1–Q2: Composite-index definitions

Q1. "Figure 1 presents a Benchmark Composite Index, but the paper does not specify how this index is measured, including how many sub-criteria each axis contains and what they are. Could the authors clarify the criteria used to measure each axis and report the grounding and agency scores for each benchmark?"

Q2. "The paper states that "Only ITBench reaches the top of the horizontal composite, and no benchmark passes half of the vertical one." However, it does not define what the horizontal and vertical composites refer to. Please clarify these terms."

The index has seven sub-criteria in two groups.

i) Agency has three: Executable (tasks run against tools rather than being answered from text), Harness (more than one agent harness is evaluated), and Underspecified (the request withholds what to retrieve).

ii) Grounding has four: Computed (references computed by a program), Fragility (per-task difficulty measured), Evidence (the execution record is scored, not only the claim), and State (the outcome is scored by the change left in the environment).

Each sub-criterion scores 1.0 satisfied, 0.5 partially satisfied, 0 not satisfied, from the released paper and artifacts as of September 2026; each composite is the mean over its group. Appendix Table 4 now reports every sub-criterion score and both composites per benchmark. The grouping reflects a split in the literature: industrial benchmarks remain largely multiple-choice over a direct LLM call, while the wider agentic community evaluates tool-using agents, and Executable is the sub-criterion that separates the two. Fragility is satisfied by no prior benchmark; averaging grounding over the other three leaves the ordering unchanged and raises the prior ceiling from 0.50 to 0.67, against 1.00 for AssetArena.

(Response Continue)

AuthorsOpenReview ↗

Response to Reviewer fHSy

Q3: Scope of agent-design conclusions (Section 5.3)

Q3. "In "Controlled Agent Interfaces" (p. 6), the authors evaluate skills, gateway access, and domain sub-agents. Since each of these is a design pattern with many possible implementations, the experiments compare one specific implementation of each pattern rather than the patterns themselves. However, the statement in Section 5.3 (p. 8) that "flat access dominates it outright" for sub-agents reads as a general comparison between approaches. I recommend revising such statements so that they are restricted to the implementations evaluated in this paper."

We agree that the experiments in Section 5.3 evaluate particular implementations, not entire design patterns. We have replaced “flat access dominates it outright” with:

“Sub-agent delegation costs 137.4% more turns and 188.3% more tool calls while yielding lower aggregate completion and mean score than flat access in the evaluated configuration” (p. 9, lines 446-447)

Q4: Solvability and sources of failure

Q4. "The paper states that "126 of 231 scenarios defeat all eight" models. However, it is difficult to gauge how many of these scenarios human practitioners could actually solve. It would be helpful to have a mechanism that distinguishes whether these failures reflect the limitations of the models or issues with the scenarios themselves, such as ambiguous queries, incorrect references, or overly strict scoring. For example, a human baseline or a root-cause analysis of the unsolved scenarios would address this concern."

We appreciate the reviewer for identifying this and the suggestion. We added five investigations based on solving WO scenarios to access the solvability (Section 5.5, pp. 9–10). i) Expert assessment ii) An expert-developed, non-generative NLP workflow iii) an enhanced coding environment that provides agents with additional NLP libraries and pretrained models iv) relaxed-pass sensitivity analysis and , and v) question-revision sensitivity.

  • In the expert assessment, five engineers reviewed all 231 Opus 5 trajectories without references or scores, identifying workflow gaps in text interpretation, classification validation, asset-hierarchy scoping, and sensor–failure-mode reasoning.

  • The expert-developed NLP workflow solved 11/65 WO scenarios (16.9%) on its development set.

  • In the enhanced-environment, agents solved 3–4/65, despite confirmed use of added resources.

  • Relaxed scoring increased overall success from 23.0% to 25.5% where eight of the 126 previously unsolved scenarios admitted a passing run under the secondary criteria.

  • Human question revisions informed by Fable feedback produced passing outputs for four previously zero-pass WO scenarios.

These results show that some scenarios can be solved with better workflows, additional tools and resources. Relaxed scoring explains only a small share of the failures. This suggests that the evaluated agents still struggle with industrial reasoning and workflow construction and that further improvements and expert review may help resolve more scenarios.

Codebases for these additional experiments are in ./Additional_experiments in the anonymized repository.

(Response Continue)

AuthorsOpenReview ↗

Response to Reviewer fHSy

Q5: Fine-grained CAR behavior

Q5: "The CAR family includes scenarios that cannot be answered from the available data, but the paper does not report fine-grained metrics for these cases, such as how often a model answers when it should have abstained. Could the authors include such analyses, for example, a mode confusion matrix?"

We thank the reviewer for the suggestion. We have now analyzed all 400 CAR trajectories using the evaluator’s normalized response-mode keys. Models returned a direct response in 78/240 cases requiring abstention (32.5%) and 48/120 requiring clarification (40.0%). Overall mode accuracy was 170/400 (42.5%), with 64 invalid-mode outputs. We report the aggregate confusion matrix in Table 1 below and Appendix F.3 (p. 48). Mode accuracy measures response-mode selection only; primary CAR success also requires the relevant decision terms, as described in Appendix F.1.

Table 1: CAR mode confusion over 400 trajectories. Rows are reference modes; columns are predicted modes.

ReferenceResponseClarificationAbstainInvalidTotal
Response2713940
Clarification4841427120
Abstain783210228240

Q6: Subset composition and difficulty

Q6: "The additional experiments (harness comparison, skills, sub-agents, gateway, and the 30- vs. 60-turn budget) are conducted on a subset of only 35 scenarios. Please disclose the family-level composition and difficulty distribution of this subset. The mean Pass@1 is 23.0% on the full set but 31.1% under the flat setting on the subset, which makes it difficult to regard the subset as representative of the full set's difficulty. For example, p. 50 states that "at the aggregate family level, the gateway improves HLT, TSFM, and FMEA completion while reducing CAR and FMSR completion." In Table 20, however, DeepSeek-V4's HLT result changes from 0 to 50. If the subset contains only two HLT scenarios, this corresponds to solving just one additional scenario, so the result may be over-interpreted. Please clarify this point."

The subset contains 9 WO, 8 CAR, 4 FCC, 2 HLT, 5 FMSR, 4 TSFM, and 3 FMEA scenarios. It is family-stratified, but not difficulty-matched to the full suite. We have added this distribution under Appendix D4: “Subset composition”. Flat-baseline Pass@1 is 87/280 (31.1%) on the 35-scenario subset, compared with 425/1,848 (23.0%) on the full 231-scenario suite.

Under the eight evaluated models, the subset contains 15 scenarios solved by none, 17 by one to seven models, and three by all eight. The corresponding full-suite counts are 126, 90, and 15. We have now reported the performance distribution in Appendix H, “Shared baseline and subset difficulty.” Ablation conclusions are restricted to the tested scenarios and implementations.

The reviewer’s HLT example is correct: DeepSeek’s change from 0% to 50% represents 0/2 to 1/2, one additional success (Appendix lines 2903-2904). We have now reported the underlying family-level counts and avoid interpreting these small changes as general advantages (in Appendix H, p. 54, lines 2902–2905). The gateway’s measured runtime and token reductions are reported separately from its small completion changes.

(Response Continue)

Reviewer_bdJUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes AssetArena, a benchmark of 231 scenarios for agents doing industrial asset maintenance work. The agent gets an operator style request, retrieves evidence through 85 MCP tools, runs some analysis, and returns an answer that is scored by script against a hidden reference. The authors build scenarios evidence first, meaning the phenomenon and the reference are fixed before the request is written, and they describe seven admission checks for this. Eight frontier models are tested and reach 16.5% to 29.9% Pass@1. On a 35 scenario subset the authors also compare domain skills, sub agents, an MCP gateway, two harnesses, a larger turn budget and repeated runs. They conclude that agent design changes cost more than accuracy, that the harness matters more than agent design, and that failures come from reasoning rather than tool access.

Critical strengths
  1. The setting is realistic and useful. Tasks require gathering evidence across work orders, telemetry and catalogues, the environment is executable and isolated, scoring is by script, and trajectories can be rescored without rerunning models. The benchmark is far from saturated. I also checked that the main results table, the Elo analysis and the Holm corrected tests are consistent with the reported counts.

  2. The authors read their own results with some care. They say Elo only groups models, that the agent design effects are not significant, and that Pass@3 is not a deployable metric. The failure analysis, and the finding that a gateway cuts the initial tool schema by about 66 times, are useful for people building agents.

Critical weaknesses
  1. I am not convinced the references are what the paper says they are. The paper says references are computed by program from a verified phenomenon. But for FMEA the appendix says the scenarios are written by experts and the catalogues do not contain the needed associations. FMSR comes from a manually transcribed standards matrix with templated synthetic assets, and work order answers come from hand written filters over short texts. That is 114 of the 231 scenarios with no reported human validation, and the work order pass rates of 2% to 6% could simply reflect ambiguous references. The paper also stresses operational action and writing back to records, and Table 4 claims state verification, yet every scoring rule I found is on the final answer and the samples I saw are read only. The 126 scenarios nobody solves are called headroom, but there is no evidence they can be solved.

  2. The conclusions on agent design and harness are not solid. The flat baseline on the 35 scenario subset is 31.1%, but Table 24 gives 22.9% for the first run of the same setup on the same subset. That gap of about 8 points is bigger than every effect the paper discusses, including Stirrup against OpenCode at 6.1 points, and it is never explained. The subset also looks easier than the full set. Several numbers do not match across the paper either, for example 333 against 425 passing runs in Appendix G versus Table 3, scenario counts in Table 7 that add up to 343, and 85 against 86 tools. The anonymous repository I looked at had only a few sample scenarios, so I could not check any of this myself.

Questions for authors
  1. Why is the first run Pass@1 in Table 24 only 22.9% when the flat baseline for the same subset is 31.1%? Do the Stirrup against OpenCode result and the agent design comparisons still hold with multiple seeds and a significance test?
  2. How were the references built for the work order, FMSR and FMEA scenarios, and has a human expert checked them? Is there any evidence that the 126 unsolved scenarios can be solved?
  3. Are any scenarios scored on the final environment state? If so, how many, and what is the basis for the check mark in Table 4?
  4. Table 7 lists 110 extra TSFM diagnosis scenarios, but the TSFM family has only 30 scenarios and the table counts add up to well over 231. Do these 110 scenarios exist, and are they part of the benchmark?
  5. Is an LLM judge used for any scenario, for instance the two free text ones, given that Figure 4 lists it? Also, does TSFM scoring use a time tolerance? Minor point. The K1 recovery condition is defined in the appendix but I could not find results for it.
5 comments
AuthorsOpenReview ↗

Response to Reviewer bdJU

We thank the reviewer for recognizing the benchmark’s practical relevance and reproducible evaluation design. We have revised the paper and all changes are marked in blue. Please see our responses to each weakness and question below.

W1,Q2: Reference construction and human validation

W1. "I am not convinced the references are what the paper says they are. The paper says references are computed by program from a verified phenomenon. But for FMEA the appendix says the scenarios are written by experts and the catalogues do not contain the needed associations. FMSR comes from a manually transcribed standards matrix with templated synthetic assets, and work order answers come from hand written filters over short texts. That is 114 of the 231 scenarios with no reported human validation, and the work order pass rates of 2% to 6% could simply reflect ambiguous references."

Q1. "How were the references built for the work order, FMSR and FMEA scenarios, and has a human expert checked them? Is there any evidence that the 126 unsolved scenarios can be solved?"

WO references were constructed using human-validated annotations. FMSR and FMEA references were derived programmatically from a validated internal library linking assets, sensors, and failure modes. Eight domain experts participated over three months. Each WO, FMSR, and FMEA reference was reviewed by two members of this team, with each review taking approximately 15–20 minutes. Fortnightly group meetings additionally examined evidence sufficiency, reference correctness, tool reachability, and operational realism (Section 3 and Appendix B).

For example, FMSR scenario 906 asks below question:

“As the PdM architect, I want to know which failure modes generalize best once assets have their full sensor suites. Once an asset has its full sensor suite installed, which failure mode is monitorable across the GREATEST number of distinct asset classes? Return only a JSON array of strings, sorted alphabetically. Use exact catalog or registry spelling and casing. Do not include explanatory text.”

Reference program is provided. Please go to anonymous repository and navigate to: Scenarios_and_trajectories/Reference_generation/fmsr906_reference.py

The reference program joins the asset–sensor–failure-mode associations, counts the distinct asset classes supporting each failure mode, and returns all maximizers. The documented reference is ["Misalignment", "Unbalance"]\color{green}{\text{["Misalignment", "Unbalance"]}}, each monitorable across 9 of the 10 asset classes.

Similarly, WO scenario 15 asks below question:

“I am checking pump-related maintenance workorders. Please check the workorders and count only jobs where the whole pump was replaced. Count a job as a whole-pump replacement when the description clearly says the pump or a pump asset was replaced, changed out, swapped, renewed, or removed and installed again. Also count descriptions such as "replace PMP..." or "replacement of PMP..." where the pump asset id itself is the object being replaced. Do not count jobs that only replace pump parts, such as seals, bearings, gaskets, hoses, motors, valves, couplings, impellers, filters, or other pump components. If the description does not clearly show that the pump itself was replaced, exclude it. Return only the final count as a single integer. No additional output is permitted.”

Reference program is provided. Please go to anonymous repository and navigate to: Scenarios_and_trajectories/Reference_generation/wo015_reference.py

The reference program counts work orders whose human-validated annotations link a replacement activity to a pump through the hasPatient relation. Each work order is counted once, yielding the reference answer 10\color{green}{\text{10}}. This computes the answer from annotated activities and their targets rather than keyword matches in the maintenance text.

Thus, the answer is computationally derived from explicit associations and checked by domain experts.

(Response Continue)

AuthorsOpenReview ↗

Response to Reviewer bdJU

W1,Q2: Solvability of the unsolved scenarios

W1: "The 126 scenarios nobody solves are called headroom, but there is no evidence they can be solved."

Q2: "Is there any evidence that the 126 unsolved scenarios can be solved?"

We agree that failure by all tested models does not establish solvability. We therefore describe the 126 scenarios as unsolved by the evaluated configurations. We added four investigations based on solving WO scenarios (Section 5.5, pp. 9–10):

i) An expert-developed, non-generative NLP workflow solved 11/65 scenarios (16.9%).

ii) Providing agents with additional NLP libraries and pretrained models yielded 3–4/65 successes per model family and resource use was confirmed in the trajectories.

iii) Relaxed scoring increased overall success from 23.0% to 25.5% where eight of the 126 previously unsolved scenarios admitted a passing run under the secondary criteria

iv) Human question revisions informed by Fable feedback produced passing outputs for four previously zero-pass WO scenarios.

We have added the details and results in Section 5.5 and in appendix K, L, M, N. Codebases for additional experiments are in ./Additional_experiments in the anonymized repository. These experiments demonstrate partial workflow coverage, but do not establish that every unsolved scenario is solvable or that every remaining failure is a reasoning failure.

W1,Q3: Operational updates and state verification

W1: "The paper also stresses operational action and writing back to records, and Table 4 claims state verification, yet every scoring rule I found is on the final answer and the samples I saw are read only."

Q3: "Are any scenarios scored on the final environment state? If so, how many, and what is the basis for the check mark in Table 4?"

Three scenarios (322, 323, and 325) request failure-code imputation and only scenario no. 325 explicitly requires database write-back. Across the 1,848 trajectories, 30 model–scenario executions returned work-order records from mutation tools: 3 for scenario 158, 1 for 164, 2 for 196, and 8 each for 322, 323, and 325. Each execution is counted once, regardless of the number of mutation-tool calls.

The environment supports persistent updates and post-execution CouchDB export through export_couchdb_state, export_state, and export_database. However, none of the 231 scenarios independently compares the final database state with a task-specific expected state. We have explicitly distinguished these capabilities: Table 5’s “Mutable state” checkmark denotes changes observable in subsequent interactions, not final-state verification (Appendix A; lines 1250-1255). Reported success remains answer-based.

(Response Continue)

AuthorsOpenReview ↗

Response to Reviewer bdJU

W2,Q1: Baseline discrepancy and scope of the comparisons

W2: "The conclusions on agent design and harness are not solid. The flat baseline on the 35 scenario subset is 31.1%, but Table 24 gives 22.9% for the first run of the same setup on the same subset. That gap of about 8 points is bigger than every effect the paper discusses, including Stirrup against OpenCode at 6.1 points, and it is never explained. The subset also looks easier than the full set."

Q1: "Why is the first run Pass@1 in Table 24 only 22.9% when the flat baseline for the same subset is 31.1%? Do the Stirrup against OpenCode result and the agent design comparisons still hold with multiple seeds and a significance test?"

Thank you for identifying this discrepancy. Table 29 (Table 24 in earlier version) used an earlier evaluation configuration that omitted the updated deterministic CAR procedure used in Tables 23–27 (Tables 18–22 in earlier version). We re-scored all three saved executions with consistent scenario-specific metadata. First-execution success now matches the flat baseline for every model: 87/280 (31.1%). Corrected Pass@3 and Pass^3 are 124/280 (44.3%) and 60/280 (21.4%). We updated the Table 29, Appendix J.2 discussion, Section 5.4, and Figure 8b. No model reruns were involved and the rescoring was based on already completed trajectories.

The repeated trials assess the flat configuration, not every harness or agent-design intervention. They therefore do not establish seed robustness for all comparisons. We restrict those conclusions to the matched 35-scenario experiments and retain the finding that agent-design effects are not statistically significant. The subset’s higher success rate also limits extrapolation to the full benchmark.

W2,Q4: Inventory, failure-analysis counts, and artifact availability

W2: "Several numbers do not match across the paper either, for example 333 against 425 passing runs in Appendix G versus Table 3, scenario counts in Table 7 that add up to 343, and 85 against 86 tools. The anonymous repository I looked at had only a few sample scenarios, so I could not check any of this myself."

Q4: "Table 7 lists 110 extra TSFM diagnosis scenarios, but the TSFM family has only 30 scenarios and the table counts add up to well over 231. Do these 110 scenarios exist, and are they part of the benchmark?"

We audited the reported inventory and corrected the following:

  • Manifest counts: the work-order–asset schema contains 125 scenarios (65 WO, 50 CAR, and 10 HLT). The evaluated categories sum to 231 (Table 8, 9 in Appendix B).

  • Additional TSFM scenarios: IDs 2001–2110 form a separate 110-scenario development partition, excluded from every reported evaluation. Within that partition, 30 are expert-verified and 80 remain under validation and audit. Table 8 in Appendix B.1 now separates this pool from the evaluated partition.

  • WO construction: references to 60 tasks were corrected to 65 (Appendix B.3).

  • Tool inventory: Figure 4 and the operating-knowledge table (Table 14) now consistently report 85 tools.

  • Failure analysis: rescoring all 1,848 trajectories confirms 425 successful and 1,423 failed runs. Appendix G and its associated figures now use this accounting.

The anonymous repository includes the evaluated scenarios (questions, manifests, and ground truths), trajectories, code, and supporting materials. The revised paper conclusion states our plan to also release these resources through Harbor Agent Eval or Kaggle Benchmarks.

(Response Continue)

Reviewer_VHwtOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

AssetArena introduces 231 industrial asset-operation scenarios over a shared MCP environment. References are established from industrial records or controlled physical simulations, with deterministic family-specific evaluation. Eight model configurations are compared, alongside interventions involving domain skills, tool discovery, sub-agents, execution harnesses, interaction budgets, and repeated trials.

Critical strengths

The effort of building such industrial asset benchmark that requires domain experts is useful to evaluate model/harness capability for the real-world tasks.

The overall benchmark design is plausible, with shared evidence, isolated execution, hidden references, and saved trajectories support controlled comparisons and retrospective analysis.

he paper reports corrected paired tests, acknowledges that leading models are not significantly separated, and distinguishes retry potential from deployable reliability.

Critical weaknesses

The suite contains only 35 distinct evidence bundles; one supports 115 scenarios. TSFM and FCC requests also exhibit substantial template overlap. This is transparently reported, but scenario-level resampling does not capture generalization uncertainty across independent assets, sites, or evidence sources.

More root cause analysis and bottleneck analysis is needed to provide insights on model, harness, environment, tools, etc.

Questions for authors

How many scenarios require an operational state change, and what independently verifies it?

4 comments
AuthorsOpenReview ↗

Response to Reviewer VHwt

We thank the reviewer for the positive assessment and constructive feedback. We have revised the paper and all changes are marked in blue. Please see our responses to each weakness and question below.

W1: Shared evidence and generalization uncertainty

W1. "The suite contains only 35 distinct evidence bundles; one supports 115 scenarios. TSFM and FCC requests also exhibit substantial template overlap. This is transparently reported, but scenario-level resampling does not capture generalization uncertainty across independent assets, sites, or evidence sources."

Thank you for pointing this out. We would like to clarify that scenario-level sampling was motivated by Table 1 (p. 3) of our review (more deployed applications for Work order (WO) and no existing support for emerging topics like CAR). This is why a large percentage of our dataset includes close to 115 scenarios capturing WO- and CAR-based scenarios (we use REI-Bench [1] from ICLR-2025 as inspiration for scenario drafting). To address your concern about generalization, we present the following three pieces of evidence:

  • Data diversity: Our data are generally spread across assets, sites, failure modes, etc.

    • For example, WO: 716 work-order records containing asset identifiers, spanning 118 distinct identifiers and 15 registry asset-type codes, including compressor, boiler, battery, etc. (as WOs are not frequent on physical assets), and WOs are not written by the same technician, as they come from various geographical locations.
    • Indirectly, the generalization is combined from our data (WO → 118 asset identifiers across 15 registry asset-type codes), Failure codes (20+), FMEA covers eight equipment/component catalog entries and 58 failure-mode rows, etc.

    We have now presented this data coverage in Appendix B.2 and described the data diversity. Specifically, Figure 10 and Table 10 in Appendix also document this coverage, which demonstrates resource overlap.

  • Task diversity: It is common in practice that different business personas and business units use the same data for different reasons (for example, reliability engineers use WO data for RUL studies, whereas Higher-level executives use it for scheduling and optimization), and this is where the diversity in our tasks comes in. This has been clarified in lines 261-269 (p. 3).

  • Sensitivity analysis: To assess sensitivity to this dependence, we also conducted a paired bootstrap over all 35 complete manifest bundles, retaining every constituent scenario whenever its bundle is sampled. We used 20,000 replicates with seed 20261106 and identical sampled bundles for all models. The resulting 95% Pass@1 intervals are 21.5–44.9% for Qwen3.8-Max and 19.3–40.2% for Claude Opus 5. Their paired difference has an interval of −4.0 to 8.9 percentage points. Please also refer to p. 7 (lines 377-395) and Appendix F.4, which report the procedure and all eight models’ intervals separately from the scenario-level analysis.

[1] Jiang, C., Zhou, C. and Yang, J., 2026, April. REI-Bench: Can embodied agents understand vague human instructions in task planning?. In International Conference on Learning Representations (Vol. 2026, pp. 31706-31738).

(Response Continue)

AuthorsOpenReview ↗

Response to Reviewer VHwt

W2: More root-cause and bottleneck analysis

W2. "More root-cause and bottleneck analysis is needed to provide insights on the model, harness, environment, tools, etc."

We agree, and we have added five analyses in “Workflow Gaps and Targeted Remediations” (Section 5.5, pp. 9–10): expert trajectory assessment (model), a deterministic reusable NLP workflow (tool), an enhanced coding environment (environment), relaxed-pass sensitivity analysis (evaluation), and question revision sensitivity (environment). More details are in Appendix K, L, M, N. The existing matched harness comparison appears in Section 5.2.

Here is a brief summary of our findings:

  1. In expert assessment, five engineers reviewed all 231 Opus 5 trajectories without references or scores, identifying gaps in maintenance-text interpretation, validation of classification decisions, asset-hierarchy scoping, and checking sensor–failure-mode relationships.

  2. The deterministic NLP workflow solved 11/65 WO scenarios (16.9%), exceeding the enhanced-environment agents’ 3–4/65.

  3. Execution Environment: Providing additional libraries and pretrained models changed aggregate WO success from 3.8% to 5.0%.

  4. The relaxed scoring increased overall success from 23.0% to 25.5%, admitting passing runs for eight previously unsolved scenarios.

  5. Human revisions informed by Fable feedback yielded passing outputs for four previously zero-pass WO scenarios.

The evaluated agents struggled with industrial domain knowledge, domain-specific text interpretation, workflow construction, and result verification. These analyses identify observed bottlenecks and sensitivity to resources and scoring and they do not establish a single cause of failure.

Codebases for additional experiments conducted during the rebuttal period are in ./Additional_experiments in the anonymized repo. The pointer to the anonymous repo is given at submission time.

(Response Continue)

AuthorsOpenReview ↗

Response to Reviewer VHwt

Q1: Operational state changes

Q1. "How many scenarios require an operational state change, and what independently verifies it?"

Three scenarios (322, 323 and 325) request failure-code imputation and scenario 325 explicitly requires writing the result back to the database. Agents can also invoke the available write tools from WO, FMSR, and TSFM MCP servers in Table 36, even when persistence is not explicitly requested.

  • How many time agents have invoked write tools? Across the 1,848 reported trajectories, 30 executions spanning six scenarios (158, 164, 196, 322, 323 and 325) returned work-order records from mutation tools. These tools in src/servers/wo/main.py support creating (generate work order), updating (update workorder), approving (approve workorder), assigning (assign technician), closing (close workorder), and cancelling work orders (cancel workorder). These responses show that agents invoked database-update tools, but we did not independently check whether the saved changes were correct (as this may also impact final performance).

  • Time series recipe and execution The run_recipe tool returned successful analysis results in 186 trajectories and 117 included fitting an algorithm to the supplied series. The implementation supports recording runs and results, as well as registering models and features for subsequent use. However, no model- or feature-registration calls were observed in the reported trajectories. These execution records do not constitute independent verification of final-state correctness.

  • Execution artifacts from MCP to coding agent We save large tool responses as workspace files to reduce context usage. Cached responses are invalidated after reported successful mutation calls to avoid reusing stale data. These files support analysis and inspection but creating them does not itself update operational database records.

  • Exporting environment state We have also released the runner (it was on different branch and anonymous git does not provide all branches) that can export CouchDB state after execution and before reset through export_couchdb_state (in src/benchmark/scenario_suite_runner.py), export_state (in src/couchdb/init_data.py), and export_database (in src/couchdb/loader.py). These JSON snapshots permit subsequent comparison, but the reported evaluators do not compare them against task-specific expected states. None of the 231 scenarios independently verifies final database state as a pass criterion. We therefore define Table 4’s checkmark as mutable-environment support, separately from final-state verification. Also, the GeneBench-Pro paper motivated us to formalize the output structure, to reduce the final answer computation.

[1] ​​Li, J., Shringarpure, S., Wong, E. and Ho, A., 2026. Genebench-pro: Evaluating multistage statistical reasoning in genomics, quantitative biology, and translational biomedicine. bioRxiv, pp.2026-06

The anonymous repository includes the evaluated scenarios (questions, manifests, and ground truths), trajectories, code, and supporting materials. The revised paper conclusion also states our plan to release these resources through Harbor Agent Eval or Kaggle Benchmarks.

If these answers resolve your concerns, we would be grateful if you could reflect them in your assessment.

AuthorsOpenReview ↗

Follow-up on our responses to Reviewer VHwt

Dear Reviewer VHwt,

Thank you for your initial comments, which helped us improve the paper. We have marked the changes in blue in the PDF and referenced in our point-by-point responses.

Please let us know if any concerns remain or further clarification would be helpful. Thank you for your time and consideration.

The Authors

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to All Reviewers

We thank all three reviewers for their careful reading and constructive feedback. The reviewers valued the realistic setting, the expert-built scenarios, the deterministic scoring, and the remaining headroom. We revised the paper and marked all changes in blue. Three points apply to every review.

1. Point-by-point revisions and consistent numbers. We answered every weakness and question point by point. Wherever a reviewer asked for evidence, we ran a new experiment:

  • a bundle-level paired bootstrap over the 35 evidence bundles (VHwt);
  • five workflow-gap analyses in Section 5.5: expert trajectory review, a deterministic NLP workflow, an enhanced coding environment, relaxed scoring and question revision (VHwt, bdJU, fHSy);
  • a confusion matrix of CAR response modes (fHSy).

Reviewers bdJU and fHSy found counts that did not match across the paper. Appendix tables generated before the CAR evaluator update caused these mismatches. We rescored all 1,848 saved trajectories under one scoring setup and regenerated every affected table and figure. The paper now reports the same values throughout: 425 passing runs, 85 tools (1 extra tool comes from agent harness), 65 WO scenarios, and 31.1% first-run Pass@1 on the subset. None of these corrections needed new model runs.

2. Experiment cost and the 35-scenario subset. Running agents is expensive. Each trajectory takes up to 30 turns of tool calls against live MCP servers, and the main evaluation alone needed 1,848 trajectories across eight models. Running every ablation on the full suite would multiply that cost by the number of configurations. We therefore ran the controlled comparisons on a 35-scenario subset stratified by family, which gives 280 paired runs per configuration. These comparisons cover skills, sub-agents, the gateway, the harness, the turn budget, and repeated trials. Standard evaluation platforms such as the Artificial Analysis leaderboard shaped the design: they report cost and token use alongside accuracy and test reliability over repeated runs. The subset is easier than the full suite (31.1% vs. 23.0% Pass@1). The paper now discloses its makeup (Appendix D.4) and difficulty spread (Appendix H), and it limits the ablation conclusions to the tested scenarios and implementations. So far we have spent close to 12,000 USD on environment building, testing, and benchmarking, excluding support of 5000 USD from an open-source community.

3. Open release. The anonymous repository now contains:

  • all 231 evaluated scenarios with questions, manifests, and ground truths; (data is withheld, but will be packaged as harbor task)
  • the programs that generate the references;
  • all trajectories and the evaluation code;
  • the runner that exports environment state;
  • the code for every rebuttal experiment (./Additional_experiments).

After the review period, we will release the complete benchmark on Harbor Agent Eval or Kaggle Benchmarks so that others can evaluate new models and agent designs on a shared, reproducible platform. The revised conclusion states this plan.

We hope these revisions address the reviewers' concerns.

AuthorsOpenReview ↗

Closing Remark from the authors to PC, SAC, AC and Reviewers

Dear conference committee,

We thank all three reviewers for their time and constructive feedback. Reviewer fHSy confirmed that the rescoring resolves the reported inconsistencies and that the engineers' review of failed trajectories and the CAR confusion matrix clarify the benchmark's failure modes. We summarize the final state of the paper below, including one new analysis.

What AssetArena contributes. AssetArena evaluates agents on end-to-end industrial asset-operation workflows rather than isolated capabilities. It provides 231 expert-built scenarios and 85 tools across six MCP servers, and a deterministic script scores every run. Its MCP-native environment lets one benchmark compare three agent-design choices: domain skills, sub-agent decomposition, and tool-access topology through a gateway, alongside the execution harness. Recent ICLR 2026 benchmarks shaped the design: REI-Bench (vague instructions), WildToolBench (tool use under real user behavior), and MCP-Bench (MCP tool orchestration). AssetArena instantiates these ideas for industrial operations on real maintenance records and physics-based simulations. Reviewer VHwt found the expert-built benchmark useful for evaluating model and harness capability on real-world tasks, and Reviewer bdJU described the setting as realistic and useful.

How the revision addresses the remaining concerns.

  • VHwt: a bundle-level paired bootstrap over the 35 evidence bundles (paired difference between the top two models: -4.0 to 8.9 points) and five workflow-gap analyses in Section 5.5.
  • bdJU: two experts reviewed every WO, FMSR, and FMEA reference, and the repository includes the reference programs. Table 5 now separates mutable state from final-state verification.
  • bdJU and fHSy: an outdated CAR evaluator caused the 22.9% vs. 31.1% gap and the count mismatches. Rescoring all 1,848 saved trajectories gives one consistent set of numbers.

New exploratory analysis (Appendix O). XYBench (Yerukola et al., 2026), the EMNLP 2026 Best Resource Paper, motivated a review of trajectories along four axes: understanding, knowledge and assumptions, approach, and fixation. The 169 failing Opus 5 runs include 12 with observed axis-related discrepancies and 116 with only candidate attributions; 16 involve scoring or output contracts and 25 require reference or evidence review. The eight-model subset review covers 280 runs, and its 193 failures include 29 observed discrepancies and 132 candidate attributions. This review separates agent workflow failures from scoring and reference issues. No independent expert has adjudicated it yet, so we report these counts as exploratory signals, not established causes.

Frontier agents reach only 16.5% to 29.9% Pass@1 on AssetArena, which leaves a substantial gap for mission-critical industrial operations. We will release the full benchmark through Harbor Agent Eval or Kaggle Benchmarks after the review period. We thank the Area Chair and all reviewers again.

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.