acceptodds
Under review as a conference paper at ICLR 2027

SmartGridBench: Cohort-Bound Evaluation of Industrial LLM Agents for Transformer Maintenance

Abstract

Power-transformer maintenance requires an agent to combine telemetry, fault interpretation, forecasting, and work-order records before recommending an action. We introduce SmartGridBench, a synthetic transformer-maintenance extension of AssetOpsBench, and use it to separate tool-interface choices from planning choices. The frozen resource contains 36 structurally validated scenarios; the principal quality comparison evaluates 31 hand-authored scenarios, while a later repository expansion to 61 scenarios was not evaluated in these experiments. On the 31-scenario core, each configuration has five trials per scenario: direct tool calls receive 67/155 recorded judge passes, MCP Agent-as-Tool receives 57/155, and Verified Plan-Execute receives 86/155. A separate six-scenario latency cohort records median completion times of 8.51 s, 12.22 s, and 37.92 s for those three configurations; the cohorts are not pooled. A 21-case, outcome-stratified cross-family judge check agrees with the historical pass label on 13/21 cases (61.9%) and assigns 4 versus 10 passes, which demonstrates evaluator sensitivity rather than judge accuracy. The synthetic fleet fails an existing DGA realism check, and independent human and domain validation remain unavailable. We therefore present SmartGridBench as a reproducible fixture for conditional systems evaluation; establishing maintenance readiness or deployment safety would require further validation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, tool use, Model Context Protocol, benchmark, power transformers, maintenance, evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_P3yNOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper adds a synthetic power-transformer maintenance domain to AssetOpsBench: 36 validated scenarios, out of which 31 are hand-authored scenarios. It compares agent configurations with 8B Llama on vLLM:

  • direct tool calls vs MCP (Agent-as-Tool);
  • Plan-Execute;
  • Verified Plan-Execute (± Self-Ask).

The scores come from a Llama-4-Maverick judge. judge passes of 67 (for direct), 57)for mcp) and 86 (for verified PE) out of 155. A separate 6-scenario cohort gives median latencies of 8.51 s, 12.22 s and 37.92 s.

Some observations from the paper:

  • the direct-vs-MCP difference is not resolved;
  • configurations bundle several confounders;
  • the judge disagrees with a second judge on 23.7% of pass labels;
  • the synthetic data fails its realism check;
  • there are no human or domain labels;
  • the code is private.

As a result, the benchmark stands as not validated and the current contribution is insufficient for a main-track ICLR paper.

Critical strengths
  • MCP servers and scenarios were contributed to AssetOpsBench (PR #287, open). This gives the resource potential community value independent of the paper.
  • Table 5 uses a paired scenario-cluster bootstrap that respects the 31-scenario clustering instead of treating 155 trials as independent showing providing good stats.
  • The reporting of the numbers is detailed and kept separate for each group allowing the examination of each pool e.g. 31/36/41/61 scenario counts are kept separate, and latency and quality numbers are kept seperate -this makes for clearer reading and evaluation
Critical weaknesses
  • The paper has only 6 references and does not benchmark itself against tool-use and agent benchmarks or the LLM-as-judge reliability literature. Its novelty relative to AssetOpsBench itself is also not argued. AssetOpsBench already compares Tool-As-Agent vs Plan-Executor . Adding a seventh domain plus an MCP wrapper is incremental.
  • Reproducibility is not possible since the code, scripts, data (analysis scripts, outputs) are private. For a benchmark paper this is a seroius shortcoming.
  • No human or expert label exists hence there is no ground -truth. This puts into question the validity of the benchmark numbers since these based on LLM as a judge that appear to be non-determinisitic and not controlled for hallucaination.
  • Section 2.1 says the historical data descriptions "do not match those tables" and without a run-specific tag, one cannot establish which inputs a historical run used. A benchmark paper whose evaluated runs cannot be tied to a definite data version falls short on the data contribution metric
Questions for authors
  • Rerun the core grid on the harness that is repaired to identify & log the data set used for each run
  • human or expert labels are needed on a stratified subset of the data which would allow paper to report LLM-judge–human agreement per configuration.
  • Release anonymized code, data and run records.
  • need a related-work section and novelty beyond AssetOpsBench.
Reviewer_PgwcOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper describes SmartGridBench, a synthetic transformer-maintenance extension of AssetOpsBench: 31 hand-authored scenarios (plus five generated ones and five invalid-input fixtures) in which an agent combines telemetry, fault interpretation, forecasting and work-order tools (Sec. 2.1–2.2). Table 1 lists eight configurations, seven of which appear in the quality results: an iterative Agent-as-Tool loop with direct tool calls, with MCP, and with an "optimized" MCP setup, and four MCP-based planning variants (Plan-Execute, with Self-Ask, Verified Plan-Execute, with Self-Ask). The actor is Llama-3.1-8B-Instruct served with vLLM; the outcome judge is Llama-4-Maverick scoring a six-dimension rubric with a 0.6 pass threshold (Sec. 2.3–2.4). On the 31-scenario core with five trials each, judge passes are 67/155 for direct calls, 57/155 for MCP and 86/155 for Verified Plan-Execute (Table 2). A separate six-scenario cohort gives median completion times of 8.51 s, 12.22 s and 37.92 s (Table 3). A cross-family judge check agrees with the historical label on 13 of 21 cases (Sec. 3.3); an offline re-scoring of 2,419 trajectory identities with a second judge agrees on 1,846 (App. B). Post-hoc bootstrap intervals for direct-vs-MCP include zero; the planning mean-score intervals exclude zero (App. D).

The paper states that the synthetic fleet fails a DGA realism check, that no human or domain validation exists, that a codebase audit found several harness defects whose effect on the recorded runs is unknown, that the configurations bundle transport, serving and prompting changes, and that the release package "remains to be finalized" (Sec. 4, App. C). It positions the resource as a "reproducible fixture for conditional systems evaluation".

Critical strengths
  • The paper is honest about its limits. Every number carries its denominator, cohorts are never pooled, the judge is called a judge rather than ground truth, and the limitations section is thorough. This is credited here and in the Presentation score below.
  • Reporting judge sensitivity (a second judge on 2,419 paired trajectory identities; a 21-case cross-family check; the finding that the historical judge is the more lenient one, with 474 pass-to-fail flips against 99 the other way) and bootstrap intervals for the headline contrasts is good practice, and the paper draws the correct conclusion from them.
  • The distinction between a tool call that succeeds and a recommendation that follows from the evidence (Sec. 1) is the right motivation for a maintenance benchmark, and the direct-call versus MCP question is a real gap: none of the existing MCP benchmarks compares MCP against native function calling on latency or quality.
  • The AI-use disclosure is detailed and clear about what was and was not machine-generated.
Critical weaknesses
  1. None of the three claimed contributions is usable by a reader today. Sec. 1 lists a benchmark extension, a cohort-bound comparison, and an auditable account of judge and data limits. The benchmark extension fails the one realism check applied to it and has no human or domain validation (Sec. 2.1, Sec. 4), and it is not released. The comparison was produced by a harness in which a full-codebase audit found call/result association, direct/MCP description parity, decoding-setting, resumed-process, mutable trial-state, verifier-fallback and empty-plan issues, and the paper cannot say which recorded trials were affected (Sec. 4). The audit itself cannot be inspected. The one result that stands on its own is the judge-sensitivity finding in App. B (the historical judge is more lenient than the second judge), but without human labels it says which judge is stricter, not which is right. The paper's own words: the results are "useful only with their cohort denominators and validity caveats" (Sec. 5), and "scoped impact assessment and revalidation remain required before causal or deployment conclusions" (Sec. 4). The right next step is the one the conclusion lists: repair the harness, re-run the grid, validate the data and the judge, release, and then submit.

  2. The two research questions remain open by the paper's own account. For direct calls versus MCP, the pass-rate difference is 6.5 points with a bootstrap interval of [−3.2, 15.5], and the mean-score interval also includes zero (Table 5). For planning, the mean-score intervals exclude zero, which the paper reports, and the pass-rate interval for Plan-Execute includes zero while the Verified Plan-Execute interval starts at 0.0; the paper then declines to attribute the planning gain to planning (Sec. 3.1). The reasons are stated in the paper: configurations bundle transport, serving and prompting; the "optimized" MCP row differs from the baseline in more than caching; the vanilla Plan-Execute client runs a different Python version from the others (Sec. 2.3, 2.5); and the latency cohort (six scenarios) cannot be joined to the quality cohort (Sec. 3.2). The paper writes: "We do not claim that these comparisons isolate a general protocol effect" (Sec. 1). Two things the paper does not discuss: the latency cohort's pass rates (6/30, 3/30, 17/30, 22/30) look nothing like the core's (67, 57, 76, 86 of 155), and MCP's 95th-percentile time is below direct calls' (13.99 s vs 20.73 s) although its median is higher (Table 3).

  3. The artifact is unavailable, so the only surviving contribution cannot be inspected. "An anonymous release package remains to be finalized"; the analysis scripts and frozen output tables are "retained in the private revision record"; the release status of the scenarios, tools, harness and judge prompt is not stated (App. C). The judge prompt is named (assetopsbench-6d-v1) and not shown. Re-running the analysis "requires the source evidence files", which are not provided, and the paper says a re-run "creates a new analysis, not a byte-identical reproduction" of this one. ICLR encourages rather than requires code at submission, so this is a reviewer expectation rather than a rule; still, a benchmark whose scenarios, evaluator and harness cannot be seen cannot be assessed as a benchmark. Release is possible during the discussion period, and question 1 asks for it.

  4. The study is small, and its one planning finding contradicts its parent without comment. One actor model (an 8B model, smaller than most models on the parent's leaderboard) on the core; the 70B rows form a separate 15-scenario cohort with unequal replication (3 versus 5 trials), and the paper reports only one of its rows (Sec. 3.3); 31 scenarios written by the authors, with one example shown (list the sensors of transformer T-011, Sec. 2.2); five trials. The second-judge re-scoring (Table 4) is broken out by workflow and never by direct versus MCP, so whether the headline ordering survives the other judge is unknown. AssetOpsBench already compares Agent-as-Tool against Plan-Execute on 141 scenarios built with reliability engineers, across seven models, and finds Agent-as-Tool stronger on task completion (about 65% versus 46% at best). This paper finds the reverse with one small model and does not mention the parent's result. The domain is new (the parent covers chillers and air handlers); the planning comparison is the parent's design on weaker data. The parent also ships an official MCP server, and the paper does not say what it reuses or why its own MCP wrapper differs.

  5. There is no related work, so the paper cannot be placed. Six references and no related-work section. MCP tool-use benchmarks exist (MCP-Bench, ICLR 2026; LiveMCPBench; MCP-Universe; MCP-RADAR; MCPEval; LiveMCP-101; MCPToolBench++) and an MCP-overhead study ("Help or Hurdle?", 2025) measures token cost with and without MCP; none measures MCP against direct calls on latency, which is exactly the gap this paper could fill once repaired. Plan-Execute and Self-Ask are used by name without citation (Plan-and-Solve, Wang et al. 2023; Self-Ask, Press et al. 2023), and the self-verification design should be positioned against ReAct and Reflexion. The judge-sensitivity analysis belongs to the LLM-as-judge reliability literature (Zheng et al. 2023; Gu et al. 2024; Ye et al. 2025; and recent work on score variability at temperature zero). The "DGA realism check" should be referenced against the standards a reader would expect (IEC 60599:2022, IEEE C57.104-2019, Duval's triangle and the IEC TC 10 database), none of which is cited.

  6. Key definitions are missing or dangling. The rubric dimensions are listed (Sec. 2.4), but the prompt text and the criteria behind each dimension are not shown. The scenarios are described by one retrieval example; the authoring and checking process for the 31 hand-written ones is not described. The DGA check is described only as a statistical check against "a labeled reference distribution" (Sec. 2.1), with no statistic, reference or failure criterion. The "audit-derived taxonomy" is defined only by its label counts (App. A). "Core-15 and core-16 cohorts" and "2,420 eligible score rows span 37 groups" (Sec. 2.5) are not explained. Configuration D (Table 1) appears in no result. The text refers to a "historical appendix" (Sec. 2.1), "the earlier presentation" (Sec. 2.3) and "the September revision" (App. C–D) that the submission does not contain. The 8,000-character truncation affects 100 of 1,085 trajectories unevenly across configurations (6 to 30 per row); the paper bounds the effect with a 28-case pilot in which the cap changed one score, which is fair, and still leaves the full-grid question open (Sec. 2.4).

Questions for authors
  1. When will the scenarios, tools, harness, judge prompt and analysis scripts be released? Can an anonymized package be made available during the discussion?
  2. Can each audit defect be mapped to the affected trials, by configuration, and can the full grid be re-run on the repaired harness? The core is 1,085 trials of an 8B model; why report historical runs rather than re-run?
  3. What is the DGA realism check: which reference distribution, which statistic, and what does "fails" mean numerically?
  4. How were the 31 scenarios authored and checked, and do any of the authors have transformer-maintenance experience? When will the prepared annotation sample (App. C) be scored?
  5. Please show the judge prompt and the per-dimension criteria. Were any trajectories scored by a human?
  6. Please report the second-judge pass counts per configuration on the 31-scenario core. Does direct-above-MCP survive the second judge?
  7. Why do the latency cohort's pass rates differ so much from the core's, and why is MCP's 95th-percentile time below direct calls' while its median is higher? Please report paired per-trial latency differences and identify the GPU.
  8. What is the isolated effect of the Python-version difference and of prefix caching? Can one matched contrast (same client environment, caching off) be run?
  9. Can a second actor (the 70B model, or a frontier model) be run on the full 31-scenario core with equal trial counts, and which rows does the 70B cohort contain?
  10. AssetOpsBench finds Agent-as-Tool stronger than Plan-Execute; this paper finds the reverse. How do the authors explain the difference, and how does their MCP wrapper relate to the parent's official MCP server?
  11. What are the "core-15" and "core-16" cohorts, and the 37 groups behind the 2,420 rows?
Reviewer_d8DmOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper adds a synthetic power-transformer maintenance domain to AssetOpsBench and uses it to compare agent setups built on Llama-3.1-8B. It compares tools called directly with tools called through MCP, and a ReAct-style iterative agent with Plan-Execute, with and without a self-verification step (plus Self-Ask variants). Quality is measured on 31 hand-written scenarios, five trials each. A Llama-4-Maverick judge scores each run on a six-item binary rubric, and 4/6 counts as a pass. Direct calls get 67/155 passes, MCP 57/155, and Verified Plan-Execute 86/155. Latency comes from a separate six-scenario set, with medians of 8.5s, 12.2s and 37.9s for those same three setups.

Most of the rest of the paper is about validity: post-hoc scenario-bootstrap intervals, a re-scoring of all stored trajectories with a second judge (DeepSeek Flash) that flips about a quarter of the pass labels, and an audit of the harness. The authors present the benchmark as a reproducible fixture for conditional systems evaluation, not as evidence of maintenance readiness.

Critical strengths

The paper is unusually candid. Every number comes with its denominator, and the quality and latency cohorts are kept apart. The intervals are clustered by scenario instead of treating 155 trials as independent. The authors also re-scored everything with a second judge and reported the result even though it weakens their findings. Agent benchmark papers rarely report this carefully, and I'd like to see more of it. It counts for less than it might, though, because once all the caveats are applied there isn't much of a claim left (see weaknesses).

The question is also a practical one. People building industrial agents need to know whether MCP costs anything compared with calling tools directly, and whether plan-and-verify is worth roughly 4x the latency.

Critical weaknesses
  1. The results come from a harness the authors say had defects. A code audit after the runs found problems including call/result association, tool descriptions that differ between direct and MCP, and decoding settings (L232-238). The authors say they don't know which trials were affected, and they also can't say which data inputs a given run used (L59-62). If the trials themselves may be invalid, I can't evaluate the comparisons.

  2. The outcome measure isn't validated, and the main effect seems to depend on the judge. There are no human labels, a run can pass while failing task completion (L130-131), and a second judge flips about a quarter of the pass labels. Direct vs. MCP isn't significant (Table 5) and is confounded anyway (L93-96, L162-164). The planning gain is the only effect that holds up, but in Table 4 it shrinks from about 0.10 to 0.03-0.04 under the second judge.

Also: there's no related work section (ReAct and Self-Ask aren't cited), only one simple example scenario is shown, and the code and data aren't released (L374-375).

Questions for authors
  1. (Clarification) Which of the 31-scenario trials ran before the fixes listed at L233-234? Did the call/result issue affect the direct and MCP runs?
  2. Could you report Tables 2 and 5 using the DeepSeek scores on the matched 31 scenarios?
  3. Could you re-run the 31-scenario core with the repaired harness?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.