- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper adds a synthetic power-transformer maintenance domain to AssetOpsBench: 36 validated scenarios, out of which 31 are hand-authored scenarios. It compares agent configurations with 8B Llama on vLLM:
- direct tool calls vs MCP (Agent-as-Tool);
- Plan-Execute;
- Verified Plan-Execute (± Self-Ask).
The scores come from a Llama-4-Maverick judge. judge passes of 67 (for direct), 57)for mcp) and 86 (for verified PE) out of 155. A separate 6-scenario cohort gives median latencies of 8.51 s, 12.22 s and 37.92 s.
Some observations from the paper:
- the direct-vs-MCP difference is not resolved;
- configurations bundle several confounders;
- the judge disagrees with a second judge on 23.7% of pass labels;
- the synthetic data fails its realism check;
- there are no human or domain labels;
- the code is private.
As a result, the benchmark stands as not validated and the current contribution is insufficient for a main-track ICLR paper.
Critical strengths
- MCP servers and scenarios were contributed to AssetOpsBench (PR #287, open). This gives the resource potential community value independent of the paper.
- Table 5 uses a paired scenario-cluster bootstrap that respects the 31-scenario clustering instead of treating 155 trials as independent showing providing good stats.
- The reporting of the numbers is detailed and kept separate for each group allowing the examination of each pool e.g. 31/36/41/61 scenario counts are kept separate, and latency and quality numbers are kept seperate -this makes for clearer reading and evaluation
Critical weaknesses
- The paper has only 6 references and does not benchmark itself against tool-use and agent benchmarks or the LLM-as-judge reliability literature. Its novelty relative to AssetOpsBench itself is also not argued. AssetOpsBench already compares Tool-As-Agent vs Plan-Executor . Adding a seventh domain plus an MCP wrapper is incremental.
- Reproducibility is not possible since the code, scripts, data (analysis scripts, outputs) are private. For a benchmark paper this is a seroius shortcoming.
- No human or expert label exists hence there is no ground -truth. This puts into question the validity of the benchmark numbers since these based on LLM as a judge that appear to be non-determinisitic and not controlled for hallucaination.
- Section 2.1 says the historical data descriptions "do not match those tables" and without a run-specific tag, one cannot establish which inputs a historical run used. A benchmark paper whose evaluated runs cannot be tied to a definite data version falls short on the data contribution metric
Questions for authors
- Rerun the core grid on the harness that is repaired to identify & log the data set used for each run
- human or expert labels are needed on a stratified subset of the data which would allow paper to report LLM-judge–human agreement per configuration.
- Release anonymized code, data and run records.
- need a related-work section and novelty beyond AssetOpsBench.