- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces MicroPerf-Bench, a benchmark for end-to-end latency optimization by coding agents in microservice systems. Because real system-level performance fixes are rare, the authors inject known performance anti-patterns into seven open-source systems, which gives 65 instances with deployable environments, functional tests and load scripts. A patch is scored by the share of the injected latency it removes, after passing correctness and rule checks. Nine configurations, from three agent scaffolds and three models, are each run three times. The best one recovers 42.0% of the injected degradation. The most common failure is a correct patch that does not reduce latency, which the authors attribute to poor bottleneck localization. A single run with a profiling tool raises one configuration from 39.4 to 47.7.
Critical strengths
1. The motivation is solid and the paper tackles a real gap. Current performance benchmarks work at the function or repository level. Here, latency comes from how services interact, and every patch is evaluated by redeploying the whole system under load. This is the main reason I see value in the paper.
2. The pattern selection and the creation of the problem set are explained very clearly. It is easy to follow how each instance was built, why it should be fixable and how it was validated. Since the benchmark relies on injected problems, this clarity matters.
Critical weaknesses
1. Agents were not instructed to test their own performance and iterate. The task prompt asks the agent to read the code, find bottlenecks, write the fix and report what changed. It never asks the agent to deploy the system, run the provided workload, check whether its change helped, and keep going if it did not. The paper itself says agents were not instructed to measure and rarely did. The budget numbers show the same thing. Agents stopped after 7.7 to 18.0 minutes on average, using at most 15% of the two-hour budget.
This undermines the main conclusion that agents fail to localize the bottleneck. An agent that never measures has to guess from the code where the time goes. The dominant failure, a correct patch with no latency gain, may therefore come from the protocol rather than from the agents. The single profiling run points the same way. A baseline where agents are told to measure and iterate within the budget is needed to support the conclusion.
2. Many well-known models used in practice are not evaluated. Only three models are tested. Widely used models such as Claude, GPT and Gemini, which power tools like Claude Code and Codex, are missing, as are recent strong models such as MiniMax. This makes it very hard to see how effective this benchmark is, and whether the low scores describe current coding agents or only these three models. It would also support the motivation to show that the evaluated models do well on other coding benchmarks and fall off on this one.
3. There is no human baseline. Engineers audited the benchmark, but the degraded systems were never given to a separate group of engineers to optimize. If a group of engineers could be found to audit the problem set, a group can be found to create a baseline. Without it, there is no way to tell what score is achievable or how hard the instances are.
Questions for authors
Clarifications
- Could the agents deploy and load-test the full system from inside their environment? How many runs executed the workload at least once, and how did those runs score?
- In the profiling run, did anything change besides the added tool, such as the instructions?
- Why do agents stop so early? Could you release the trajectories and report reasoning tokens per configuration, so readers can see the agents' reasoning?
Suggestions
- A baseline where agents are instructed to measure and iterate.
- Agents based on other widely used models, such as Claude, GPT, Gemini and MiniMax.
- A human baseline. At least on a subset of instances.
Your related work is missing close work in:
- Between Promise and Pain: The Reality of Automating Failure Analysis in Microservices with LLMs. A. Cornacchia, I. Alabdulaal, I. Saghier, A. Mirdad, O. Fayoumi, M. Canini. APSys'25, Oct 2025.
Concurrent work, for context only. It appeared on arXiv shortly before the submission deadline and is not yet published, so it does not count against this submission, but it would be worth discussing in a revision:
- DDBench: A Benchmark for Agentic Debugging on Distributed Systems. Y. Yan, H. Wang, J. He, Y. Liang, S. Wang, H. Sun, S. J. Park. arXiv:2608.14863, Aug 2026; to appear at NeurIPS'26.
4 comments
Response to Reviewer eWY3 (Submission 39580) — Part 2/2: Suggestions & New Experiments
Suggestions & New Experiments
-
Suggestion 1. Baseline instructed to measure and iterate. Following your suggestion, we ran Claude Code + Qwen on all 65 tasks keeping the CLAUDE.md measurement-and-revision instructions while removing measure.py. We report both unconstrained iteration within the 2-hour cap and the matched (≤2 measurements) checkpoint score:
- Unconstrained iteration: Mean score rises from 39.4 (no-measurement baseline) to 40.4, a paired gain of +1.0 point. The modest gain reflects more fairness-violating optimizations than in the matched conditions, possibly because more iterations led agents to attempt riskier optimizations. Approximately 9% of tasks complied with the ≤2-measurement budget; agents averaged 8.0 measurements and reached up to 10 measurement-revision iterations.
- Matched ≤2-measurement budget: Evaluating the saved checkpoints at ≤2 measurements yields 44.0 (+4.6 over baseline 39.4), vs. 47.7 (+8.3, 95% CI [-1.6, +18.3]) with measure.py.
- Revised takeaway: This shows that unconstrained measurement and revision alone yield only a +1.0-point net gain, and we will revise our main conclusion accordingly. At the same time, the unconstrained prompt-only run scores 40.4 with an average of 8.0 measurements and up to 10 measurement-revision iterations, below both 44.0 at the matched ≤2-measurement checkpoint and 47.7 for measure.py, and even when agents measure and iterate freely, >59% of injected system-level degradation remains unrecovered.
-
Suggestion 2. Stronger, widely used frontier model. We evaluated Codex + GPT-6.1 Sol across all 65 instances on the exact same dedicated host, container resource limits, and frozen , baselines (single-pass run). It achieves a mean score of 49.5, exceeding the best 3-run baseline (OpenHands + GLM, 42.0) and Claude Code + Qwen with profiling (47.7). System means are PetClinic (86.5), HotelReservation (84.8), MartianBank (58.5, improving all 8 tasks), SocialNetwork (40.5), Pitstop (37.5), RobotShop (20.1), and MediaService (14.4). Complex multi-service systems remain challenging and overall recovery stays near 50%, showing that low scores are not an artifact of the initial three models. We will also add a table contrasting evaluated models' high scores on repo-level benchmarks (e.g., SWE-bench Verified) against their drop on MicroPerf-Bench.
-
Suggestion 3. Human baseline. We agree that an independent human baseline helps calibrate achievable scores. To avoid data contamination, participants cannot be the engineers who audited the repos or inspected ground-truth patches in our provenance/difficulty studies (where 3 source-blinded engineers rated sampled tasks at 3.0 to 4.0 median difficulty on a 5-point scale, Krippendorff's = 0.81). Recruiting a separate, unexposed group of microservice engineers for isolated 2-hour container sessions across a representative subset could not be completed within the rebuttal window without compromising rigor. Every task already ships with a verified human-constructed clean reference scoring 100.0 under all gates, confirming full solvability, and we will include an unexposed 15-task human baseline if completed for the final version.
-
Related work. Thank you for both references; we will discuss them in Related Work, citing Cornacchia et al. (APSys'25) on practical gaps in LLM microservice failure analysis and contrasting concurrent work DDBench (Yan et al., NeurIPS'26) (debugging functional/operational distributed faults) with MicroPerf-Bench (end-to-end latency optimization under functional and fairness gates).
Thanks to the authors for addressing the points I raised; as a result the paper will be stronger. I maintain my positive support for this paper and think it should be accepted.
We sincerely thank the reviewer for the constructive feedback and continued support, and we are pleased that our responses have addressed the points raised.
Response to Reviewer eWY3 (Submission 39580) — Part 1/2: Clarifications (Q1–Q3)
We thank the reviewer for recognizing the benchmark's motivation and clarity, and for the constructive suggestions. Below we address your clarifications first, followed by our new experiments and revisions.
Clarifications
Q1. Shell runtime access and spontaneous workload runs. Yes, agents can build, deploy, and load-test full systems from the shell: all 7 systems expose load scripts and logs, and 5/7 expose distributed traces out of the box. Across all 1,755 baseline runs, however, only 7 runs (0.40%) executed the workload at least once: six OpenHands + GLM runs on Pitstop (which provides a single command for build, startup, health checks, seeding, and load testing), scoring 98.5, 11.6, 63.2, 22.5, 100.0, and 33.5 (mean = 54.9, including two variant-best scores of 98.5 and 100.0), plus one MartianBank run that measured twice and regressed (score 0; 7-run mean = 47.0). Multi-step systems saw zero spontaneous load tests.
Q2. What changed in the profiling condition? Yes, workspace instructions also changed. Alongside the read-only measure.py probe (unifying workload, trace, and docker stats outputs into one CLI report with fallback to wrk/logs), the profiling run added a workspace file (CLAUDE.md) requiring pre-edit baseline and post-edit validation measurements (MAX_MEASUREMENTS = 2, no intermediate runs), rollback/revision on regression, and reporting of measured data. The base prompt, task text, and 2-hour cap were unchanged. We state this dual change explicitly in the revision and isolate the instruction effect in Suggestion 1 below.
Q3. Early stopping, trajectories, and reasoning tokens. Without a runtime-measurement requirement, baseline agents form a static checklist (read code, edit, compile/build, summarize) and stop once all self-defined items are checked. In audited logs, successful compilation or build serves as their final check (Claude Code + Qwen averages 8.5 min, median 7.9 min, vs. ~14 min with profiling due to deployment and two load cycles). We will release all full trajectories and add per-configuration token and reasoning-token statistics to the manuscript.