acceptodds
Under review as a conference paper at ICLR 2027

Benchmarking Coding Agents for System-Level Performance Optimization

Abstract

Coding agents are becoming an increasingly important paradigm for automating software engineering tasks. However, existing coding benchmarks primarily evaluate agents at the function or repository level, leaving their capabilities in more complex distributed software systems largely unexplored. In practice, many modern applications adopt microservice architectures, where end-to-end performance emerges from interactions among multiple independently running services, requiring agents to reason beyond local code changes. To study this capability, we introduce MicroPerf-Bench, a benchmark for evaluating coding agents on end-to-end performance optimization in microservice systems. Constructing such a benchmark is challenging because reproducible system-level performance fixes are sparse and difficult to attribute. We address this challenge by injecting empirically grounded performance anti-patterns into real-world open-source systems while preserving functionality and ensuring measurable performance degradation. The resulting benchmark contains 65 optimization instances across seven systems and nine anti-patterns, each with an executable environment, functional tests, and mixed-workload evaluation.Across 1,755 runs covering nine agent–model configurations, the best configuration recovers only 42.0% of the injected performance degradation. The dominant failure is not invalid code, but functionally correct and constraint-compliant changes that fail to improve end-to-end performance, highlighting system-level bottleneck localization as a key challenge for current coding agents. The source code and data are available at https://anonymous.4open.science/r/microperf-45B5/.

Primary area:
datasets and benchmarks
Keywords:
Benchmark, microservice systems, code performance optimization, LLM agents, per- formance anti-patterns, fault injection

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_eWY3OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces MicroPerf-Bench, a benchmark for end-to-end latency optimization by coding agents in microservice systems. Because real system-level performance fixes are rare, the authors inject known performance anti-patterns into seven open-source systems, which gives 65 instances with deployable environments, functional tests and load scripts. A patch is scored by the share of the injected latency it removes, after passing correctness and rule checks. Nine configurations, from three agent scaffolds and three models, are each run three times. The best one recovers 42.0% of the injected degradation. The most common failure is a correct patch that does not reduce latency, which the authors attribute to poor bottleneck localization. A single run with a profiling tool raises one configuration from 39.4 to 47.7.

Critical strengths

1. The motivation is solid and the paper tackles a real gap. Current performance benchmarks work at the function or repository level. Here, latency comes from how services interact, and every patch is evaluated by redeploying the whole system under load. This is the main reason I see value in the paper.

2. The pattern selection and the creation of the problem set are explained very clearly. It is easy to follow how each instance was built, why it should be fixable and how it was validated. Since the benchmark relies on injected problems, this clarity matters.

Critical weaknesses

1. Agents were not instructed to test their own performance and iterate. The task prompt asks the agent to read the code, find bottlenecks, write the fix and report what changed. It never asks the agent to deploy the system, run the provided workload, check whether its change helped, and keep going if it did not. The paper itself says agents were not instructed to measure and rarely did. The budget numbers show the same thing. Agents stopped after 7.7 to 18.0 minutes on average, using at most 15% of the two-hour budget.

This undermines the main conclusion that agents fail to localize the bottleneck. An agent that never measures has to guess from the code where the time goes. The dominant failure, a correct patch with no latency gain, may therefore come from the protocol rather than from the agents. The single profiling run points the same way. A baseline where agents are told to measure and iterate within the budget is needed to support the conclusion.

2. Many well-known models used in practice are not evaluated. Only three models are tested. Widely used models such as Claude, GPT and Gemini, which power tools like Claude Code and Codex, are missing, as are recent strong models such as MiniMax. This makes it very hard to see how effective this benchmark is, and whether the low scores describe current coding agents or only these three models. It would also support the motivation to show that the evaluated models do well on other coding benchmarks and fall off on this one.

3. There is no human baseline. Engineers audited the benchmark, but the degraded systems were never given to a separate group of engineers to optimize. If a group of engineers could be found to audit the problem set, a group can be found to create a baseline. Without it, there is no way to tell what score is achievable or how hard the instances are.

Questions for authors

Clarifications

  1. Could the agents deploy and load-test the full system from inside their environment? How many runs executed the workload at least once, and how did those runs score?
  2. In the profiling run, did anything change besides the added tool, such as the instructions?
  3. Why do agents stop so early? Could you release the trajectories and report reasoning tokens per configuration, so readers can see the agents' reasoning?

Suggestions

  1. A baseline where agents are instructed to measure and iterate.
  2. Agents based on other widely used models, such as Claude, GPT, Gemini and MiniMax.
  3. A human baseline. At least on a subset of instances.

Your related work is missing close work in:

  • Between Promise and Pain: The Reality of Automating Failure Analysis in Microservices with LLMs. A. Cornacchia, I. Alabdulaal, I. Saghier, A. Mirdad, O. Fayoumi, M. Canini. APSys'25, Oct 2025.

Concurrent work, for context only. It appeared on arXiv shortly before the submission deadline and is not yet published, so it does not count against this submission, but it would be worth discussing in a revision:

  • DDBench: A Benchmark for Agentic Debugging on Distributed Systems. Y. Yan, H. Wang, J. He, Y. Liang, S. Wang, H. Sun, S. J. Park. arXiv:2608.14863, Aug 2026; to appear at NeurIPS'26.
4 comments
AuthorsOpenReview ↗

Response to Reviewer eWY3 (Submission 39580) — Part 1/2: Clarifications (Q1–Q3)

We thank the reviewer for recognizing the benchmark's motivation and clarity, and for the constructive suggestions. Below we address your clarifications first, followed by our new experiments and revisions.

Clarifications

  • Q1. Shell runtime access and spontaneous workload runs. Yes, agents can build, deploy, and load-test full systems from the shell: all 7 systems expose load scripts and logs, and 5/7 expose distributed traces out of the box. Across all 1,755 baseline runs, however, only 7 runs (0.40%) executed the workload at least once: six OpenHands + GLM runs on Pitstop (which provides a single command for build, startup, health checks, seeding, and load testing), scoring 98.5, 11.6, 63.2, 22.5, 100.0, and 33.5 (mean = 54.9, including two variant-best scores of 98.5 and 100.0), plus one MartianBank run that measured twice and regressed (score 0; 7-run mean = 47.0). Multi-step systems saw zero spontaneous load tests.

  • Q2. What changed in the profiling condition? Yes, workspace instructions also changed. Alongside the read-only measure.py probe (unifying workload, trace, and docker stats outputs into one CLI report with fallback to wrk/logs), the profiling run added a workspace file (CLAUDE.md) requiring pre-edit baseline and post-edit validation measurements (MAX_MEASUREMENTS = 2, no intermediate runs), rollback/revision on regression, and reporting of measured data. The base prompt, task text, and 2-hour cap were unchanged. We state this dual change explicitly in the revision and isolate the instruction effect in Suggestion 1 below.

  • Q3. Early stopping, trajectories, and reasoning tokens. Without a runtime-measurement requirement, baseline agents form a static checklist (read code, edit, compile/build, summarize) and stop once all self-defined items are checked. In audited logs, successful compilation or build serves as their final check (Claude Code + Qwen averages 8.5 min, median 7.9 min, vs. ~14 min with profiling due to deployment and two load cycles). We will release all full trajectories and add per-configuration token and reasoning-token statistics to the manuscript.

AuthorsOpenReview ↗

Response to Reviewer eWY3 (Submission 39580) — Part 2/2: Suggestions & New Experiments

Suggestions & New Experiments

  • Suggestion 1. Baseline instructed to measure and iterate. Following your suggestion, we ran Claude Code + Qwen on all 65 tasks keeping the CLAUDE.md measurement-and-revision instructions while removing measure.py. We report both unconstrained iteration within the 2-hour cap and the matched (≤2 measurements) checkpoint score:

    • Unconstrained iteration: Mean score rises from 39.4 (no-measurement baseline) to 40.4, a paired gain of +1.0 point. The modest gain reflects more fairness-violating optimizations than in the matched conditions, possibly because more iterations led agents to attempt riskier optimizations. Approximately 9% of tasks complied with the ≤2-measurement budget; agents averaged 8.0 measurements and reached up to 10 measurement-revision iterations.
    • Matched ≤2-measurement budget: Evaluating the saved checkpoints at ≤2 measurements yields 44.0 (+4.6 over baseline 39.4), vs. 47.7 (+8.3, 95% CI [-1.6, +18.3]) with measure.py.
    • Revised takeaway: This shows that unconstrained measurement and revision alone yield only a +1.0-point net gain, and we will revise our main conclusion accordingly. At the same time, the unconstrained prompt-only run scores 40.4 with an average of 8.0 measurements and up to 10 measurement-revision iterations, below both 44.0 at the matched ≤2-measurement checkpoint and 47.7 for measure.py, and even when agents measure and iterate freely, >59% of injected system-level degradation remains unrecovered.
  • Suggestion 2. Stronger, widely used frontier model. We evaluated Codex + GPT-6.1 Sol across all 65 instances on the exact same dedicated host, container resource limits, and frozen LRL_R, LDL_D baselines (single-pass run). It achieves a mean score of 49.5, exceeding the best 3-run baseline (OpenHands + GLM, 42.0) and Claude Code + Qwen with profiling (47.7). System means are PetClinic (86.5), HotelReservation (84.8), MartianBank (58.5, improving all 8 tasks), SocialNetwork (40.5), Pitstop (37.5), RobotShop (20.1), and MediaService (14.4). Complex multi-service systems remain challenging and overall recovery stays near 50%, showing that low scores are not an artifact of the initial three models. We will also add a table contrasting evaluated models' high scores on repo-level benchmarks (e.g., SWE-bench Verified) against their drop on MicroPerf-Bench.

  • Suggestion 3. Human baseline. We agree that an independent human baseline helps calibrate achievable scores. To avoid data contamination, participants cannot be the engineers who audited the repos or inspected ground-truth patches in our provenance/difficulty studies (where 3 source-blinded engineers rated sampled tasks at 3.0 to 4.0 median difficulty on a 5-point scale, Krippendorff's α\alpha = 0.81). Recruiting a separate, unexposed group of microservice engineers for isolated 2-hour container sessions across a representative subset could not be completed within the rebuttal window without compromising rigor. Every task already ships with a verified human-constructed clean reference RR scoring 100.0 under all gates, confirming full solvability, and we will include an unexposed 15-task human baseline if completed for the final version.

  • Related work. Thank you for both references; we will discuss them in Related Work, citing Cornacchia et al. (APSys'25) on practical gaps in LLM microservice failure analysis and contrasting concurrent work DDBench (Yan et al., NeurIPS'26) (debugging functional/operational distributed faults) with MicroPerf-Bench (end-to-end latency optimization under functional and fairness gates).

Thanks to the authors for addressing the points I raised; as a result the paper will be stronger. I maintain my positive support for this paper and think it should be accepted.

We sincerely thank the reviewer for the constructive feedback and continued support, and we are pleased that our responses have addressed the points raised.

Reviewer_UvgkOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

MicroPerf-Bench evaluates performance optimization in deployed microservice systems. Structural anti-patterns are injected into seven open-source systems to create 65 tasks requiring agents to modify, rebuild, and redeploy applications. Patches are scored by end-to-end latency recovery after passing functional and fairness gates. Three agent frameworks, three models, and three repeats produce 1,755 runs; the best configuration has a mean normalized score of 42.0. The paper interprets frequent failures to achieve a performance gain as difficulty locating bottlenecks and reports 47.7 from one profiling-augmented run.

Critical strengths
  1. The task and evaluation infrastructure have practical value. Multi-container deployment, functional validation, and end-to-end workloads form an evaluation closer to systems engineering than function optimization. Finding only 15 reproducible fixes among more than ten thousand commits also motivates controlled injection.
  2. The evaluation scale and reporting transparency are strengths. The 1,755 runs, repeated trials, failure labels, variability analysis, and threshold sensitivity support reuse. Case studies explicitly distinguish measurement noise from actually surpassing the reference, improving interpretability.
Critical weaknesses
  1. Evidence for the main intervention and configuration rankings is weaker than the wording suggests. Profiling augmentation is run once and also enforces two measurement cycles, so the change from 39.4 to 47.7 mixes runtime information with continued iteration. Table 9's Best@3 is 51.7, but it selects the best result per instance from three attempts, with a different budget and statistic. It therefore cannot directly disprove the effect of one profiling run; it highlights the need to control sampling budgets and variability. Top configurations differ by only a few points, while reported run standard deviations range from 1.0 to 5.9. Without appropriate paired intervals, definitive rankings are unwarranted.
  2. The measurements and construct underlying the scores are insufficiently verifiable. Functional-test provenance and coverage and the fairness-violation procedure are inadequately documented. Frozen reference/degraded latencies versus live optimized measurements may introduce environmental drift, and per-task noise, workload intensity, and timeout details are missing. Recovering the pre-injection architecture while forbidding parameter tuning cannot represent all system optimization. Coverage is also uneven: ESB has no instances and Mega service has only one among the nine advertised categories. Infrastructure failures are not sufficiently separated from cross-service diagnosis failures. Protocol details and existing-data analyses are needed to support the capability-bottleneck and generality claims; these gaps currently lead me to lean toward rejection.
Questions for authors
  1. Intervention protocol: What call, iteration, and measurement budgets apply to profiling and ordinary conditions? Are repeated results or records allowing ordinary agents the same iteration budget already available?
  2. Acceptance gates: Who supplied the functional tests, how many exist per task, what do they cover, and were they validated on reference and degraded versions? Are fairness violations identified by rules, models, or humans, and are they independently checked?
  3. Measurement: Please state workload rate, duration, utilization, timeout handling, and the timing relationship between frozen baselines and live measurements. Can existing repeats estimate score noise?
  4. Existing statistics: Please report uncertainty for paired configuration differences by task and the rate of actual performance regressions. Separate build, deployment, and violation failures from failures to improve performance.
  5. Coverage and reproducibility: Please identify anti-patterns with actual instances, explain the difference between cleaned references and original upstrea
4 comments
AuthorsOpenReview ↗

Response to Reviewer Uvgk — Part 1/2: Q1–Q3

We thank the reviewer for the rigorous methodological feedback. Below we provide the exact protocol specifications, matched-budget and unconstrained ablations, fixed-patch noise measurements, and paired bootstrap statistics, all of which will be incorporated into the revised manuscript.

Q1. Intervention Protocol, Budgets, and Prompt-Only Ablation

  • Protocol comparison. Both baseline and profiling runs share the 7,200s wall-clock cap per task and 20-min command timeout, with no cap on API calls or edits. The baseline prompt specifies one static pass without measurement instructions, whereas our profiling run (Claude Code + Qwen) adds workspace instructions (CLAUDE.md, requiring pre/post-edit measurements and rollback/revision on regression) plus a read-only script (measure.py, unifying workload, trace, and docker stats outputs under MAX_MEASUREMENTS = 2, with functional-repair exemptions disclosed). We will revise the main text to present this explicitly as a combined measurement-and-tooling intervention.

  • Single-pass vs. Best@3 (51.7). Table 9's Best@3 (51.7) selects the best score per task post hoc across 3 runs using ground-truth evaluation (an oracle selector unavailable at deployment). Individual baseline rounds score 40.1, 40.2, and 37.9 (mean 39.4, SD 1.33). Against the 65-task baseline mean, single-pass profiling (47.7) achieves a paired gain of +8.3 (95% task-bootstrap CI [-1.6, +18.3]), beating all three individual rounds (+7.6, +7.5, +9.9).

  • Prompt-only ablation (removing measure.py). Running Claude Code + Qwen on all 65 tasks with the same CLAUDE.md protocol but without measure.py:

    • Unconstrained end-of-run: Agents averaged 40.4 overall. The relatively low score reflects many fairness-violating optimizations, possibly because more iterations led agents to attempt riskier optimizations. Approximately 9% of tasks complied with the ≤2-measurement budget; agents averaged 8.0 measurements and reached up to 10 measurement-revision iterations.
    • Matched ≤2-measurement budget: Restoring over-budget runs from revision logs to their ≤2-measurement code state and re-evaluating yields 44.0 (+4.6 over baseline 39.4, 95% CI [-6.4, +15.8]).
    • Takeaway: At matched ≤2 measurements, instructions add +4.6 points (39.4 to 44.0) and measure.py adds +3.7 points (44.0 to 47.7). The unconstrained prompt-only run scores 40.4, using substantially more measurement and revision than the matched conditions.

Q2. Acceptance Gates: Provenance, Scale, and Accuracy

Both gates execute as deterministic scripts without humans or LLMs in the loop.

  • Correctness. Suites adapt upstream integration workloads and cover all injected paths, totaling 3,594 HTTP requests (3 to 245 per task, median 24) and 8,178 assertions (6 to 3,920 per task, median 72, of which 91.1% verify database persistent state, and every one of the 65 tasks includes at least one database-backed assertion). Dynamic fields (timestamps, UUIDs/IDs, auto-keys, unordered sets) use per-endpoint masks derived from repeated runs on RR. Across 3 redeployments of all 65 tasks, RR and DD pass 100% (0% false positives).

  • Fairness (quick_scan.py). Ten static diff rules check OO vs. DD (avoiding false positives from comparing OO to RR on valid refactorings). Across 1,755 runs, quick_scan.py flagged 154 patches; manual review confirmed 146 violations (precision = 94.8%). An LLM-assisted audit of all 1,601 unflagged patches (with manual review of borderline candidates) found 4 missed violations (FNR = 2.7%, recall = 97.3%). Table 7 reports the 146 + 4 = 150 verified violations; standalone quick_scan.py differs on only 12/1,755 runs (0.68%).

Q3. Measurement Protocol and Score Noise

  • Protocol. Load is set to 90% of RR's saturation rate on fixed hardware, with a 90s warm-up and three 60s measurement iterations (timeouts count toward error rate, excluded from mean latency; absence of continuous evaluator CPU/memory logging will be noted). LRL_R and LDL_D were measured 3 times, averaged, and frozen across all 27 configuration-rounds; each OO is measured live across 3 iterations on the same dedicated host with fixed container limits and no co-located load.

  • Pure measurement noise vs. agent variance. Across the 3 consecutive 60s iterations on fixed code, median latency CV is 1.18% (IQR 0.56% to 2.76%) for LRL_R, 0.90% (0.42% to 2.29%) for LDL_D, and 1.23% (0.47% to 3.15%) for valid LOL_O. Propagating SE(LˉO)=SD/3\mathrm{SE}(\bar{L}_O) = \mathrm{SD}/\sqrt{3} onto [0, 100] gives a median pure measurement SE of 0.36 points (IQR 0.00 to 1.56; 0.87 on positive-score runs), far below the cross-round within-cell agent SD (median 4.12, round-mean SD 0.97 to 5.93; 39.5% of 585 cells mix 0 and >0, and 15.4% contain both 0 and 100).

AuthorsOpenReview ↗

Response to Reviewer Uvgk — Part 2/2: Q4–Q5

Q4. Paired CIs, Regressions, and Failure Breakdown

  • Paired CIs (20,000 task-cluster bootstraps). OpenHands + GLM (42.0) leads the next four configurations by +1.8 [-4.2, +8.2], +2.6 [-4.6, +9.7], +2.8 [-3.7, +9.6], and +3.7 [-1.5, +8.9]; as adjacent and scaffold CIs include zero, we temper fine-grained ranking claims. GLM and Qwen beat DeepSeek by +12.7 [+5.8, +19.5] and +9.3 [+3.7, +15.2] (GLM - Qwen: +3.4 [-0.6, +7.4]).

  • Regressions and zero-score separation. Among 1,251 runs with numeric latency pairs, 773 (61.8%) reduce latency, 68 (5.4%) leave it exactly unchanged, and 410 (32.8%) regress it (388 by >5%, 372 by >10%). Table 7 separately reports 478 zero-score trials with no performance gain:

\textbf{Zero-Score Mode} & \textbf{Count} & \textbf{\\% Zero-Score} & \textbf{\\% All (1{,}755)} \\\\ \text{No performance gain} & 478 & 48.7\\% & 27.2\\% \\\\ \text{Correctness / Fairness violation} & 201\,/\,150 & 20.5\\%\,/\,15.3\\% & 11.5\\%\,/\,8.5\\% \\\\ \text{Build / Deploy failure / Timeout} & 111\,/\,31\,/\,11 & 11.3\\%\,/\,3.2\\%\,/\,1.1\\% & 6.3\\%\,/\,1.8\\%\,/\,0.6\\% \end{array}$$ **Q5. Pattern Coverage and Cleaned Reference ($U \to R \to D \to O$)** - **Coverage.** Eight of nine patterns are instantiated (97 occurrences across 65 tasks): Chatty (24), Nano-service (21), Hub-like (17), Service chain (11), On-line only (10), Shared persistency (10), Empty messages (3), and Mega service (1). ESB misuse (0) lacks ESB middleware in modern suites. Mega service is currently difficult to instantiate with reproducible degradation under mixed load, so we implemented only one instantiated task; we will scope claims strictly to instantiated patterns. - **Lineage ($U \to R \to D \to O$).** Three senior engineers audited upstream $U$ (92.3% agreement, Fleiss' $\kappa$ = 0.87) and fixed 15 native defects while preserving equivalence to form $R$ (removing client think time across $R, D, O$), so native bottlenecks do not mask injected degradations and 100 means recovering $R$. Commit hashes, U--R/R--D diffs, and scripts will be released.

Thank you for the detailed response. My concerns have been fully addressed, and I have raised my score accordingly.

We sincerely thank the reviewer for the constructive feedback and careful consideration of our responses, and we are grateful that all concerns have been fully addressed.

Reviewer_isBnOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2(+1)
Read review
Paper summary

This paper presents a benchmark suite to evaluate the coding agents capability on end-to-end performance optimization in microservice systems. The benchmark suite constructed by injecting different types of performance issues and anti-patterns while preserving the functionality. Each instance provides an executable multi-service environment, functional tests, and mixed-workload performance evaluation. An agent is required to generate changes to the services, which are then rebuilt, deployed, and checked for correctness and compliance with optimization constraints. The authors validate the injected instances through performance measurements and expert review, then evaluate different agent–model configurations including three models and three harnesses. The highest recovery score is 42.0 across different configurations, indicating the potential of model and harness improvements.

Critical strengths
  • The paper presents a useful benchmark for end-to-end performance optimizations for distributed microservice systems. This appears to be an under explored setting that differs from repository-level code optimization and operational incident recovery that some prior papers have been focusing on.
Critical weaknesses
  • Although executable workloads are provided, the default diagnostic tools and agent-visible telemetry are insufficiently documented. This limits reproducibility and complicates attributing failures to optimization reasoning rather than difficulty obtaining runtime evidence.

  • The industrially grounded anti-patterns provide reasonable controlled test cases, but the expert validation does not establish comparable diagnostic and optimization difficulty to daily problems that SWEs and SREs faces everyday.

Questions for authors
  • Which observability tools, logs, traces, and service-level metrics are available to the agents out of the box for each system? Do they require additional setup or instrumentation for human engineer to do performance optimization and debugging?

  • How are correctness and fairness violations detected and adjudicated? Are those rule based or do they require manual judgement or LLM-as-a-judge? Are there any evidence that those detections are accurate enough?

  • Beyond the expert provenance-discrimination study, what existing evidence supports comparable diagnostic and repair complexity between injected and naturally occurring performance problems? Is it possible to port some real performance bugs from other repos to the microservice systems included in the paper?

  • How much of the issues requires comprehensive monitoring and reasoning about interactions across services rather than local and static code analysis?

4 comments
AuthorsOpenReview ↗

Response to Reviewer isBn — Part 1/2: Q1–Q2

We thank the reviewer for the constructive feedback. Below we address each question with concrete protocol details, audit statistics, and empirical analyses, all of which will be added to the revised manuscript.

Q1. Out-of-the-Box Observability and Agent Access

  • Native system telemetry (no code instrumentation required). Each benchmark instance retains its upstream observability stack alongside a load script (loadtest.py or Locust) reporting end-to-end and per-endpoint latency distributions:
\textbf{System} & \textbf{Tracing / Metrics} & \textbf{Logs \\& Health} & \textbf{Latency Telemetry} & \textbf{CLI / API Access} \\\\ \text{HotelRes. / MediaServ. / SocialNet.} & \text{Jaeger (distributed tracing)} & \text{stdout/stderr} & \text{loadtest.py (mean, P50–P99)} & \text{Jaeger HTTP API} \\\\ \text{PetClinic} & \text{Zipkin + Grafana / Micrometer} & \text{Spring logs} & \text{loadtest.py (mean, P50–P99)} & \text{Zipkin/Actuator API} \\\\ \text{RobotShop} & \text{OpenTracing / Instana hooks} & \text{Container logs} & \text{Locust / loadtest.py} & \text{trace endpoints} \\\\ \text{Pitstop} & \text{Seq + RabbitMQ UI} & \text{Health endpoints + Seq} & \text{loadtest.py (min/max/mean, P50–P99)} & \text{Seq/RabbitMQ HTTP API} \\\\ \text{MartianBank} & \text{None built-in} & \text{stderr, Morgan logs} & \text{Locust (avg, P50–P99)} & \text{Locust CLI} \end{array}$$ *(All 7 systems also support docker logs and the load CLI in the sandbox.)* - **Human vs. agent access.** Once a workload runs, 5/7 systems expose distributed traces and all 7 provide logs and endpoint latency breakdowns out of the box. Neither humans nor agents need code instrumentation to localize service-level bottlenecks (manual spans are only needed for fine-grained intra-function profiling in Pitstop/MartianBank). In our **baseline setting**, although logs, tracing APIs, and load scripts were accessible via shell, the prompt tested whether agents would autonomously initiate runtime profiling---and most defaulted to static code reading. In our **profiling-augmented run** (39.4 $\rightarrow$ 47.7), packaging this existing telemetry into direct CLI feedback separates failure to invoke runtime tools from failure to reason over traces. **Q2. Correctness and Fairness Gates: Mechanism and Accuracy** At evaluation time, both gates run as deterministic evaluator-side scripts without humans or LLMs in the loop. - **Correctness Gate:** We replay deterministic request suites on the clean reference $R$ and record golden HTTP responses and post-request database states; the optimized system $O$ must match both. Dynamic fields (timestamps, random UUIDs/IDs, auto-incremented keys, unordered sets) are normalized via per-endpoint canonicalization masks derived from repeated runs on $R$. Across 3 redeployments of all 65 instances, both $R$ and degraded $D$ achieve a **100% pass rate (0% false positives)**. - **Fairness Gate (quick\_scan.py):** To catch shortcut edits (e.g., hardcoding outputs, bypassing auth/checks, deleting telemetry, or altering test scripts), quick\_scan.py checks 10 static diff rules anchored on the agent's patch ($O$ vs. $D$, avoiding false positives from comparing $O$ directly to $R$ when legitimate refactorings occur). - **Accuracy across 1,755 runs:** Across all 1,755 trials, quick\_scan.py flagged **154** patches. Manual verification by the authors confirmed **146** genuine violations (**precision = 94.8%**). Conversely, an LLM-assisted semantic audit over all **1,601** unflagged patches---followed by manual inspection of all flagged borderline candidates by the authors---found **4** missed violations (**FNR = 2.7%** relative to this audit). Full rules and edge cases will be added to the appendix.
AuthorsOpenReview ↗

Response to Reviewer isBn — Part 2/2: Q3–Q4

Q3. Difficulty Relative to Natural Bugs & Cross-Repo Porting

  • Structural overlap with real fixes. Comparing the 15 reproducible natural performance fixes (mined from >10,000 real microservice commits in our pilot study) against the ground-truth fixes (RR vs. DD) of our 65 injected instances shows substantial overlap:
\textbf{Metric} & \textbf{15 Natural Fixes (Min / Med / Max)} & \textbf{65 Injected Instances (Min / Med / Max)} & \textbf{Injected within Natural Range} \\\\ \textbf{Files changed} & \text{2 / 8 / 27} & \text{1 / 9 / 121} & \text{50 / 65 (77\\%)} \\\\ \textbf{Services touched} & \text{1 / 3 / 8} & \text{1 / 4 / 12} & \text{59 / 65 (91\\%)} \\\\ \textbf{LOC changed} & \text{23 / 349 / 836} & \text{33 / 673 / 4,232} & \text{42 / 65 (65\\%)} \\\\ \textbf{Within all 3 ranges simultaneously} & \text{---} & \text{---} & \textbf{39 / 65 (60\\%)} \end{array}$$ Thus, the injected tasks are not trivial single-file edits and share substantial structural overlap with real microservice fixes. - **Source-blinded difficulty evaluation.** Three independent engineers scored all 15 natural fixes and a stratified sample of 15 injected instances on a 5-point scale across five dimensions (*localization, cross-service reasoning, observability needs, repair design, validation*), blinded to provenance. Inter-rater reliability was **0.81** (ordinal Krippendorff's $\alpha$), and ratings overlapped substantially across dimensions (natural vs. injected medians: localization, 3.67 vs. 3.00; cross-service reasoning, 3.00 vs. 4.00; observability needs, 3.33 vs. 3.33; repair design, 4.00 vs. 3.67; validation, 2.67 vs. 3.67; median absolute difference **0.67**). - **Porting real bugs from other repos.** - *From other microservice repos:* Cross-service bottlenecks (e.g., N+1 RPC cascades, synchronous call-chain blocking, shared-store contention) are tightly coupled to the donor system's topology, RPC schemas, and workloads. One cannot copy-paste such bugs across architectures; **porting them requires abstracting the root cause into an architectural anti-pattern and re-instantiating it along the target system's call graph---which is exactly our controlled injection methodology (§3.2)**. - *From monolithic repos:* Transplanting isolated function/algorithmic bugs is straightforward but yields purely intra-service issues solvable by local static analysis (already covered by SWE-Perf), rather than testing distributed system reasoning. **Q4. Cross-Service Reasoning vs. Local Static Analysis** Classifying all 65 instances by the scope required to localize and verify a fix yields: 1. **Cross-service interaction required --- 45/65 (69.2%):** Root cause or fix spans service boundaries (e.g., N+1 RPC amplification, cross-service blocking, redundant payload serialization); single-service code inspection cannot establish the bottleneck. 2. **Local defect requiring cross-service confirmation --- 13/65 (20.0%):** Code edit is intra-service, but identifying which service/endpoint dominates end-to-end latency requires runtime traces or service-level perspective. 3. **Locally & statically analyzable --- 7/65 (10.8%):** Self-contained intra-service inefficiency identifiable via static code reading. Thus, **58/65 instances (89.2%)** require cross-service reasoning or system-level telemetry. Empirically across our 1,755 baseline runs, agents score **59.0** on the 7 local instances vs. **28.8** on the 13 confirmation instances and **32.9** on the 45 cross-service instances; in the profiling-augmented configuration (39.4 $\rightarrow$ 47.7 overall), profiling yields its largest gain (**+13.0**) on these 45 cross-service instances.

I thank the authors for the replies. The authors have addressed my primary concerns and I am willing to increase my score. Nevertheless, I recommend the authors to add more details in correctness and fairness gates to the main context of the manuscript.

Thank you very much for your follow-up and willingness to raise your score. We will move the key details of both the correctness and fairness gates, including their mechanisms and audit results, into the main text of the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.