acceptodds
Under review as a conference paper at ICLR 2027

The Evaluation Substrate Matters: Benchmarking Multi-Agent Serving Substrates

Abstract

Multi-agent systems (MAS) usually coordinate heterogeneous models and tools to solve complex tasks, creating a growing need for reliable evaluation. However, a multi-agent workflow that is fastest in one serving environment can lose its advantage in another, even when its models, prompts, and task remain fixed. This makes a benchmark result conditional on how the workflow is executed. We introduce MASSys, a benchmark for measuring this dependence in multi-agent large language model (LLM) systems. MASSys executes the same candidate–task pairs across serving substrates and load regimes, using five workload families spanning software repair, research synthesis, financial analysis, interactive control, and diagram generation. It records task quality alongside delivery, latency, cost, resource pressure, and failure evidence. In a controlled motivation diagnostic, the Star workflow falls from first to fourth in mean-latency rank under KV-cache pressure, while Chain rises from fourth to first. The substrate sweeps further show that increasing load creates long runtime tails and timeout pressure. In several task families, quality among evaluated outputs changes less than effective quality over submitted jobs: the system delivers fewer usable results even when the surviving outputs remain competitive. Tool and runtime traces reveal that these outcomes involve both model serving and non-model execution. Together, the results motivate evaluating MAS designs over explicit operating conditions and reporting which tasks are delivered, how well they are solved, and where execution breaks down.

Primary area:
datasets and benchmarks
Keywords:
LLM-based Multi-agent System, Serving Substrate, Performance Evaluation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_xXpxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces MASSys, a substrate-aware benchmark for multi-agent LLM systems. Rather than treating the serving environment as a fixed implementation detail, MASSys evaluates identical candidate-task pairs across multiple serving backends, load regimes, and resource conditions. It records both task-level outcomes and service-level metrics such as completion, latency, queueing, KV-cache pressure, timeout behavior, and failure traces. Across heterogeneous workloads, the experiments show that serving conditions can change workflow latency rankings, generate severe tail-latency and timeout effects under load, and alter effective task quality by preventing otherwise competitive jobs from completing.

Critical strengths
  1. The paper identifies an important and under-measured confound in MAS evaluation. Treating the serving substrate as an experimental variable rather than hidden infrastructure is a useful conceptual extension of existing system-level agent evaluation.

  2. The paired design and dual task/service traces are well motivated. Replaying the same candidate-task pairs across substrate conditions is substantially more informative than comparing aggregate scores from unrelated runs, and distinguishing evaluated-output quality from delivery/effective quality is particularly valuable for practical agent systems.

Critical weaknesses

The key substrate cells are represented by only one run. This is a major concern because several central claims concern workflow rank reversals, P95 latency, queueing, timeout rates, and reliability envelopes, all of which can exhibit non-negligible systems variance. Without replicated runs or uncertainty estimates, it is difficult to determine how stable the reported substrate differences and ranking inversions are.

Questions for authors
  1. Can the authors replicate the key substrate/rank-inversion experiments with multiple independent runs and report uncertainty intervals for latency, completion, and effective quality?

  2. Do the reported rank reversals persist when replaying a fixed deterministic workload trace across repeated substrate runs, rather than resampling arrivals?

  3. How sensitive are the main conclusions to restricting analysis to workload families with execution-grounded quality metrics?

6 comments
AuthorsOpenReview ↗

Rebuttal for Q1 - part 1

We thank Reviewer xXpx for the constructive feedback and thoughtful questions, which have helped us strengthen the statistical support for our findings, further investigate the robustness of rank reversals, and clarify the implications of our evaluation methodology.

Q1: Repeated Substrate Experiments and Statistical Uncertainty

We thank the reviewer for highlighting the importance of experimental reproducibility and statistical uncertainty. To address this concern, we conducted additional repetitions for the main substrate sweeps and computed uncertainty estimates for the original repeated workflow-ranking experiment in Fig. 1(b). We report confidence intervals for completion, latency, and effective quality below.

Main benchmark repetitions. We have completed two additional workload-seed sweeps (seeds 43 and 44) across all five serving cells and five offered loads (0.005, 0.01, 0.02, 0.05, and 0.10 JPS) on 2×H200, totaling 50 additional runs. Together with the original seed 42, these experiments provide three workload realizations per serving cell and load. Each seed independently resamples the task mixture and Poisson arrivals, while candidate–task pairs and arrival schedules are held fixed across serving cells within each seed–load block.

We report untruncated 95% Student-t confidence intervals across workload seeds. Some intervals extend beyond the natural metric bounds due to the limited number of repetitions and sparse per-family samples at low loads. P95 is computed separately for each run before aggregation, and effective quality is reported separately for each task family.

Table 1. Completion rate (%). Mean [95% CI] across three seeds.

Serving cell0.005 JPS0.01 JPS0.02 JPS0.05 JPS0.10 JPS
vLLM94.59 [82.70, 106.48]90.22 [57.04, 123.40]90.70 [82.46, 98.93]83.23 [71.82, 94.64]51.25 [42.70, 59.81]
SGLang94.59 [82.70, 106.48]91.88 [65.75, 118.01]92.32 [87.08, 97.57]83.73 [77.84, 89.62]47.52 [36.11, 58.93]
Continuum94.59 [82.70, 106.48]91.88 [65.75, 118.01]90.66 [78.39, 102.93]83.77 [73.64, 93.90]43.39 [30.20, 56.59]
ORLA+vLLM94.59 [82.70, 106.48]93.33 [64.65, 122.02]91.49 [82.76, 100.22]88.24 [78.71, 97.76]51.91 [48.36, 55.47]
ORLA+SGLang94.59 [82.70, 106.48]93.33 [64.65, 122.02]91.67 [87.97, 95.37]85.75 [69.53, 101.97]52.27 [41.93, 62.61]

Table 2. P95 end-to-end runtime (s). Mean [95% CI] across three run-level P95 values.

Serving cell0.005 JPS0.01 JPS0.02 JPS0.05 JPS0.10 JPS
vLLM273.6 [172.1, 375.1]318.7 [160.0, 477.4]383.8 [252.5, 515.1]645.2 [473.1, 817.3]1139.6 [1090.2, 1189.0]
SGLang253.8 [153.6, 353.9]337.2 [93.5, 580.9]384.9 [127.7, 642.1]675.6 [237.9, 1113.3]1084.1 [1011.5, 1156.7]
Continuum268.2 [178.3, 358.2]325.8 [160.5, 491.0]360.8 [264.3, 457.3]656.4 [480.1, 832.7]1119.0 [1031.5, 1206.4]
ORLA+vLLM265.2 [126.6, 403.8]323.1 [186.8, 459.4]371.5 [292.4, 450.5]596.1 [459.0, 733.2]1127.9 [1052.8, 1203.0]
ORLA+SGLang269.6 [135.4, 403.8]342.3 [124.0, 560.6]413.8 [189.9, 637.6]726.1 [195.1, 1257.1]1039.1 [913.5, 1164.8]
AuthorsOpenReview ↗

Rebuttal for Q1 - part 2

Table 3. Per-family effective quality. Mean [95% CI] across three seeds, using the submitted-job denominator.

JPSServing cellSWE-BenchResearchBenchPinchBenchPaperBanana
0.005vLLM0.400 [-0.915, 1.715]0.764 [0.410, 1.119]0.592 [-0.712, 1.895]0.588 [0.452, 0.723]
0.005SGLang0.467 [-0.784, 1.717]0.767 [0.459, 1.075]0.667 [-0.768, 2.101]0.696 [0.435, 0.956]
0.005Continuum0.400 [-0.915, 1.715]0.764 [0.372, 1.157]0.600 [-0.715, 1.915]0.538 [0.171, 0.904]
0.005ORLA+vLLM0.333 [-1.101, 1.768]0.773 [0.730, 0.817]0.533 [-0.717, 1.784]0.483 [0.017, 0.950]
0.005ORLA+SGLang0.233 [-0.392, 0.859]0.775 [0.615, 0.935]0.600 [-0.715, 1.915]0.821 [0.539, 1.103]
0.01vLLM0.472 [-0.049, 0.993]0.788 [0.740, 0.837]0.600 [-0.715, 1.915]0.560 [0.433, 0.688]
0.01SGLang0.472 [-0.049, 0.993]0.782 [0.743, 0.821]0.595 [-0.706, 1.897]0.733 [0.527, 0.938]
0.01Continuum0.472 [-0.049, 0.993]0.791 [0.747, 0.835]0.595 [-0.705, 1.895]0.531 [0.435, 0.628]
0.01ORLA+vLLM0.264 [-0.616, 1.144]0.794 [0.750, 0.837]0.529 [-0.706, 1.763]0.549 [0.276, 0.823]
0.01ORLA+SGLang0.347 [-0.357, 1.052]0.791 [0.747, 0.834]0.595 [-0.706, 1.897]0.702 [0.487, 0.918]
0.02vLLM0.485 [0.420, 0.550]0.780 [0.764, 0.796]0.589 [-0.689, 1.866]0.487 [0.355, 0.619]
0.02SGLang0.388 [0.235, 0.541]0.779 [0.767, 0.791]0.626 [-0.723, 1.974]0.743 [0.546, 0.939]
0.02Continuum0.443 [0.286, 0.600]0.782 [0.772, 0.792]0.593 [-0.694, 1.879]0.545 [0.451, 0.640]
0.02ORLA+vLLM0.183 [-0.143, 0.509]0.767 [0.762, 0.772]0.533 [-0.688, 1.755]0.527 [0.412, 0.642]
0.02ORLA+SGLang0.213 [-0.054, 0.481]0.776 [0.751, 0.801]0.556 [-0.670, 1.782]0.726 [0.670, 0.782]
0.05vLLM0.367 [0.041, 0.693]0.788 [0.781, 0.795]0.812 [0.591, 1.034]0.521 [0.456, 0.585]
0.05SGLang0.436 [0.128, 0.744]0.786 [0.778, 0.794]0.830 [0.554, 1.105]0.690 [0.636, 0.743]
0.05Continuum0.434 [0.185, 0.682]0.787 [0.766, 0.808]0.833 [0.575, 1.090]0.505 [0.453, 0.558]
0.05ORLA+vLLM0.289 [-0.284, 0.863]0.785 [0.767, 0.803]0.779 [0.488, 1.069]0.497 [0.375, 0.618]
0.05ORLA+SGLang0.224 [-0.304, 0.751]0.790 [0.784, 0.795]0.777 [0.469, 1.086]0.710 [0.553, 0.868]
0.10vLLM0.122 [-0.068, 0.312]0.786 [0.761, 0.810]0.580 [0.365, 0.795]0.093 [0.015, 0.171]
0.10SGLang0.173 [0.060, 0.287]0.785 [0.764, 0.806]0.625 [0.570, 0.681]0.072 [-0.192, 0.336]
0.10Continuum0.065 [-0.173, 0.303]0.329 [0.020, 0.637]0.308 [0.148, 0.469]0.006 [-0.019, 0.031]
0.10ORLA+vLLM0.181 [0.036, 0.326]0.748 [0.659, 0.838]0.567 [0.513, 0.621]0.111 [-0.068, 0.289]
0.10ORLA+SGLang0.100 [-0.059, 0.259]0.602 [0.163, 1.040]0.508 [0.221, 0.794]0.034 [-0.079, 0.146]

At 0.005 JPS, ResearchBench uses n=2 (df=1), because seed 43 sampled no ResearchBench jobs.

The additional experiments support our main load-sensitivity findings. Across the complete five-load sweep, mean completion generally declines, while mean run-level P95 runtime increases monotonically across all five serving cells as offered load rises. Across all 15 matched backend–seed comparisons from 0.05 to 0.10 JPS, completion decreases and run-level P95 runtime increases. The mean completion reduction ranges from 31.98 to 40.38 percentage points across the five serving cells, with all five paired 95% t confidence intervals excluding zero. P95 runtime increases in all five cells, with three of five paired confidence intervals excluding zero. For effective quality, PaperBanana and PinchBench decrease in all 15 comparisons, SWE-bench decreases in 13 of 15, and ResearchBench exhibits backend-dependent behavior. These results provide additional evidence for load-dependent degradation while highlighting differences in uncertainty across metrics and task families.

Workflow-ranking repetitions (Fig. 1b). In the original experiment, each of the 25 workflow–condition combinations was executed 10 times, yielding 250 workflow executions under fixed workflow definitions and synthetic context. The rankings in Fig. 1(b) were computed using the mean end-to-end workflow latency across these ten executions. We now additionally report the sample standard deviations and 95% confidence intervals to quantify the variability of these measurements.

AuthorsOpenReview ↗

Rebuttal for Q1 - part 3

Table 4. Workflow end-to-end latency (s). Mean [95% CI] across ten repeated executions.

WorkflowCleanKV-tightCPU-tightGPU pressureMixed-tight
Star5.012 [4.954, 5.069]12.490 [12.346, 12.634]5.174 [5.139, 5.210]5.590 [5.576, 5.605]18.109 [18.032, 18.186]
DAG5.479 [5.450, 5.509]11.559 [11.535, 11.584]5.659 [5.585, 5.732]5.562 [5.523, 5.601]19.295 [18.828, 19.762]
Tool-ReAct5.768 [5.688, 5.848]11.978 [11.903, 12.054]12.977 [12.747, 13.208]6.356 [6.292, 6.420]15.193 [15.092, 15.295]
Chain6.180 [6.175, 6.186]6.039 [6.034, 6.044]6.511 [6.462, 6.559]6.357 [6.342, 6.372]27.301 [27.265, 27.337]
Debate7.930 [7.922, 7.938]12.896 [12.889, 12.903]8.409 [8.329, 8.489]8.225 [8.201, 8.249]21.134 [21.074, 21.194]

The Star/Chain ranking reversal remains evident in the repeated measurements. Under Clean, Star achieves 5.0118 ± 0.0804 s versus 6.1803 ± 0.0081 s for Chain. Under KV-tight, Star increases to 12.4900 ± 0.2018 s, whereas Chain remains at 6.0392 ± 0.0068 s. Their mean-latency rankings reverse from first/fourth under Clean to fourth/first under KV-tight. Notably, their 95% latency confidence intervals are well separated under both conditions, providing further evidence for the robustness of this rank reversal. These repeated measurements provide additional support for the topology–substrate interaction illustrated in Fig. 1(b).

Overall, the additional multi-seed experiments and the uncertainty analysis of the original workflow-ranking experiment strengthen the statistical support for our central findings: serving conditions affect delivery reliability, tail latency, effective quality, and the relative latency of different workflow structures. We will incorporate the full multi-seed statistics and workflow-ranking uncertainty estimates into the revised manuscript and Appendix.

Reviewer_HMGhOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces MASSYS, a benchmark and evaluation framework for studying how serving substrates affect multi-agent LLM systems. The proposed protocol fixes candidate–task pairs while varying the backend/runtime, offered load, and deployment conditions. It combines five workload families—SWE-Bench, ResearchBench, DeepFund, PinchBench, and PaperBanana—with heterogeneous agent topologies, model assignments, and tools. Experiments compare five serving cells across two H200 deployment configurations using Poisson job arrivals. Its primary contribution is an evaluation protocol and integrated benchmark infrastructure for examining these dependencies.

Critical strengths
  1. The paired substrate-evaluation protocol addresses an important source of ambiguity in MAS benchmarking. Holding the candidate–task pair fixed while varying execution conditions is a sensible way to investigate how reported performance depends on the serving environment. This matters for multi-agent workflows because multiple dependent model calls, tool invocations, and aggregation stages can amplify queueing and resource constraints. The proposed evaluation unit makes these conditions explicit and could improve the comparability and deployment relevance of future MAS evaluations. The combination of five workload families and heterogeneous model/tool requirements gives the framework useful breadth beyond a single workload or isolated inference request.

  2. The benchmark explicitly separates task quality from result delivery and collects relevant execution evidence. Table 2 and Figure 7 distinguish usable-artifact generation, task correctness, and quality normalized over submitted jobs. This is consequential because evaluating only surviving outputs can conceal substantial losses in the usefulness of a deployed service. The tool profiles and runtime decomposition in Figures 8–9 further help connect aggregate outcomes to execution behavior. Although the accounting needs clarification, the intended joint measurement of quality, delivery, and traces is a substantive strength of the benchmark design.

Critical weaknesses
  1. Unclear outcome accounting. Figure 6 reports 50–70% execution success alongside 95–100% timeouts at 0.10 JPS on 2×H200. Please clarify whether these outcomes overlap.

  2. Inconsistent completion rates. Figures 1(c) and 6(a) show approximately 50% versus 86% completion for the apparently matching 4×H200/SGLang/0.10 JPS setting. This discrepancy needs reconciliation.

  3. Ambiguous quality losses. Figure 7 assigns zero to unevaluated jobs. Please separate service-delivery failures from failed or missing evaluations when interpreting the effective-quality gap.

  4. Insufficient ranking-experiment details. Figure 1(b) lacks clear task, configuration, pressure-setting, and trial-count specifications. The Star/Chain reversal’s connection to the main benchmark is also unclear.

  5. Limited statistical support. Each cell uses one run ((n=1)), with only 9–18 expected jobs at low load. Actual counts, independent repetitions, and uncertainty estimates are needed, particularly for P95 and per-family quality.

Questions for authors

See Critical Weaknesses above.

8 comments

I would appreciate it if the authors could provide a response to my comments, particularly the first four questions, as their clarification will be important for my final evaluation.

AuthorsOpenReview ↗

Rebuttal for Q1

We thank Reviewer HMGh for the careful reading and constructive feedback, and for recognizing MASSYS's paired evaluation protocol and joint analysis of task quality, result delivery, and execution traces. We have conducted 50 additional runs on 2×H200 to address the concern regarding statistical support and provide our detailed responses below.

Q1: Clarification of Timeout and Execution Success

Yes, these two outcomes overlap. In Fig. 6(h), the metric labeled "Timeout Rate" reports post-hoc runtime-threshold exceedance rather than enforced timeout termination, so a job can exceed the reporting threshold and still finish with a valid output.

To make the accounting clear, our pipeline tracks three distinct runtime and delivery concepts:

  • Runtime-threshold exceedance (Fig. 6h): whether a job's end-to-end runtime exceeds a task-family reference threshold (50 s for DeepFund, 130 s for ResearchBench and PinchBench, 350 s for SWE-bench, and 360 s for PaperBanana). These thresholds were empirically selected by inspecting the upper-tail runtime distributions of each task family in the low-load 2×H200 runs (0.005 and 0.01 JPS), comparing candidate cutoffs, examining how candidate cutoffs behaved across load levels. The same fixed thresholds were then applied uniformly across all backends, loads, and GPU configurations. They are post-hoc reporting cutoffs and never terminate a running job.
  • Execution success vs. deadline termination: each job has an enforced hard deadline of 1,200 s. Jobs that produce usable task artifacts before 1,200 s are counted as execution successes, regardless of whether they exceeded the softer reporting threshold above.
  • Window completion (Fig. 6a): whether a submitted job finishes before the 30-minute submission window closes. Jobs still running at the 30-minute mark are allowed to finish during the post-window drain period (still subject to the 1,200 s per-job deadline).

For the reviewer's example (2×H200, vLLM, 0.10 JPS), out of 170 submitted jobs, 162 (95.29%) exceeded the reporting threshold and 102 (60.00%) achieved execution success. Specifically, 94 jobs belonged to both groups (queueing under 0.10 JPS pushed their runtimes past the reference threshold, yet they finished before the 1,200 s deadline with valid artifacts), alongside 8 jobs that succeeded within the threshold and 68 jobs that failed after exceeding it (94 + 8 + 68 = 170). Separately, when the 30-minute submission window closed, 91 jobs had already finished (73 succeeded) and 79 remained active in the drain period, of which 29 subsequently succeeded (73 + 29 = 102). The same overlap occurs in all four other serving cells at 2×H200 / 0.10 JPS: 79, 114, 112, and 89 jobs both exceeded the reporting threshold and achieved execution success under Continuum, SGLang, ORLA+vLLM, and ORLA+SGLang, respectively.

We agree that calling the threshold-exceedance metric "Timeout Rate" was misleading. We will rename Fig. 6(h) to "Runtime Threshold Exceedance Rate", list the per-family thresholds alongside the 1,200 s hard deadline in Table 2 and Fig. 6, and clearly separate threshold exceedance, window completion, and deadline termination in Section 3.

AuthorsOpenReview ↗

Rebuttal for Q2

Q2: Inconsistency Between Figures 1(c) and 6(a)

Thank you for catching this discrepancy. Fig. 6(a) reports the authoritative benchmark numbers: for 4×H200 / SGLang / 0.10 JPS, the completion rate is 146/170 = 85.88% and the P95 runtime is 492.31 s. Tracing Fig. 1(c) back to its underlying experiment logs and plotting exports revealed two compounding issues in how Fig. 1(c) was generated, both of which we will fix so the paper is fully consistent:

  • Mislabeled y-axis metric (plotting Success Rate as Completion Rate): Even against its own source export, Fig. 1(c)'s plotting script mapped execution success rate (succeeded / submitted) onto a y-axis labeled "Completion Rate (%)", rather than using the 30-minute window completion rate (completed_within_window / submitted) reported in Fig. 6(a). For example, at 0.10 JPS in that export, Fig. 1(c) plotted the execution success rates (50.59% for SGLang, 90.00% for ORLA+vLLM, and 82.94% for Continuum) rather than window completion rates.
  • Outdated pre-fix source runs for SGLang and ORLA+SGLang: While the vLLM, ORLA+vLLM, and Continuum points in Fig. 1(c) came from the same 4×H200 runs as Fig. 6(a), Fig. 1(c) inadvertently retained an earlier May 2026 export for SGLang and ORLA+SGLang—prior to our September 2026 fix for a Qwen3-Coder-30B-A3B-Instruct serving configuration issue under SGLang (where a transformers upgrade moved rope_theta=10000000 into rope_scaling, causing SGLang to fall back to 10000 until explicitly overridden). In that earlier SGLang 0.10 JPS run, 84/170 jobs failed, giving an execution success rate of 86/170 = 50.59% (which appeared as the ~50% dot in Fig. 1c), even though 163/170 (95.88%) finished within the 30-minute window. After applying the rope_theta override in the finalized 45-run benchmark used in Fig. 6(a), 4×H200 / SGLang / 0.10 JPS achieves 146/170 = 85.88% window completion, 163/170 = 95.88% execution success, and 492.31 s P95 runtime.

We have recomputed all 15 points in Fig. 1(c) from the finalized 45-run records using the exact 30-minute window completion definition matching Fig. 6(a):

  • 0.01 JPS: 100.00% (23/23) completion across all five cells, with P95 runtimes of 203.51 s (SGLang), 231.88 s (ORLA+SGLang), 247.39 s (Continuum), 269.20 s (ORLA+vLLM), and 269.50 s (vLLM).
  • 0.05 JPS: 96.47% (82/85) / 282.18 s for SGLang, 96.47% (82/85) / 292.42 s for ORLA+SGLang, 91.76% (78/85) / 433.81 s for Continuum, 91.76% (78/85) / 437.21 s for vLLM, and 91.76% (78/85) / 446.58 s for ORLA+vLLM.
  • 0.10 JPS: 85.88% (146/170) / 492.31 s for SGLang, 79.41% (135/170) / 883.70 s for ORLA+SGLang, 77.06% (131/170) / 946.54 s for ORLA+vLLM, 75.88% (129/170) / 972.11 s for vLLM, and 71.76% (122/170) / 1,114.42 s for Continuum.

We will regenerate Fig. 1(c) from these finalized records so that its points match Fig. 6(a) identically, and update the caption and corresponding discussion in Section 1 and Section 4.2. Fig. 6(a) and all main-benchmark results in Section 4 already use these finalized records, and the regenerated Fig. 1(c) continues to demonstrate the motivating takeaway that completion drops and tail latency separates substantially across serving cells as offered load rises.

Reviewer_taDcOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper argues that multi-agent system (MAS) benchmark results depend on the serving substrate: backend/runtime, load, queueing, KV cache, resource budget, and timeout/retry policy. Existing benchmarks hold this layer fixed.

It introduces MASSYS, which:

  • samples jobs from five task families (SWE-Bench, ResearchBench, DeepFund, PinchBench, PaperBanana; 876 candidate tasks, Fig. 4);
  • has a GPT-5.4 plan agent build MAS candidates (Fig. 3);
  • runs them on five serving cells (vLLM-style, SGLang-style, Continuum-style, Orla+vLLM, Orla+SGLang) on 2×2\timesH200 and 4×4\timesH200 hardware (Table 3);
  • submits jobs with Poisson arrivals at 0.005–0.10 jobs/s over a 30-minute window.

It records task quality and service metrics (Table 2).

Reported findings:

  • In a motivation diagnostic (Fig. 1b), Star falls from 1st to 4th in mean-latency rank under KV pressure, while Chain rises from 4th to 1st.
  • Increasing load causes tail latency, timeouts and KV pressure (Result 1, Fig. 6).
  • Substrates trade off latency against completion (Result 2).
  • Effective quality (failures count as zero) drops more than evaluated-only quality (Fig. 7).
  • Tool mix and queue wait shape runtime (Results 3–4, Figs. 5, 8, 9).
Critical strengths
  1. A relevant question, framed with the right metric separation. Treating the serving substrate as an explicit evaluation variable is well motivated; the paper cites a ~6-point Terminal-Bench swing across resource configurations (Sec. 1). Separating completion, execution success, effective quality and evaluated-only quality (Table 2, Fig. 7) is the correct way to report MAS results under failures.
  2. Useful tracing infrastructure with released code. Per-job traces of queue wait, prefill/decode time, tool calls and latencies, and KV usage (Sec. 3.3, Figs. 8–9) come with an anonymized code link, software versions and evaluator configurations (Appendix A).
Critical weaknesses
  1. The statistical basis is too thin to support any comparative claim.
    • "Each substrate cell is represented by one run (n=1n = 1)" (Appendix A), with seed 42 (Sec. 4.1).
    • A 30-minute window at 0.005–0.10 jobs/s yields roughly 9–180 jobs per cell, split across five families with probability 0.2 each. Low-load per-family points in Fig. 7 therefore rest on a handful of jobs.
    • There are no CIs, repeated runs, or tests anywhere.
    • The 2×2\timesH200 and 4×4\timesH200 regimes also use different model sets (Table 3), which confounds hardware with model allocation.
  2. The central claim is supported only by an undocumented diagnostic. The abstract's key example (Star 1st →\to 4th, Chain 4th →\to 1st under KV pressure; Fig. 1b) is described only in one caption sentence ("controlled serving bottlenecks", Fig. 1): there is no definition of KV-tight/CPU-tight/GPU-pressure/Mixed-tight, no tasks, no models, and no run counts. Its workflows (Star, DAG, Tool-ReAct, Chain, Debate) also differ from the Sec. 3.1 candidate topologies (chain, star, fan-in/fan-out, FSM, composite). Meanwhile, the main experiments explicitly do not compare MAS candidates (Sec. 4: "not to produce a single leaderboard").
Questions for authors
  1. Please fully document the Fig. 1b diagnostic: tasks, models, concrete pressure settings, number of runs, and variance. Why do its workflows differ from the Sec. 3.1 topologies?
  2. Can you re-run each cell with ≥3\geq 3 seeds and report CIs for completion, P95 runtime, and effective quality?
  3. In the main sweeps, for a fixed set of MAS candidates, do quality or latency rankings change across serving cells and loads?
  4. Sec. 3.2 states that the same candidate–task pairs run across cells. Are the GPT-5.4 plans generated once and replayed verbatim (identical plans and arrival offsets) in every cell? Are "vLLM-/SGLang-/Continuum-style" the original engines or custom re-implementations (Appendix A mentions custom source trees)?
7 comments
AuthorsOpenReview ↗

Rebuttal for Q1

We thank Reviewer taDc for the constructive feedback and insightful questions, which have helped us strengthen the experimental evidence and clarify the evaluation methodology of MASSYS.

Q1: Experimental Details of Figure 1(b) and Connection to the Main Benchmark

Fig. 1(b) is a controlled single-GPU motivation experiment designed to show that even when workflow prompts and decoding are completely deterministic, changing serving resource constraints alone can reverse the latency ranking of fixed multi-agent topologies, motivating why MASSYS evaluates workflows across serving substrates at scale. Below are the full specifications and how this finding connects to the main benchmark:

  • Tasks and workflow structures: Rather than using PlanAgent, the diagnostic runs five fixed workflows over a single deterministic synthetic telemetry text (target length 2,048 tokens), constructed once and reused across all conditions:
    • Star: 14 parallel long-context worker calls followed by an aggregation call;
    • Chain: 8 sequential steps with reduced intermediate context after the first step;
    • DAG: two parallel dependency waves followed by a reducer;
    • Tool-ReAct: repeated function calls interleaved with local JSON processing;
    • Debate: parallel long-generation branches followed by synthesis.
  • Model and hardware: Qwen2.5-14B-Instruct served via vLLM on a single NVIDIA H200 GPU (TP=1, 8,192-token context limit, temperature=0, prefix caching disabled).
  • Serving conditions: Five pressure settings relative to Clean:
ConditionConfiguration
Clean32 GiB KV-cache allocation and max_num_seqs=16, with no injected CPU or background-request pressure.
KV-tight2 GiB KV-cache allocation and max_num_seqs=1, jointly constraining KV capacity and concurrent-sequence admission.
CPU-tightServer and orchestrator pinned to one CPU core, with three competing CPU workers on that core.
GPU pressureBackground long-generation requests every 20 ms (concurrency cap 8, output cap 1,536 tokens).
Mixed-tight2 GiB KV allocation, max_num_seqs=1, single-core affinity with one competing CPU worker, and background requests every 500 ms (concurrency cap 1, output cap 256 tokens).
  • Trial count and latency shift: Each workflow–condition pair was run 10 times (n=10) with greedy decoding (temperature=0), measuring end-to-end workflow latency across all model calls and local tool work (reported as mean ± sample standard deviation). The Star/Chain reversal reflects a large structural gap rather than run-to-run noise: in Clean, Star finishes in 5.0118 ± 0.0804 s vs. Chain in 6.1803 ± 0.0081 s because max_num_seqs=16 lets Star's branches run in a parallel batch; under KV-tight (max_num_seqs=1), Star's parallel long-context branches are serialized at admission and rise to 12.4900 ± 0.2018 s, whereas Chain (6.0392 ± 0.0068 s) avoids concurrent KV contention through sequential compact-context steps.

Connection to the main benchmark: Fig. 1(b) does not rank PlanAgent-generated benchmark candidates; it isolates topology–substrate interaction without task or tool confounders. In the main benchmark, PlanAgent candidates combine these same structural patterns (parallel fan-out vs. sequential refinement) with heterogeneous models and tools, and we observe the same underlying behavior in our substrate sweeps (Section 4.2, Fig. 6): workflows with heavier parallel fan-out and aggregation suffer sharper queueing inflation and larger completion/effective-quality drops when moving from 4×H200 to 2×H200 or from low load to 0.10 JPS. We will add all Fig. 1(b) setup details and clarify this connection in Section 1 and the Appendix.

AuthorsOpenReview ↗

Rebuttal for Q2 - part 1

Q2: Repeated Runs and Statistical Uncertainty

We conducted two additional workload seeds (43 and 44) across all five serving cells and all five offered loads under 2×H200, supplementing the original seed 42. This adds 50 runs and provides three independent workload realizations for each of the 25 serving-cell/load configurations. Candidate–task assignments and arrival offsets are identical across serving cells within each seed/load block. We report the mean and 95% Student-t confidence interval across run-level measurements (n=3, df=2), unless otherwise noted.

Table 1. Completion rate (%). Mean [95% CI] across three seeds.

Serving cell0.005 JPS0.01 JPS0.02 JPS0.05 JPS0.10 JPS
vLLM94.59 [82.70, 106.48]90.22 [57.04, 123.40]90.70 [82.46, 98.93]83.23 [71.82, 94.64]51.25 [42.70, 59.81]
SGLang94.59 [82.70, 106.48]91.88 [65.75, 118.01]92.32 [87.08, 97.57]83.73 [77.84, 89.62]47.52 [36.11, 58.93]
Continuum94.59 [82.70, 106.48]91.88 [65.75, 118.01]90.66 [78.39, 102.93]83.77 [73.64, 93.90]43.39 [30.20, 56.59]
ORLA+vLLM94.59 [82.70, 106.48]93.33 [64.65, 122.02]91.49 [82.76, 100.22]88.24 [78.71, 97.76]51.91 [48.36, 55.47]
ORLA+SGLang94.59 [82.70, 106.48]93.33 [64.65, 122.02]91.67 [87.97, 95.37]85.75 [69.53, 101.97]52.27 [41.93, 62.61]

Table 2. P95 end-to-end runtime (s). Mean [95% CI] across three run-level P95 values.

Serving cell0.005 JPS0.01 JPS0.02 JPS0.05 JPS0.10 JPS
vLLM273.6 [172.1, 375.1]318.7 [160.0, 477.4]383.8 [252.5, 515.1]645.2 [473.1, 817.3]1139.6 [1090.2, 1189.0]
SGLang253.8 [153.6, 353.9]337.2 [93.5, 580.9]384.9 [127.7, 642.1]675.6 [237.9, 1113.3]1084.1 [1011.5, 1156.7]
Continuum268.2 [178.3, 358.2]325.8 [160.5, 491.0]360.8 [264.3, 457.3]656.4 [480.1, 832.7]1119.0 [1031.5, 1206.4]
ORLA+vLLM265.2 [126.6, 403.8]323.1 [186.8, 459.4]371.5 [292.4, 450.5]596.1 [459.0, 733.2]1127.9 [1052.8, 1203.0]
ORLA+SGLang269.6 [135.4, 403.8]342.3 [124.0, 560.6]413.8 [189.9, 637.6]726.1 [195.1, 1257.1]1039.1 [913.5, 1164.8]
AuthorsOpenReview ↗

Rebuttal for Q2 - part 2

Table 3. Per-family effective quality. Mean [95% CI] across three seeds, using the submitted-job denominator.

JPSServing cellSWE-BenchResearchBenchPinchBenchPaperBanana
0.005vLLM0.400 [-0.915, 1.715]0.764 [0.410, 1.119]0.592 [-0.712, 1.895]0.588 [0.452, 0.723]
0.005SGLang0.467 [-0.784, 1.717]0.767 [0.459, 1.075]0.667 [-0.768, 2.101]0.696 [0.435, 0.956]
0.005Continuum0.400 [-0.915, 1.715]0.764 [0.372, 1.157]0.600 [-0.715, 1.915]0.538 [0.171, 0.904]
0.005ORLA+vLLM0.333 [-1.101, 1.768]0.773 [0.730, 0.817]0.533 [-0.717, 1.784]0.483 [0.017, 0.950]
0.005ORLA+SGLang0.233 [-0.392, 0.859]0.775 [0.615, 0.935]0.600 [-0.715, 1.915]0.821 [0.539, 1.103]
0.01vLLM0.472 [-0.049, 0.993]0.788 [0.740, 0.837]0.600 [-0.715, 1.915]0.560 [0.433, 0.688]
0.01SGLang0.472 [-0.049, 0.993]0.782 [0.743, 0.821]0.595 [-0.706, 1.897]0.733 [0.527, 0.938]
0.01Continuum0.472 [-0.049, 0.993]0.791 [0.747, 0.835]0.595 [-0.705, 1.895]0.531 [0.435, 0.628]
0.01ORLA+vLLM0.264 [-0.616, 1.144]0.794 [0.750, 0.837]0.529 [-0.706, 1.763]0.549 [0.276, 0.823]
0.01ORLA+SGLang0.347 [-0.357, 1.052]0.791 [0.747, 0.834]0.595 [-0.706, 1.897]0.702 [0.487, 0.918]
0.02vLLM0.485 [0.420, 0.550]0.780 [0.764, 0.796]0.589 [-0.689, 1.866]0.487 [0.355, 0.619]
0.02SGLang0.388 [0.235, 0.541]0.779 [0.767, 0.791]0.626 [-0.723, 1.974]0.743 [0.546, 0.939]
0.02Continuum0.443 [0.286, 0.600]0.782 [0.772, 0.792]0.593 [-0.694, 1.879]0.545 [0.451, 0.640]
0.02ORLA+vLLM0.183 [-0.143, 0.509]0.767 [0.762, 0.772]0.533 [-0.688, 1.755]0.527 [0.412, 0.642]
0.02ORLA+SGLang0.213 [-0.054, 0.481]0.776 [0.751, 0.801]0.556 [-0.670, 1.782]0.726 [0.670, 0.782]
0.05vLLM0.367 [0.041, 0.693]0.788 [0.781, 0.795]0.812 [0.591, 1.034]0.521 [0.456, 0.585]
0.05SGLang0.436 [0.128, 0.744]0.786 [0.778, 0.794]0.830 [0.554, 1.105]0.690 [0.636, 0.743]
0.05Continuum0.434 [0.185, 0.682]0.787 [0.766, 0.808]0.833 [0.575, 1.090]0.505 [0.453, 0.558]
0.05ORLA+vLLM0.289 [-0.284, 0.863]0.785 [0.767, 0.803]0.779 [0.488, 1.069]0.497 [0.375, 0.618]
0.05ORLA+SGLang0.224 [-0.304, 0.751]0.790 [0.784, 0.795]0.777 [0.469, 1.086]0.710 [0.553, 0.868]
0.10vLLM0.122 [-0.068, 0.312]0.786 [0.761, 0.810]0.580 [0.365, 0.795]0.093 [0.015, 0.171]
0.10SGLang0.173 [0.060, 0.287]0.785 [0.764, 0.806]0.625 [0.570, 0.681]0.072 [-0.192, 0.336]
0.10Continuum0.065 [-0.173, 0.303]0.329 [0.020, 0.637]0.308 [0.148, 0.469]0.006 [-0.019, 0.031]
0.10ORLA+vLLM0.181 [0.036, 0.326]0.748 [0.659, 0.838]0.567 [0.513, 0.621]0.111 [-0.068, 0.289]
0.10ORLA+SGLang0.100 [-0.059, 0.259]0.602 [0.163, 1.040]0.508 [0.221, 0.794]0.034 [-0.079, 0.146]

At 0.005 JPS, ResearchBench uses n=2 (df=1), because seed 43 sampled no ResearchBench jobs.

Across all five serving cells, increasing load from 0.005 to 0.10 JPS reduces mean completion by 42.3–51.2 percentage points and increases mean run-level P95 runtime by approximately 3.9–4.2×. Effective quality also drops substantially at high load for SWE-Bench, PinchBench, and PaperBanana, while ResearchBench shows backend-dependent changes. These repeated-run results reinforce the original findings and quantify their uncertainty across the full 2×H200 load sweep.

Comments on the paper

3 comments

Dear Reviewers,

Could you please review the authors’ rebuttals and follow up with any further comments or questions?

Many thanks for your time and efforts.

Best regards, AC

Dear Reviewers,

As the AC–reviewer discussion phase is about to begin, please highlight any disagreements or inconsistencies across the reviews, as well as any concerns that remain unresolved after the author rebuttal.

Please share your perspectives on these points and engage with one another to help us reach a well-supported assessment of the paper.

Thank you for your time and contributions.

Best regards, AC

AuthorsOpenReview ↗

Global Comment: Uploaded Revised Manuscript, New 4xH200 Multi-Seed Results, and Color-Coded Guide

Dear Area Chair and Reviewers,

We sincerely thank the Area Chair and all reviewers for their constructive feedback. We have uploaded the revised manuscript incorporating additional experiments, statistical analyses, corrected figures, and methodological clarifications in response to the reviews.

In addition to the 50 multi-seed runs on 2×H200 reported in our initial responses, we have now completed 12 further runs (seeds 43 and 44) on 4×H200 across three native serving backends (vLLM, SGLang, and Continuum) at 0.05 and 0.10 JPS. This brings the total to 62 additional runs and 107 benchmark runs overall.

What the New Experiments Establish

  1. Robustness of load-dependent service behavior. The expanded multi-seed experiments strengthen the statistical support for load-sensitive completion and tail latency, while revealing important differences across hardware/model configurations, serving backends, and task families. The additional 4×H200 results also demonstrate why tail latency must be interpreted alongside execution success and the composition of completed jobs.

  2. Substrate-dependent workflow-ranking reversals. The controlled motivation experiment confirms that the key Star/Chain ranking reversal persists across repeated executions. Additional matched-workload analyses identify workflow-ranking reversals in the main benchmark, demonstrating that these effects are not limited to the synthetic motivation setting, while also revealing variability across workload seeds.

  3. Execution and delivery failures dominate the effective-quality gap. Our evaluation audit shows that missing quality scores arise overwhelmingly from execution and delivery failures rather than evaluator errors. Sensitivity analyses using execution- and artifact-grounded metrics further confirm that evaluated-only quality can substantially overstate the useful quality delivered per submitted job.

Manuscript Updates and Revision Guide

To facilitate verification, revisions in the uploaded PDF are color-coded by the primary reviewer concern: blue for Reviewer HMGh, dark orange for Reviewer taDc, and violet for Reviewer xXpx. Some revisions and shared appendices address overlapping concerns across multiple reviews.

The major manuscript updates include:

  • Corrected figures and metric definitions (Sections 1 and 3, Table 2, Figs. 1(c) and 6): Regenerated Fig. 1(c) using the finalized benchmark traces, corrected the runtime-threshold exceedance labeling, and clarified window completion, execution success, and enforced deadline termination.
  • Expanded experimental and statistical evidence (Section 4.2, Appendices D–F): Added workflow-ranking uncertainty and stability analyses, multi-seed confidence intervals, paired load comparisons across both hardware configurations, and main-benchmark workflow-ranking evidence.
  • Evaluation audits and quality sensitivity (Section 4.2, Appendix G): Added the execution/evaluation failure breakdown and judge-free quality analyses with explicit evaluation denominators.
  • Reproducibility and experimental comparability (Section 4.1, Appendix A): Clarified hardware/model allocations, candidate replay, serving-engine implementations, and the scope and limitations of the experimental comparisons.

We hope these additional experiments and revisions address the reviewers' central concerns and provide a clearer, more rigorous account of MASSYS's contributions and experimental evidence. We welcome any follow-up questions during the remainder of the discussion period and sincerely appreciate your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.