- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces MASSys, a substrate-aware benchmark for multi-agent LLM systems. Rather than treating the serving environment as a fixed implementation detail, MASSys evaluates identical candidate-task pairs across multiple serving backends, load regimes, and resource conditions. It records both task-level outcomes and service-level metrics such as completion, latency, queueing, KV-cache pressure, timeout behavior, and failure traces. Across heterogeneous workloads, the experiments show that serving conditions can change workflow latency rankings, generate severe tail-latency and timeout effects under load, and alter effective task quality by preventing otherwise competitive jobs from completing.
Critical strengths
-
The paper identifies an important and under-measured confound in MAS evaluation. Treating the serving substrate as an experimental variable rather than hidden infrastructure is a useful conceptual extension of existing system-level agent evaluation.
-
The paired design and dual task/service traces are well motivated. Replaying the same candidate-task pairs across substrate conditions is substantially more informative than comparing aggregate scores from unrelated runs, and distinguishing evaluated-output quality from delivery/effective quality is particularly valuable for practical agent systems.
Critical weaknesses
The key substrate cells are represented by only one run. This is a major concern because several central claims concern workflow rank reversals, P95 latency, queueing, timeout rates, and reliability envelopes, all of which can exhibit non-negligible systems variance. Without replicated runs or uncertainty estimates, it is difficult to determine how stable the reported substrate differences and ranking inversions are.
Questions for authors
-
Can the authors replicate the key substrate/rank-inversion experiments with multiple independent runs and report uncertainty intervals for latency, completion, and effective quality?
-
Do the reported rank reversals persist when replaying a fixed deterministic workload trace across repeated substrate runs, rather than resampling arrivals?
-
How sensitive are the main conclusions to restricting analysis to workload families with execution-grounded quality metrics?
6 comments
Rebuttal for Q1 - part 2
Table 3. Per-family effective quality. Mean [95% CI] across three seeds, using the submitted-job denominator.
| JPS | Serving cell | SWE-Bench | ResearchBench | PinchBench | PaperBanana |
|---|---|---|---|---|---|
| 0.005 | vLLM | 0.400 [-0.915, 1.715] | 0.764 [0.410, 1.119] | 0.592 [-0.712, 1.895] | 0.588 [0.452, 0.723] |
| 0.005 | SGLang | 0.467 [-0.784, 1.717] | 0.767 [0.459, 1.075] | 0.667 [-0.768, 2.101] | 0.696 [0.435, 0.956] |
| 0.005 | Continuum | 0.400 [-0.915, 1.715] | 0.764 [0.372, 1.157] | 0.600 [-0.715, 1.915] | 0.538 [0.171, 0.904] |
| 0.005 | ORLA+vLLM | 0.333 [-1.101, 1.768] | 0.773 [0.730, 0.817] | 0.533 [-0.717, 1.784] | 0.483 [0.017, 0.950] |
| 0.005 | ORLA+SGLang | 0.233 [-0.392, 0.859] | 0.775 [0.615, 0.935] | 0.600 [-0.715, 1.915] | 0.821 [0.539, 1.103] |
| 0.01 | vLLM | 0.472 [-0.049, 0.993] | 0.788 [0.740, 0.837] | 0.600 [-0.715, 1.915] | 0.560 [0.433, 0.688] |
| 0.01 | SGLang | 0.472 [-0.049, 0.993] | 0.782 [0.743, 0.821] | 0.595 [-0.706, 1.897] | 0.733 [0.527, 0.938] |
| 0.01 | Continuum | 0.472 [-0.049, 0.993] | 0.791 [0.747, 0.835] | 0.595 [-0.705, 1.895] | 0.531 [0.435, 0.628] |
| 0.01 | ORLA+vLLM | 0.264 [-0.616, 1.144] | 0.794 [0.750, 0.837] | 0.529 [-0.706, 1.763] | 0.549 [0.276, 0.823] |
| 0.01 | ORLA+SGLang | 0.347 [-0.357, 1.052] | 0.791 [0.747, 0.834] | 0.595 [-0.706, 1.897] | 0.702 [0.487, 0.918] |
| 0.02 | vLLM | 0.485 [0.420, 0.550] | 0.780 [0.764, 0.796] | 0.589 [-0.689, 1.866] | 0.487 [0.355, 0.619] |
| 0.02 | SGLang | 0.388 [0.235, 0.541] | 0.779 [0.767, 0.791] | 0.626 [-0.723, 1.974] | 0.743 [0.546, 0.939] |
| 0.02 | Continuum | 0.443 [0.286, 0.600] | 0.782 [0.772, 0.792] | 0.593 [-0.694, 1.879] | 0.545 [0.451, 0.640] |
| 0.02 | ORLA+vLLM | 0.183 [-0.143, 0.509] | 0.767 [0.762, 0.772] | 0.533 [-0.688, 1.755] | 0.527 [0.412, 0.642] |
| 0.02 | ORLA+SGLang | 0.213 [-0.054, 0.481] | 0.776 [0.751, 0.801] | 0.556 [-0.670, 1.782] | 0.726 [0.670, 0.782] |
| 0.05 | vLLM | 0.367 [0.041, 0.693] | 0.788 [0.781, 0.795] | 0.812 [0.591, 1.034] | 0.521 [0.456, 0.585] |
| 0.05 | SGLang | 0.436 [0.128, 0.744] | 0.786 [0.778, 0.794] | 0.830 [0.554, 1.105] | 0.690 [0.636, 0.743] |
| 0.05 | Continuum | 0.434 [0.185, 0.682] | 0.787 [0.766, 0.808] | 0.833 [0.575, 1.090] | 0.505 [0.453, 0.558] |
| 0.05 | ORLA+vLLM | 0.289 [-0.284, 0.863] | 0.785 [0.767, 0.803] | 0.779 [0.488, 1.069] | 0.497 [0.375, 0.618] |
| 0.05 | ORLA+SGLang | 0.224 [-0.304, 0.751] | 0.790 [0.784, 0.795] | 0.777 [0.469, 1.086] | 0.710 [0.553, 0.868] |
| 0.10 | vLLM | 0.122 [-0.068, 0.312] | 0.786 [0.761, 0.810] | 0.580 [0.365, 0.795] | 0.093 [0.015, 0.171] |
| 0.10 | SGLang | 0.173 [0.060, 0.287] | 0.785 [0.764, 0.806] | 0.625 [0.570, 0.681] | 0.072 [-0.192, 0.336] |
| 0.10 | Continuum | 0.065 [-0.173, 0.303] | 0.329 [0.020, 0.637] | 0.308 [0.148, 0.469] | 0.006 [-0.019, 0.031] |
| 0.10 | ORLA+vLLM | 0.181 [0.036, 0.326] | 0.748 [0.659, 0.838] | 0.567 [0.513, 0.621] | 0.111 [-0.068, 0.289] |
| 0.10 | ORLA+SGLang | 0.100 [-0.059, 0.259] | 0.602 [0.163, 1.040] | 0.508 [0.221, 0.794] | 0.034 [-0.079, 0.146] |
At 0.005 JPS, ResearchBench uses n=2 (df=1), because seed 43 sampled no ResearchBench jobs.
The additional experiments support our main load-sensitivity findings. Across the complete five-load sweep, mean completion generally declines, while mean run-level P95 runtime increases monotonically across all five serving cells as offered load rises. Across all 15 matched backend–seed comparisons from 0.05 to 0.10 JPS, completion decreases and run-level P95 runtime increases. The mean completion reduction ranges from 31.98 to 40.38 percentage points across the five serving cells, with all five paired 95% t confidence intervals excluding zero. P95 runtime increases in all five cells, with three of five paired confidence intervals excluding zero. For effective quality, PaperBanana and PinchBench decrease in all 15 comparisons, SWE-bench decreases in 13 of 15, and ResearchBench exhibits backend-dependent behavior. These results provide additional evidence for load-dependent degradation while highlighting differences in uncertainty across metrics and task families.
Workflow-ranking repetitions (Fig. 1b). In the original experiment, each of the 25 workflow–condition combinations was executed 10 times, yielding 250 workflow executions under fixed workflow definitions and synthetic context. The rankings in Fig. 1(b) were computed using the mean end-to-end workflow latency across these ten executions. We now additionally report the sample standard deviations and 95% confidence intervals to quantify the variability of these measurements.
Rebuttal for Q1 - part 3
Table 4. Workflow end-to-end latency (s). Mean [95% CI] across ten repeated executions.
| Workflow | Clean | KV-tight | CPU-tight | GPU pressure | Mixed-tight |
|---|---|---|---|---|---|
| Star | 5.012 [4.954, 5.069] | 12.490 [12.346, 12.634] | 5.174 [5.139, 5.210] | 5.590 [5.576, 5.605] | 18.109 [18.032, 18.186] |
| DAG | 5.479 [5.450, 5.509] | 11.559 [11.535, 11.584] | 5.659 [5.585, 5.732] | 5.562 [5.523, 5.601] | 19.295 [18.828, 19.762] |
| Tool-ReAct | 5.768 [5.688, 5.848] | 11.978 [11.903, 12.054] | 12.977 [12.747, 13.208] | 6.356 [6.292, 6.420] | 15.193 [15.092, 15.295] |
| Chain | 6.180 [6.175, 6.186] | 6.039 [6.034, 6.044] | 6.511 [6.462, 6.559] | 6.357 [6.342, 6.372] | 27.301 [27.265, 27.337] |
| Debate | 7.930 [7.922, 7.938] | 12.896 [12.889, 12.903] | 8.409 [8.329, 8.489] | 8.225 [8.201, 8.249] | 21.134 [21.074, 21.194] |
The Star/Chain ranking reversal remains evident in the repeated measurements. Under Clean, Star achieves 5.0118 ± 0.0804 s versus 6.1803 ± 0.0081 s for Chain. Under KV-tight, Star increases to 12.4900 ± 0.2018 s, whereas Chain remains at 6.0392 ± 0.0068 s. Their mean-latency rankings reverse from first/fourth under Clean to fourth/first under KV-tight. Notably, their 95% latency confidence intervals are well separated under both conditions, providing further evidence for the robustness of this rank reversal. These repeated measurements provide additional support for the topology–substrate interaction illustrated in Fig. 1(b).
Overall, the additional multi-seed experiments and the uncertainty analysis of the original workflow-ranking experiment strengthen the statistical support for our central findings: serving conditions affect delivery reliability, tail latency, effective quality, and the relative latency of different workflow structures. We will incorporate the full multi-seed statistics and workflow-ranking uncertainty estimates into the revised manuscript and Appendix.
Rebuttal for Q1 - part 1
We thank Reviewer xXpx for the constructive feedback and thoughtful questions, which have helped us strengthen the statistical support for our findings, further investigate the robustness of rank reversals, and clarify the implications of our evaluation methodology.
Q1: Repeated Substrate Experiments and Statistical Uncertainty
We thank the reviewer for highlighting the importance of experimental reproducibility and statistical uncertainty. To address this concern, we conducted additional repetitions for the main substrate sweeps and computed uncertainty estimates for the original repeated workflow-ranking experiment in Fig. 1(b). We report confidence intervals for completion, latency, and effective quality below.
Main benchmark repetitions. We have completed two additional workload-seed sweeps (seeds 43 and 44) across all five serving cells and five offered loads (0.005, 0.01, 0.02, 0.05, and 0.10 JPS) on 2×H200, totaling 50 additional runs. Together with the original seed 42, these experiments provide three workload realizations per serving cell and load. Each seed independently resamples the task mixture and Poisson arrivals, while candidate–task pairs and arrival schedules are held fixed across serving cells within each seed–load block.
We report untruncated 95% Student-t confidence intervals across workload seeds. Some intervals extend beyond the natural metric bounds due to the limited number of repetitions and sparse per-family samples at low loads. P95 is computed separately for each run before aggregation, and effective quality is reported separately for each task family.
Table 1. Completion rate (%). Mean [95% CI] across three seeds.
Table 2. P95 end-to-end runtime (s). Mean [95% CI] across three run-level P95 values.