- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper asks whether coding agents can synthesize a complete LLM serving stack for each deployment target, rather than relying on one general-purpose engine. That is a good question, and the answer on offer is a system called VibeServe: a coding-agent harness wrapped in an outer planning loop and an inner implementation loop. The outer loop keeps state in a structured issue backlog, a long-term memory file, and git history, and dispatches one concrete task per round (independent issues can be dispatched to parallel Implementer instances). The inner loop runs three agents in fresh contexts: an Implementer that writes the candidate system, an Accuracy Judge that runs the user-supplied checker and watches for reward hacking, and a Performance Evaluator that profiles the accepted build and produces hints for the next round.
The evaluation covers seven scenarios: Llama-3.1-8B on an H100 as the standard case, Qwen3.5-397B-A17B on four MI300As as the hard AMD MoE case, and Show-o2 image generation on H100 and MacBook as the case where no off-the-shelf engine works at all. The other four — predicted-output speculative decoding for code editing, prompt caching for a hybrid DeltaNet/attention model, streaming ASR, constrained JSON decoding on a MacBook — are the same story on easier targets, and I took them more or less as reported.
On the standard case the generated system is roughly at vLLM parity and about 5% above SGLang. On the harder targets the paper reports 1.69x to 6.27x wins over various baselines, plus 2.33x goodput over a VibeServe-tuned SGLang on the AMD MoE case. The ablations replace the issue tracker with OpenEvolve, remove the skills library, and collapse the three inner agents into one.
Critical strengths
The strongest evidence is on the targets where a general-purpose stack genuinely does not work, and Scenario B is the one I keep coming back to.
Stock SGLang does not serve the 397B MXFP4 MoE on MI300A; the generated engine reaches 2,242 tok/s at 96 concurrent sessions against 961 tok/s at 48 for the tuned baseline. Scenario G is similar: neither vLLM nor vLLM-Omni supports Show-o2, and the generated system gets 6.27x over the PyTorch-MPS reference on a MacBook while staying within 7% of the paper's per-component fp16 floor. Those are not microbenchmark deltas. If they hold, they are a real existence proof that a multi-agent loop can build a complete serving stack — schedulers, caches, kernels, communication, the parts that usually do not survive contact with an agent paper — and that is a more ambitious artifact than most of this literature attempts. I would like it to be true.
The second strength is the design of the loop, and specifically the honesty of the failure analysis. The fresh-context Accuracy Judge is a sensible answer to the obvious failure mode of an agent relaxing its own gates, and the single-agent ablation gives concrete evidence for it: 617 tok/s, with the agent repeatedly skipping its own accuracy and benchmark checks. The skills-library ablation is the more interesting result, at 1,100 tok/s against 10k for the full pipeline.
Critical weaknesses
The conclusion of this paper is economic, and the economics are missing. The premise is that per-target engineering cost is what kept bespoke serving impractical. The paper then reports wall-clock per scenario and nothing else. No token cost. No dollar cost. No accounting for the human effort behind the skills library, the accuracy checkers, the benchmarks or the reference implementations. No comparison against what it would cost a human to get the same performance out of a generic stack. The conclusion says VibeServe relies on a non-trivial per-target compute budget (Appendix A), but Appendix A gives call counts and active time, not compute cost.
The no-skill ablation makes the omission harder to ignore. If removing the skills library costs 9x, then much of the demonstrated capability is human knowledge encoded in advance, and the per-target marginal cost is being subsidized by a fixed human cost the paper does not report.
Questions for authors
Please give the full cost of each scenario: agent tokens, dollar cost, wall-clock, and human hours for the skills library, checker, benchmark and reference implementation. Who authors the skills for a new target, and how long does that take?
For B, Give the agent budget spent making SGLang functional and tuning it, and the budget spent on the generated system. If the two differ, what does 2.33x mean per unit of cost? More directly, what happens if you run the same issue-tracker loop on SGLang for the same 105 hours? That is the comparison I would need before accepting the paper.
6 comments
W2. Is the per-target cost subsidized by the cost of making the Skills Library?
Thanks for raising a sharp question. Because removing the Skills Library reduces Scenario A throughput by ~9x (10,000 tok/s → 1,100 tok/s, Table 8), we must explicitly account for the fixed human cost of creating that library so readers can evaluate the true economics.
We would like to clarify three key facts:
- The only human cost is preparing common skills, which is small and amortized across all targets, with no per-target marginal cost. As described above, we built our skill library ourselves with the help of agents over a few days, summarizing public documentation and open-source software.
- Skills encode general primitives, not target-specific solutions. We used the same skills library for all scenarios. In Scenario A, the No-skill agent reached 1,100 tok/s by writing a working continuous-batching PyTorch server, but stalled because it did not know the exact API contracts for integrating third-party C++/CUDA backend libraries like FlashInfer paged attention. The Skills Library provides these reusable library/concept pointers (much like documentation for a human systems engineer), while the agents still must synthesize, wire together, and debug the bespoke architecture for workloads the Skills Library has never seen (e.g., dual Gated DeltaNet/KV caching on MI300A in Scenario B, predicted-output verification in C, or Show-o2 AR/diffusion interleaving in G).
- In another ablation, alternative agents with the same skills but without VibeServe’s orchestration fail to implement an efficient engine (Table 8). While removing the Skills Library drops throughput from 10,000 to 1,100 tok/s (9x), keeping the full Skills Library while removing VibeServe’s orchestration drops throughput even further, to 617 tok/s (16x drop) in the Single-Agent ablation and 537 tok/s (18.6x drop) in the OpenEvolve ablation. Thus, encoded domain knowledge alone cannot build a serving system, but both the reusable Skills Library and VibeServe's outer/inner loop orchestration are essential.
Q2. What did each side of Scenario B cost, and what happens at a matched time budget?
Thank you very much for pointing out the apples-to-oranges comparison in our Scenario B comparison. Table 2 now reports both budgets. Making SGLang functional on MI300A and tuning it took 19 MI300A hours and cost about USD 585 (USD 433 in tokens, USD 152 in GPU time). The generated system took 108 hours and cost about USD 3,289 (USD 2,425 in tokens, USD 864 in GPU time), about 5.6x more. So the 2.33x goodput gain (2,242 vs. 961 tok/s) came at a higher cost. The tuned SGLang still cannot express some of the generated system's optimizations as configuration or kernel substitutions, such as joint placement of DeltaNet state and MoE expert shards in MI300A's unified memory (Appendix A), so a matched budget alone may not close the gap. Because running the experiment with the same 105-hour time budget takes a long time, we first provide a similar, smaller-scale experiment at a matched time budget (Appendix B, Table 7, Figure 7).
In this experiment, we build two serving systems for Qwen3.6-35B-A3B on 8x AMD MI250 GPUs, using coding agent traces from TraceLab with the same time budget of 25 hours: one starting from SGLang and another built from scratch. SGLang first had to be ported to gfx90a.
After about 25 hours, the bespoke engine reached 1,375 tok/s vs. 815 (TPOT p50 3.1 vs. 6.6 ms), at a total cost of USD 1,348 vs. USD 980. The bespoke run used more tokens but fewer GPU node-hours (67 vs. 77), so it delivered 1.69x the throughput for 1.38x the cost. SGLang improved through changes inside single operations, then plateaued. The bespoke engine landed changes spanning several components: whole-layer decode fusion, speculative decoding with double-buffered recurrent state, and a fully graph-captured prefill. The SGLang agents attempted versions of these but did not land them. Please refer to Table 7 and Figure 7 in Appendix B for more details.
Additionally, regarding Scenario B (Qwen3.5-397B-A17B-
MXFP4 on 4x AMD MI300A), we are running a full 105-hour ablation now. We will post another response once the data are ready.
Follow-up: Scenario B ablation with equal time budget
Thank you again for the reviews. In our first response we said we were running a full 105-hour ablation for Scenario B (Qwen3.5-397B-A17B-MXFP4 on 4x AMD MI300A), and it has finished. We ran a new VibeServe campaign that starts from SGLang, with the same agents, skills library, accuracy checker, workload, and 105-hour budget as the bespoke run in the paper. SGLang does not run out of the box on this model and hardware, so the campaign first had to port it. The porting time counts toward the time budget. The bespoke curve in the figure is the original run from the paper, replotted against hours since start. We added both runs to Appendix B (Figure 8, a new row in Table 3) and to the cost table (Table 2).
Result. At 96 concurrent sessions (the highest concurrency before overload), the bespoke engine reaches 2,242 tok/s. The best measurement from the SGLang-seeded campaign is 1,637 tok/s, so the bespoke engine is 1.37x higher at an equal time budget. This is consistent with the smaller 25-hour MI250 experiment in our first response (1,375 vs. 815 tok/s). The 1.37x is the more appropriate number for comparing the two starting points under the same budget, and we will revise the paper to use this speedup number for comparison.
Performance difference. Most of the throughput gap comes from three things.
- Speculative decoding (MTP). The bespoke engine implements MTP speculative decoding, whereas the SGLang port does not. It gives the bespoke engine +16-20%. The SGLang agents tried twice to land it and failed: the first attempt did not pass the task's accuracy checks, and the second gave no speedup because of a poor kernel implementation.
- Sequence-parallel all-reduce. The bespoke engine implemented it to reduce time spent in all-reduce. The SGLang agent did not optimize all-reduce operation.
- More aggressive kernel fusion. The bespoke agents fused prefill kernels (in-place DeltaNet state, fused DeltaNet norm, fused prefill attention), RoPE with KV writes, routing, elementwise and norm operations, and the all-reduce with the residual add and RMSNorm. The SGLang agents attempted far less fusion.
As in the MI250 experiment, the SGLang agents made progress inside individual operations, while the bespoke engine landed changes that span components. The takeaway is that agents make cross-cutting optimizations more easily in a bespoke engine, where they own the whole forward path and request loop, than in a generic framework whose abstractions must serve a wide range of hardware, models, and workloads.
Cost. The SGLang-seeded campaign cost about 2,725 USD at API list prices: 1,733 USD for Claude Opus 5 tokens, plus 992 USD for 124 MI300A node-hours. The bespoke run in Table 2 cost 3,289 USD (2,425 USD in tokens plus 864 USD for 108 node-hours). The bespoke engine therefore delivered 1.37x the goodput for about 1.2x the cost. As in Table 2, dollar figures are API-list-price estimates, since we ran on subscriptions.
Please refer to Appendix B (Figure 8, Table 3) and Appendix A (Table 2) in the revised PDF for details.
Thank you for the review. We agree the economic argument needs numbers; the revision adds them.
W1/Q1. What are the costs of VibeServe?
Token and GPU costs for running experiments. We added Table 2 in Appendix A to show detailed costs in terms of token consumption, dollar cost, and GPU hours. Since we ran all experiments on Codex or Claude subscriptions, the dollar costs in the table are not what we were billed, but were calculated from token consumption at API list prices as an estimate of what each run would cost under per-token billing.
Because agents repeatedly re-read long, mostly unchanged contexts, 94--99% of input tokens are served from the prompt cache, and these account for a majority of the costs. Even with GPU time included, the runs are not prohibitively expensive. Scenarios A and C–G each cost a few hundred dollars. Scenario B is the outlier, at about $3,900 including its SGLang tuning run and GPU costs, because it is the longest run, consumes the most tokens, and occupies MI300A hardware for over 100 hours.
Human effort. We built most of our skills library ourselves, with agents helping over a few days, distilling our knowledge from existing open-source serving systems like vLLM or SGLang. Users can extend the skills library in different dimensions, with the following estimate of the effort:
models/directory, which is usually a few hundred lines of code.However, these are not always necessary. For example, for new model architectures, agents can often figure out the architecture and optimization methods from reference implementations, even without explicit explanation in the skills. Moreover, it is common for hardware vendors to provide some skills, like for AMD or AWS Trainium, and users can integrate such skills into VibeServe. Note that these are a one-time effort, and the same skills files can be reused across multiple VibeServe runs. Please refer to Appendix C for more details on the skills library.
The reference implementation, accuracy checker, and benchmark are easy to prepare. In our experiments, we use implementations in the Hugging Face Transformers library for each model as reference implementations, which show how model inference should run but with little performance optimization. The accuracy checker and the benchmark are both short, standalone Python scripts: the checker compares the generated system's outputs against the reference on a fixed set of inputs, and the benchmark replays the target workload against the server and reports the target metric (e.g., throughput, TTFT, or latency). Across our scenarios, each is about 50 to 700 lines of Python, and most of that code is shared across targets (request generation, timing, and output comparison), so a new target typically adapts an existing script rather than writing one from scratch. Writing them is a one-time effort per target, comparable to the evaluation harness an engineer would write before tuning any serving system by hand.