acceptodds
Under review as a conference paper at ICLR 2027

VibeServe: Can AI Agents Build Bespoke LLM Serving Systems?

Abstract

For years, we have built LLM serving systems like any other critical infrastructure, with a single general-purpose stack hand-tuned over many engineer-years to support every model and workload. In this paper, we take the opposite bet and let coding agents automatically synthesize bespoke serving systems for different usage scenarios. We propose VibeServe, the first agentic loop that generates entire LLM serving stacks end-to-end. VibeServe uses an outer loop to plan and track the search over system designs, and an inner loop to implement candidates, check correctness, and measure performance on the target benchmark. In the standard deployment setting, where existing stacks are highly optimized, VibeServe remains competitive with vLLM, showing that generation-time specialization need not come at the cost of performance. More interestingly, VibeServe outperforms existing systems in six non-standard scenarios by exploiting opportunities that generic systems miss. These scenarios involve non-standard model architectures, workload knowledge, and hardware-specific optimizations. Together, these results argue that infrastructure software should favor generation-time specialization over runtime generality.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
LLM serving system, agent

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_QEGLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether coding agents can synthesize a complete LLM serving stack for each deployment target, rather than relying on one general-purpose engine. That is a good question, and the answer on offer is a system called VibeServe: a coding-agent harness wrapped in an outer planning loop and an inner implementation loop. The outer loop keeps state in a structured issue backlog, a long-term memory file, and git history, and dispatches one concrete task per round (independent issues can be dispatched to parallel Implementer instances). The inner loop runs three agents in fresh contexts: an Implementer that writes the candidate system, an Accuracy Judge that runs the user-supplied checker and watches for reward hacking, and a Performance Evaluator that profiles the accepted build and produces hints for the next round.

The evaluation covers seven scenarios: Llama-3.1-8B on an H100 as the standard case, Qwen3.5-397B-A17B on four MI300As as the hard AMD MoE case, and Show-o2 image generation on H100 and MacBook as the case where no off-the-shelf engine works at all. The other four — predicted-output speculative decoding for code editing, prompt caching for a hybrid DeltaNet/attention model, streaming ASR, constrained JSON decoding on a MacBook — are the same story on easier targets, and I took them more or less as reported.

On the standard case the generated system is roughly at vLLM parity and about 5% above SGLang. On the harder targets the paper reports 1.69x to 6.27x wins over various baselines, plus 2.33x goodput over a VibeServe-tuned SGLang on the AMD MoE case. The ablations replace the issue tracker with OpenEvolve, remove the skills library, and collapse the three inner agents into one.

Critical strengths

The strongest evidence is on the targets where a general-purpose stack genuinely does not work, and Scenario B is the one I keep coming back to.

Stock SGLang does not serve the 397B MXFP4 MoE on MI300A; the generated engine reaches 2,242 tok/s at 96 concurrent sessions against 961 tok/s at 48 for the tuned baseline. Scenario G is similar: neither vLLM nor vLLM-Omni supports Show-o2, and the generated system gets 6.27x over the PyTorch-MPS reference on a MacBook while staying within 7% of the paper's per-component fp16 floor. Those are not microbenchmark deltas. If they hold, they are a real existence proof that a multi-agent loop can build a complete serving stack — schedulers, caches, kernels, communication, the parts that usually do not survive contact with an agent paper — and that is a more ambitious artifact than most of this literature attempts. I would like it to be true.

The second strength is the design of the loop, and specifically the honesty of the failure analysis. The fresh-context Accuracy Judge is a sensible answer to the obvious failure mode of an agent relaxing its own gates, and the single-agent ablation gives concrete evidence for it: 617 tok/s, with the agent repeatedly skipping its own accuracy and benchmark checks. The skills-library ablation is the more interesting result, at 1,100 tok/s against 10k for the full pipeline.

Critical weaknesses

The conclusion of this paper is economic, and the economics are missing. The premise is that per-target engineering cost is what kept bespoke serving impractical. The paper then reports wall-clock per scenario and nothing else. No token cost. No dollar cost. No accounting for the human effort behind the skills library, the accuracy checkers, the benchmarks or the reference implementations. No comparison against what it would cost a human to get the same performance out of a generic stack. The conclusion says VibeServe relies on a non-trivial per-target compute budget (Appendix A), but Appendix A gives call counts and active time, not compute cost.

The no-skill ablation makes the omission harder to ignore. If removing the skills library costs 9x, then much of the demonstrated capability is human knowledge encoded in advance, and the per-target marginal cost is being subsidized by a fixed human cost the paper does not report.

Questions for authors

Please give the full cost of each scenario: agent tokens, dollar cost, wall-clock, and human hours for the skills library, checker, benchmark and reference implementation. Who authors the skills for a new target, and how long does that take?

For B, Give the agent budget spent making SGLang functional and tuning it, and the budget spent on the generated system. If the two differ, what does 2.33x mean per unit of cost? More directly, what happens if you run the same issue-tracker loop on SGLang for the same 105 hours? That is the comparison I would need before accepting the paper.

6 comments

Thank you for the review. We agree the economic argument needs numbers; the revision adds them.

W1/Q1. What are the costs of VibeServe?

Token and GPU costs for running experiments. We added Table 2 in Appendix A to show detailed costs in terms of token consumption, dollar cost, and GPU hours. Since we ran all experiments on Codex or Claude subscriptions, the dollar costs in the table are not what we were billed, but were calculated from token consumption at API list prices as an estimate of what each run would cost under per-token billing.

Because agents repeatedly re-read long, mostly unchanged contexts, 94--99% of input tokens are served from the prompt cache, and these account for a majority of the costs. Even with GPU time included, the runs are not prohibitively expensive. Scenarios A and C–G each cost a few hundred dollars. Scenario B is the outlier, at about $3,900 including its SGLang tuning run and GPU costs, because it is the longest run, consumes the most tokens, and occupies MI300A hardware for over 100 hours.

Human effort. We built most of our skills library ourselves, with agents helping over a few days, distilling our knowledge from existing open-source serving systems like vLLM or SGLang. Users can extend the skills library in different dimensions, with the following estimate of the effort:

  • A new model architecture is about adding just one file under the models/ directory, which is usually a few hundred lines of code.
  • A new workload usually does not require a specialized skills file, unless users want to teach new optimization techniques, which usually takes a few hundred lines of code. VibeServe already provides a comprehensive set of optimization techniques that commonly appear in model serving system optimizations (continuous batching, CUDA graphs, speculative decoding, etc.).
  • A new hardware would require at least three files on hardware architecture, profiling, and theoretical performance floors, which can take a few thousand lines of code at most.

However, these are not always necessary. For example, for new model architectures, agents can often figure out the architecture and optimization methods from reference implementations, even without explicit explanation in the skills. Moreover, it is common for hardware vendors to provide some skills, like for AMD or AWS Trainium, and users can integrate such skills into VibeServe. Note that these are a one-time effort, and the same skills files can be reused across multiple VibeServe runs. Please refer to Appendix C for more details on the skills library.

The reference implementation, accuracy checker, and benchmark are easy to prepare. In our experiments, we use implementations in the Hugging Face Transformers library for each model as reference implementations, which show how model inference should run but with little performance optimization. The accuracy checker and the benchmark are both short, standalone Python scripts: the checker compares the generated system's outputs against the reference on a fixed set of inputs, and the benchmark replays the target workload against the server and reports the target metric (e.g., throughput, TTFT, or latency). Across our scenarios, each is about 50 to 700 lines of Python, and most of that code is shared across targets (request generation, timing, and output comparison), so a new target typically adapts an existing script rather than writing one from scratch. Writing them is a one-time effort per target, comparable to the evaluation harness an engineer would write before tuning any serving system by hand.

W2. Is the per-target cost subsidized by the cost of making the Skills Library?

Thanks for raising a sharp question. Because removing the Skills Library reduces Scenario A throughput by ~9x (10,000 tok/s → 1,100 tok/s, Table 8), we must explicitly account for the fixed human cost of creating that library so readers can evaluate the true economics.

We would like to clarify three key facts:

  1. The only human cost is preparing common skills, which is small and amortized across all targets, with no per-target marginal cost. As described above, we built our skill library ourselves with the help of agents over a few days, summarizing public documentation and open-source software.
  2. Skills encode general primitives, not target-specific solutions. We used the same skills library for all scenarios. In Scenario A, the No-skill agent reached 1,100 tok/s by writing a working continuous-batching PyTorch server, but stalled because it did not know the exact API contracts for integrating third-party C++/CUDA backend libraries like FlashInfer paged attention. The Skills Library provides these reusable library/concept pointers (much like documentation for a human systems engineer), while the agents still must synthesize, wire together, and debug the bespoke architecture for workloads the Skills Library has never seen (e.g., dual Gated DeltaNet/KV caching on MI300A in Scenario B, predicted-output verification in C, or Show-o2 AR/diffusion interleaving in G).
  3. In another ablation, alternative agents with the same skills but without VibeServe’s orchestration fail to implement an efficient engine (Table 8). While removing the Skills Library drops throughput from 10,000 to 1,100 tok/s (9x), keeping the full Skills Library while removing VibeServe’s orchestration drops throughput even further, to 617 tok/s (16x drop) in the Single-Agent ablation and 537 tok/s (18.6x drop) in the OpenEvolve ablation. Thus, encoded domain knowledge alone cannot build a serving system, but both the reusable Skills Library and VibeServe's outer/inner loop orchestration are essential.

Q2. What did each side of Scenario B cost, and what happens at a matched time budget?

Thank you very much for pointing out the apples-to-oranges comparison in our Scenario B comparison. Table 2 now reports both budgets. Making SGLang functional on MI300A and tuning it took 19 MI300A hours and cost about USD 585 (USD 433 in tokens, USD 152 in GPU time). The generated system took 108 hours and cost about USD 3,289 (USD 2,425 in tokens, USD 864 in GPU time), about 5.6x more. So the 2.33x goodput gain (2,242 vs. 961 tok/s) came at a higher cost. The tuned SGLang still cannot express some of the generated system's optimizations as configuration or kernel substitutions, such as joint placement of DeltaNet state and MoE expert shards in MI300A's unified memory (Appendix A), so a matched budget alone may not close the gap. Because running the experiment with the same 105-hour time budget takes a long time, we first provide a similar, smaller-scale experiment at a matched time budget (Appendix B, Table 7, Figure 7).

In this experiment, we build two serving systems for Qwen3.6-35B-A3B on 8x AMD MI250 GPUs, using coding agent traces from TraceLab with the same time budget of 25 hours: one starting from SGLang and another built from scratch. SGLang first had to be ported to gfx90a.

After about 25 hours, the bespoke engine reached 1,375 tok/s vs. 815 (TPOT p50 3.1 vs. 6.6 ms), at a total cost of USD 1,348 vs. USD 980. The bespoke run used more tokens but fewer GPU node-hours (67 vs. 77), so it delivered 1.69x the throughput for 1.38x the cost. SGLang improved through changes inside single operations, then plateaued. The bespoke engine landed changes spanning several components: whole-layer decode fusion, speculative decoding with double-buffered recurrent state, and a fully graph-captured prefill. The SGLang agents attempted versions of these but did not land them. Please refer to Table 7 and Figure 7 in Appendix B for more details.

Additionally, regarding Scenario B (Qwen3.5-397B-A17B-
MXFP4 on 4x AMD MI300A), we are running a full 105-hour ablation now. We will post another response once the data are ready.

Follow-up: Scenario B ablation with equal time budget

Thank you again for the reviews. In our first response we said we were running a full 105-hour ablation for Scenario B (Qwen3.5-397B-A17B-MXFP4 on 4x AMD MI300A), and it has finished. We ran a new VibeServe campaign that starts from SGLang, with the same agents, skills library, accuracy checker, workload, and 105-hour budget as the bespoke run in the paper. SGLang does not run out of the box on this model and hardware, so the campaign first had to port it. The porting time counts toward the time budget. The bespoke curve in the figure is the original run from the paper, replotted against hours since start. We added both runs to Appendix B (Figure 8, a new row in Table 3) and to the cost table (Table 2).

Result. At 96 concurrent sessions (the highest concurrency before overload), the bespoke engine reaches 2,242 tok/s. The best measurement from the SGLang-seeded campaign is 1,637 tok/s, so the bespoke engine is 1.37x higher at an equal time budget. This is consistent with the smaller 25-hour MI250 experiment in our first response (1,375 vs. 815 tok/s). The 1.37x is the more appropriate number for comparing the two starting points under the same budget, and we will revise the paper to use this speedup number for comparison.

Performance difference. Most of the throughput gap comes from three things.

  1. Speculative decoding (MTP). The bespoke engine implements MTP speculative decoding, whereas the SGLang port does not. It gives the bespoke engine +16-20%. The SGLang agents tried twice to land it and failed: the first attempt did not pass the task's accuracy checks, and the second gave no speedup because of a poor kernel implementation.
  2. Sequence-parallel all-reduce. The bespoke engine implemented it to reduce time spent in all-reduce. The SGLang agent did not optimize all-reduce operation.
  3. More aggressive kernel fusion. The bespoke agents fused prefill kernels (in-place DeltaNet state, fused DeltaNet norm, fused prefill attention), RoPE with KV writes, routing, elementwise and norm operations, and the all-reduce with the residual add and RMSNorm. The SGLang agents attempted far less fusion.

As in the MI250 experiment, the SGLang agents made progress inside individual operations, while the bespoke engine landed changes that span components. The takeaway is that agents make cross-cutting optimizations more easily in a bespoke engine, where they own the whole forward path and request loop, than in a generic framework whose abstractions must serve a wide range of hardware, models, and workloads.

Cost. The SGLang-seeded campaign cost about 2,725 USD at API list prices: 1,733 USD for Claude Opus 5 tokens, plus 992 USD for 124 MI300A node-hours. The bespoke run in Table 2 cost 3,289 USD (2,425 USD in tokens plus 864 USD for 108 node-hours). The bespoke engine therefore delivered 1.37x the goodput for about 1.2x the cost. As in Table 2, dollar figures are API-list-price estimates, since we ran on subscriptions.

Please refer to Appendix B (Figure 8, Table 3) and Appendix A (Table 2) in the revised PDF for details.

Reviewer_xtKEOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

VibeServe uses coding agents to build serving systems specialized to a model, workload, and hardware target. An outer loop plans optimization tasks using persistent project state, while an inner loop separates implementation, correctness checking, and performance measurement. A serving-systems skills library guides the agents. Across seven scenarios, the generated systems match established engines in a standard setting and outperform the selected baselines in several specialized settings.

Critical strengths
  • Generating an entire serving system is a meaningful step beyond optimizing individual kernels or policies. The paper presents substantial engineering evidence that agents can coordinate changes across multiple components.
  • The scenarios cover different models, workloads, and hardware platforms. The AMD MoE result is particularly compelling: the generated system achieves 2.33× the goodput of a tuned SGLang baseline on a difficult model–hardware combination.
  • Separating implementation from correctness checking is a sensible design. Read-only reference artifacts, independent review of candidate changes, and explicit checks for reward hacking address practical problems in long-running coding agents. The ablations provide useful support for these choices.
  • The paper describes failed attempts and unsuccessful optimizations, rather than reporting only the final results. The experiment starting from existing engines also helps clarify when building from scratch may be useful.
Critical weaknesses
  1. Are the comparisons fair? In several non-standard scenarios, the baseline is a plain configuration of an existing tool rather than the strongest alternative available. Some of the gains might come from simply enabling features that already exist, such as prompt-based speculation for code editing or speculative decoding on the Mac. Others come from approximations that change the output, like the shortcut in image generation, rather than from bespoke system design. The paper would be much stronger if each win were measured against the best thing a practitioner could do today.
  2. The cost of specialization isn’t shown. The thesis is that specialization beats generality, but the paper only measures the upside. Each generated system is evaluated on the same benchmark it was optimized for. So we don’t know: o how it behaves when traffic patterns change; o how much of a real serving system’s functionality it implements; o how hard it would be to maintain. These are exactly the reasons people build general systems in the first place.
  3. Economics and reproducibility. The central argument is that agents make per-target engineering cheap enough to be worthwhile, but the paper doesn’t really quantify the cost. That includes compute, model usage and the human effort behind the skills library. o The skills library seems to carry much of the result, yet it is barely described. o The planner’s mechanics are underspecified. o Each result comes from a single run, and different scenarios use different underlying models, so it’s hard to know how repeatable the results are.
  4. Bespoke vs. agent-improved generic. Several of the wins could just as well be added to a general engine. The paper’s own experiment seeding the agents with vLLM suggests that patching an existing engine can get there faster. The evidence supports “agents can specialize serving systems,” which is already interesting. It supports that more than “bespoke systems should replace general ones.”
Questions for authors
  1. How do the gains hold up against the strongest existing alternative in each non-standard scenario, including simple configuration changes?
  2. What happens to a generated system when the workload changes, and how much standard serving functionality does it support?
7 comments

Thank you for the thoughtful review.

W1/Q1. How do the gains hold up against the strongest existing alternative?

We would like to clarify that the baselines are not plain configurations of existing systems. As we describe in Appendix A, we tuned the baseline to ensure a fair comparison. Here, we provide an overview for each scenario:

  • A: We specified sequence length and chose throughput-oriented mode in vLLM. We tried adding speculative decoding as in Appendix H, but it did not improve the overall throughput for this workload.
  • B: We tuned SGLang with VibeServe to make it functional on MI300A and improve the performance, reaching +42% goodput. Both systems use MTP with k=3, and SGLang also received workload-specific changes with multiple flags for choosing better GPU kernels.
  • C: We tried using speculative decoding in vLLM with EAGLE3 (7 draft tokens), which boosted inference speed from 41 token/s to 122 token/s, but VibeServe’s implementation with predicted-output-based speculation reached even faster at 245 token/s. No serving systems provide an interface for predicted outputs compatible with this workload.
  • D: In the baseline, we enabled prefix caching. However, vLLM’s support of prefix caching for a hybrid attention model was not flexible enough for the workload in this scenario, which has a long common prefix and a short unique suffix. Unlike that, VibeServe’s system has knowledge about the workload at generation time.
  • E: There is no serving engine that supports Moonshine Streaming, so we wrote a vLLM plugin based on the Hugging Face implementation, based on the recommendation in an official document.
  • F: We apologize for the incorrect description in Appendix A. As the reviewer notes, mlx_lm supports draft-model speculation, but it has no grammar-constrained decoding; the VibeServe gain comes from making grammar masking and speculative rollback compose. We revised Appendix A to reflect this change.
  • G: There are no serving engines that support Show-o2; hence, we used model developers’ official PyTorch implementation as the baseline.

W2/Q2. What happens when the traffic pattern changes?

For Scenario B (AMD MoE), as noted in Appendix A, all performance evaluations in the paper already use a 20% held-out conversation set distinct from the optimization prompts.

For other scenarios, we provide additional performance evaluations under different workload patterns. We get the following insights:

  • A: VibeServe’s bespoke system achieves comparable performance to vLLM for different input/output shapes and concurrencies.
  • C: The inference engine generated by VibeServe consistently outperforms the baseline with speculative decoding for different input distributions with correct predictions. However, it does not generalize to higher concurrency or missing/wrong predictions, which is expected since those cases are out of scope for the agents implemented.
  • D: The bespoke engine can handle the case where requests are arriving one at a time instead of all at once. However, for other prefix lengths, output lengths, or numbers of requests, its performance is worse than vLLM, or it fails to execute. This is expected since agents were instructed to optimize for a particular request shape.
  • E: VibeServe’s system shows generalizability across different chunk sizes, number of concurrent streams, and input datasets, outperforming the baseline across all the cases tested.
  • F: On JSONSchemaBench's held-out set, VibeServe retains an advantage over the baseline, with 3,563 ms for mlx_lm to 1,576 ms for VibeServe, which is a 2.26x speedup.
  • G: We test the VibeServe implementation with 50 denoising steps instead of 20, and the speedup from the baseline is 6.26x, consistent with the case of 20 steps.

Overall, bespoke systems generated by VibeServe agents show some generalizability for cases covered by the initial workload assumption, but not for cases outside the instructions given to the agents. Please refer to Appendix H for more details.

W2/Q2. How much of the serving systems’ functionality do bespoke systems implement?

Each generated system implements only the functionality its target requires, rather than reproducing a general-purpose engine. The two scenarios whose targets resemble standard serving build the most generic machinery.

  • Scenario A has a paged KV cache, FlashInfer attention, CUDA graphs, mixed prefill/decode batching with CPU scheduling that overlaps the GPU, and n-gram and draft-model speculative decoding.
  • Scenario B adds what a 397B MoE on four MI300A GPUs needs: tensor and expert parallelism, custom all-reduce kernels, fused MXFP4 MoE and Gated DeltaNet kernels, and a multi-turn prefix cache holding KV and DeltaNet state snapshots.

The specialized scenarios implement much less, and what they implement follows the workload.

  • Scenario C serves a single-batch latency benchmark, so it has no batching or concurrency; its effort goes into speculative decoding from the user-supplied prediction, a paged KV cache, and CUDA graphs.
  • Scenario D implements a prefix cache for the hybrid architecture that holds both attention KV and recurrent state, plus batching for the concurrent requests that share one 32k-token prefix.
  • Scenario E keeps per-session streaming state and microbatch decoding across sessions.
  • Scenario F needs no batching or KV paging on a laptop. Instead, it implements grammar-constrained decoding with jump-forward, a prompt prefix cache, and grammar-aware speculative decoding with a 1B draft model.
  • Scenario G implements no request-level serving machinery at all, but changes at the model level by reducing guidance passes, removing dead work, porting the denoiser and VAE to MLX, and running the GPU and Neural Engine in parallel.

None of the generated systems includes the general-purpose layer it does not exercise. This is the source of both their advantage and their limitation. Removing generic machinery lets the systems outperform general-purpose engines on their target, but it also means they are not drop-in replacements and can fail on traffic outside the target.

W2/Q2. How hard will it be to maintain bespoke systems?

Bespoke systems shift maintenance from editing code to re-running the optimization. Each generated system is small (3–4k lines for Scenarios C–E, 14k for the MI250 engine, versus orders of magnitude more in vLLM or SGLang) because it implements only the functionality its target needs. Every change is a git commit tied to an issue, a profile, a benchmark result, and a Judge verdict, so a maintainer can see why each optimization exists. The accuracy checker and benchmark stay with the system as regression tests.

When the target changes, the same loop can resume from the current system rather than from scratch, as in our seeded-start experiments (Appendix B), and a run costs hours and tens to hundreds of dollars.

We acknowledge that generated systems usually have thin test suites and assumptions fitted to the target traffic, and they become unusable outside the scope they were originally designed for (Appendix H).

W3. What are the costs for bespoke systems and detailed system design?

Token and GPU costs for running experiments. We added Table 2 in Appendix A to show detailed costs in terms of token consumption, dollar cost, and GPU hours. Since we ran all experiments on Codex or Claude subscriptions, the dollar costs in the table are not what we were billed, but were calculated from token consumption at API list prices as an estimate of what each run would cost under per-token billing.

Because agents repeatedly re-read long, mostly unchanged contexts, 94--99% of input tokens are served from the prompt cache, and these account for a majority of the costs. Even with GPU time included, the runs are not prohibitively expensive. Scenarios A and C–G each cost a few hundred dollars. Scenario B is the outlier, at about $3,900 including its SGLang tuning run and GPU costs, because it is the longest run, consumes the most tokens, and occupies MI300A hardware for over 100 hours.

Details of skills library construction. The serving-systems skills library in VibeServe is a single Agent Skill. Its 400-character description stays in the agent's context at all times. Its body points the agent to 82 reference files (about 12.7k lines), which the agent opens only when they're relevant. The files are grouped into six tiers: model architectures, serving algorithms, frameworks, hardware platforms with their kernel libraries, reference engines, and tooling. Entries are distilled from the source code of vLLM, SGLang, and TensorRT-LLM, from kernel-library documentation, and from the serving literature. The engine entries give the path of each component in that engine's source. Each run receives only its target backend's files, so agents never follow another platform's guidance.

We provide the codebase in supplementary material, and you can inspect the skills under resources/skills/serving-systems. We also added Appendix C on details about the skills library.

Human effort behind skills library. We built most of our skills library ourselves, with agents helping over a few days, distilling our knowledge from existing open-source serving systems like vLLM or SGLang. Users can extend the skills library in different dimensions, with the following estimate of the effort:

  • A new model architecture is about adding just one file under the models/ directory, which is usually a few hundred lines of code.
  • A new workload usually does not require a specialized skills file, unless users want to teach new optimization techniques, which usually takes a few hundred lines of code. VibeServe already provides a comprehensive set of optimization techniques that commonly appear in model serving system optimizations (continuous batching, CUDA graphs, speculative decoding, etc.).
  • A new hardware would require at least three files on hardware architecture, profiling, and theoretical performance floors, which can take a few thousand lines of code at most.

However, these are not always necessary. For example, for new model architectures, agents can often figure out the architecture and optimization methods from reference implementations, even without explicit explanation in the skills. Moreover, it is common for hardware vendors to provide some skills, like for AMD or AWS Trainium, and users can integrate such skills into VibeServe. Note that these are a one-time effort, and the same skills files can be reused across multiple VibeServe runs. Please refer to Appendix C for more details on the skills library.

Planner mechanism. We added Appendix D to describe how the outer loop's planning mechanism works in detail. It has an issue-tracker policy that exposes its backlog to the inner-loop agents. The backlog is a single JSON file of issues served by a small stdio MCP server. Each issue records an id, a type (bug, feature, or perf), a status, a title, a Markdown description, the agent and round that created it, an attempt counter, and an append-only history of state changes. The server offers four tools, three of which are read-only and unrestricted (list_issues, get_issue, and search_issues), and the last one is create_issue. The policy fixes the issue types the agent may file and an optional per-round limit. The Accuracy Judge may file at most one bug per round, so it can report an unrelated defect without turning correctness review into planning. The Performance Evaluator may file bug, feature, and perf issues up to a configurable limit, and it is the main source of new optimization directions.

Reviewer_5Y8fOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents VibeServe, an agentic framework for building LLM serving systems tailored to particular models, hardware, and workloads. It combines an outer planning loop with implementation, correctness checking, and performance evaluation, supported by persistent state and a skills library. The evaluation demonstrates competitive performance across standard serving and several specialized scenarios.

Critical strengths
  1. I find the focus on complete serving systems compelling. Coordinating scheduling, caching, model execution, and communication is a meaningful extension of agentic optimization, and the experiments provide useful evidence that this is feasible.

  2. The paper explains the mechanisms behind the improvements and discusses unsuccessful attempts. This makes the results informative and helps clarify where specialization offers practical benefits.

Critical weaknesses
  1. I am not yet convinced that the comparisons isolate the contribution of the design. The single-agent baseline changes several aspects of the workflow, while the OpenEvolve baseline has a restricted editing scope. Given the substantial contribution of the skills library, it remains unclear how much the proposed orchestration adds beyond a well-equipped coding agent.

  2. Correctness and generalization are less thoroughly established in some scenarios. The image-generation evaluation uses a fixed prompt and seed while allowing approximate computation, and the JSON scenario retains unresolved constraint failures. I think broader validation is needed to establish whether the gains hold under consistent quality requirements.

Questions for authors
  1. How does VibeServe compare with a coding agent given the same skills, tools, persistent memory, independent validation, and compute budget? Would the advantage hold across repeated runs?

  2. Do the main gains persist on inputs held out from optimization? Could the authors report performance alongside correctness or quality, separating semantics-preserving optimizations from those that allow approximation?

6 comments

Thank you for the careful review.

W1/Q1. What does the orchestration add beyond a well-equipped coding agent?

We would like to clarify that for our ablation studies for scenario A (Table 8), the single-agent baseline is equivalent to using existing coding agents, with the same skills library, profilers, read-only reference/checker, persistent git/memory state, and a larger time budget (35 hours, longer than 25 hours for VibeServe to reach the final performance). Only the multi-agent formation has been changed from the VibeServe pipeline. Similarly, Outer-loop OpenEvolve had the inner loop and skills library, isolating the benefit of the issue-tracker Orchestrator (537 tok/s). No-Skill Ablation (1,100 tok/s over 35 rounds) keeps the full multi-agent orchestration (Orchestrator + fresh-context Implementer/Judge/Evaluator), but removes the Skills Library. Together, Table 8 demonstrates that neither a well-equipped agent with the Skills Library alone (617 tok/s) nor the multi-agent orchestration alone (1,100 tok/s) is sufficient. Both are required to synthesize a competitive end-to-end engine (~10,000 tok/s).

Regarding the restricted editing scope of the OpenEvolve baseline: this restriction comes from OpenEvolve's design rather than from our setup. Evolutionary frameworks such as OpenEvolve and AlphaEvolve mutate a user-marked region of a fixed program and keep only a scalar score per candidate. We gave it the same inner loop, skills library, and compute budget, but its mutations do not scale past about 1k lines of code, while a complete serving system spans thousands of lines across several files. We include it because it is the strongest existing search policy for agentic optimization, and its limited scope is exactly the gap that VibeServe's outer loop addresses. To isolate the outer loop's planning state without changing the editing scope, the new ablations below keep VibeServe's full editing scope and remove only the issue tracker or the long-term memory.

Additionally, to further isolate how much the Orchestrator's planning state contributes, we reran Scenario A (Llama-3.1-8B on H100) with Codex GPT-5.6-Sol and compared five configurations over their first 15 rounds: the full pipeline, no skills library, OpenEvolve, and two new ablations that remove planning state: (1) no issue tracker drops the issue roadmap but keeps the round history, and (2) no long-term memory also removes the round history, the plateau summary and the pending claims, and keeps only the best passing idea, the current frontier and the latest profile. (Note that for these experiments, we used newer models since GPT-5.5, which we used in our original evaluations, is excluded from the Codex subscription, and we ran only for a limited number of rounds for this ablation)

The proposed full pipeline climbs fastest and most steadily, reaching 6,963 tok/s by round 7 and 7,374 tok/s by round 13. Removing either form of planning state causes long plateaus. Without memory, the run starts strong but stays at 5,111 tok/s from round 7 to 13 and ends at 5,698. Without the issue tracker, the run stalls hardest: it sits at 1,554 tok/s for eight rounds, even though two of those rounds produced faster candidates that failed the Judge, and it finally reaches 4,395 at round 15. This aligns with the tracker's role in separating implementation failures from unsuitable directions. Without skills, the run plateaus at 1,156 tok/s. OpenEvolve evaluates about twice as many candidates per round but improves only from 2,739 to 3,180 tok/s, ahead of only the no-skill run.

Collectively, these results show that all the components in VibeServe, including multi-agent formation, orchestrator agent-based search policy, and skills library, are necessary to achieve good performance.

Q1. Does the advantage hold across repeated runs?

We provide additional evaluation on Scenarios D, E, and F to show the stability of VibeServe experiments by repeating the same configuration three times. Overall, we confirmed that while there is variance across runs, VibeServe consistently outperforms the baseline systems across multiple runs for all three scenarios:

  • D: 4.78x, 5.43x, and 6.66x over vLLM.
  • E: 2.45x, 2.90x, and 4.59x over vLLM plugin.
  • F: 2.08x, 2.25x, and 2.30x over mlx_lm.

Note that for these experiments, we used newer models (GPT-6.1 Sol) since GPT-5.5, which we used in our original evaluations, is excluded from the Codex subscription. Therefore, these results are not directly comparable to those in Figures 5 and 6. Still, this data shows the stability of VibeServe’s approach across multiple runs.

Please refer to Appendix G for more details.

W2/Q2. Concerns about optimizations that involve approximation?

We would like to clarify that almost all the optimizations implemented by VibeServe agents are semantic-preserving, meaning there is no approximation. The only exception was in Scenario G on MacBook, where agents implemented “CFG striding” in round 11, which reduces the number of diffusion steps for the unconditional half of CFG denoising, since agents found it does not affect the output for the cases in the benchmarks provided, maintaining PSNR > 35dB and SSIM >= 0.98.

To make sure this approximation does not hurt output quality, we additionally test the bespoke implementation on 60 unseen VBench prompts (10 from each of six suites) at a larger 512x512 resolution, with the same seed for VibeServe and the baseline. The bespoke engine generates each image in 20.9s vs. 91.9s for the baseline (4.4x speedup), and its images match their prompts as well as the baseline's do on both SigLIP and CLIP-L image-text scores. On VBench's image metrics, its aesthetic quality is 2% lower and its imaging quality is 3.5% higher. Moreover, even without this optimization, the agent’s implementation achieves ~3.5x lossless speedup over the baseline at round 10. We added this evaluation to Appendix H.

Regarding the unresolved failures in the JSON scenario (F), the “residual failures” that we mentioned in Appendix A are a known limitation of upstream XGrammar itself on the Ultra/Hard tail of JSONSChemaBench, and not a regression introduced by VibeServe. VibeServe’s output matches the exact schema coverage of XGrammar while running 2.6x faster. We have clarified this point in our revised paper. We apologize for the unclear description.

W2/Q2. Do the gains persist on inputs held out from optimization?

For Scenario B (AMD MoE), as noted in Appendix A, all performance evaluations in the paper already use a 20% held-out conversation set distinct from the optimization prompts.

For other scenarios, we provide additional performance evaluations under different workload patterns. We get the following insights:

  • A: VibeServe’s bespoke system achieves comparable performance to vLLM for different input/output shapes and concurrencies.
  • C: The inference engine generated by VibeServe consistently outperforms the baseline with speculative decoding for different input distributions with correct predictions. However, it does not generalize to higher concurrency or missing/wrong predictions, which is expected since those cases are out of scope for the agents implemented.
  • D: The bespoke engine can handle the case where requests are arriving one at a time instead of all at once. However, for other prefix lengths, output lengths, or numbers of requests, its performance is worse than vLLM, or it fails to execute. This is expected since agents were instructed to optimize for a particular request shape.
  • E: VibeServe’s system shows generalizability across different chunk sizes, number of concurrent streams, and input datasets, outperforming the baseline across all the cases tested.
  • F: On JSONSchemaBench's held-out set, VibeServe retains an advantage over the baseline, with 3,563 ms for mlx_lm to 1,576 ms for VibeServe, which is a 2.26x speedup.
  • G: We test the VibeServe implementation with 50 denoising steps instead of 20, and the speedup from the baseline is 6.26x, consistent with the case of 20 steps.

Overall, bespoke systems generated by VibeServe agents show some generalizability for cases covered by the initial workload assumption, but not for cases outside the instructions given to the agents. Please refer to Appendix H for more details.

Regarding reporting performance alongside quality: for Scenario G, the one scenario with an approximate optimization, we report quality together with latency on the unseen prompts (above). For the other scenarios, each result in the main paper passed that scenario's accuracy checker during optimization, while the perturbed-workload runs in Appendix H measure performance only and were not re-gated by the accuracy checker.

AuthorsOpenReview ↗

Following up

Dear Reviewer 5Y8f,

Thank you again for your thoughtful review and constructive feedback. We would like to kindly follow up and ask whether our rebuttal has adequately addressed the concerns raised in your original review.

If any questions or concerns remain, we would be very happy to provide additional experimental results or further clarification.

We greatly appreciate your time and consideration and look forward to hearing your feedback.

Comments on the paper

2 comments

We thank the reviewers for their detailed and constructive reviews. We have uploaded a revised PDF with changes in blue. The main additions are as follows:

  1. Framing (§1, §3.1, §6). We revised the framing of VibeServe to say that VibeServe can start from either an empty repository or an existing engine. We no longer claim that synthesis should replace extending existing engines.
  2. Cost (Appendix A, Table 2). We added Table 2, which summarizes the costs for VibeServe runs for LLM APIs and for GPUs.
  3. Comparison against an existing engine with equivalent time budget (Appendix B, Table 7, Fig. 7). We added new experiments on two VibeServe campaigns with the same agents and time budget on Qwen3.6-35B-A3B on 8x MI250, replaying coding-agent traces, from different starting points (SGLang and from scratch).
  4. Detailed explanation of VibeServe design (Appendix C and D). We added detailed explanations of skills library construction (Appendix C) and issue-tracker MCP tools (Appendix D).
  5. Repeated runs (Appendix G). We added three additional runs for Scenarios D, E, and F to show VibeServe's variance across multiple runs.
  6. Modified traffic and held-out inputs (Appendix H). For six scenarios, we evaluated the VibeServe-generated engines on different workload patterns to show generalizability under perturbed workloads.

Additionally, we are running an ablation regarding Scenario B that starts from SGLang and runs for an equivalent amount of time as the bespoke system we presented in the paper (105h). We will update the rebuttal once the data are ready.

We give detailed rebuttals in individual responses to each reviewer.

Update: Scenario B at an equal 105-hour budget. As promised, we ran VibeServe seeded from SGLang on Scenario B, with the same agents, skills, workload and 105-hour budget as the bespoke run (SGLang porting time included). The bespoke engine reaches 2,242 tok/s vs. 1,637 for the SGLang-seeded run, so it is 1.37x higher at an equal budget (about 1.2x the cost). This matches the 25-hour MI250 result in our first response. The 2.33x in the paper is against a baseline tuned with a smaller budget, so the intro and §4 now report both numbers. The gap comes mainly from MTP speculative decoding (SGLang agents failed to land it), sequence-parallel all-reduce, and more aggressive kernel fusion. Details are in Appendix B (Fig. 8) and Table 2.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.