acceptodds
Under review as a conference paper at ICLR 2027

SEIS: Self-Evolving Inference Systems

Abstract

The concept of "intelligence per watts" indicates the significance of efficient large language model inference systems, which has been an important research area. However, prior works focus mostly on optimizing certain parts, such as kernels or memory, within the large system. In this work, we take a holistic approach where we apply an agentic self-evolution approach to optimize the whole system end-to-end. Our SEIS (Self-Evolving Inference Systems) autonomously optimizes the entire mini-SGLang engine without any human interventions through iterative sessions with inherited experiences and code changes. Serving Qwen3-0.6B on H100, the resulting engine reaches 3.27x throughput than the original mini-SGLang implementation and beats SoTA engines like vLLM, TensorRT, and SGLang in the single-request workload. The correctness of the optimized inference engine by SEIS is tested in terms of numerical difference and downstream accuracy on math and long-context retrieval tasks. To further understand the mechanisms behind the speedup, we conduct detailed trajectory and final optimized code artifact analysis, which provides significant insight on the evolution dynamics and concrete system optimization techniques. The success of SEIS implies that agentic self-evolution, when done right, can achieve impressive results given a complex objective, and the core challenge might shift to the meta-evolution of the concrete objective given an abstract task.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Self-Evolving, Inference Engine, Agent, Harness

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_3GzuOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The authors present SEIS, a framework for automatic optimization of inference systems. The framework applies an iterative evolution-esque process that builds on past sessions for more effective optimizations. The authors show some successful trajectories for specifically serving a Qwen3.0-0.6B on a H100 when starting with an existing mini-sglang engine and show that current frontier agents are able to perform inference optimization on the specific settings presented to varying degrees of success.

Critical strengths

I believe the main strength of this paper is the way that it presents a cumulative research codebase across "generations". I think there are many implications that are not discussed in depth in this particular paper regarding inheritance of cumulative knowledge in agentic systems that I think may be valuable for the community under a different work.

Critical weaknesses

I suppose the main problems that I have with the paper are that:

  1. The claims made in the paper are too grandiose for the scale of experiments presented. The main experiments is done with a 0.6B model on a H100 meaning the optimizations made are likely to be overfitting to the specific configuration with no evidence of this generalizing to production level settings.
  2. This makes it somewhat less interesting as a standalone work in my opinion, especially with many other adjacent works that specifically analyze the idea of automatically optimizing inference systems. It is currently not clear to me what this paper offers that has not been previously explored in previous works, both cited and not cited in the paper's bibliography.
Questions for authors
  1. [Clarification] Do the authors expect the SEIS procedure itself to transfer to more "production-level" settings that use larger models, multiple GPUs, batching/concurrent serving? Could the authors provide evidence that the gains and the analyses are not primarily overfitted to this particular configuration? I am aware of the existing study on Llama 3.2 1B, but production inference is something that introduces meaningfully different problems than the models tested.

  2. [Clarification] There are already many existing works that focus on automating inference optimization such as 2602.19594 or 2607.20468, along with the other works already present in the bibliography. Could the authors more clearly articulate what SEIS contributes beyond prior work?

  3. [Clarification] The strongest result appears to come from a specific Opus Sync-9 trajectory while other results show substantially weaker gains or even worse task behavior. This leads to be a bit concerned about the sensitivity of the conclusions, especially at the scale of the experiments shown.

  4. [Clarification] SEIS uses many long-running agent sessions to obtain its final result but does not report a cost-matched comparison with other non-agentic methods. Claims such as "impressive results" thus feel overly subjective. Having more insights on what having frameworks such as SEIS changes and where with numbers to back it up would help better express the impact of the work.

Reviewer_RSqyOpenReview ↗
Rating
3
Confidence
3
Informativeness
2(−1)
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a framework named SEIS (Self-Evolving Inference Systems). Different from previous works that mainly focus on improving certain parts of an inference system, the proposed SEIS employs an agent self-evolution method to directly and autonomously optimize the entire inference system. Experimental results show that the proposed SEIS have better performance than many widely-used pipelines such as vLLM. It further conducts analysis to provide extra insights.

Critical strengths

This submission has the following critical strengths:

  • The proposed method is clear and easy to understand.
  • The exploration of optimizing the whole inference system using the idea of self-evolution is novel and interesting.
  • Experiments validate the effectiveness of SEIS in refining inference systems and extra analysis is also insightful.
Critical weaknesses

This submission has the following critical weaknesses:

  • The current related work section could be improved to better clarify the contributions and uniqueness of SEIS.
  • The two models used for evaluation, Qwen3-0.6B and Llama-3.2-1B-Instruct, are too small. They can hardly be considered large language models. In real-world inference systems, we typically need to serve much larger models.
Questions for authors

I have the these questions or suggestions:

  • It would be better to add more direct comparisons in related work section.
  • It would be better to conduct experiments using large language models rather than models below 1B size.
  • Some typos could be corrected:
    • Line 150, 'the success an attempt’ -> 'the success of an attempt'
    • Line 205-206, 'More discusses can be found' -> 'more discussion can be found'
Reviewer_Kqp1OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SEIS uses coding agents to optimize a whole LLM inference engine (mini-sglang) over many two-hour sessions. Each session inherits the code and written reports of the previous one. In some conditions, parallel chains also share their work every N sessions. The agent stays fixed; only the engine changes. The main study runs two agents in four conditions. The best engine reaches 3.27× vanilla throughput on natural-length requests, and it is 2.81–3.10× faster than tuned vLLM, SGLang, and TensorRT-LLM in this setting.

Critical strengths

Strong engineering result. The best engine changes 30 files across kernels, attention, KV cache, and scheduling. Most of the gain holds on natural-length requests and with speculation turned off. The audit shows it keeps the full output head and attention.

Informative session histories. For example, speculative decoding only paid off after the kernels became faster. Also, an off-by-one error in inherited reports made it look useless for several sessions.

Honest reporting of failures. Section 6 and Appendix F clearly document engines that passed the checks while dropping vocabulary rows, removing EOS, or reusing stale state.

Critical weaknesses

The lessons about organizing sessions rest on one run per condition. Each condition is run only once, and the results are not consistent. For Opus, Sync-9 is clearly best, while Sync-3 is no better than plain Chain. For GPT, Sync-3 is best, while Sync-9 is about the same as Chain. The large Opus Sync-9 gain mainly comes from one event: one chain got speculative decoding to work and the other two adopted it. Branching is never run as its own condition; its reference is built from only nine first sessions per agent. There is also no code-only condition, so the value of written reports is not measured. The general claims, such as Branching being "often the worst design", are not yet supported.

The pipeline is not reliable without human checks. According to Appendix F, 13 of the 24 final engines either skip part of the computation or fall outside the paper's accuracy margin. The authors found these problems by auditing the code by hand; the gates did not catch them. The checks were also redesigned by the authors after pilot failures. This weakens the abstract's claim of optimization "without any human interventions". It also makes the proposal to let agents evolve their own evaluation premature.

The setting is narrow and favors this kind of optimization. The study uses a 0.6B model (plus a 1B model in the appendix), one GPU, and batch size 1. At this scale, a large share of the time goes to kernel launches and host overhead, which is mainly what the agents removed. vLLM, SGLang, and TensorRT-LLM are built mainly for batched serving, so beating them here says little about real serving workloads.

The cost is not reported. The study runs 480 sessions of up to two hours each, but the paper gives no total GPU hours or API cost. So the speedup cannot be weighed against the search effort, or against simpler setups given the same budget.

Minor: The top-1 agreement threshold is ≥90% in Sec. 3.2 but ≥0.88 in Appendix D.

Questions for authors
  1. Can you repeat key conditions (e.g., Opus Chain vs. Sync-9) and add a code-only condition?
  2. Why is GSM8K not a gate? Would a cheap automatic check, such as comparing full-vocabulary logits on more varied prompts, catch the restricted engines without a manual audit?
  3. How much of the speedup remains for larger models or with concurrent requests?
  4. What was the total GPU time and API cost of the main study?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.