- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors present SEIS, a framework for automatic optimization of inference systems. The framework applies an iterative evolution-esque process that builds on past sessions for more effective optimizations. The authors show some successful trajectories for specifically serving a Qwen3.0-0.6B on a H100 when starting with an existing mini-sglang engine and show that current frontier agents are able to perform inference optimization on the specific settings presented to varying degrees of success.
Critical strengths
I believe the main strength of this paper is the way that it presents a cumulative research codebase across "generations". I think there are many implications that are not discussed in depth in this particular paper regarding inheritance of cumulative knowledge in agentic systems that I think may be valuable for the community under a different work.
Critical weaknesses
I suppose the main problems that I have with the paper are that:
- The claims made in the paper are too grandiose for the scale of experiments presented. The main experiments is done with a 0.6B model on a H100 meaning the optimizations made are likely to be overfitting to the specific configuration with no evidence of this generalizing to production level settings.
- This makes it somewhat less interesting as a standalone work in my opinion, especially with many other adjacent works that specifically analyze the idea of automatically optimizing inference systems. It is currently not clear to me what this paper offers that has not been previously explored in previous works, both cited and not cited in the paper's bibliography.
Questions for authors
-
[Clarification] Do the authors expect the SEIS procedure itself to transfer to more "production-level" settings that use larger models, multiple GPUs, batching/concurrent serving? Could the authors provide evidence that the gains and the analyses are not primarily overfitted to this particular configuration? I am aware of the existing study on Llama 3.2 1B, but production inference is something that introduces meaningfully different problems than the models tested.
-
[Clarification] There are already many existing works that focus on automating inference optimization such as 2602.19594 or 2607.20468, along with the other works already present in the bibliography. Could the authors more clearly articulate what SEIS contributes beyond prior work?
-
[Clarification] The strongest result appears to come from a specific Opus Sync-9 trajectory while other results show substantially weaker gains or even worse task behavior. This leads to be a bit concerned about the sensitivity of the conclusions, especially at the scale of the experiments shown.
-
[Clarification] SEIS uses many long-running agent sessions to obtain its final result but does not report a cost-matched comparison with other non-agentic methods. Claims such as "impressive results" thus feel overly subjective. Having more insights on what having frameworks such as SEIS changes and where with numbers to back it up would help better express the impact of the work.