- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper provides a novel approach to enabling robust agentic optimization for serving engines, and provides benchmark and its results for optimizing off-the-shelf inference engines using different inference workloads, models, and workloads using agents composed of different LLMs and agentic harnesses.
Critical strengths
The paper provides an interesting agentic workflow design for optimizing inference engines and provides a wide range of results across various inference workloads, hardware, and agent configurations.
Critical weaknesses
-
It is not clear what it is focusing on. Is it the benchmark results? Is it the agentic serving engine optimization protocol design? Is it the optimization techniques and the findings from the benchmarks? Or the problems that arise while running agentic optimizations? I assume the main contribution is the benchmark results, following the title.
-
The benchmark focuses on outdated hardware (A100) and models (Llama 3, Qwen 2.5). This setup lacks state-of-the-art optimization opportunities (native FP8 tensor cores, NVFP4 tensor cores, Tensor Memory Accelerator, Warp-specialization, etc.) and model architecture design (MoE, hybrid attention, etc.). This makes the benchmark results less relevant for evaluating the performance of the suggested agent configurations on modern inference deployment scenarios.
-
The benchmark results are weak. The presented results are from single-shot search results where many of the agent configurations failed to produce a meaningful improvement. Also, the benchmark provides no real information on how inference engines can be optimized. Case study results seem to suggest that the improvements are mostly from hyperparameter search.
Questions for authors
-
Why did you select such outdated hardware and models for benchmarking? It nullifies the relevance of the benchmark results.
-
The design of the agentic optimization infrastructure for the inference engine is quite interesting. However, despite such a sophisticated design, the improvements that the agents found seem to be mostly from hyperparameter tuning. Were there any other architectural or algorithmic improvements found by the agents?