FusionAgent: Joint Search over Fusion Boundaries and Kernel Implementations for Whole-Model Inference

Kernel fusion can reduce launch and data-movement overhead in model inference, but optimizing a complete model requires jointly deciding which operations to fuse and how to implement each fused region. Larger regions expose more optimization opportunities but may increase resource pressure and reduce GPU parallelism; their implementation spaces are also difficult to explore under a limited budget, particularly when LLM-generated decisions and code are unreliable. We present FusionAgent, an agentic system that addresses these challenges to optimize full-model inference. GraphAgent searches fusion boundaries over multiple phases, while KernelAgent implements the proposed regions; only verified, measured improvements rewrite the graph and expose larger regions for subsequent phases. KernelAgent uses ECHO search to rank a diverse candidate pool from profiles and search history, decompose plans into independently evaluated units, distinguish an ineffective hypothesis from an inefficient implementation, and reuse experimental outcomes across rounds. Generated kernels are checked against reference values at their graph boundaries and validated again after model integration. Thus, LLMs make the semantic decisions, while deterministic verification and hardware measurements control acceptance. FusionAgent automatically produces optimized models that can be integrated with production inference engines while preserving distributed tensor-parallel execution; we implement and evaluate this integration with SGLang. Across GLM-4.7-358B and LLaDA2.2-Flash workloads on eight NVIDIA H20-3e GPUs, FusionAgent improves end-to-end execution by $1.54$–$1.89\times$ over SGLang v0.5.8 and $1.18$–$1.50\times$ over the more optimized SGLang v0.5.17. On the fused-region benchmarks, KernelAgent outperforms torch.compile in every case and achieves the best latency on three out of four, with up to a $2.17\times$ advantage over the strongest competing kernel agent.