- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents DitronGen, a torch.compile backend that generates distributed kernels in which computation and communication overlap at tile level, starting from single-device PyTorch programs with tensor placements. A compiler infers the required communication and generates communication and computation kernels from predefined descriptions and templates. An LLM agent, guided by expert-written skills, then inserts Triton-distributed synchronization. On a single 8-GPU node, it reports speedups over Torch Inductor of 1.13–3.70× (GEMM), 4.07–15.3× (MoE Group GEMM), 3.99× (attention), and up to 1.14× and 5.22× for dense MLP and MoE layers.
Critical strengths
- Well-motivated problem. Hand-writing kernels that overlap computation and communication takes a lot of expertise. Optimizing them is useful in practice.
- Sensible division of labor. The deterministic parts (inferring communication, generating communication code and computation kernels) go to the compiler. The LLM only inserts synchronization, which limits how much it can get wrong. This is a good attempt to integrate LLM to an existing system. Though I am not sure whether it's a good way since no ablation study is provided.
Critical weaknesses
-
Weak baselines. The main baseline is Torch Inductor, with no overlap and no fusion. Much of the 4–15× Group GEMM speedup comes from fusion. Against the authors' non-overlapped Triton baseline, the gain is only 1.06–1.49×, and dense MLP layers gain just 1.04–1.14×. There are no comparisons with highly optimized MoE layers in vLLM, SGLang or TensorRT-LLM, or with overlap systems such as FLUX and Comet. I would raise my score if these were added.
-
No end-to-end evaluation. All results are kernel- or layer-level on a single node, with no end-to-end training or inference throughput.
-
The LLM's role is weakly justified and evaluated. The expert-written synchronization rules are explicit enough that a compiler pass could likely implement them. There is no ablation and no success-rate or failure analysis. Correctness is checked only against a reference output, which cannot rule out nondeterministic race conditions.
-
A more direct alternative is not explored. The agent could write CUDA/PTX or CuTeDSL directly. In our experience, lower-level interfaces usually yield faster kernels, but the paper does not compare against this approach.
Questions for authors
An end-to-end comparison against existing frameworks would help establish the value of this work. For example, the authors could replace the MoE kernels in vLLM with DitronGen-generated ones and compare against the original vLLM. MoE layers in state-of-the-art frameworks (vLLM, SGLang, TensorRT-LLM) are usually highly optimized. In practice, producing overlapped kernels that beat these is more important than having a system that generates such kernels automatically but performs worse than those frameworks.
2 comments
I thank the authors for the additional experiments, which follow my suggestions. However, they confirm my concern. On the evaluated configurations, the generated kernels reach only 0.91–0.95× of FLUX, and replacing the MoE layers in vLLM gives a 1.03× forward speedup on a single model and sequence length. The large speedups in the paper therefore mainly reflect the weak Torch Inductor baseline. I hope the comparison with state-of-the-art implementations could show a clear advantage, which would make the practical value of this work more convincing. The role of the LLM also remains unclear: the agent-inserted synchronization matches the expert reference, but without an ablation against a rule-based pass it is not clear that an LLM is needed.
I keep my score.
Thank you for suggesting a comparison within an optimized inference framework. We have carried out additional specialized-kernel and vLLM experiments.
1. Specialized kernels.
On the full-mesh Hopper GPU (H200) platform, for GEMM and Group GEMM configurations in Table 3, DitronGen achieves 0.95×(GEMM) and 0.91×(Group GEMM) average performance of the corresponding FLUX implementations. This demonstrates near-specialized performance of the automatically generated kernels on the evaluated configurations.
2. Model-level baselines.
Following your suggestion, we replaced the MoE implementations in vLLM with generated ones. On Qwen3-30B-A3B model, TP=8, we observe 1.03× speedup in full model forwarding over native vLLM under sequence length of 8192. The original layer experiments quantify gains relative to Inductor, and the new model-level experiments show comparable performance to native vLLM under the stated workload conditions.
3. Attribution of overlap.
We agree that a controlled ablation should hold the computation and communication implementations fixed while varying their execution schedule. With the same fused computation and the same communication implementation, overlapped execution gives speedups of 1.08×-1.53× over serial execution. The DitronGen serial implementation is slower than Triton+NCCL, so we retain Triton+NCCL as a stronger serial alternative.
All-Gather + MoE Group GEMM
MoE Group GEMM + Reduce-Scatter
4. LLM agent design.
The two examples in §5.4 illustrate synchronization dependencies and their application to generated kernel code. However, they do not cover all synchronization scenarios. The skill is also used for Group GEMM and attention, showing that it can be generalized to other scenarios. Agent-generated synchronization matches the expert-inserted reference in the tested cases, supporting the quality of the generated kernel.
5. CUDA/PTX/CuTeDSL generation.
These interfaces provide relevant alternative routes for kernel generation. Due to time constraints of the discussion phase, we are temporarily unable to use LLM to generate them for comparison. Our evaluated workflow combines compiler-generated Triton-distributed computation and communication with LLM-generated synchronization. The current results establish that this compiler–LLM division can produce implementations approaching hand-written specialized kernels and can be integrated into an optimized inference framework.