acceptodds
Under review as a conference paper at ICLR 2027

DitronGen: Compiler–LLM Coordination for Adaptive Kernel Generation and Computation–Communication Overlap

Abstract

Optimizing training and inference of large artificial intelligence models on distributed platforms is challenging because inter-device communication can limit efficiency of computation. High-level frameworks typically optimize computation and communication separately, while manually implementing overlapped kernels demands substantial expertise. We present DitronGen, a framework that coordinates a compiler and an LLM agent for adaptive kernel generation and computation–communication overlap. Starting from single-device programs with user-specified tensor placements, the compiler infers communication requirements and adapts computation and communication implementations to hardware architecture, interconnect topology, and communication dimensions. The LLM agent generates synchronization code using Triton-distributed primitives, adapting synchronization operations and their placement to the data dependencies and control flow of the generated kernels to enable tile-level overlap across different communication scenarios. This division of responsibilities coordinates compiler-based generation of computation and communication kernels with LLM-based generation confined to synchronization. DitronGen can be used as a custom backend for \tt torch.compile. On NVIDIA Hopper GPUs, it achieves speedups of 1.13×–3.70×, 4.07×–15.3×, and 2.55×–5.59× over Torch Inductor for GEMM, Group GEMM, and Attention, respectively, and up to 1.14× and 5.22× for dense MLP and MoE layers.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Distributed Computing, ML Compiler, Computation-Communication Overlap, LLM for Code Generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_HQACOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents DitronGen, a torch.compile backend that generates distributed kernels in which computation and communication overlap at tile level, starting from single-device PyTorch programs with tensor placements. A compiler infers the required communication and generates communication and computation kernels from predefined descriptions and templates. An LLM agent, guided by expert-written skills, then inserts Triton-distributed synchronization. On a single 8-GPU node, it reports speedups over Torch Inductor of 1.13–3.70× (GEMM), 4.07–15.3× (MoE Group GEMM), 3.99× (attention), and up to 1.14× and 5.22× for dense MLP and MoE layers.

Critical strengths
  1. Well-motivated problem. Hand-writing kernels that overlap computation and communication takes a lot of expertise. Optimizing them is useful in practice.
  2. Sensible division of labor. The deterministic parts (inferring communication, generating communication code and computation kernels) go to the compiler. The LLM only inserts synchronization, which limits how much it can get wrong. This is a good attempt to integrate LLM to an existing system. Though I am not sure whether it's a good way since no ablation study is provided.
Critical weaknesses
  1. Weak baselines. The main baseline is Torch Inductor, with no overlap and no fusion. Much of the 4–15× Group GEMM speedup comes from fusion. Against the authors' non-overlapped Triton baseline, the gain is only 1.06–1.49×, and dense MLP layers gain just 1.04–1.14×. There are no comparisons with highly optimized MoE layers in vLLM, SGLang or TensorRT-LLM, or with overlap systems such as FLUX and Comet. I would raise my score if these were added.

  2. No end-to-end evaluation. All results are kernel- or layer-level on a single node, with no end-to-end training or inference throughput.

  3. The LLM's role is weakly justified and evaluated. The expert-written synchronization rules are explicit enough that a compiler pass could likely implement them. There is no ablation and no success-rate or failure analysis. Correctness is checked only against a reference output, which cannot rule out nondeterministic race conditions.

  4. A more direct alternative is not explored. The agent could write CUDA/PTX or CuTeDSL directly. In our experience, lower-level interfaces usually yield faster kernels, but the paper does not compare against this approach.

Questions for authors

An end-to-end comparison against existing frameworks would help establish the value of this work. For example, the authors could replace the MoE kernels in vLLM with DitronGen-generated ones and compare against the original vLLM. MoE layers in state-of-the-art frameworks (vLLM, SGLang, TensorRT-LLM) are usually highly optimized. In practice, producing overlapped kernels that beat these is more important than having a system that generates such kernels automatically but performs worse than those frameworks.

2 comments

Thank you for suggesting a comparison within an optimized inference framework. We have carried out additional specialized-kernel and vLLM experiments.

1. Specialized kernels.

On the full-mesh Hopper GPU (H200) platform, for GEMM and Group GEMM configurations in Table 3, DitronGen achieves 0.95×(GEMM) and 0.91×(Group GEMM) average performance of the corresponding FLUX implementations. This demonstrates near-specialized performance of the automatically generated kernels on the evaluated configurations.

2. Model-level baselines.

Following your suggestion, we replaced the MoE implementations in vLLM with generated ones. On Qwen3-30B-A3B model, TP=8, we observe 1.03× speedup in full model forwarding over native vLLM under sequence length of 8192. The original layer experiments quantify gains relative to Inductor, and the new model-level experiments show comparable performance to native vLLM under the stated workload conditions.

3. Attribution of overlap.

We agree that a controlled ablation should hold the computation and communication implementations fixed while varying their execution schedule. With the same fused computation and the same communication implementation, overlapped execution gives speedups of 1.08×-1.53× over serial execution. The DitronGen serial implementation is slower than Triton+NCCL, so we retain Triton+NCCL as a stronger serial alternative.

All-Gather + MoE Group GEMM

Case#TorchTriton+NCCLDitronGen(Non-Overlap)
14.23×1.10×1.12×
26.07×1.15×1.28×
39.01×1.21×1.38×
49.35×1.49×1.53×

MoE Group GEMM + Reduce-Scatter

Case#TorchTriton+NCCLDitronGen(Non-Overlap)
16.34×1.06×1.08×
27.43×1.13×1.18×
34.07×1.19×1.24×
415.3×1.27×1.35×

4. LLM agent design.

The two examples in §5.4 illustrate synchronization dependencies and their application to generated kernel code. However, they do not cover all synchronization scenarios. The skill is also used for Group GEMM and attention, showing that it can be generalized to other scenarios. Agent-generated synchronization matches the expert-inserted reference in the tested cases, supporting the quality of the generated kernel.

5. CUDA/PTX/CuTeDSL generation.

These interfaces provide relevant alternative routes for kernel generation. Due to time constraints of the discussion phase, we are temporarily unable to use LLM to generate them for comparison. Our evaluated workflow combines compiler-generated Triton-distributed computation and communication with LLM-generated synchronization. The current results establish that this compiler–LLM division can produce implementations approaching hand-written specialized kernels and can be integrated into an optimized inference framework.

I thank the authors for the additional experiments, which follow my suggestions. However, they confirm my concern. On the evaluated configurations, the generated kernels reach only 0.91–0.95× of FLUX, and replacing the MoE layers in vLLM gives a 1.03× forward speedup on a single model and sequence length. The large speedups in the paper therefore mainly reflect the weak Torch Inductor baseline. I hope the comparison with state-of-the-art implementations could show a clear advantage, which would make the practical value of this work more convincing. The role of the LLM also remains unclear: the agent-inserted synchronization matches the expert reference, but without an ablation against a rule-based pass it is not clear that an LLM is needed.

I keep my score.

Reviewer_BVQLOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents DitronGen, a framework that combines compiler-based kernel generation with LLM-assisted synchronization to enable tile-level computation–communication overlap in distributed GPU workloads. The compiler handles communication inference, communication implementation, and computation kernel adaptation, while an LLM agent inserts synchronization operations using Triton-distributed primitives. The framework is integrated as a custom backend for torch.compile and evaluated on GEMM, Group GEMM, self-attention, MLP, and MoE workloads.

The results show improvements over Torch Inductor across the evaluated workloads, with particularly large gains for Group GEMM and MoE. The idea of separating structured code generation from synchronization generation is reasonable, as synchronization placement depends on data dependencies and control flow that can be cumbersome to handle through fixed compiler transformations alone. However, the current evaluation does not fully establish how much of the performance improvement comes from overlap, kernel fusion, and implementation-specific optimizations, nor does it sufficiently characterize the reliability and generality of the LLM-based approach.

Critical strengths

A reasonable division of responsibilities. The compiler handles communication inference and kernel generation, while the LLM focuses on synchronization insertion. This avoids relying on the LLM to generate the entire kernel and makes the approach more structured.

Attention to practical synchronization issues. The paper discusses producer–consumer dependencies across different tile partitions and the placement of waits along reduction and non-reduction dimensions. These are important implementation details for achieving correct tile-level overlap.

Support for different communication patterns and hardware configurations. The framework covers several collective communication patterns and tensor dimensions, with experiments on Hopper and Ada Lovelace GPUs. The use of predefined communication descriptions provides a practical way to accommodate different interconnect topologies.

Evaluation beyond isolated kernels. In addition to GEMM and attention, the paper evaluates Group GEMM, dense MLP, and MoE workloads. The comparison against a fused Triton implementation without overlap is also useful for separating the benefit of overlap from that of operator fusion.

Critical weaknesses

The contribution of the LLM component is not sufficiently established. The paper shows that the agent can generate synchronization code that passes validation, but it does not compare this approach with rule-based synchronization insertion, manually written synchronization, or other automated methods. It is therefore unclear whether an LLM is necessary for the proposed task or whether a smaller set of compiler analyses and synchronization templates could achieve similar results.

The performance evaluation does not fully isolate the source of the gains. The reported speedups over Torch Inductor are substantial in some cases, but the comparison is affected by differences in kernel fusion and implementation. For Group GEMM, the paper acknowledges that Torch Inductor does not fuse local routing operations in the evaluated configuration. The comparison against Triton without overlap is more informative, but the additional gains are relatively modest, at 1.06×–1.49×. More consistent baselines are needed to establish the benefit attributable specifically to the proposed overlap mechanism.

Correctness and robustness are evaluated on a limited set of cases. The agent reportedly succeeds within one or two calls for all evaluated cases, but the paper does not provide the number of independent generation runs, success rates across repeated runs, or sensitivity to prompt and model changes. Validation against a PyTorch reference is useful, but it does not establish correctness across a broader range of shapes, control-flow structures, and synchronization dependencies.

The scope of generality remains limited. The framework relies on predefined communication patterns, computation templates, and expert-authored synchronization skills. Self-attention is represented as a custom FX operator, and the conclusion identifies multi-operator graph overlap as future work. These choices make the current implementation understandable, but also limit the strength of the claim that the approach generalizes to diverse distributed workloads.

The evaluation does not account for compilation and agent overhead in end-to-end performance. All reported timings exclude compilation and LLM-side code generation, which can take from a few seconds to approximately two minutes. This may be acceptable for workloads with substantial reuse, but the paper does not quantify the break-even point or discuss how generated implementations are cached and reused in realistic execution settings.

The performance portability story is incomplete. The evaluation covers two NVIDIA GPU architectures, but it is unclear how much effort is required to add a new communication topology, collective pattern, or computation operator. Since predefined descriptions and templates are central to the framework, the maintenance and extension cost is important to its practical value.

Questions for authors

How does DitronGen compare with a rule-based synchronization insertion method or manually implemented synchronization, both in generation effort and runtime performance? Can the authors provide an ablation that isolates the contribution of the LLM?

For the Group GEMM and MoE experiments, how much of the speedup comes from operator fusion, and how much comes from computation–communication overlap? Could the authors provide a consistent comparison using the same fused computation kernels, with and without overlap?

What are the synchronization-generation success rates across repeated runs and unseen workload configurations? How often does the agent produce code that passes numerical validation but fails under different shapes or execution conditions?

Since compilation and agent generation are excluded from the reported latency, how many executions are required to amortize this overhead? How are generated kernels cached, and what happens when input shapes or hardware configurations change?

How much manual effort is required to add a new collective communication pattern or computation operator? Which parts of the process can be handled automatically, and which require new expert-authored descriptions, templates, or synchronization skills?

1 comment

Thank you for distinguishing the framework's design from the evidence needed to establish its benefits. We have addressed the questions below.

1. Synchronization generation versus alternatives.

The synchronization examples in §5.4 illustrate how dependency principles are applied to generated kernel code. However, they do not cover all synchronization scenarios. We also use the same skill for Group GEMM and attention, and the agent can handle them correctly. Expert-inserted synchronization matches agent-inserted synchronization and has the same performance on the evaluated cases. This comparison supports the quality of the agent-generated implementations relative to the expert reference.

2. Fusion versus overlap.

We add a direct comparison using the same fused computation and the same communication implementation, executed serially or with tile-level overlap. For cases listed in Table 3, DitronGen with overlap gains speedups of 1.08×-1.53× over DitronGen without overlap. We also retain the Triton(fused-computation)+NCCL baseline, since it is faster than our serial communication. These controls quantify overlap within our implementation and its net benefit over the NCCL-based alternative.

All-Gather + MoE Group GEMM

Case#TorchTriton+NCCLDitronGen(Non-Overlap)
14.23×1.10×1.12×
26.07×1.15×1.28×
39.01×1.21×1.38×
49.35×1.49×1.53×

MoE Group GEMM + Reduce-Scatter

Case#TorchTriton+NCCLDitronGen(Non-Overlap)
16.34×1.06×1.08×
27.43×1.13×1.18×
34.07×1.19×1.24×
415.3×1.27×1.35×
Reviewer_WHB8OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

DitronGen generates computation and communication kernels from templates, then uses an LLM to insert synchronization for tile-level overlap. It is exposed as a torch.compile backend with user-specified tensor placements. Experiments on eight-GPU machines report up to 5.22x speedup for MoE layers and 1.14x for dense MLP layers over Torch Inductor.

Critical strengths
  • Using a compiler to generate kernels and limiting the LLM to synchronization is a reasonable design.
Critical weaknesses
  • The performance comparisons are quite weak. The large MoE gains include fusion of local routing operations that the Torch Inductor baseline leaves separate. For grouped GEMM, the reported 4.07x-15.3x speedups become 1.06x-1.49x against fused Triton computation with NCCL communication. Even this comparison changes both the communication implementation and overlap. Comet and FLUX are cited but not benchmarked, while relevant MoE systems such as MoonEP (Chen et al., 2026) and Mixture-of-Kittens (Sul et al., 2026) are not discussed. There is no indication where the generated kernels stand against specialized implementations on comparable hardware. The evaluation also stops at kernels and layers, without full-model training or inference results.

  • The benefit of using an LLM agent is not established. Code generation happens at compilation time, and the layer benchmarks explicitly use kernels generated offline. The shapes, placements, and target hardware are available when the code is generated. An expert implementation or a general coding agent could optimize the entire kernel and schedule, yet neither is compared. There is also no comparison with a compiler pass implementing the stated synchronization rules. Speedups over Torch Inductor do not establish the value of restricting an agent to synchronization around predefined templates.

  • The experimental reporting is inadequate for an ML systems paper. The platforms are described only as eight Hopper GPUs with a full-mesh topology and eight Ada Lovelace GPUs with a NUMA topology. Exact GPU models, memory capacities, the actual interconnect and its bandwidth are not specified. Software versions, Torch Inductor settings, and a detailed timing protocol are also missing. For a paper about computation-communication overlap, these are basic experimental details. Their omission makes the performance results difficult to interpret or reproduce.

The idea is reasonable, but there are significant flaws in the experimental design and reporting. The weak baselines do not establish competitive performance, and the value of the LLM agent remains unclear. I therefore recommend a strong reject.

Chen et al. (2026). MoonEP: A Perfectly Balanced Expert Parallelism Library via Dynamic Redundant Experts. https://github.com/MoonshotAI/MoonEP

Sul et al. (2026). Mixture-of-Kittens: our open-source MoE megakernel for NVL72s. https://cursor.com/blog/mixture-of-kittens

Questions for authors

I understood the authors' intent and the manuscript. I do not have specific questions that need to be addressed.

1 comment

Thank you for highlighting the importance of controlled comparisons and specialized baselines. The original comparisons measure the combined effects of computation implementation/fusion, communication implementation, and overlap. Our additional measurements separate these effects.

1. Reproducibility.

The platform of eight Hopper GPUs is composed of eight H200 GPUs, each with 141GB HBM. They are connected by full-mesh NVLink with 900GB/s bandwidth. The platform of eight Ada Lovelace GPUs is composed of eight L20 GPUs, each with 48GB memory. They are divided into two NUMA groups, each with 4 GPUs. Inside a NUMA group, GPUs are connected by PCIe with 64GB/s bandwidth. The connection between NUMA groups is SMP interconnect. Timing uses CUDA events and are measured with 5 to 10 warmups and 20 to 100 repetitions.

2. Performance attribution.

We now compare three configurations with the same fused computation: (i) NCCL communication with serial execution, (ii) DitronGen communication with serial execution, and (iii) the same DitronGen computation and communication with tile-level overlap. The direct overlap gain (speedup of (iii) over (ii)) is 1.08×-1.53×. The comparison with Triton(fused computation)+NCCL is 1.06×-1.49×. The DitronGen serial implementation is slower than NCCL, giving a larger speedup. We retain both controls: the DitronGen serial execution isolates overlap within our generated implementation, and Triton+NCCL measures the benefit against a stronger serial implementation.

All-Gather + MoE Group GEMM

Case#TorchTriton+NCCLDitronGen(Non-Overlap)
14.23×1.10×1.12×
26.07×1.15×1.28×
39.01×1.21×1.38×
49.35×1.49×1.53×

MoE Group GEMM + Reduce-Scatter

Case#TorchTriton+NCCLDitronGen(Non-Overlap)
16.34×1.06×1.08×
27.43×1.13×1.18×
34.07×1.19×1.24×
415.3×1.27×1.35×

3. Specialized implementations.

On the full-mesh H200 platform, for GEMM and Group GEMM configurations in Table 3, DitronGen achieves 0.95×(GEMM) and 0.91×(Group GEMM) average performance of the corresponding FLUX implementations. These results place the generated implementations close to the specialized baseline.

For specialized MoE systems, MoonEP focuses on expert parallelism and load balancing through dynamic redundant experts, while Mixture-of-Kittens targets NVL72 training megakernels. Our current evaluation covers tensor-parallel workloads on a single machine with 8 GPUs. Thus, fair comparison between DitronGen and these two MoE systems cannot be covered in current evaluation.

4. Model-level evaluation.

We also integrate the generated MoE implementations into vLLM, evaluating Qwen3-30B-A3B model with TP=8 on the full-mesh H200 platform. We observe 1.03× speedup in full model forwarding over native vLLM under sequence length of 8192.

5. Role of the LLM.

The examples in §5.4 illustrate synchronization dependencies and their application to GEMM. However, they do not cover all synchronization scenarios. The agent also handles Group GEMM and Attention using the same skill, demonstrating its generalizability. In the evaluated cases, agent-inserted synchronization matches the expert reference, providing evidence of generated output quality.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.