- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies joint compilation of co-located draft and target models for speculative decoding on a single GPU. It exposes concurrency across independent batches and represents cross-model optimization choices through operation groups, kernel configurations, and scheduling dependencies. Compatible operations can be combined into interleaved kernels, while an LLM proposes plan edits and MCTS organizes the search using compiler feedback and measured execution performance. The implementation is evaluated across six model/GPU configurations on NVIDIA B300 and RTX 5090. The paper reports a geometric-mean per-round speedup of 1.37× without interleaved kernels and 1.45× with them over conventional sequential execution.
Critical strengths
-
A practical systems problem with a working implementation. Jointly optimizing draft and target execution on a shared GPU is relevant to inference efficiency. The paper implements cross-model operation grouping and interleaved kernels for GEMM and non-GEMM operations.
-
Promising results on recent hardware. Figures 4 and 6 report improvements across several model pairs on B300 and RTX 5090, with additional gains from interleaving. The batch-size and draft-length experiments provide useful sensitivity results.
-
Attention to measurement reliability. Section 3.3 and Appendix A.2 describe alternating timing, cache rotation, paired final measurements, and checks that retained transformations contribute measurable gains. These are useful safeguards for a measurement-driven optimization system.
Critical weaknesses
-
The mechanism and technical positioning of co-compilation are not sufficiently clear. Section 2.2 includes concurrent kernel-configuration selection, join/fork scheduling, and interleaved kernels under the same abstraction. Figure 2 illustrates their execution timelines, but does not show a concrete compiler transformation that makes the technical distinction clear. Section 5 and Appendix B mainly categorize related systems, with limited discussion of the alternatives and their tradeoffs. In particular, the relationship to concurrency-aware compilation such as ConCo [1], which the paper already cites, needs a more specific explanation. An operator-level example showing the original computations, lowering, work layout, and resulting execution would help establish what is new. This is my most significant concern.
-
The evaluation does not establish the advantage over existing solutions or sufficiently isolate co-compilation. Figures 4 and 6 compare conventional sequential execution with the proposed noninterleaved and interleaved configurations. There is no direct comparison with an existing co-serving or concurrency-aware compilation approach. The with/without-interleaving comparison is useful, but does not separate the benefit of cross-batch concurrency from joint configuration selection and compiler transformations. Section 3.3 and Appendix A.2 describe a plain two-stream anchor and a placement control, yet their results are not separately presented in the main evaluation. The reported gains therefore leave the incremental benefit over relevant existing approaches unresolved.
-
The motivation and empirical value of LLM-guided search are not established. Section 3 motivates the agent through a combinatorial search space, expensive measurements, and informative feedback. These properties motivate search, but do not explain why an LLM is particularly suitable for proposing the structured plan edits. The evaluation provides no matched-budget comparison with a conventional optimizer or ablation of the feedback and reuse mechanisms supporting the self-improvement claim. Total compilation time, GPU measurement effort, and LLM costs are also needed to assess the practical value of this choice.
Questions for authors
-
Please explain the main alternatives for same-GPU co-serving and show which limitations require the proposed compiler transformations. Could you provide a concrete lowering example and clarify the technical differences from concurrency-aware compilation such as ConCo [1]?
-
Can you compare with a representative existing approach, or a justified implementation of its relevant mechanism, under the same workload and resources? Please also report the existing two-stream and placement controls, and separate the gains from concurrent execution, joint configuration tuning, and interleaved kernels under matched tuning budgets.
-
With the compiler choices and measurement budget held fixed, how does LLM-guided search compare with a conventional optimizer? What do feedback and reuse contribute, and what are the total optimization costs?
-
As a secondary scope question, which optimization opportunities remain with native MTP, as in DeepSeek-V3 and Qwen3-Next [2, 3], rather than a separate draft Transformer? An analysis of the changed dependencies would help clarify the intended deployment scope.
minors:
The introduction repeats the speculative-decoding motivation and the operand-layout contribution on pages 2–3. Removing these repetitions would leave more space for the mechanism explanation and comparison with existing approaches.
7 comments
Response to W1/Q1 - Continued
ConCo selects resources and code at model granularity; we optimize cross-model operation groups. ConCo selects a code variant and runtime resource limit for each model, applying that choice across its operators. We instead select individual draft and target operations at specific layers and draft steps, and can lower each selected group into a single interleaved kernel. Operations outside those groups retain their separate kernels. This exposes optimization choices that a model-wide resource setting cannot express. (ConCo, §3.5.)
ConCo accommodates whichever operators overlap; we explicitly coordinate cross-model execution. ConCo’s authors identify their inability to control which operators overlap as a reason its full-model gains fall below its single-operator gains: one model-wide configuration must accommodate different concurrent operator pairs. We instead encode execution dependencies within a single CUDA graph, permitting overlap, ordering operations when necessary, and bringing selected draft and target operations into one interleaved launch. The compiler can therefore change the execution relationships it optimizes, rather than only tune code around overlap determined by the GPU scheduler. (ConCo, §4.5.)
Co-location does not determine how the models can be execute together. MIG partitions compute and memory resources, which prevents one model from using idle capacity assigned to the other. Plain two-stream execution permits sharing but leaves overlap between ready kernels to the GPU scheduler. Conjugate Compilation uses CUDA streams and events to realize execution dependencies selected by the compiler, and can combine selected operations into interleaved kernels.
FASER’s adaptive speculation and early pruning reduce unnecessary verification work, while its phase management exposes overlap. Conjugate Compilation instead exposes cross-batch overlap—drafting batch (b+1) while verifying batch (b)—and can recompile selected draft and target operations into interleaved kernels. These mechanisms need not be alternatives: FASER’s phase management could supply the remaining work and its dependencies, and our compiler could optimize how that work executes together. Integrating the two would require adapting compilation plans to those runtime-dependent phases and evaluating the resulting shared execution.
For every alternative to Conjugate Compilation, a launch still serves one model and the pairing of overlapping operations is uncontrolled.
For Conjugate Compilation's operator-level example, we show the configuration with Llama-3.1-8B as the target model and Llama-3.2-1B as the draft model. For its MLP down-projection, the original computations are: Target: Y_T = X_T W_T, with X_T of size 40 x 14336 and W_T of size 14336 x 4096. Draft: Y_D = X_D W_D, with X_D of size 8 x 8192 and W_D of size 8192 x 2048. FP16 inputs, FP32 accumulation. Then it goes through separate lowering: There are two tensor core GEMM kernels and one launch each.
Interleaved lowering and work layout is that we decompose GEMMs from different models into output-tile and reduction-slice units and execute them within one launch through a descriptor table. The GEMMs can have unrelated shapes and separately stored operands, with each contributing work in proportion to its arithmetic cost. Each unit accesses and accumulates only its own operation’s data, preserving the original computations. The approach also supports non-GEMM operations, including RMSNorm and residual addition. This changes the contents of the generated kernels, not merely when existing kernels are launched.
Response to W2/Q2
To clarify the terminology: the "co-scheduling anchor" of Section 3.3 and Appendix A.2 is not a separate system, and instead co-scheduling has to work with Conjugate Compilation. The co-scheduler exists only as part of Conjugate Compilation. Plain two-stream execution is different: it has no co-scheduler and simply launches the two separately compiled models from the host on two CUDA streams. There is no separate number for placement controls, and placement controls are part of the implementation of Conjugate Compilation with interleaved kernels.
The paper evaluates two configurations of Conjugate Compilation against Conventional:
-
Conjugate Compilation. The compiler builds one program for the whole round instead of two independently launched models. It partitions both lowered models into ordered segments, down to individual projection stages, assigns target and draft segments to two CUDA streams, inserts CUDA events only where a dependency requires one, and captures the complete round as a single CUDA graph that is replayed as a unit.
-
Conjugate Compilation with Interleaved Kernels. The compiler additionally generates kernels that span both models. For each draft–target operation group selected by the search of Section 3, it interleaves the operands of both models within common tiles, emits one kernel that executes the group in a single launch, and tunes it with the compiler. This is also called the final-plan in the paper. The blue revisions to Section 3.3, Section 4.1, and Appendix A.2 make this part explicit.
We conducted additional experiments against plain two-stream execution, ConCo, and GPU MIG baselines. MIG is unavailable on RTX 5090. The Llama-3.1-70B / 8B configuration does not fit within the available B300 MIG partitioning. These comparisons test whether the speedups of Conjugate Compilation persist beyond simply co-locating the models. Conjugate Compilation outperforms every measured alternative across all six configurations. Conjugate Compilation with Interleaved Kernels achieves 1.12–1.47× speedup over the strongest alternative in each configuration.
| Configuration | Conventional | Two-Stream | ConCo | GPU MIG | Conjugate Compilation | Conjugate Compilation with Interleaved Kernels | Conjugate Compilation with Interleaved Kernels over Strongest Alternative |
|---|---|---|---|---|---|---|---|
| Llama-3.2-3B / 1B (RTX 5090) | 1.00× | 1.22× | 1.02× | - | 1.33× | 1.38× | 1.13× |
| Qwen3-4B / 0.6B (RTX 5090) | 1.00× | 1.18× | 1.05× | - | 1.37× | 1.41× | 1.19× |
| Llama-3.1-8B / 1B (B300) | 1.00× | 1.04× | 1.08× | 1.06× | 1.42× | 1.59× | 1.47× |
| Qwen3-14B / 0.6B (B300) | 1.00× | 1.09× | 1.12× | 1.28× | 1.45× | 1.55× | 1.21× |
| Qwen3-32B / 4B (B300) | 1.00× | 1.17× | 1.07× | 1.09× | 1.38× | 1.43× | 1.22× |
| Llama-3.1-70B / 8B (B300) | 1.00× | 1.14× | 1.20× | - | 1.27× | 1.34× | 1.12× |
MIG partitions compute and memory resources, preventing one model from using idle capacity assigned to the other. Plain two-stream execution permits sharing but leaves overlap between ready kernels to the GPU scheduler. Conjugate Compilation uses CUDA streams and events to realize compiler-selected execution dependencies, and can combine selected operations into interleaved kernels.
Response to W1/Q1
We thank the reviewer for the reviews, and we address the three weaknesses and four questions below:
Response to W1/Q1: We explain the relationship between Conjugate Compilation and other co-serving methods in detail. For Conjugate Compilation, we make the following contributions:
The two models of speculative decoding as one compilation unit. The compiler treats draft and target computations as a single optimization boundary. At specific layers and draft steps, it selects operations for conjugation, determines their placement in the co-scheduled round through joint nodes, and selects their kernel configurations. These are cross-model compiler decisions, rather than one resource allocation and code variant applied across each model.
Interleaved cross-model kernels. We decompose GEMMs from different models into output-tile and reduction-slice units and execute them within one launch through a descriptor table. The GEMMs can have unrelated shapes and separately stored operands, with each contributing work in proportion to its arithmetic cost. Each unit accesses and accumulates only its own operation’s data, preserving the original computations. The approach also supports non-GEMM operations, including RMSNorm and residual addition. This changes the contents of the generated kernels, not merely when existing kernels are launched.
Measured shared execution guides cross-model compilation. Plans are ranked by the measured latency of the complete co-scheduled round, subject to legality and numerical checks. This objective accounts for contention, overlap, and synchronization introduced by the transformations. An interleaved kernel that improves isolated latency can still worsen the complete round; consequently, isolated kernel improvements cannot determine which compilation plan is best.
Execution-guided agentic co-compilation. The contribution is not merely attaching an LLM to MCTS. We formulate cross-model optimization as executable plans with structured edits that add, merge, split, or remove conjugated operation groups. The compiler checks and realizes those edits, and GPU execution returns full-round performance alongside kernel verdicts and compiler feedback. MCTS uses measured rewards to select plans for refinement, while the LLM uses accumulated feedback to propose subsequent transformations. The contribution lies in this cross-model search space and its coupling to the execution that the compiler changes, building on prior LLM–MCTS search methods.
The intellectual contribution is to make cooperating models a single compilation unit. Prior approaches treat co-location as the context in which separately compiled models execute: they tune resource-constrained variants, allocate GPU resources, or coordinate execution phases. We instead bring the draft and target computations into a shared compiler optimization boundary, allowing the compiler to transform operations across the two models rather than only optimize each model for sharing. Moving from compiling models individually and coordinating their execution to co-compiling their shared execution is a significant intellectual leap forward.
ConCo optimizes resource-constrained variants; we optimize cross-model execution plans. ConCo generates code under MPS resource limits in 10-percentage-point increments, then selects a code variant and runtime resource limit for each model using concurrent throughput. Its compilation feedback comes from operator measurements under those resource limits, rather than measurements with the particular co-running model. ConCo also identifies uncontrolled operator overlap as a limitation of its full-model optimization. (ConCo, §§3.3–3.5 and §4.5.)
Resource limits alone do not characterize all interference: kernels on separate SMs can still contend for L2 cache and DRAM bandwidth, while kernels sharing SMs can additionally contend for shared-memory bandwidth, warp schedulers, and compute pipelines. (Elvinger et al.)
Conjugate Compilation measures the execution that its transformations actually change. Our LLM–MCTS search selects draft and target operations at particular layers and draft steps, forms conjugated groups, and evaluates the resulting complete shared execution, including synchronization and interleaving overhead. MCTS uses measured round performance to select plans for refinement. The LLM uses that performance, kernel verdicts, and compiler feedback to propose subsequent edits. Selected groups can become single interleaved kernels, while other operations retain separate kernels. Thus, the measured advantage is not merely over serial execution: it persists against concurrent execution, resource partitioning, and concurrency-aware compilation.