acceptodds
Under review as a conference paper at ICLR 2027

Conjugate Compilation: Self-Improving Agentic Co-Compilation for Speculative Decoding

Abstract

As LLM adoption surges, scaling model serving requires not only adding more GPUs but also unlocking the substantial capacity stranded in those already deployed. Speculative decoding brings two models into the serving process: a small draft model proposes tokens that a larger target model verifies. Yet compiling these models individually leaves their shared execution outside the compiler’s optimization boundary. We therefore ask whether co-locating the draft and target models on the same GPU and co-compiling their execution can reclaim this stranded capacity. We introduce CONJUGATE COMPILATION, which reframes speculative decoding as a cross-model compilation problem and jointly optimizes the draft and target models within their shared execution rather than treating each in isolation. One of our key insights is that cross-model compilation should be informed by the measured performance of the co-located shared execution it produces: an optimization that improves a kernel in isolation need not improve the models when they execute together. To expand cross-model opportunities, we devise operand layouts that interleave blocks from both models within common tiles, co-locating GEMM and non-GEMM work while preserving their mathematics. We realize CONJUGATE COMPILATION through a self-improving agentic search system in which an LLM edits cross-model optimization plans using the compiler and GPU execution as its environment. Each proposed plan is compiled and evaluated through its co-located shared execution, whose measured score, together with kernel-level verdicts and compiler feedback, informs subsequent proposals, while Monte Carlo tree search accumulates rewards to guide exploration. The system also expands its search space and tuning budget when progress stalls and retains tuned kernels and verdicts for reuse across runs. Across six single-GPU speculative-decoding configurations on NVIDIA B300 and RTX 5090 GPUs, CONJUGATE COMPILATION achieves a geometric-mean per-round speedup of 1.37× over conventional execution. Interleaved kernels raise this speedup to 1.45×, reaching 1.59× in the best case. CONJUGATE COMPILATION also improves GPU utilization on average by 1.46× on RTX 5090 and 1.50× on B300. These results demonstrate the potential of cross-model conjugate compilation guided by measurements of the co-located shared execution.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Compiler, Optimization, LLM, Speculative Decoding, Monte Carlo Tree Search

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_nHHAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies joint compilation of co-located draft and target models for speculative decoding on a single GPU. It exposes concurrency across independent batches and represents cross-model optimization choices through operation groups, kernel configurations, and scheduling dependencies. Compatible operations can be combined into interleaved kernels, while an LLM proposes plan edits and MCTS organizes the search using compiler feedback and measured execution performance. The implementation is evaluated across six model/GPU configurations on NVIDIA B300 and RTX 5090. The paper reports a geometric-mean per-round speedup of 1.37× without interleaved kernels and 1.45× with them over conventional sequential execution.

Critical strengths
  • A practical systems problem with a working implementation. Jointly optimizing draft and target execution on a shared GPU is relevant to inference efficiency. The paper implements cross-model operation grouping and interleaved kernels for GEMM and non-GEMM operations.

  • Promising results on recent hardware. Figures 4 and 6 report improvements across several model pairs on B300 and RTX 5090, with additional gains from interleaving. The batch-size and draft-length experiments provide useful sensitivity results.

  • Attention to measurement reliability. Section 3.3 and Appendix A.2 describe alternating timing, cache rotation, paired final measurements, and checks that retained transformations contribute measurable gains. These are useful safeguards for a measurement-driven optimization system.

Critical weaknesses
  • The mechanism and technical positioning of co-compilation are not sufficiently clear. Section 2.2 includes concurrent kernel-configuration selection, join/fork scheduling, and interleaved kernels under the same abstraction. Figure 2 illustrates their execution timelines, but does not show a concrete compiler transformation that makes the technical distinction clear. Section 5 and Appendix B mainly categorize related systems, with limited discussion of the alternatives and their tradeoffs. In particular, the relationship to concurrency-aware compilation such as ConCo [1], which the paper already cites, needs a more specific explanation. An operator-level example showing the original computations, lowering, work layout, and resulting execution would help establish what is new. This is my most significant concern.

  • The evaluation does not establish the advantage over existing solutions or sufficiently isolate co-compilation. Figures 4 and 6 compare conventional sequential execution with the proposed noninterleaved and interleaved configurations. There is no direct comparison with an existing co-serving or concurrency-aware compilation approach. The with/without-interleaving comparison is useful, but does not separate the benefit of cross-batch concurrency from joint configuration selection and compiler transformations. Section 3.3 and Appendix A.2 describe a plain two-stream anchor and a placement control, yet their results are not separately presented in the main evaluation. The reported gains therefore leave the incremental benefit over relevant existing approaches unresolved.

  • The motivation and empirical value of LLM-guided search are not established. Section 3 motivates the agent through a combinatorial search space, expensive measurements, and informative feedback. These properties motivate search, but do not explain why an LLM is particularly suitable for proposing the structured plan edits. The evaluation provides no matched-budget comparison with a conventional optimizer or ablation of the feedback and reuse mechanisms supporting the self-improvement claim. Total compilation time, GPU measurement effort, and LLM costs are also needed to assess the practical value of this choice.

Questions for authors
  • Please explain the main alternatives for same-GPU co-serving and show which limitations require the proposed compiler transformations. Could you provide a concrete lowering example and clarify the technical differences from concurrency-aware compilation such as ConCo [1]?

  • Can you compare with a representative existing approach, or a justified implementation of its relevant mechanism, under the same workload and resources? Please also report the existing two-stream and placement controls, and separate the gains from concurrent execution, joint configuration tuning, and interleaved kernels under matched tuning budgets.

  • With the compiler choices and measurement budget held fixed, how does LLM-guided search compare with a conventional optimizer? What do feedback and reuse contribute, and what are the total optimization costs?

  • As a secondary scope question, which optimization opportunities remain with native MTP, as in DeepSeek-V3 and Qwen3-Next [2, 3], rather than a separate draft Transformer? An analysis of the changed dependencies would help clarify the intended deployment scope.

minors:

The introduction repeats the speculative-decoding motivation and the operand-layout contribution on pages 2–3. Removing these repetitions would leave more space for the mechanism explanation and comparison with existing approaches.

7 comments
AuthorsOpenReview ↗

Response to W1/Q1

We thank the reviewer for the reviews, and we address the three weaknesses and four questions below:

Response to W1/Q1: We explain the relationship between Conjugate Compilation and other co-serving methods in detail. For Conjugate Compilation, we make the following contributions:

  1. The two models of speculative decoding as one compilation unit. The compiler treats draft and target computations as a single optimization boundary. At specific layers and draft steps, it selects operations for conjugation, determines their placement in the co-scheduled round through joint nodes, and selects their kernel configurations. These are cross-model compiler decisions, rather than one resource allocation and code variant applied across each model.

  2. Interleaved cross-model kernels. We decompose GEMMs from different models into output-tile and reduction-slice units and execute them within one launch through a descriptor table. The GEMMs can have unrelated shapes and separately stored operands, with each contributing work in proportion to its arithmetic cost. Each unit accesses and accumulates only its own operation’s data, preserving the original computations. The approach also supports non-GEMM operations, including RMSNorm and residual addition. This changes the contents of the generated kernels, not merely when existing kernels are launched.

  3. Measured shared execution guides cross-model compilation. Plans are ranked by the measured latency of the complete co-scheduled round, subject to legality and numerical checks. This objective accounts for contention, overlap, and synchronization introduced by the transformations. An interleaved kernel that improves isolated latency can still worsen the complete round; consequently, isolated kernel improvements cannot determine which compilation plan is best.

  4. Execution-guided agentic co-compilation. The contribution is not merely attaching an LLM to MCTS. We formulate cross-model optimization as executable plans with structured edits that add, merge, split, or remove conjugated operation groups. The compiler checks and realizes those edits, and GPU execution returns full-round performance alongside kernel verdicts and compiler feedback. MCTS uses measured rewards to select plans for refinement, while the LLM uses accumulated feedback to propose subsequent transformations. The contribution lies in this cross-model search space and its coupling to the execution that the compiler changes, building on prior LLM–MCTS search methods.

The intellectual contribution is to make cooperating models a single compilation unit. Prior approaches treat co-location as the context in which separately compiled models execute: they tune resource-constrained variants, allocate GPU resources, or coordinate execution phases. We instead bring the draft and target computations into a shared compiler optimization boundary, allowing the compiler to transform operations across the two models rather than only optimize each model for sharing. Moving from compiling models individually and coordinating their execution to co-compiling their shared execution is a significant intellectual leap forward.

ConCo optimizes resource-constrained variants; we optimize cross-model execution plans. ConCo generates code under MPS resource limits in 10-percentage-point increments, then selects a code variant and runtime resource limit for each model using concurrent throughput. Its compilation feedback comes from operator measurements under those resource limits, rather than measurements with the particular co-running model. ConCo also identifies uncontrolled operator overlap as a limitation of its full-model optimization. (ConCo, §§3.3–3.5 and §4.5.)

Resource limits alone do not characterize all interference: kernels on separate SMs can still contend for L2 cache and DRAM bandwidth, while kernels sharing SMs can additionally contend for shared-memory bandwidth, warp schedulers, and compute pipelines. (Elvinger et al.)

Conjugate Compilation measures the execution that its transformations actually change. Our LLM–MCTS search selects draft and target operations at particular layers and draft steps, forms conjugated groups, and evaluates the resulting complete shared execution, including synchronization and interleaving overhead. MCTS uses measured round performance to select plans for refinement. The LLM uses that performance, kernel verdicts, and compiler feedback to propose subsequent edits. Selected groups can become single interleaved kernels, while other operations retain separate kernels. Thus, the measured advantage is not merely over serial execution: it persists against concurrent execution, resource partitioning, and concurrency-aware compilation.

AuthorsOpenReview ↗

Response to W1/Q1 - Continued

ConCo selects resources and code at model granularity; we optimize cross-model operation groups. ConCo selects a code variant and runtime resource limit for each model, applying that choice across its operators. We instead select individual draft and target operations at specific layers and draft steps, and can lower each selected group into a single interleaved kernel. Operations outside those groups retain their separate kernels. This exposes optimization choices that a model-wide resource setting cannot express. (ConCo, §3.5.)

ConCo accommodates whichever operators overlap; we explicitly coordinate cross-model execution. ConCo’s authors identify their inability to control which operators overlap as a reason its full-model gains fall below its single-operator gains: one model-wide configuration must accommodate different concurrent operator pairs. We instead encode execution dependencies within a single CUDA graph, permitting overlap, ordering operations when necessary, and bringing selected draft and target operations into one interleaved launch. The compiler can therefore change the execution relationships it optimizes, rather than only tune code around overlap determined by the GPU scheduler. (ConCo, §4.5.)

Co-location does not determine how the models can be execute together. MIG partitions compute and memory resources, which prevents one model from using idle capacity assigned to the other. Plain two-stream execution permits sharing but leaves overlap between ready kernels to the GPU scheduler. Conjugate Compilation uses CUDA streams and events to realize execution dependencies selected by the compiler, and can combine selected operations into interleaved kernels.

FASER’s adaptive speculation and early pruning reduce unnecessary verification work, while its phase management exposes overlap. Conjugate Compilation instead exposes cross-batch overlap—drafting batch (b+1) while verifying batch (b)—and can recompile selected draft and target operations into interleaved kernels. These mechanisms need not be alternatives: FASER’s phase management could supply the remaining work and its dependencies, and our compiler could optimize how that work executes together. Integrating the two would require adapting compilation plans to those runtime-dependent phases and evaluating the resulting shared execution.

For every alternative to Conjugate Compilation, a launch still serves one model and the pairing of overlapping operations is uncontrolled.

For Conjugate Compilation's operator-level example, we show the configuration with Llama-3.1-8B as the target model and Llama-3.2-1B as the draft model. For its MLP down-projection, the original computations are: Target: Y_T = X_T W_T, with X_T of size 40 x 14336 and W_T of size 14336 x 4096. Draft: Y_D = X_D W_D, with X_D of size 8 x 8192 and W_D of size 8192 x 2048. FP16 inputs, FP32 accumulation. Then it goes through separate lowering: There are two tensor core GEMM kernels and one launch each.

Interleaved lowering and work layout is that we decompose GEMMs from different models into output-tile and reduction-slice units and execute them within one launch through a descriptor table. The GEMMs can have unrelated shapes and separately stored operands, with each contributing work in proportion to its arithmetic cost. Each unit accesses and accumulates only its own operation’s data, preserving the original computations. The approach also supports non-GEMM operations, including RMSNorm and residual addition. This changes the contents of the generated kernels, not merely when existing kernels are launched.

AuthorsOpenReview ↗

Response to W2/Q2

To clarify the terminology: the "co-scheduling anchor" of Section 3.3 and Appendix A.2 is not a separate system, and instead co-scheduling has to work with Conjugate Compilation. The co-scheduler exists only as part of Conjugate Compilation. Plain two-stream execution is different: it has no co-scheduler and simply launches the two separately compiled models from the host on two CUDA streams. There is no separate number for placement controls, and placement controls are part of the implementation of Conjugate Compilation with interleaved kernels.

The paper evaluates two configurations of Conjugate Compilation against Conventional:

  1. Conjugate Compilation. The compiler builds one program for the whole round instead of two independently launched models. It partitions both lowered models into ordered segments, down to individual projection stages, assigns target and draft segments to two CUDA streams, inserts CUDA events only where a dependency requires one, and captures the complete round as a single CUDA graph that is replayed as a unit.

  2. Conjugate Compilation with Interleaved Kernels. The compiler additionally generates kernels that span both models. For each draft–target operation group selected by the search of Section 3, it interleaves the operands of both models within common tiles, emits one kernel that executes the group in a single launch, and tunes it with the compiler. This is also called the final-plan in the paper. The blue revisions to Section 3.3, Section 4.1, and Appendix A.2 make this part explicit.

We conducted additional experiments against plain two-stream execution, ConCo, and GPU MIG baselines. MIG is unavailable on RTX 5090. The Llama-3.1-70B / 8B configuration does not fit within the available B300 MIG partitioning. These comparisons test whether the speedups of Conjugate Compilation persist beyond simply co-locating the models. Conjugate Compilation outperforms every measured alternative across all six configurations. Conjugate Compilation with Interleaved Kernels achieves 1.12–1.47× speedup over the strongest alternative in each configuration.

ConfigurationConventionalTwo-StreamConCoGPU MIGConjugate CompilationConjugate Compilation with Interleaved KernelsConjugate Compilation with Interleaved Kernels over Strongest Alternative
Llama-3.2-3B / 1B (RTX 5090)1.00×1.22×1.02×-1.33×1.38×1.13×
Qwen3-4B / 0.6B (RTX 5090)1.00×1.18×1.05×-1.37×1.41×1.19×
Llama-3.1-8B / 1B (B300)1.00×1.04×1.08×1.06×1.42×1.59×1.47×
Qwen3-14B / 0.6B (B300)1.00×1.09×1.12×1.28×1.45×1.55×1.21×
Qwen3-32B / 4B (B300)1.00×1.17×1.07×1.09×1.38×1.43×1.22×
Llama-3.1-70B / 8B (B300)1.00×1.14×1.20×-1.27×1.34×1.12×

MIG partitions compute and memory resources, preventing one model from using idle capacity assigned to the other. Plain two-stream execution permits sharing but leaves overlap between ready kernels to the GPU scheduler. Conjugate Compilation uses CUDA streams and events to realize compiler-selected execution dependencies, and can combine selected operations into interleaved kernels.

Reviewer_HxYzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies how to use spare GPU capacity during speculative decoding by co-scheduling the draft and target models. Conjugate Compilation overlaps verification of one batch with drafting of the next on 2 CUDA streams and captures the round in a single CUDA graph. The compiler jointly schedules operations from both models and can combine some of them into interleaved kernels that run work from both models in one launch. An LLM agent proposes plan edits, Monte Carlo tree search selects plans to refine, and full-round latency is used to evaluate them. Across 6 configurations on NVIDIA B300 and RTX 5090 GPUs, the authors report geometric-mean per-round speedups of 1.37×1.37\times without interleaved kernels and 1.45×1.45\times with them, with a maximum of 1.59×1.59\times. The comparison is against CONVENTIONAL, which compiles the 2 models independently and runs them sequentially.

Critical strengths

S1. The interleaved kernel design is useful. It combines GEMMs of different shapes from the 2 models in one launch, and the same approach also supports row operations. The authors check correctness at every draft step and use repeated, controlled timings to decide whether a change helps.

S2. The gains are reported for every draft and target pair on both GPUs. They become smaller at larger batch sizes and longer speculation lengths, but remain positive. This is consistent with the method benefiting from spare GPU capacity. The paper also reports the extra gain from interleaving separately.

Critical weaknesses

Major

W1. Unreported co-scheduling baseline. I could not find results for the co-scheduling anchor, even though it is the MCTS root and is measured repeatedly during the search (L283-284, L350-351). Appendix A.2 also re-measures plain two-stream baselines without reporting their results (L612-613). This makes it hard to tell how much of the 1.37×1.37\times speedup comes from simply overlapping the 2 models, especially since interleaved kernels add only 3 to 7% on average (L415, L451-452). I am also unclear how Conjugate Compilation without interleaved kernels differs from the placement control. Both appear to retain separate kernels with a changed schedule, and the paper does not explain how the separate-kernel configurations are selected (L228-233). Reporting the anchor, placement-control and final-plan latencies for each configuration would help distinguish these contributions.

W2. Baseline efficiency and matched concurrency. I am not convinced that CONVENTIONAL is sufficiently optimized. Figure 5 places its down-projection kernels at a combined 17.9 TFLOP/s (L429-430), about an order of magnitude below the 8 TB/s HBM memory-bandwidth roof at the same arithmetic intensity. The interleaved kernel also remains well below that roof. Section 4.1 treats CONVENTIONAL as current practice (L376-377), but gives no comparison with production engines such as vLLM or TensorRT-LLM, or with cuBLAS and CUTLASS GEMMs. Measured draft and verification latencies (γTD\gamma T_D, TVT_V), tokens per second and achieved bandwidth for complete rounds would help assess the baseline; Figure 2's 31.3 ms round is labeled an illustration. It is also unclear which of the 2 conventional baselines in L357-358 is plotted in Figures 4 to 8. The tuning comparison is also unclear. Appendix A.1 gives each interleaved GEMM at least the budget of the kernels it replaces (L604-607), but the progressively larger budgets and extra kernel templates appear to apply only to the searched plans (L599-602). Please clarify whether CONVENTIONAL receives the same tuning budget and template choices. I am also unsure whether concurrency is matched. Co-scheduling overlaps drafting for batch b+1 with verification of batch b (L151-153). If this keeps 2 batches in flight while CONVENTIONAL runs 1, the comparison should match the total number of active sequences. In Figure 7, the speedup falls from 1.34×1.34\times to 1.22×1.22\times at batch size 16 (L459-460), and no larger batch is reported.

W3. Unmeasured contribution of search. The paper does not show whether the LLM agent and MCTS improve the search. Section 4 gives no search trajectories, optimization costs or comparisons with random search or a greedy strategy. Reusing tuned kernels and verdicts across phases and resumed runs (L362-364) does not by itself support the claim of a self-improving system. I also do not understand why Equation 11 requires an isolated kernel-level gain before a group is tested in the full round. Could this reject a kernel that is slower on its own but overlaps better with the other model? Equation 7 motivates accounting for cross-model runtime interactions. Is there an example where full-round measurements led the search to choose a different plan than isolated kernel timings would suggest?

Minor

W4. Unclear utilization metric. I am unsure what the reported utilization numbers add to the throughput results. The paper does not name the Nsight Compute metric used (L393-395). For the Llama pair on B300, utilization rises 61%, compared with a 59% throughput gain (L421-425). On the RTX 5090, utilization rises from 50.4 to 73.2% and from 46.2 to 67.9%, compared with throughput gains of 1.38×1.38\times and 1.41×1.41\times (L446-451). Please specify the metric, explain these differences, and describe how it was measured while preserving concurrent execution.

Questions for authors

Q1. Could the authors report the co-scheduling anchor, plain two-stream, placement-control and final-plan latencies across the 6 configurations? Please clarify what Conjugate Compilation without interleaved kernels adds to the anchor.

Q2. Could the authors compare CONVENTIONAL and the proposed method with a well-optimized external baseline at matched concurrency, hardware, models, speculation length and input length? Please report absolute draft, verification and full-round latencies, and clarify whether tuning budgets and kernel-template choices are matched.

Q3. Under the same evaluation budget, how does LLM + MCTS compare with random search or a simple greedy strategy? Please also report the search cost per configuration in plan evaluations, LLM calls and GPU-hours.

Please provide an example of a final accepted plan, state γ\gamma for every configuration, and report the values of δ\delta, zz, β\beta, κ\kappa, α\alpha (Equation 9), mm and Kmax⁡K_{\max}.

For presentation, I recommend adding uncertainty intervals to Figures 4 and 6 to 8, redrawing Figure 2 to show the intended overlap between draft and target segments, and removing the duplicated paragraphs in Section 1 (L058-074 and L075-083, L104-124 and L125-132).

10 comments
AuthorsOpenReview ↗

Response to W1

We thank the reviewer for the reviews. Below, we address the four weaknesses and questions.

Response to W1:

To clarify the terminology: the "co-scheduling anchor" of Section 3.3 and Appendix A.2 is not a separate system, and instead co-scheduling has to work with Conjugate Compilation. The co-scheduler exists only as part of Conjugate Compilation. Plain two-stream execution is different: it has no co-scheduler and simply launches the two separately compiled models from the host on two CUDA streams. There is no separate number for placement controls, and placement controls are part of the implementation of Conjugate Compilation with interleaved kernels.

The paper evaluates two configurations of Conjugate Compilation against Conventional:

Conjugate Compilation. The compiler builds one program for the whole round instead of two independently launched models. It partitions both lowered models into ordered segments, down to individual projection stages, assigns target and draft segments to two CUDA streams, inserts CUDA events only where a dependency requires one, and captures the complete round as a single CUDA graph that is replayed as a unit.

Conjugate Compilation with Interleaved Kernels. The compiler additionally generates kernels that span both models. For each draft–target operation group selected by the search of Section 3, it interleaves the operands of both models within common tiles, emits one kernel that executes the group in a single launch, and tunes it with the compiler. This is also called the final-plan in the paper. The blue revisions to Section 3.3, Section 4.1, and Appendix A.2 make this part explicit.

We report the following table, where all columns except the last report performance relative to conventional serial execution. The last column normalizes Conjugate Compilation with Interleaved Kernels to the strongest of Two-Stream, ConCo, and GPU MIG. Conjugate Compilation with Interleaved Kernels achieves 1.12–1.47× speedup over the strongest alternative in each configuration.

ConfigurationConventionalTwo-StreamConCoGPU MIGConjugate CompilationConjugate Compilation with Interleaved KernelsConjugate Compilation with Interleaved Kernels over Strongest Alternative
Llama-3.2-3B / 1B (RTX 5090)1.00×1.22×1.02×-1.33×1.38×1.13×
Qwen3-4B / 0.6B (RTX 5090)1.00×1.18×1.05×-1.37×1.41×1.19×
Llama-3.1-8B / 1B (B300)1.00×1.04×1.08×1.06×1.42×1.59×1.47×
Qwen3-14B / 0.6B (B300)1.00×1.09×1.12×1.28×1.45×1.55×1.21×
Qwen3-32B / 4B (B300)1.00×1.17×1.07×1.09×1.38×1.43×1.22×
Llama-3.1-70B / 8B (B300)1.00×1.14×1.20×-1.27×1.34×1.12×

MIG is unavailable on RTX 5090. The Llama-3.1-70B / 8B configuration does not fit within the available B300 MIG partitioning.

AuthorsOpenReview ↗

Response to W2

CONVENTIONAL is sufficiently optimized. The 8 TB/s roof is a hardware upper bound, not the bandwidth every GEMM shape can sustain. Small-row GEMMs can be limited by insufficient parallel work and execution overhead as well as memory bandwidth; NVIDIA’s performance guidance explicitly distinguishes these constraints. We compare our kernels with cuBLAS and CUTLASS and our kernels achieve speedup on different configurations:

KernelGEMMs it computesSpeedup over CUTLASS
Llama-3.2-3B / 1B (RTX 5090)
q,k,v, both models48x3072x{3072, 1024, 1024} + 16x2048x{2048, 512, 512}1.99x
o, both models48x3072x3072 + 16x2048x20481.17x
Qwen3-4B / 0.6B (RTX 5090)
q,k,v, both models32x2560x{4096, 1024, 1024} + 8x1024x{2048, 1024, 1024}1.89x
o, both models32x4096x2560 + 8x2048x10241.28x
down, both models32x9728x2560 + 8x3072x10241.12x
draft gate,up8x1024x{3072, 3072}1.26x
Llama-3.1-8B / 1B (B300)
target q,k,v40x4096x{4096, 1024, 1024}1.64x
draft q,k,v8x2048x{2048, 512, 512}3.70x
o, both models40x4096x4096 + 8x2048x20481.21x
draft down8x8192x20481.46x
Qwen3-14B / 0.6B (B300)
target q,k,v40x5120x{5120, 1024, 1024}1.92x
draft q,k,v8x1024x{2048, 1024, 1024}3.02x
draft o8x2048x10241.94x
draft gate,up8x1024x{3072, 3072}1.86x
draft down8x3072x10241.70x
Qwen3-32B / 4B (B300)
q,k,v, both models48x5120x{8192, 1024, 1024} + 8x2560x{4096, 1024, 1024}1.34x
draft o8x4096x25601.48x
draft gate,up8x2560x{9728, 9728}1.27x
Llama-3.1-70B / 8B (B300)
draft q,k,v8x4096x{4096, 1024, 1024}2.79x
draft o8x4096x40961.10x
draft down8x14336x40961.26x
KernelGEMMs it computesSpeedup over cuBLAS
Llama-3.2-3B / 1B (RTX 5090)
q,k,v, both models48x3072x{3072, 1024, 1024} + 16x2048x{2048, 512, 512}1.44x
o, both models48x3072x3072 + 16x2048x20481.20x
Qwen3-4B / 0.6B (RTX 5090)
q,k,v, both models32x2560x{4096, 1024, 1024} + 8x1024x{2048, 1024, 1024}1.30x
Llama-3.1-8B / 1B (B300)
draft q,k,v8x2048x{2048, 512, 512}2.24x
draft down8x8192x20481.30x
Qwen3-14B / 0.6B (B300)
target q,k,v40x5120x{5120, 1024, 1024}1.11x
draft q,k,v8x1024x{2048, 1024, 1024}2.08x
draft o8x2048x10241.13x
draft gate,up8x1024x{3072, 3072}1.39x
Llama-3.1-70B / 8B (B300)
draft q,k,v8x4096x{4096, 1024, 1024}1.89x

These measurements show gains of 1.10–3.70× over CUTLASS and 1.11–2.24× over cuBLAS in the listed cases. They demonstrate that the generated implementations can outperform optimized libraries on the actual operation shapes.

We are currently integrating Conjugate Compilation into vLLM with our industry partners. The compiler transformations can operate within a serving engine: vLLM manages requests and serving state, while Conjugate Compilation optimizes how cooperating models execute on the shared GPU.

Batch-size sensitivity. Figure 7 already tests increased batching: the proposed method retains a 1.22× improvement at batch size 16, compared with 1.34× at batch size 8. As each model supplies more work, less unused capacity remains for cross-model optimization, but the measured benefit persists.

There is a single Conventional baseline, where the draft and target models are compiled independently and executed one after the other. The sentence in §3.3 refers to two ways of launching this same execution, both of which we time in every measurement: (i) the two compiled models invoked from the host one after the other, and (ii) the same sequential round captured once into a single CUDA graph and replayed, which removes host launch overhead exactly as our co-scheduled round does (§2.1). We report Conventional as the faster of the two. In every configuration in the paper this is (ii). Figures 4 and 6–8 are therefore normalized to the captured-graph execution.

CONVENTIONAL receives the same tuning budget and template choices.

As §2.1 describes, what our comparison holds equal is the batch size of each model invocation: every system executes the same work per round, one verification of B(γ+1) token positions and γ draft steps for B sequences, and the reported speedup is the rate at which that work completes. It is a throughput result.

AuthorsOpenReview ↗

Response to W3

Our LLM+MCTS improves the search and we report the search trajectories, optimization costs and comparisons with random search as requested by the reviewer.

As search progresses, Conjugate Compilation with Interleaved Kernels’ speedup over CONVENTIONAL also increases, as shown in the following table:

ConfigurationSearch StartAfter 25 StepsAfter 100 StepsFinal
Llama-3.2-3B/1B1.3251.3571.3651.379
Qwen3-4B/0.6B1.3681.3691.3691.411
Llama-3.1-8B/1B1.4741.4741.4761.590
Qwen3-14B/0.6B1.4551.4581.4591.548
Qwen3-32B/4B1.3851.4221.4221.433
Llama-3.1-70B/8B1.2651.2691.2701.339

We also report optimization costs in the following table:

ConfigurationLLM callsLLM costGPU-hours
Llama-3.2-3B / 1B (RTX 5090)68$0.260.6
Qwen3-4B / 0.6B (RTX 5090)77$0.290.7
Llama-3.1-8B / 1B (B300)68$0.261
Qwen3-14B / 0.6B (B300)78$0.301.3
Qwen3-32B / 4B (B300)69$0.561.0
Llama-3.1-70B / 8B (B300)51$0.252.1

Conjugate Compilation with Interleaved Kernels using LLM + MCTS as the search method is much more sample efficient than Conjugate Compilation with Interleaved Kernels using Random Search. In the following table, we report the sample efficiency of Conjugate Compilation with Interleaved Kernels using Random Search, and the sample efficiency of Conjugate Compilation with Interleaved Kernels using LLM + MCTS. The improvement in sample efficiency using LLM + MCTS is obvious. Sample efficiency is computed by speedup divided by the number of search steps, then multiplied by 10^3.

ConfigurationConjugate Compilation with interleaved kernels (Random Search)Conjugate Compilation with interleaved kernels (LLM + MCTS)Improvement in Sample Efficiency
Llama-3.2-3B / 1B (RTX 5090)0.89.612.0
Qwen3-4B / 0.6B (RTX 5090)0.79.914.1
Llama-3.1-8B / 1B (B300)1.111.610.5
Qwen3-14B / 0.6B (B300)1.59.26.2
Qwen3-32B / 4B (B300)0.910.111.2
Llama-3.1-70B / 8B (B300)1.212.910.8

Equation 11's kernel-level gain check only removes kernels that are not faster than the kernels they replace. An example where the full round overrides isolated timings is the Llama-3.1-8B / 1B configuration. Summing isolated kernel savings, as on the right-hand side of Eq. 7, predicts that interleaving the attention projections (q,k,v and o) on every layer saves 4.7 ms per round, and that interleaving the down projections saves 3.8 ms. The full round reverses the ranking and, for the first plan, the sign. The attention plan lengthens the round by 2.8 ms, while the down-projection plan, which the search accepts, shortens it by 2.3 ms. The attention plan places 54 joint nodes against 28, and each one adds a join and a fork that isolated timings do not see. The accepted plans of Qwen3-4B / 0.6B, Qwen3-32B / 4B and Llama-3.1-70B / 8B likewise leave out admitted kernels with higher isolated speedups than the ones they use.

Reviewer_d7zJOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper aims to address GPU under-utilization (in the context of speculative decoding). Here the draft and target models are typically compiled independently and may be executed sequentially. Its key idea is to jointly optimize their execution assuming they are scheduled concurrently. This includes overlapping phases and tuning kernel configurations for the shared workload, and selectively combining operations from both models into interleaved kernels. An LLM agent guided by Monte Carlo tree search proposes and refines these optimizations using compiler feedback and measurements of the complete execution.

Critical strengths
  • The proposed mechanism seems to have practical use
  • Is a clear case of co-optimizing two different model for utilization
  • The paper is clearly written and easy to read
Critical weaknesses

The major concern I have with this work is that the baseline for comparison is too weak. The baseline assumes both models are run sequentially when there are independent parts that could be co-located without this compiler optimizations for each. This should be the real baseline.

The underutilization reported is also somewhat surprising. Typically, this is handled by increasing the batch size, colocating other workloads, or approaches such as NanoFlow, so the motivation of the work should be more specific about the cases where the proposed technique makes sense.

The novelty of the work is limited. Tuning applications for co-location is not new (e.g., ConCo, which is cited by the authors). The authors also do not compare quantitatively against any other approach such as ConCo or scheduling approaches like FASER, which makes it difficult to gauge the merit of their proposed approach. The way the agents are used for this compilation task is also not particularly novel.

The authors initially motivate the work as the key contribution being agentic tuning for cross-model compilation. But the only aspect of this evaluated is for speculative decoding. It is unclear if this approach works beyond this. The authors should scope the contributions more clearly, especially relative to prior work.

Overall, while these concerns are not addressed, it is difficult to recommend the paper for acceptance.

Questions for authors

Please comment on the points under weaknesses.

6 comments
AuthorsOpenReview ↗

Response to W1

We thank the reviewer for the review and we address the four weaknesses in the comments below:

Response to W1:

We conducted additional experiments against plain two-stream execution, ConCo, and GPU MIG baselines. These comparisons test whether the speedups of Conjugate Compilation persist beyond simply co-locating the models. Conjugate Compilation outperforms every measured alternative across all six configurations. Conjugate Compilation with Interleaved Kernels achieves 1.12–1.47× speedup over the strongest alternative in each configuration.

All columns except the last report performance relative to conventional serial execution. The last column normalizes Conjugate Compilation with Interleaved Kernels to the strongest of Two-Stream, ConCo, and GPU MIG.

ConfigurationConventionalTwo-StreamConCoGPU MIGConjugate CompilationConjugate Compilation with Interleaved KernelsConjugate Compilation with Interleaved Kernels over Strongest Alternative
Llama-3.2-3B / 1B (RTX 5090)1.00×1.22×1.02×-1.33×1.38×1.13×
Qwen3-4B / 0.6B (RTX 5090)1.00×1.18×1.05×-1.37×1.41×1.19×
Llama-3.1-8B / 1B (B300)1.00×1.04×1.08×1.06×1.42×1.59×1.47×
Qwen3-14B / 0.6B (B300)1.00×1.09×1.12×1.28×1.45×1.55×1.21×
Qwen3-32B / 4B (B300)1.00×1.17×1.07×1.09×1.38×1.43×1.22×
Llama-3.1-70B / 8B (B300)1.00×1.14×1.20×-1.27×1.34×1.12×

MIG is unavailable on RTX 5090. The Llama-3.1-70B / 8B configuration does not fit within the available B300 MIG partitioning.

Co-location does not determine how the models can be execute together. MIG partitions compute and memory resources, which prevents one model from using idle capacity assigned to the other. Plain two-stream execution permits sharing but leaves overlap between ready kernels to the GPU scheduler. Conjugate Compilation uses CUDA streams and events to realize execution dependencies selected by the compiler, and can combine selected operations into interleaved kernels.

ConCo optimizes resource-constrained variants, while our Conjugate Compilation optimizes cross-model execution plans. ConCo generates code under MPS resource limits in 10-percentage-point increments, then selects a code variant and runtime resource limit for each model using concurrent throughput. Its compilation feedback comes from operator measurements under those resource limits, rather than measurements with the particular co-running model. ConCo also identifies uncontrolled operator overlap as a limitation of its full model optimization. (ConCo, §§3.3–3.5 and §4.5)

Resource limits alone do not characterize all interference: kernels on separate SMs can still contend for L2 cache and DRAM bandwidth, while kernels sharing SMs can additionally contend for shared-memory bandwidth, warp schedulers, and compute pipelines. (Elvinger et al.)

Conjugate Compilation measures the execution that its transformations change. Our LLM–MCTS search selects draft and target operations at particular layers and draft steps, forms conjugated groups, and evaluates the resulting complete shared execution, including synchronization and interleaving overhead. MCTS uses measured round performance to select plans for refinement. The LLM uses that performance, kernel verdicts, and compiler feedback to propose subsequent edits. Selected groups can become single interleaved kernels, while other operations remain separate kernels. Thus, the measured advantage is not merely over serial execution: it persists against concurrent execution, resource partitioning, and concurrency-aware compilation.

ConCo selects resources and code at model granularity; we optimize cross-model operation groups. ConCo selects a code variant and runtime resource limit for each model, applying that choice across its operators. We instead select individual draft and target operations at specific layers and draft steps, and can lower each selected group into a single interleaved kernel. Operations outside those groups retain their separate kernels. This exposes optimization choices that a model-wide resource setting cannot express. (ConCo, §3.5.)

ConCo accommodates whichever operators overlap, while Conjugate Compilation explicitly coordinates cross-model execution. ConCo’s authors identify their inability to control which operators overlap as a reason its full-model gains fall below its single-operator gains: one model-wide configuration must accommodate different concurrent operator pairs. We instead encode execution dependencies within a single CUDA graph, permitting overlap, ordering operations when necessary, and bringing selected draft and target operations into one interleaved launch. The compiler can therefore change the execution relationships it optimizes, rather than only tune code around overlap determined by the GPU scheduler. (ConCo, §4.5.)

AuthorsOpenReview ↗

Response to W2

Underutilization is well documented. Microsoft Research reports compute utilization below 10% for decode attention kernels in POD-Attention. Orion finds that DNN applications underutilize GPUs even with large batches because successive operators have different compute and memory requirements. NanoFlow likewise attributes unused capacity to heterogeneous compute-, memory-, and network-intensive operations executing sequentially within a device. These observations motivate both runtime coordination and our complementary cross-model compiler optimizations. (POD-Attention, Orion, NanoFlow.)

Additional utilization and energy measurements. We report measurements to distinguish SM activity, tensor-pipe utilization, memory throughput, and energy consumption. Each paired entry below reports Conventional / Conjugate Compilation with Interleaved Kernels.

Configuration (Conventional / Conjugate Compilation with Interleaved Kernels)DRAM Throughput (% of peak)sustained DRAM read (% of peak)SMs active (%)tensor pipe (%)board power (W)energy per round (J)throughput (sustained)
Llama-3.2-3B/1B (5090)50 / 7348 / 6770 / 859.7 / 14.0483 / 5617.85 / 6.451.38×
Qwen3-4B/0.6B (5090)46 / 6841 / 6071 / 836.7 / 9.8453 / 5519.37 / 7.801.41×
Llama-3.1-8B/1B (B300)12 / 1911 / 1871 / 805.1 / 8.2428 / 55613.4 / 10.91.59×
Qwen3-14B/0.6B (B300)12 / 1812 / 1864 / 806.0 / 9.4460 / 60319.7 / 16.71.55×
Qwen3-32B/4B (B300)13 / 1913 / 1779 / 845.8 / 8.0464 / 55553.5 / 47.61.43×
Llama-3.1-70B/8B (B300)18 / 2518 / 2484 / 8710.1 / 14.3562 / 673100.7 / 89.61.34×

Under conventional execution on B300, tensor-pipe utilization is only 5.1–10.1%, and sustained DRAM-read throughput is 11–18% of peak. Substantial SM activity therefore does not imply that compute pipelines or memory bandwidth are saturated. Conjugate Compilation turns more of that available capacity into useful work: tensor-pipe utilization rises by approximately 38–61%, and sustained DRAM-read throughput rises by 31–64%. Although instantaneous board power increases, shorter execution reduces energy per round by approximately 11–19% across the six configurations.

Larger batches. Larger batches amortize weight loading, but each request still accesses its own KV cache. As batch size and context length increase, KV-cache traffic can become the dominant bottleneck rather than disappearing through batching. MagicDec explicitly analyzes this bottleneck shift and uses sparse draft KV caches to sustain speculative-decoding benefits in high-throughput regimes. Thus, batching changes the optimization opportunity; it does not universally remove it. (MagicDec.)

A production-grade vLLM study likewise finds that speculative-decoding gains generally shrink as batch size grows. (Speculative Decoding: Performance or Illusion?) Our Figure 7 examines the interaction: Conjugate Compilation with interleaved kernels retains a 1.22× improvement at batch size 16, compared with 1.34× at batch size 8. Batching and joint compilation therefore provide complementary benefits in the evaluated configurations.

Co-locating other workloads. Co-location exposes concurrent work, but does not by itself determine which kernel implementations, execution dependencies, or cross-model combinations best exploit the shared GPU. The additional comparisons in our response to W1 evaluate this alternative: Conjugate Compilation with interleaved kernels remains 1.12–1.47× faster than the strongest measured two-stream, ConCo, or MIG baseline. Our method also uses work already present in speculative decoding rather than depending on another application being available for co-location.

Combining runtime techniques with cross-model compilation. Runtime coordination and compiler transformations address different decisions and can be combined. Runtime techniques determine which work is available and how it is scheduled; Conjugate Compilation can change how selected draft and target operations are implemented together. The opportunities for combination are concrete: FASER manages speculative-decoding phases dynamically. It adapts speculative length per request, prunes rejected tokens during verification, divides verification into frontiers, and overlaps verification with drafting through spatial multiplexing. Offline profiles of draft and target latency under different batch sizes, token lengths, and SM allocations guide its runtime decisions. (FASER, §§4.1–4.4.)

AuthorsOpenReview ↗

Response to W2 - Continued

FASER’s adaptive speculation and early pruning reduce unnecessary verification work, while its phase management exposes overlap. Conjugate Compilation instead exposes cross-batch overlap—drafting batch (b+1) while verifying batch (b)—and can recompile selected draft and target operations into interleaved kernels. These mechanisms need not be alternatives: FASER’s phase management could supply the remaining work and its dependencies, and our compiler could optimize how that work executes together. Integrating the two would require adapting compilation plans to those runtime-dependent phases and evaluating the resulting shared execution.

NanoFlow exposes intra-device parallelism within a model by dividing work into nano-batches and overlapping operations with different resource demands. Its search uses MILP to choose pipeline structure and resource allocations, informed by kernel profiles and measured interference; it also selects kernel implementations. (NanoFlow, §4.1.)

Nano-batching creates independent operations within the target model, exposing additional work that our compiler could combine with draft operations. Our measurement-guided LLM–MCTS search could be extended to explore nano-batch composition and decide which nano-batch operations to conjugate with draft work. Complete shared-execution measurements would then evaluate the combined effects of intra-model pipelining, cross-model overlap, and interleaved kernels. This offers a direct path to combining NanoFlow’s execution structure with our cross-model transformations, rather than treating the systems as mutually exclusive. Exploring these combinations is promising future work. The present results establish that cross-model compilation improves the evaluated speculative-decoding workloads, including at increased batch size and against measured co-location alternatives; combining it with adaptive phase management or nano-batching could expose further optimization opportunities.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.