acceptodds
Withdrawn from ICLR 2027

MOSAIC: More LLMs, Less Cost through Collaborative Reasoning for Compiler Optimizations

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

LLM-guided compiler optimization has recently shown promise. However, current LLM-guided optimizers query a single LLM at every search step, so their cost grows with both that model’s price and the search budget. A smaller model is cheaper but degrades the search: in our experiments, a recent single-LLM optimizer reaches 10.2% (6.0%) lower speedup on GPU (CPU) with gpt-5-mini than with GPT-5.2, which gives up the output quality that motivates using LLMs in the first place. Agentic multi-LLM frameworks add controllers, concurrent calls, and shared memory, whose overhead can offset the savings of cheaper models. This paper shows that the optimization search tree itself can coordinate heterogeneous LLMs: the tree already records which proposals improved the program, and we can augment the tree to record which model made them. That evidence can then be used to decide which model should propose next. To this end, we introduce MOSAIC, in which each Monte Carlo tree search (MCTS) node records a program and the LLM that expands it, and the active LLM proposes both a compiler transformation and the LLM for the next step. We also contribute an LLM-aware UCT that favors smaller models and a course-alteration rule that invokes the largest model only after repeated small-model regressions. Against that single-LLM optimizer run with GPT-5.2 alone and the same number of search iterations and tree parameters, on an NVIDIA GPU (Intel CPU), MOSAIC with eight LLMs improves the geometric-mean speedup on five kernels by 7.2% (14.4%) and the end-to-end Llama-3-8B speedup by 1.61× (1.41×), while reducing kernel compilation time by 1.95× (1.74×) and API cost by 4.47× (4.32×), with GPT-5.2 making only 23.1% (23.9%) of LLM calls. With the same eight models, random or round-robin model choice improves over the single-LLM baseline by only 2.5% on CPU, showing that the gains come from letting the LLMs choose the next model rather than from mixing the pool at random.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
compiler optimization, multi-LLM collaboration, model serving

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_LhehOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MOSAIC extends Reasoning Compiler by allowing several LLMs to share one MCTS tree. Each model proposes both a compiler transformation and the model that should expand the resulting child. A modified UCT rule favors smaller models, while a course-alteration mechanism calls the largest model after repeated regressions. The aim is to reduce optimization cost without sacrificing the performance of the compiled program. Across five kernels, the eight-model configuration reports 7.2% higher geometric-mean speedup on GPU and 14.4% on CPU than Reasoning Compiler with GPT-5.2, while reducing API cost by 4.47× and 4.32×, respectively. The results are promising, although the evaluation leaves open whether contextual model selection is necessary to obtain these gains.

Critical strengths
  1. The paper addresses a practical problem with a relatively simple design. Reusing the search tree to coordinate models avoids a separate controller and lets model selection use information already collected during optimization. Returning the transformation and next-model choice in the same response is an economical implementation choice.
  2. The reported improvements cover both optimization cost and generated-program performance. MOSAIC improves over the GPT-5.2 baseline on all five kernels on both platforms. Compilation time includes hardware measurement and additional course-alteration calls, so the reported savings account for more than regular LLM requests alone.
  3. Table 4 provides useful evidence about the individual components. Removing course alteration reduces the CPU improvement over Reasoning Compiler from 14.4% to 8.3%. Intervening after every regression improves performance slightly further but increases cost substantially. This supports the decision to use selective intervention.
  4. Final speedups are measured on hardware rather than inferred from the search cost model. The experiments with Llama-3.3-70B-Instruct also provide evidence that the approach can work with a different largest model, although those experiments are limited to CPU.
Critical weaknesses
  1. The most relevant adaptive baselines are missing. Random and round-robin selection establish that uniform mixing is insufficient, but they do not establish an advantage over a bandit selector or a cheap-first cascade with large-model fallback. These alternatives are particularly relevant because the paper identifies model selection as its main contribution. The absence of individual results for the small models also leaves unclear how much improvement requires collaboration rather than access to a particularly effective model in the pool.
  2. The routing ablations do not isolate the benefit of using program context. MOSAIC changes both which models are used and how frequently they are used. A fixed, nonuniform mixture could potentially recover part of the improvement by favoring the same models. A useful control would sample models according to MOSAIC’s observed invocation frequencies while removing the dependence on the current program. Without this comparison, the claim that the gains come from contextual next-model decisions is stronger than the evidence supports.
  3. The paper reports means over ten runs but gives no standard deviations, confidence intervals, or individual-run results. This matters for comparisons between nearby configurations: the GPU geometric-mean speedups of the four-model and eight-model variants differ by only about 1.1%. In addition, λ and the intervention threshold are selected using the same CPU kernels on which the main CPU results are reported. Their transfer to GPU is useful evidence, but the CPU evaluation still needs a clearer separation between tuning and testing.
  4. The cost-aware tree policy is not evaluated directly enough. Parameter count is an imperfect proxy for hosted inference cost, and the sizes assigned to GPT-5.2 and gpt-5-mini are assumptions. Replacing GPT-5.2 with an open-weight model reduces this uncertainty but does not show that size is the right quantity to optimize. Table 4 also omits compilation time and API cost for the variant without the size term, making it difficult to assess whether that term achieves its intended purpose. Results under fixed time and dollar budgets would further clarify the tradeoff, especially because course alteration adds calls beyond the nominal sample budget.
  5. The end-to-end results need substantially more experimental detail. Table 3 reports gains of up to 1.86× over Reasoning Compiler, much larger than the relative improvements on the evaluated kernels. This is possible, but the paper does not identify which operators account for the difference. Batch size, sequence length, numerical precision, prefill versus decoding, memory placement, and the allocation of tuning trials across operators are also needed to interpret the result. More broadly, five kernels and one model on a single consumer GPU and CPU provide limited evidence for the wider serving claims.
Questions for authors
  1. How does MOSAIC compare with a bandit-based model selector and a cheap-first cascade using the same model pool and intervention budget? How well do the most frequently selected small models perform individually?
  2. Does contextual routing still improve performance over a fixed sampling distribution matched to MOSAIC’s model invocation frequencies? This would help distinguish the value of program context from the value of selecting a better overall mixture.
  3. Can you provide confidence intervals or per-run results for the main comparisons and ablations? Were the runs paired by random seed, and do the selected hyperparameters retain their advantage on workloads excluded from tuning?
  4. What are the compilation time and API cost of the λ = 0 variant? How sensitive are the results to the assumed closed-model sizes, and what happens if the size term is replaced with measured token cost?
  5. Under equal wall-clock or API-cost budgets, how do MOSAIC and the baselines compare? Please also report the number of hardware-measured candidates, since equal MCTS iteration counts do not necessarily imply equal evaluation work.
  6. What exactly is measured in the end-to-end Llama-3-8B experiment, and which operators explain the 1.61×–1.86× GPU gains? Please provide the workload settings, absolute latencies, memory configuration, and tuning budget per operator.
Reviewer_FBPQOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

MOSAIC aims to reduce the cost of LLM-guided compiler optimization by sharing one search tree across several models. Each model proposes compiler transformations and assigns a model to expand the resulting child. A modified tree policy favors smaller models (LA-UCT), while repeated predicted regressions trigger corrective proposals from the largest model. Experiments on five kernels and end-to-end Llama-3-8B are presented to support improvements in generated-code performance, tuning time, and API cost.

Critical strengths

The paper addresses the cost of LLM-guided search through a simple integration of model selection into existing proposal calls. The evaluation measures generated programs on hardware, covers CPU and GPU targets, and tests an alternative, larger model. Random and round-robin controls, component ablations, and explicit accounting for additional correction calls provide useful evidence about the design.

Critical weaknesses

The paper does not establish whether LLM-selected routing improves on adaptive statistical selection or the other small models used alone. Parameter count is an unvalidated proxy for cost, and the fixed-model baseline lacks a matched correction mechanism to explain the full performance gap. Large end-to-end gains lack an operator-level explanation. Missing workload and baseline details, limited hardware coverage, and unreported variability make the results difficult to assess. Comparisons under equal time or API budgets are also needed to establish the quality-cost tradeoff.

Questions for authors
  1. Unclear basis for model selection

The prompt supplies model identities, sizes, and historical success/error statistics and asks for the smallest suitable model, but the paper does not validate whether the resulting choices match the needs of particular compiler tasks. Table 2 reports invocation frequencies without establishing whether those choices are appropriate. Table 4 compares against random and round-robin selection, but not adaptive selectors based on historical performance, such as those used in Multi-LLM AB-MCTS (NeurIPS 2025) and ShinkaEvolve (ICLR 2026). Although these missing comparisons are acknowledged, the evidence does not distinguish task-specific selection from repeatedly favoring a few effective models. Beyond gpt-5-mini, the other small models are not evaluated individually.

  1. Unvalidated cost preference

LA-UCT favors smaller parameter counts, assuming 300B for GPT-5.2 and 20B for gpt-5-mini. Actual expense depends on token lengths and provider prices, while latency also depends on serving conditions. Using Llama-3.3-70B as the largest model reduces the size assumptions but does not establish their relationship to cost. Compare the size preference with observed per-call cost and test sensitivity to the assumed sizes. Table 4 also omits time and cost for the variants without the size term or correction, leaving their efficiency contributions unclear.

  1. Incomplete explanation of performance gains

MOSAIC uses GPT-5.2 for correction, yet Figure 2 shows that GPT-5.2 alone finds slower programs than MOSAIC with eight models. Table 4 reports an 8.3% CPU gain even without correction, so extra corrective calls cannot explain the entire advantage. However, a fixed-model baseline with the same correction mechanism under matched budgets is still needed to explain the full gap. Table 3 reports a 1.61x GPU end-to-end gain, compared with a 7.2% geometric-mean improvement on the five GPU kernels. These workloads differ; operator timings and schedule comparisons are needed to explain the larger end-to-end gain.

  1. Incomplete quality-cost evaluation

Figure 2 and Table 1 together show better final code at lower reported cost, and extra correction calls are included in cost accounting. However, performance curves use iteration counts rather than time or expenditure. Plot the best measured performance against cumulative tuning time and API cost to show which method reaches a target quality more cheaply across budgets. Report per-model token totals, API versions, generation settings, and the prices used. Separating API waiting, compiler work, and hardware measurement would clarify how much of the time savings depends on provider behavior.

  1. Limited workload and baseline evidence

The paper does not clearly specify benchmark input dimensions, numerical precision, the exact CPU model, or the execution stages covered by end-to-end timing. Report execution latencies and how the end-to-end tuning budget is allocated across tasks. Figure 2's roughly 30x speedups use an insufficiently specified unoptimized IRModule; define its initial schedules and enabled optimizations. Reasoning Compiler is a relevant direct baseline, but a matched MetaSchedule comparison is needed to establish practical competitiveness. The earlier CPU comparison cited in Appendix D does not establish this on the current GPU setup. GPU evaluation is limited to an RTX 2080 Ti; transfer to newer GPU architectures remains untested.

  1. Limited uncertainty and generalization evidence

Ten repetitions are useful, but means alone do not establish whether differences between model pools and ablations are robust. Report variability or confidence intervals and independently remeasure selected programs, since selecting the lowest measured latency can favor timing noise. The same five CPU kernels are used to choose the size weight and correction threshold and to report the main CPU comparisons. Transferring these settings to GPU and another large model provides some evidence, but held-out shapes or workloads would better establish generalization. Separate parameter selection from independent evaluation when claiming a preferred quality-cost balance.

Reviewer_LBSNOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes MOSAIC that builds on Reasoning Compiler and uses several LLMs of different sizes inside one MCTS search for TVM schedule optimization. The main purpose of models of different sizes is to save cost with smaller and less competitive models for faster and easier tasks. Specifically, each tree node of MCTS stores the program and the LLM that will expand it, and the LLM expanding a node also picks which LLM handles the next one. A modified UCT rule adds a bonus for smaller models based on parameter count, and if two small-model steps in a row make things worse, GPT-5.2 is called to propose a replacement. On five kernels and end-to-end Llama-3-8B, on an RTX 2080 Ti and an Intel CPU, an 8-LLM pool is reported to find slightly faster code than Reasoning Compiler with GPT-5.2 alone, at about a quarter of the API cost and roughly half the tuning time.

Critical strengths
  • The core idea is simple and easy to follow.
  • The cost reporting is careful. Absolute dollars and time are given, results are averaged over 10 runs, and there are ablations for the size term and the course alteration rule.
Critical weaknesses

W1. The cost-saving problem doesn’t appear to really matter Based on the paper’s own numbers, tuning one kernel with GPT-5.2 costs around 14 dollars and about 40 minutes, and MOSAIC saves roughly 10 dollars of that. This is a one-time cost, while the tuned kernel runs on every inference after deployment (which the introduction also points out), so even a small speedup would likely be worth much more than the API savings. It would help to see a realistic setting where tuning cost is actually the bottleneck.

W2. it’s not clear how much the LLMs contribute. On top of that, the LLM’s role here is limited: it only names a transformation type, TVM randomly samples the actual parameters, and scoring comes from TVM’s cost model. A few results suggest the model choice may not matter much. gpt-5-mini alone is only a bit behind GPT-5.2 alone, Llama-3.3-70B alone and gpt-5-mini alone end up almost the same on CPU, and even lambda = 1, where the tree ignores reward and just prefers small models, still beats the GPT-5.2 baseline. However, there’s no search without an LLM in the same setup (plain MetaSchedule, or the same MCTS with random transformation choice). If a no-LLM search turns out to be close, then reducing the cost of LLM calls becomes much less important. I’m also not sure how good the final kernels are in absolute terms. All speedups are measured against unoptimized TVM code, which is why they’re so large, and there’s no comparison with what people actually use, such as PyTorch with cuBLAS/cuDNN, torch.compile or TensorRT, or with code-generating kernel agents such as CUDA Agent, KernelAgent or Astra. Since the LLMs here can only choose from a fixed set of TVM schedule transformations, while code-generating agents can also change the algorithm or the dataflow, the gap could be significant, although TVM’s correctness-by-construction is a real advantage. Without these comparisons, it’s hard to tell how much performance is being given up when compared to SOTA kernel agent systems. If the gap is large, saving a few dollars on tuning doesn’t mean much.

W3. The contribution over Reasoning Compiler is incremental, and the added heuristics haven’t been proved to be really effective. Overall the technical contribution over Reasoning Compiler feels quite incremental: it’s the same MCTS setup with a model label on each node, a size bonus in the selection rule, and a rule for calling the large model after regressions. These additions are heuristics, and the evidence doesn’t really show which of them matter. The size bonus uses parameter counts, with the GPT-5.2 and gpt-5-mini sizes assumed, even though the paper has the actual API prices. Lambda and the “call GPT-5.2 after two bad steps” rule are tuned on the same CPU kernels used for reporting. The main novelty, letting the LLM pick the next model, basically asks the LLM to do bandit-style selection from the hit rates and counts in its prompt. But it’s compared only with random and round-robin choice, and only on CPU, not with a simple bandit (e.g. UCB or Thompson sampling) over the same statistics. The paper also doesn’t evaluate any of the routing or cascade methods it cites, such as RouteLLM or AutoMix.

W4. Insufficient explanation of why MOSAIC works. There is not sufficient explanation for why mixing in 7B models gives better code than GPT-5.2 alone. One possible reason is that course-alteration calls come on top of the 1000-sample budget and bad children get replaced, so MOSAIC gets extra proposals and something like a best-of-two at those nodes. Removing course alteration reduces the gain quite a bit, which seems consistent with that.The end-to-end Llama-3-8B gains are much larger than any per-kernel gain, and there the 4-LLM pool beats the 8-LLM pool, which doesn’t fit the kernel results. Finally, the evaluation covers only five kernels, one older consumer GPU and one CPU, which feels narrow for the claims being made.

Questions for authors
  1. How does a search with no LLM perform in your exact setup, for example plain MetaSchedule or the same MCTS with random transformation choice at the same budget?
  2. How do the final kernels compare in absolute latency with PyTorch (cuBLAS/cuDNN), torch.compile or TensorRT on the same hardware, and ideally with a code-generating kernel agent?
  3. How does a simple bandit (UCB or Thompson sampling) over the same per-model statistics compare with letting the LLM pick the next model, on both GPU and CPU?
  4. Why does a pool with 7B models beat GPT-5.2 alone? Is there any analysis of where the better programs come from?
  5. Why does end-to-end Llama-3-8B improve so much more than any single kernel, and why does the 4-LLM pool beat the 8-LLM pool there?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.