- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
MOSAIC extends Reasoning Compiler by allowing several LLMs to share one MCTS tree. Each model proposes both a compiler transformation and the model that should expand the resulting child. A modified UCT rule favors smaller models, while a course-alteration mechanism calls the largest model after repeated regressions. The aim is to reduce optimization cost without sacrificing the performance of the compiled program. Across five kernels, the eight-model configuration reports 7.2% higher geometric-mean speedup on GPU and 14.4% on CPU than Reasoning Compiler with GPT-5.2, while reducing API cost by 4.47× and 4.32×, respectively. The results are promising, although the evaluation leaves open whether contextual model selection is necessary to obtain these gains.
Critical strengths
- The paper addresses a practical problem with a relatively simple design. Reusing the search tree to coordinate models avoids a separate controller and lets model selection use information already collected during optimization. Returning the transformation and next-model choice in the same response is an economical implementation choice.
- The reported improvements cover both optimization cost and generated-program performance. MOSAIC improves over the GPT-5.2 baseline on all five kernels on both platforms. Compilation time includes hardware measurement and additional course-alteration calls, so the reported savings account for more than regular LLM requests alone.
- Table 4 provides useful evidence about the individual components. Removing course alteration reduces the CPU improvement over Reasoning Compiler from 14.4% to 8.3%. Intervening after every regression improves performance slightly further but increases cost substantially. This supports the decision to use selective intervention.
- Final speedups are measured on hardware rather than inferred from the search cost model. The experiments with Llama-3.3-70B-Instruct also provide evidence that the approach can work with a different largest model, although those experiments are limited to CPU.
Critical weaknesses
- The most relevant adaptive baselines are missing. Random and round-robin selection establish that uniform mixing is insufficient, but they do not establish an advantage over a bandit selector or a cheap-first cascade with large-model fallback. These alternatives are particularly relevant because the paper identifies model selection as its main contribution. The absence of individual results for the small models also leaves unclear how much improvement requires collaboration rather than access to a particularly effective model in the pool.
- The routing ablations do not isolate the benefit of using program context. MOSAIC changes both which models are used and how frequently they are used. A fixed, nonuniform mixture could potentially recover part of the improvement by favoring the same models. A useful control would sample models according to MOSAIC’s observed invocation frequencies while removing the dependence on the current program. Without this comparison, the claim that the gains come from contextual next-model decisions is stronger than the evidence supports.
- The paper reports means over ten runs but gives no standard deviations, confidence intervals, or individual-run results. This matters for comparisons between nearby configurations: the GPU geometric-mean speedups of the four-model and eight-model variants differ by only about 1.1%. In addition, λ and the intervention threshold are selected using the same CPU kernels on which the main CPU results are reported. Their transfer to GPU is useful evidence, but the CPU evaluation still needs a clearer separation between tuning and testing.
- The cost-aware tree policy is not evaluated directly enough. Parameter count is an imperfect proxy for hosted inference cost, and the sizes assigned to GPT-5.2 and gpt-5-mini are assumptions. Replacing GPT-5.2 with an open-weight model reduces this uncertainty but does not show that size is the right quantity to optimize. Table 4 also omits compilation time and API cost for the variant without the size term, making it difficult to assess whether that term achieves its intended purpose. Results under fixed time and dollar budgets would further clarify the tradeoff, especially because course alteration adds calls beyond the nominal sample budget.
- The end-to-end results need substantially more experimental detail. Table 3 reports gains of up to 1.86× over Reasoning Compiler, much larger than the relative improvements on the evaluated kernels. This is possible, but the paper does not identify which operators account for the difference. Batch size, sequence length, numerical precision, prefill versus decoding, memory placement, and the allocation of tuning trials across operators are also needed to interpret the result. More broadly, five kernels and one model on a single consumer GPU and CPU provide limited evidence for the wider serving claims.
Questions for authors
- How does MOSAIC compare with a bandit-based model selector and a cheap-first cascade using the same model pool and intervention budget? How well do the most frequently selected small models perform individually?
- Does contextual routing still improve performance over a fixed sampling distribution matched to MOSAIC’s model invocation frequencies? This would help distinguish the value of program context from the value of selecting a better overall mixture.
- Can you provide confidence intervals or per-run results for the main comparisons and ablations? Were the runs paired by random seed, and do the selected hyperparameters retain their advantage on workloads excluded from tuning?
- What are the compilation time and API cost of the λ = 0 variant? How sensitive are the results to the assumed closed-model sizes, and what happens if the size term is replaced with measured token cost?
- Under equal wall-clock or API-cost budgets, how do MOSAIC and the baselines compare? Please also report the number of hardware-measured candidates, since equal MCTS iteration counts do not necessarily imply equal evaluation work.
- What exactly is measured in the end-to-end Llama-3-8B experiment, and which operators explain the 1.61×–1.86× GPU gains? Please provide the workload settings, absolute latencies, memory configuration, and tuning budget per operator.