acceptodds
Under review as a conference paper at ICLR 2027

Hardware-Aware Features for CUTLASS Kernel Selection

Abstract

GPU libraries such as CUTLASS expose tens of thousands of semantically equivalent kernels for a single operation, making exhaustive autotuning expensive and execution-free selection difficult. Existing analytical selectors require hand-designed performance rules, while learned selectors operate on raw configuration parameters and must infer hardware consequences from data. We introduce a hardware-aware representation for CUTLASS kernel selection that augments candidate configurations with statically computable estimates of induced hardware behavior. We construct a dataset of 4.9 million CUTLASS kernels and train gradient-boosted and neural learning-to-rank models to rank candidates within each problem. On held-out exhaustive evaluation problems, hardware-aware representations reduce selection regret by up to 40% relative to structural baselines and 64.2% relative to NVIDIA’s matrix-multiply heuristics. We further evaluate data-efficient cross-precision and epilogue-fusion transfer within CUTLASS GEMM, showing that explicitly representing candidate-induced hardware behavior provides a useful inductive bias for learned kernel selection.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
machine learning, CUTLASS, GPU, GEMM, autotuning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_WriYOpenReview ↗
Rating
4
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses how to choose a fast GPU kernel from the many implementations available in CUTLASS without exhaustively benchmarking every option. The main idea describe each candidate through its expected hardware behavior, such as data reuse, memory demands, and GPU utilization, rather than relying only on configuration settings. A learned model then uses these descriptions to rank candidates and select an implementation with potentially good performance. The main contribution is a set of statically computed features that estimate each kernel's data reuse, memory traffic, resource usage, and pipeline behavior. These features augment the kernel configuration parameters, allowing a learned model to rank candidates without executing them. Experiments show that the approach reduces mean selection regret, the throughput lost relative to the fastest candidate, from 17.3% with NVIDIA’s heuristic to 6.2%, and supports adaptation to different numerical precisions and fused operations.

Critical strengths
  • Searching the large CUTLASS kernel space requires substantial overhead for both compilation and profiling. The proposed approach uses statically computed hardware features with standard learning models, making it relatively straightforward to implement while avoiding candidate kernel benchmarking at selection time. The evaluation shows very clear gain over SOTA approaches.
Critical weaknesses

The paper claims that hardware-aware features enable data-efficient transfer across numerical precisions. However, the transfer experiments only fine-tune pretrained BF16 selectors without comparing against models trained from scratch using the same target-domain data. This makes it difficult to determine whether the reported performance improvements arise from transferable knowledge or simply from the available target-domain supervision. Moreover, data efficiency is evaluated in terms of the fraction of GEMM shapes rather than the actual number of kernel measurements, even though each shape may involve thousands of measured configurations. A comparison against from-scratch training under identical measurement budgets would provide stronger evidence for the claimed transfer efficiency.

Questions for authors

How does fine-tuning a BF16-pretrained selector compare with training the same model from scratch using exactly the same target-domain measurements? This comparison is important for determining whether the reported data efficiency comes from knowledge transferred across precisions or simply from the available target-domain training data.

1 comment

We thank the reviewer for their feedback and for recognizing the practical relevance of our selection approach and the performance improvements over existing approaches.

How does BF16 pretraining compare with training from scratch under identical target-domain budgets?

Our transfer experiments are intended to demonstrate that the hardware-aware representation remains effective across numerical precisions, with strong performance even when fine-tuning uses a limited fraction of target-domain shapes. To quantify the additional benefit of pretrained parameters independently of the representation, we are currently preparing and running the requested experiments; we will submit a revision in the next few days including the requested from-scratch comparison under identical target-domain measurement budgets. We will also report the exact number of target-domain kernel measurements used at each training fraction, ensuring that data efficiency is evaluated in terms of measurement cost and not just shape coverage.

Reviewer_qgsDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies execution-free selection of CUTLASS GEMM kernels on Hopper: for a given problem (M, N, K, operand layout, data type), pick one kernel from the roughly 61,000 statically valid BF16 configurations per layout without running any of them. Each candidate is described by its configuration plus statically computed estimates of how it would use the hardware, in four groups: work decomposition (edge-tile waste, wave efficiency, Stream-K applicability), memory behavior (arithmetic intensity, operand re-streaming, working set relative to L2), resource pressure (shared memory, a register proxy, estimated occupancy) and pipeline behavior (TMA versus WGMMA time, pipeline fill and amortization). The choice of quantities is guided by within-problem correlations in 408 Nsight Compute profiles. XGBoost and MLP rankers are trained on about 4.9 million BF16 measurements on GH200 (593 shapes × 4 layouts, 1,000–3,000 candidates per group, three quarters drawn from the top of a static score), and the top-scored kernel is used. On 68 held-out shape–layout groups in which every valid kernel was measured (3.9 million measurements), the best MLP has 6.2% mean selection regret and XGBoost 6.4%, against 7.4% and 10.7% with configuration-only features, 17.3% for the top-1 recommendation of nvMatmulHeuristics (nvMMH) and 71.5% for a random pick. On 8,000 further BF16 problems the selectors are 1.13–1.14× faster than nvMMH (geometric mean), and 1.12× on 151 DeepBench GEMMs. Fine-tuned on 5–100% of a target corpus, they reach 1.20–1.24× over nvMMH on FP32 and 1.12–1.19× on FP8, and about parity with nvMMH on FP16 GEMMs with fused epilogues.

Critical strengths

The representation is well motivated, and it helps. The within-problem correlation analysis is a useful lesson in itself: counters that track throughput across problems, such as SM activity (+0.910 globally), carry no ranking signal once the problem is fixed (−0.001), while DRAM throughput and tensor-core activity do. The features are built as static proxies for the mechanisms behind these two. Adding them cuts mean regret from 10.7% to 6.4% for XGBoost and from 60.8% to 23.7% for a ridge regressor, and lets much smaller models reach low regret (Figure 3). This is the paper's central claim, and for tree models and small networks the evidence supports it.

The ground truth is strong. In the 68 held-out groups every valid kernel was measured, so regret is measured against the true best kernel in the catalogue. Against it, the learned selectors clearly beat nvMMH: 6.2% versus 17.3% mean regret, a kernel within 5% of the best in 63.2% versus 10.3% of the groups, and lower regret than even the best measured kernel of nvMMH's shortlist (12.0%). On 8,000 further problems the MLP is 1.14× faster than nvMMH and faster on 79.6% of them, although nvMMH is given its best measured schedule variant there. This makes the improvement over nvMMH credible.

Critical weaknesses
  1. The paper does not show that execution-free top-1 selection is a better use of resources than measuring a few candidates. This weighs most in my rating: the paper motivates execution-free selection by the cost of autotuning (Section 1) but gives no end-to-end cost accounting or break-even comparison.

Cost. Training uses about 4.9 million BF16 measurements, plus 593-shape FP32, FP8 and fusion corpora whose numbers of measurements are not reported, yet the paper neither estimates how many downstream problems repay this nor compares spending the same budget on tuning the target problems directly.

Regret. The best model's kernel is within 5% of the best in only 63.2% of the groups and loses more than 15% in over one group in ten (Table 13, Figure 2b). For a fixed shape that runs many times, as many deployed GEMMs do, measuring once removes a loss that model selection repeats on every call.

Measured shortlists. Measuring the top 250 candidates of the paper's static sampling score gives about 2.5% mean regret on the same 68 groups (Appendix G), against 6.2% for the model. Yet nvMMH is the only external baseline: neither shortlist-then-measure with the model's own top-K, the workflow of the NVIDIA blog post the paper cites (Barclay et al., 2025), nor cited tuners such as Ansor (Zheng et al., 2020) are compared at equal cost.

  1. The evidence for generality is narrower than the claims.

Scope. All data come from one GPU model (GH200) and CUTLASS SM90 GEMM, so the architecture descriptors are constant in training and no other GPU or operator is tested. Yet Section 6 contrasts the method with device-transfer frameworks, and the conclusion suggests reuse "across precisions, operators, and architectures".

Transfer. A new dtype needs target-dtype measurements, and at 5% of the target shapes the selected kernel runs on only 77.1% (FP32, MLP) and 68.2% (FP8, XGBoost) of the problems (Table 15), which Section 5.4 does not report. With fused epilogues the selectors only match nvMMH (1.00× for XGBoost and 0.97× for the MLP with all fusion data).

Size of the representation effect. For the MLP, the full-versus-structural gap under the same objective lies inside the error bars and reverses for LambdaRank (Figure 11), and the 7.4% → 6.2% headline compares two different objectives. On DeepBench the structural MLP matches the full one (1.13× versus 1.12×, Table 16), so the effect is clear only for XGBoost and small models.

Questions for authors

Shortlists (clarification, from existing data). From the exhaustive measurements, what are the mean regret and the within-5% rate of the best measured kernel among the model's top-K, for K = 1, 4, 16, 64 and 256, next to the static score's and nvMMH's top-K? If a few measurements on the model's shortlist close most of the gap to the oracle, that would change my view of the method's practical value.

Cost (clarification). How many measurements and GPU-hours did the BF16, FP32, FP8 and fusion corpora take, and how many problems, weighted by how often they run, are needed before this cost beats measuring a few candidates for each problem?

Transfer (clarification). How do problems whose selected kernel fails to run enter the speedups and win rates of Figure 4b–d and Table 15, and what are the speedups if each failure falls back to nvMMH? As a suggestion rather than a requirement, even a small test on a second GPU would show whether the architecture descriptors help.

2 comments

We thank the reviewer for the feedback and for recognizing the value of our representation, exhaustive evaluation, and improvements over nvMMH. We address the practical costs of kernel selection, shortlist comparisons, and transfer evaluation below.

Why prefer execution-free selection over measuring a shortlist?

The central motivation is the compilation of CUTLASS candidates: in our environment, compiling one program typically takes between 25-35 seconds, substantially exceeding the GPU benchmarking time; selection (including shortlisting, inference, and scoring) on our models and nvMMH, in comparison, takes less than 65 milliseconds. The time taken to measure a small shortlist therefore grows linearly with the number of kernels in the shortlist that must be compiled. Thus, it is always beneficial to compile as few candidates as possible, motivating the best top-K selector for any K. For example, our new top-K experiments show that at K=32, our selectors achieve a median regret of 0 and mean of 0.34% – meaning they hit the oracle more than 50% of the time. nvMMH, on the other hand, plateaus around 11%. At K=8, MLP selects within 5% of oracle on 89.7% of groups, versus 26.5% for nvMMH. We will submit a revision in the next few days including a comprehensive summary of the top-K experiments.

How are offline training and deployment costs amortized?

The measurement corpus represents a one-time training investment of around 200 GPU hours, which was our data collection budget, whereas compilation overhead is incurred whenever new candidates must be evaluated. At the scale of a hardware vendor such as NVIDIA, constructing a training corpus is a modest investment, amortized across potentially millions of subsequent kernel selections, making this approach attractive for vendor-scale deployment in libraries such as nvMMH itself. In contrast, empirical autotuning repeatedly incurs expensive compilation and benchmarking costs. We will include these per-corpus measurement counts and GPU-hours explicitly in the revision.

How are unsuccessful selections and cross-precision transfer handled?

Our transfer experiments evaluate adaptation across numerical precisions and epilogue configurations using the same hardware-aware representation. We report speedup only on successful selections. Execution coverage is reported separately to distinguish model selection quality from the ability of a generated configuration to execute successfully. In deployment, unsuccessful selections can fall back to nvMMH, preserving a valid kernel choice. We will clarify the reported denominators and win rates in the revision. We will also report the overall speedups and win rates when unsuccessful selections fall back to nvMMH. Cross-device transfer is a separate question from the demonstrated cross-precision adaptation; we will check and restrict our claims to avoid implying otherwise; however, the architecture descriptors provide the mechanism for parameterizing the representation across devices.

AuthorsOpenReview ↗

Follow-up Before the Discussion Closes

Thank you once again for your constructive feedback. We have expanded the evaluation with additional top-K comparisons, measurement-cost details, and methodological clarifications. Could you please let us know if the revision resolves your concerns? We would be glad to address any remaining concerns.

Reviewer_KcU6OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes execution-free selection of CUTLASS GEMM kernels using learned models augmented with static estimates of memory behavior, resource pressure, work decomposition, and pipeline behavior. It validates the approach using a large measurement corpus, exhaustive candidate evaluation, broader workloads, and adaptation across precisions and fused operations.

Critical strengths

A real systems problem. Selecting among many equivalent kernels without repeated benchmarking addresses a practical compilation and deployment bottleneck.

Substantial empirical groundwork. The measurement corpus and exhaustive evaluation provide a useful reference for assessing selection quality against the best measured candidate.

Critical weaknesses

The technical advance over prior work is insufficiently established. Hardware-derived features and learned ranking already appear in existing works (e.g., AutoTVM), while Ansor incorporates extensive memory, reuse, and arithmetic-intensity features. The paper needs to isolate the new modeling insight beyond adapting this established approach to Hopper/CUTLASS.

The practical justification for the added representation remains unclear. Larger structural models recover much of the accuracy gap. The paper therefore needs to demonstrate what benefit justifies the manual feature engineering and full-catalogue scoring, considering measurement cost, selection latency, and maintenance across hardware or library changes.

Questions for authors

\Which specific features or modeling insights are new relative to AutoTVM, Ansor, and prior learned CUTLASS selectors? Please distinguish new execution mechanisms from established performance descriptors.

With the training data and predictor held fixed, how does your representation compare with features adapted from prior learned cost models? Comparison against raw configuration inputs alone does not establish this advance.

Since a larger structural MLP largely closes the accuracy gap, what practical advantage justifies the proposed features? Please demonstrate the advantage under matched data and computational budgets.

What are the complete costs of catalogue enumeration, feature extraction, and scoring? When do the resulting execution savings amortize these costs for compile-time versus dynamic selection?

When the GPU architecture or CUTLASS implementation changes, which features require redesign and which measurements must be repeated? Please delimit the adaptability claims accordingly.

4 comments

We thank the reviewer for their feedback and for recognizing the importance of kernel selection and the strength of our empirical evaluation. We clarify the novelty, representation benefits, deployment costs, and adaptability below.

What is novel relative to AutoTVM, Ansor, and prior CUTLASS selectors?

AutoTVM and Ansor use generic program descriptors such as memory accesses, reuse, and parallelism. These approaches iteratively generate, compile, and measure candidate programs; in contrast, our selector evaluates the entire valid CUTLASS catalogue without compiling or executing any candidates, compiling only the final selection. The novelty lies in combining mechanism-derived features with execution-free selection. Additionally, AutoTVM and Ansor extract features from lowered program representations, whereas our features are computed statically from problem dimensions, configuration parameters, and hardware constants. They require only simple arithmetic calculations, and no code transformations, program analysis, profiling, or hardware-aware passes. Consequently, the implementation and maintenance costs are small.

Why are the features useful if larger structural models recover the accuracy gap?

Our features are deterministic transformations of configuration parameters and hardware properties; they simply provide an inductive bias rather than fundamentally new information. It is therefore expected that very large, sufficiently expressive structural models can eventually learn these relationships implicitly; however, there is a tradeoff between model size and ease of computation/portability, which motivates the use of smaller models. Larger models increase training and inference costs and are thus less suitable for universal deployment at vendor-scale. Thus, the important result is that explicit hardware behavior substantially reduces the model complexity required for accurate selection. Our model-size ablations demonstrate this across smaller MLPs, while XGBoost retains a significant advantage across model depths, including the largest models. The representation simplifies the performance-learning problem without requiring increasingly complex predictors. We will make this clear in the revision.

What are the selection costs and when are they amortized?

End-to-end selection, including catalogue enumeration, static arithmetic feature extraction, inference, and scoring, itself takes less than 65 milliseconds end-to-end for both our selectors and nvMMH, making the final cost simply the cost of compilation; compilation is the dominant cost of CUTLASS autotuning, taking between 25-35 seconds per candidate in our environment. Empirical top-K selection requires compiling and benchmarking K candidates. Reducing the number of compilations required always directly eliminates the dominant selection time, motivating better top-K selection approaches for any K. For example, our new top-K experiments show that at K=32, our selectors achieve a median regret of 0 and mean of 0.34% – meaning they hit the oracle more than 50% of the time. nvMMH, on the other hand, plateaus around 11%. The one-time training cost is borne by the library developer or hardware developer, and is amortized across millions of subsequent selections, making this approach attractive for vendor-scale deployment in libraries such as nvMMH itself. We will clarify these numbers and add our new top-K experiments to the revision.

How does the representation adapt to new architectures and CUTLASS versions?

Most features, such as work decomposition, memory traffic, resource occupancy, and pipeline balance, describe general execution effects present in modern GPUs. Adapting them to another device primarily requires only updating hardware constants such as processor count, memory capacity, instruction throughput for different data types, memory usage estimates, etc. as long as the GPU’s execution model is not fundamentally changed between architectures, which happens very rarely. Model training measurements are hardware-specific, but the feature-construction methodology and selection framework remain unchanged; moreover, due to the bounded nature of the task, the simple feature map construction for new hardware can easily also be passed on to agents. We will make these points clearer in the revision.

Thank you for the clarifications. The reported selection costs strengthen the practical case for compile-time deployment. My concern is whether the paper establishes a sufficiently distinct contribution over prior hardware-informed learned cost models and direct CUTLASS selectors, and whether the performance improvement evidence is sufficient. As such, I still lean toward a reject. Please see below for the detail.

AutoTVM and Ansor already incorporate engineered execution features, and CUTLASS-tailor provides precedent for direct learned selection. Computing features directly from configuration parameters is a useful implementation distinction. However, the rebuttal does not yet identify which modeling insights or selection capabilities constitute the advance over these approaches. Improvements over nvMMH and raw configuration baselines do not resolve that comparison. Could the authors identify the specific advance and clarify how a representative prior feature set would compare under the same training data and predictor?

The performance evidence also needs to distinguish the complete selector’s benefit from the incremental contribution of the representation. I acknowledge the stronger benefits for smaller models and XGBoost. However, a structural selector already captures much of the broad-workload improvement, and the hardware features do not improve the MLP’s aggregate performance on DeepBench. The relevant systems question is therefore the accuracy–cost tradeoff: at comparable selection quality, how much training or inference cost does the representation actually save over a competitive structural model? Quantifying that benefit would make the justification for the additional feature engineering substantially more convincing.

Thank you for the clarification. We address the comparison with prior work and the accuracy-cost tradeoff more directly below.

Novelty relative to Related Work.

CUTLASS-tailor predicts GEMM tile parameters from problem dimensions and GPU hardware descriptors, but not candidate-specific execution mechanisms. These inputs are already covered by our structural ablation, which also includes the parameters of each candidate kernel. The distinction vs CUTLASS-tailor is therefore not simply learned selection, but candidate-wise modeling of execution mechanisms to rank the complete catalogue, including mainloop, epilogue, cluster, and scheduling choices. We will make this distinction explicit in the revision.

AutoTVM and Ansor extract execution features from lowered program representations using their IR, whereas our approach deliberately avoids generating or lowering candidate programs. Directly applying their feature extractors would therefore require a fundamentally different selection pipeline and would not provide a meaningful matched comparison. However, their general program-analysis based performance descriptors can be adapted to our setting. We will extend our model-size ablation with an adapted prior-work feature set covering computation, memory traffic, reuse, parallelism, and resource requirements, holding the training data and predictors fixed, and present an analysis of the feature sets. This will directly distinguish the contribution of our execution-effects-based features from established performance descriptors.

Accuracy-cost tradeoff.

We will extend the model-size ablation to report selection latency, including feature extraction and inference, alongside model size and selection regret. This will explicitly quantify the selection-time differences from using smaller hardware-aware models compared with larger structural models achieving comparable regret.

We will include these comparisons and clarifications in the upcoming revision.

AuthorsOpenReview ↗

Follow-up Before the Discussion Closes

Thank you once again for your detailed feedback. The revision now includes the additional feature comparisons, related work clarifications, selection-cost analysis, and other clarifications discussed in our responses. Could you please let us know whether these changes resolve your concerns? We would be glad to address any remaining concerns.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Revision 1 Uploaded

We thank all reviewers for their constructive feedback. We have uploaded a revised manuscript, with changes highlighted in blue. The main updates are:

  • Novelty and related work: Clarified our contributions relative to AutoTVM, Ansor, and CUTLASS-tailor, particularly emphasizing the difference between feature sets and execution-free ranking of complete CUTLASS configurations.
  • Prior-work feature comparison: Added a program-analysis-inspired feature baseline to the model-size ablation, using the same training data, predictors, objectives, and three seeds. We also included a top-K study.
  • Accuracy–cost tradeoff: Added selection-latency measurements, including feature computation and inference. At comparable regret, the compact hardware-aware MLP achieves 3.0–8.8× lower CPU selection latency than the largest structural MLP.

We also expanded the corresponding methodological explanations and analysis in Sections 5–6 and Appendix H. We also added a scratch-training baseline for the transfer learning experiments.

We hope these additions address the concerns about novelty, comparisons with prior work, and practical deployment costs. We would appreciate any further feedback.

AuthorsOpenReview ↗

Revision 2 Uploaded

We have uploaded a second revision, with additional changes highlighted in blue italics. The main updates since Revision 1 are:

  • Static-score comparison: Added a top-K comparison against the sampling heuristic used for dataset collection, including explicit treatment of tied configurations.
  • Measurement costs: Added per-corpus training measurements, collection costs, and representative CUTLASS compilation times.
  • Methodological clarifications: Clarified nvMMH's schedule-selection treatment and transfer-data budgets.

We hope these additions address the remaining questions. We welcome any further feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.