- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates whether coding agents search more effectively when they choose high-level sparse strategies and leave implementation details to a compiler backend. The authors propose SABER, an agent-facing interface that exposes sparse strategy and coarse hardware intent while delegating low-level realization to operator-specific compiler backends. They compare SABER with Direct CUDA generation on 26 workloads from 6 sparse operator groups, primarily on an RTX 3090. They also run a controlled study in which SABER and a lower-level matched schedule give agents access to the same 32 implementations per workload. SABER produces fewer failures and reaches its best candidates earlier in the full-system comparison. The matched study also finds that agents need fewer responses with SABER. However, SABER does not consistently produce faster final kernels than Direct CUDA, and its performance depends on the available backend implementations.
Critical strengths
S1. The matched study is well designed. Both interfaces search the same fixed set of backend implementations. This makes the comparison more convincing, since a better result cannot simply come from access to a better kernel. I also like that the authors measure search progress against the fastest correct implementation in this set, rather than only against each trajectory's own best (L356-359). SABER needs fewer responses to reach its best candidate across all three coding models, with paired-bootstrap results supporting the difference.
S2. I appreciate that the authors are upfront about the performance limitations. SABER only matches Direct CUDA overall and performs considerably worse on SpGEMM. They also acknowledge that the SDDMM gap against SparseTIR cannot be explained by abstraction level alone. These qualifications make the paper's claims more credible. The paper is clear that the matched study is a controlled interface experiment rather than a production-scale benchmark.
Critical weaknesses
Major
W1. The headline search metrics may largely reflect small action domains. The 26-workload search comparison is difficult to interpret without the SABER action-domain sizes. SABER can stop once it has exhausted its finite action domain (L305-307), while Direct CUDA always makes 8 proposals. Across the 78 trajectories, SABER makes 189 proposals and Direct CUDA makes 624. Figure 3(c) also shows that every SABER trajectory for SDDMM and SpGEMM finds its best implementation on the first proposal. This makes me wonder how many of these workloads offer very few choices to begin with. The headline metrics, 92.3 vs. 47.4% of trajectories within 10% of final best by proposal 2 and mean proposals-to-best of 1.628 vs. 3.986, are measured against each trajectory's own final best (L312-314). A search over only 1 or 2 candidates scores well on these metrics by construction. Such small domains do occur in the separate comparison of independently configured systems: SABER has only 2 candidates for SpMM and 1 for SDDMM there (Table 22). The manuscript reports neither the 26-workload domain sizes nor the number of trajectories ended by exhaustion. The reduction from 69 hard failures to 0 is useful, but the comparison does not tell us how much comes from the abstraction level. SABER chooses among backend implementations and never generates source code (L51-52), which already removes many opportunities for failure.
W2. Matched-schedule schema not disclosed. The matched study is the paper's strongest evidence, but the paper never shows what the lower-level representation looks like. SABER's schema is summarized in Table 7 and illustrated in Figures 1 and 2, whereas the matched schedule is described only as a fixed physical scheduling schema (Appendix B.3), with no fields, prompt text or example action. Table 2 suggests the 2 schemas differ in how hard they are to write: matched-schedule responses are about 1.8 to 2.4 times longer (348, 320 and 620 vs. 148, 142 and 340 output tokens) and contain more invalid actions (9, 8 and 18 vs. 3, 2 and 10). It's hard to tell how much of the difference comes from the abstraction level versus the schema itself. For example, we don't know how many different descriptions can map to the same candidate, or how often the agents generate duplicate candidates. Providing the schema would help, but I think an ablation is still needed to distinguish these explanations. For example, the authors could try a matched-schedule interface with more semantic field names. The results show that agents do better with the SABER schema used here. I am less convinced that they establish where the abstraction boundary should be, as claimed at L472-474.
Minor
W3. The practical benefit in the matched study is small. SABER's speedup over the matched schedule is only to (L370-372). With shared-reference geometric-mean latencies of 8.74 µs for SpMM and 6.21 µs for SDDMM (Table 19), the absolute gains appear to be below a microsecond. I would like to see the latency range across the 32 candidates and repeated timing measurements to understand how reliably these differences can be measured. It would also help to include the Direct CUDA latencies in the main text, which currently compares output-token counts on these workloads (L118-119). Table 15 shows that SABER's selected kernels are slower under 2 of the 3 models ( and speedup over Direct CUDA).
Questions for authors
Q1. Could the authors show both prompt templates and the SABER and matched-schedule actions corresponding to all 32 candidates for one SpMM workload and one SDDMM workload? I would also like to know which SABER fields vary and how often agents revisit the same candidate under each interface.
Q2. What are the SABER action-domain sizes in the 26-workload experiment, and how many trajectories end by exhaustion? How do attainment and proposals-to-best compare when measured against a fixed reference, or restricted to workloads whose finite action domain contains at least 8 actions?
Q3. What numerical tolerance applies in the 26-workload experiment? When a numerical failure occurs, does the agent receive an error magnitude or only a pass/fail flag?
Q4. For the ten workloads of the controlled study, what is the latency range across the 32 candidates, and how large is the measurement variation across repeated timing samples? How does cuSPARSE 13.3 perform on these workloads under the same timing protocol?
2 comments
I thank the authors for addressing my main concerns about the search space and the matched comparison.
Q1. Prompts, candidate encodings, and revisits
Please see Reviewer YseR Q1 and Q2.
Q2. Action-domain sizes, exhaustion, and search metrics
The 26 workloads expose 1, 2, or 4 legal SABER actions each, as shown below.
In total, there are 56 workload-action combinations. Across 78 trajectories, SABER makes 189 proposals: 118 improving candidates, 50 correct but non-improving candidates, and 21 duplicates or no-ops. The recorded action visits are consistent with domain exhaustion in all 78 trajectories. None of the workloads has eight or more legal actions.
We also report two search metrics at response 2, both relative to each trajectory's final best candidate:
SABER finds its final best candidate after 1.628 responses on average, compared with 3.986 for Direct CUDA, over 78 and 74 successful trajectories, respectively. SABER also records 0 hard failures in 189 proposals, compared with 69 in 624 for Direct CUDA.
For comparison against a common reference, the separate matched study shows that SABER finds a kernel within 10% of the fastest shared candidate in 80.0% versus 56.7% of trajectories at R2, and 97.8% versus 84.4% at R8 (Table 21).
Q3. Numerical tolerance and correctness feedback
Both the 26-workload experiment and the ten-workload matched study use atol = rtol = 1e-4 for numerical correctness. Candidate outputs are checked against the corresponding reference results.
Agents receive numerical error magnitudes in their feedback. For example, Direct CUDA on
stage8:spmm-bsr-059(replicate 0, response 6) receivedNUMERICAL_CORRECTNESS_FAILUREwithmax_abs=0.955982andmax_rel=0.142661. The 26-workload Direct CUDA results contain 54 numerical failures.Q4. Candidate latency, timing variation, and cuSPARSE
Table 15 reports the geometric-mean speedups of SABER-selected kernels:
We benchmark all 32 candidates on each of the ten CSR workloads. The geometric means of the fastest and slowest candidate latencies across workloads are 8.74-17.48 μ s for SpMM and 6.21-12.42 μ s for SDDMM, respectively.
Table 20 reports paired workload-bootstrap 95% confidence intervals of [1.018, 1.129], [1.015, 1.118], and [1.009, 1.115] for GPT-5.6, GPT-6, and MiniMax-M3, respectively.
We also benchmark cuSPARSE 13.3 on the same ten CSR workloads using the same FP32 inputs and GPU timing protocol. Its geometric-mean latencies are 18.6 μ s for SpMM and 14.4 μ s for SDDMM. The fastest shared implementations outperform cuSPARSE by 2.13x for SpMM and 2.32x for SDDMM.