acceptodds
Under review as a conference paper at ICLR 2027

Scheduling Kernel-Generating Agents at Scale in Decoupled Agent–Worker Systems

Abstract

Programming agents develop GPU kernels by repeatedly generating, compiling, and running code. In production, the large volume of operator-generation tasks motivates a decoupled Agent–Worker architecture that avoids dedicating GPUs to individual agents during model inference and tool use. Agents run on CPU clusters and submit code under test to a shared GPU execution pool when compilation and testing are needed. In this setting, each batch’s Skill, the types and number of operators to generate, and model inference speed jointly determine how frequently the batch requires GPU execution. Too few execution requests leave GPUs idle, whereas too many cause queues to accumulate, delaying execution feedback and reducing useful iterations, which can compromise kernel quality. We propose a Scheduling Agent that controls the concurrency and admission of operator-generation tasks. It reads each batch’s Skill to identify the stages that require GPUs, then combines resource utilization, the queue of code awaiting execution, and model-API feedback to select the next tasks to launch. It also uses file semantics to promptly identify and remove residual artifacts created at incorrect output paths, reclaiming disk space and inodes. Compared with FIFO alone, adding the Scheduling Agent increases Worker utilization from 59.9% to 85.2%. It also removes approximately 2.2 TB of residual files from incorrect paths and frees about 380,000 inodes. The system has operated in production for several months, providing scheduling support for over ten thousand operator-generation tasks each month.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
LLM agents, GPU kernel generation, agent–worker systems, resource-aware scheduling, shared GPU clusters

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_nxxzOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies a decoupled system in which CPU-hosted programming agents generate GPU kernels and submit code to shared GPU/NPU Workers for compilation, testing, and profiling. It proposes a Scheduling Agent that retrieves historical runs using Skill lineage and execution configuration, predicts future Worker demand, and adjusts task admission and API concurrency under capacity constraints. The same agent also identifies residual files for deletion. Based on production traces, the paper reports median Worker utilization of 85.2% with FIFO plus the Scheduling Agent versus 59.9% with FIFO alone, as well as cleanup of approximately 2.2 TB and 380,000 inodes.

Critical strengths
  1. The paper identifies an operational problem in decoupled agent-worker systems: agent-side generation creates bursty execution demand, and excessive Worker queues can delay feedback. The deployment scale and long-running production telemetry make the setting potentially valuable to systems researchers.

  2. The retrieval-prediction-admission workflow is intuitive and auditable. Conservative fallback behavior and executor-side revalidation are sensible safeguards for a production resource controller.

Critical weaknesses
  1. The importance and robustness of Skill information are not established. There is no evaluation showing that Skill text or Git diffs add value beyond the model, Harness, and task composition. A deployment using one fixed, strongest Skill would receive no lineage signal. And even for the same Skill, the resource utilization patterns can be very different, as Figure 23 shows that two batches with the same displayed Skill commit and configuration can produce 1,644 versus 433 submissions.

  2. The paper does not justify the need for an LLM agent, and the baseline is too weak. The method mainly retrieves and averages traces, checks queue and API states, and applies capacity thresholds. These operations could be handled by a conventional controller. The paper compares only against head-of-line-blocking FIFO, not adaptive scheduling algorithms such as moving averages, or direct profile reuse. It also omits the agent's model, latency, cost, and scalability.

  3. The production ON/OFF comparison is heavily confounded. The two settings cover different workloads, Skills, models, service conditions and batch stages. The paper also uses an unspecified "sufficient task supply" filter. Thus, the reported 85.2% versus 59.9% utilization difference is not a controlled estimate of the scheduler's effect.

  4. The evaluation does not measure the claimed end-to-end benefit. The paper does not report completed tasks, deadline attainment, useful iterations, or final kernel correctness and performance. The file-cleanup component is also only evaluated by bytes and inodes removed, without comparison to rule-based cleanup or analysis of false deletions.

Questions for authors
  1. Can you run a controlled replay or randomized A/B experiment with identical task arrivals, Skills, model-service traces, and Worker capacities, comparing the Scheduling Agent against last-run profile reuse, and a simple adaptive scheduling policy? Which component of the proposed system produces the gain?

  2. What is the incremental predictive value of Skill text and Git diffs? Can you run an ablation that removes Skill information while retaining model, Harness, task count, operator composition and workload traces? How would the method help when the deployment uses one fixed, strongest Skill?

  3. Why is normalized Wasserstein similarity an adequate forecasting metric for admission control when it discards total submission volume? What are the absolute errors in submission count, peak demand, and per-hardware-pool demand?

  4. Under matched workloads, what are the effects on completed tasks, deadline attainment, number of useful generation/test iterations, and final kernel correctness and performance? Does any policy improve utilization by doing more low-value or slow work?

  5. What LLM and prompt are used for each scheduling decision, and what is the decision latency and token cost? How many batches can one Scheduling Agent control? Which safeguards prevent erroneous admission or deletion decisions?

1 comment
AuthorsOpenReview ↗

Response to Reviewer nxxz

Thank you for the review. We have revised the paper accordingly.

Q1.1 Can you run a controlled replay or a randomized A/B experiment with the same arrival sequence and capacity? Section 4.3 is a controlled replay: 25 scenarios, each with the same real load, capacity, and submission sequence, where each of the four methods decides for itself when batches start. We did not run an online randomized A/B experiment, and the production results in Section 4.4 are only observations.

Q1.2 Which component brings the gain? The four methods build on each other: first-come-first-served, then threshold-based feedback control, then reuse of historical curves, and finally semantic analysis of the Skill and the Workflow. The median batch scores are 0.39, 0.55, 0.49, and 0.64, and the Scheduling Agent scores highest in 20 scenarios. So the Scheduling Agent clearly helps system scheduling, and threshold-based feedback control also helps to some extent.

Q2.1 What does the Skill text and Git diff add to prediction? In each group of Tables 1 and 2, only one piece of Skill or Workflow semantics changes, and we compare against Baseline 3, which keeps version retrieval but does not read the diff or the script. The median Pearson similarity is 0.95 against 0.14 for Workflow changes and 0.90 against 0.60 for Skill changes. This shows that by analyzing the Git diff, the Scheduling Agent predicts the future GPU usage distribution reasonably well.

Q2.2 If a deployment always uses one fixed Skill, how does the method help? When the Skill is fixed, reading the Skill diff is of no use, but the rest of the method does not depend on it. First, a fixed Skill does not mean fixed GPU demand: with the same Skill, different Workflows launch different numbers of agents per task, and GPU usage can differ a lot. The parallelism is written in a Python script, which has to be read to be known. Table 2 is this case: the Skill and everything else stay the same and only the Workflow changes, and the median Pearson similarity is 0.95 against 0.14, because the historical curve does not contain the added agents. Second, when a batch starts and how many tasks each model API serves at once are decided from the current pool capacity, the queue, and the state of the model services, which also does not depend on whether the Skill changes.

Q3.1 Why is a normalized similarity enough for admission control? You are right that normalization discards information about the total. So we replaced the Wasserstein similarity with Pearson and cosine similarity, and added the volume ratio, the ratio of predicted to actual total GPU usage.

Q3.2 What are the absolute errors in submission count, peak demand, and demand per hardware pool? On the volume ratio, the Scheduling Agent is between 0.89 and 1.11 and Baseline 3 is between 0.09 and 1.29, as shown in Tables 1 and 2. We do not report the error in peak size, peak time, or per hardware pool.

Q4.1 What is the effect on completed tasks, deadline attainment, and final kernel correctness and performance? Appendix F gives a real case: when the queue backs up and the code an Agent submits is not tested promptly, the number of iterations drops and the final results get worse. This is easy to understand, since kernel generation keeps optimizing on top of the current kernel, so far fewer iterations hurt the final quality. Besides, completed tasks, correctness, and performance depend mostly on the model, the Skill, and similar factors, while our focus is system scheduling. It is enough to show that queue backlog hurts quality, and that Section 4.3 shows scheduling can reduce queueing.

Q4.2 Could a policy raise utilization just by doing more low-value work? The batch score looks at whether a queue builds up while the batch runs, not at utilization. Submitting more tasks only adds to the queue, so it cannot raise the score.

Q5.1 Which LLM and prompt are used for each scheduling decision? A model from the DeepSeek-Flash series. A simplified version of the prompts is in Appendix G.

Q5.2 What are the decision latency and the token cost? The token usage is about 0.07% of that of the kernel-generation agents. The latency of each round is low, and negligible compared with a kernel-generation task that runs for dozens of hours.

Q5.3 How many batches can one Scheduling Agent control? In the real deployment, a separate Scheduling Agent is started for the prediction of each batch. Once tasks have started, one Scheduling Agent keeps analyzing the batches waiting in the queue and decides whether each should be submitted. This is sufficient for our setting.

Reviewer_vT53OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper addresses GPU scheduling for large-scale agentic kernel generation, where coding agents run on CPUs and submit code to a shared accelerator pool. Because Skills, models, and API conditions change when agents submit code, a fixed concurrency setting either idles GPUs or causes backlogs. The authors propose an LLM-based Scheduling Agent that predicts each batch's GPU demand from Skill version history and past runs, and admits batches under a capacity constraint. In production, they report median daily Worker utilization of 85.2% with the agent vs. 59.9% with FIFO, plus reclaimed storage from residual files.

Critical strengths

Strengths

  1. The problem is real and timely. Fig. 2 clearly shows that equal task counts can produce very different GPU demand depending on the Skill.
  2. The system is evaluated in a real deployment at meaningful scale (28 days, ~29K tasks, ~1.4M submissions), which is rare for this topic.
  3. The observation that consecutive Skill versions on the same branch share similar demand profiles (Table 1) is useful on its own for scheduling agent workloads.
Critical weaknesses
  1. Lack of baselines, and the value of the LLM is unclear. The only comparison is FIFO, which never adjusts concurrency. The controlled quantity, the queue of code awaiting execution, is directly observable, so a simple feedback controller on queue length and utilization is the natural baseline. The deployed prompt (App. G) largely encodes such a controller as threshold rules (queue near 0–12 items, utilization 95–97%). The agent's cost (model, tokens, latency) is never reported, so the claim in Sec. 5 that gains outweigh inference overhead is unsupported. Related work on feedback-based admission control and on compiler autotuning, which also pairs CPU-side search with a shared device pool, is not discussed.

  2. The utilization comparison is not controlled. ON and OFF observations come from different periods with different workloads, Skills, and models. According to the daily notes, many OFF intervals coincided with service restarts or outages. Since utilization depends directly on task supply and system health, the 85.2% vs. 59.9% gap cannot be attributed to the agent. “Sufficient task supply” is not defined, outliers are excluded post hoc, and no statistical test is reported. A matched-load comparison, e.g., alternating ON/OFF on the same batch, is needed.

  3. Task outcomes are never reported. The paper motivates scheduling by its effect on kernel quality and states its objective as increasing tasks completed by their deadlines (Sec. 3). Yet it reports only utilization and submission counts. The only evidence that backlog hurts agents is two historical batches compared by submission count (App. E). The agent's own prompt (App. G) records compilation and accuracy results per batch, so this data appears to exist.

  4. The role of Skills, the paper's central idea, is unclear and untested. Skill-aware prediction is the main novelty, yet the paper never explains how a Skill diff becomes a demand estimate (Sec. 3.1–3.3), nor who authors Skills or how large diffs typically are. Prediction is evaluated on only six forecasts, all on consecutive versions within a single branch, using a metric that normalizes away submission volume. Reusing the previous batch's observed profile scores only 0.004–0.042 below the agent (Table 1), so the evidence does not show that Skill semantics add much. The quantity admission actually uses, demand in Worker capacity (Eq. 4), is never validated.

Questions for authors
  1. How does the Scheduling Agent compare to (a) a rule-based controller using the same thresholds as its prompt, and (b) admission based on per-Skill historical profiles without an LLM? What does the agent cost per scheduling round?

  2. Can you provide an ON/OFF comparison under matched load?

  3. What are the compile rate, correctness rate, and kernel speedup for tasks run with vs. without the agent?

  4. How is a Skill diff converted into a demand prediction, and how does accuracy compare to reusing the previous batch's profile as diffs grow larger?

1 comment
AuthorsOpenReview ↗

Response to Reviewer vT53

Thank you for the review. We have revised the paper accordingly.

Q1.1 How does the Scheduling Agent compare with a rule-based controller that uses the same thresholds as its prompt? We added this as Baseline 2. It adjusts concurrency using thresholds on pool utilization and API availability, the same thresholds as in the prompt. Across 25 scenarios, its median batch score is 0.55 against 0.64 for the Scheduling Agent, and the Scheduling Agent scores highest in 20 of them. Details are in Section 4.3.

Q1.2 How does it compare with a method that uses no LLM and relies only on each Skill's historical curves? This is Baseline 3. On top of Baseline 2, it reuses the historical curve of the same configuration and does not read the semantics of the Skill or the Workflow. Its median batch score is 0.49, lower than Baseline 2, so reusing historical curves alone does not help. On prediction accuracy, over 60 batches the median Pearson similarity is 0.79 for the Scheduling Agent and 0.34 for Baseline 3, and the Scheduling Agent is higher on 46 of the 60.

Q1.3 What is the Agent overhead per scheduling round? The scheduler uses a model from the DeepSeek-Flash series, and its token usage is about 0.07% of that of the kernel-generation agents. We did not measure the latency of each round.

Q2 The ON/OFF utilization comparison does not control the experimental conditions. We agree. In a real industrial production environment it is very hard to run an ablation on a long-running service, so we treat the 85.0% versus 59.9% only as an observation, and the paper says so. The comparison that does control the conditions is the 25 scenarios in Section 4.3, where all four methods see the same real load, the same capacity, and the same submission sequence.

Q3 Why did you not evaluate the compile pass rate, accuracy pass rate, and speedup of the final kernels for each batch? Appendix F gives a real case showing that when the queue backs up and the code an Agent submits is not tested promptly, the final kernel generation results get worse. This is also easy to understand: kernel generation improves the current kernel step by step, so if the number of iterations drops sharply because of queueing, the final quality is bound to suffer. Besides, the quality of generated kernels depends mostly on the model, the Skill, and similar factors, while our paper focuses on system scheduling. Showing that queue backlog hurts kernel quality is enough for this purpose.

Q4.1 How does a Skill diff turn into a demand prediction? The Agent reads the Git diff between the candidate batch and the reference batch, together with the Workflow script. If only the wording changed, the curve stays as it is. If a new stage was added, the reference curve is split into segments that are shifted back and stretched. If the number of parallel agents changed, the curve is scaled in proportion. The method is described in Section 3.1.

Q4.2 As the diff grows, how does prediction accuracy change compared with directly reusing the previous batch's curve? We did not bin results by diff size, but Tables 1 and 2 are consistent with this trend. In the second group of Skill changes, the change has little effect on the curve, so reusing the previous curve is already accurate and the Agent does about the same: Pearson 0.82, 0.90, 0.89 for the Agent against 0.85, 0.89, 0.85. In the first group of Skill changes and in both groups of Workflow changes, the change clearly alters the shape or height of the curve, and reusing the previous curve is far off, with Pearson from 0.35 down to slightly below zero, while the Agent stays between 0.71 and 0.97.

Reviewer_uqXhOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

In real world production, GPU kernel development tasks are a major task. This motivates an Agent–Worker architecture in this paper where agents run on CPU clusters and submit code to a shared GPU execution pool. Because the frequency of these GPU requests varies by task complexity and inference speed, unmanaged queues can either leave GPUs idle or cause massive bottlenecks that degrade kernel quality. To balance this workload, the authors propose a scheduling agentic module that analyzes task requirements, resource utilization, and system feedback to intelligently control task concurrency and admission. The authors test this out in production and report substantial efficiency gains.

Critical strengths
  • important problem, GPU inference optimization (via kernels and mega kernels) is a major topic of research interest.
  • simple enough method, primary contribution is really the identification of core bottlenecks.
  • production grade deployment, large scale efficiency gains.
Critical weaknesses
  • simple method, but works at scale. Not much of a negative.
Questions for authors

None.

1 comment
AuthorsOpenReview ↗

Response to Reviewer uqXh

God bless you! Thank you for seeing the most important value of this paper: finding a new problem in a new setting and proposing a simple solution that works. We are especially grateful that you gave a valuable review based on your own judgment.

We work on the front line of kernel production in industry, and the number of GPUs we use has recently grown to several hundred. Since we started using workflows like Humanize, the number of agents that a batch of tasks launches concurrently is no longer fixed, and neither is the GPU demand that comes with it. This calls for an intelligent scheduler that reads the Humanize script code to work it out. We are glad that the Scheduling Agent in the paper fits this need, while other methods, such as those that estimate resource demand from a fixed number of concurrent agents, have a hard time with this kind of change. In our runs over the past month, the predicted curves have also matched actual demand quite well.

We have also polished and improved the paper a great deal, adding engineering and ablation experiments that further show the method works.

Thank you again for your recognition. We hope this paper can help colleagues in the community who are working on kernel generation!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.