- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies a decoupled system in which CPU-hosted programming agents generate GPU kernels and submit code to shared GPU/NPU Workers for compilation, testing, and profiling. It proposes a Scheduling Agent that retrieves historical runs using Skill lineage and execution configuration, predicts future Worker demand, and adjusts task admission and API concurrency under capacity constraints. The same agent also identifies residual files for deletion. Based on production traces, the paper reports median Worker utilization of 85.2% with FIFO plus the Scheduling Agent versus 59.9% with FIFO alone, as well as cleanup of approximately 2.2 TB and 380,000 inodes.
Critical strengths
-
The paper identifies an operational problem in decoupled agent-worker systems: agent-side generation creates bursty execution demand, and excessive Worker queues can delay feedback. The deployment scale and long-running production telemetry make the setting potentially valuable to systems researchers.
-
The retrieval-prediction-admission workflow is intuitive and auditable. Conservative fallback behavior and executor-side revalidation are sensible safeguards for a production resource controller.
Critical weaknesses
-
The importance and robustness of Skill information are not established. There is no evaluation showing that Skill text or Git diffs add value beyond the model, Harness, and task composition. A deployment using one fixed, strongest Skill would receive no lineage signal. And even for the same Skill, the resource utilization patterns can be very different, as Figure 23 shows that two batches with the same displayed Skill commit and configuration can produce 1,644 versus 433 submissions.
-
The paper does not justify the need for an LLM agent, and the baseline is too weak. The method mainly retrieves and averages traces, checks queue and API states, and applies capacity thresholds. These operations could be handled by a conventional controller. The paper compares only against head-of-line-blocking FIFO, not adaptive scheduling algorithms such as moving averages, or direct profile reuse. It also omits the agent's model, latency, cost, and scalability.
-
The production ON/OFF comparison is heavily confounded. The two settings cover different workloads, Skills, models, service conditions and batch stages. The paper also uses an unspecified "sufficient task supply" filter. Thus, the reported 85.2% versus 59.9% utilization difference is not a controlled estimate of the scheduler's effect.
-
The evaluation does not measure the claimed end-to-end benefit. The paper does not report completed tasks, deadline attainment, useful iterations, or final kernel correctness and performance. The file-cleanup component is also only evaluated by bytes and inodes removed, without comparison to rule-based cleanup or analysis of false deletions.
Questions for authors
-
Can you run a controlled replay or randomized A/B experiment with identical task arrivals, Skills, model-service traces, and Worker capacities, comparing the Scheduling Agent against last-run profile reuse, and a simple adaptive scheduling policy? Which component of the proposed system produces the gain?
-
What is the incremental predictive value of Skill text and Git diffs? Can you run an ablation that removes Skill information while retaining model, Harness, task count, operator composition and workload traces? How would the method help when the deployment uses one fixed, strongest Skill?
-
Why is normalized Wasserstein similarity an adequate forecasting metric for admission control when it discards total submission volume? What are the absolute errors in submission count, peak demand, and per-hardware-pool demand?
-
Under matched workloads, what are the effects on completed tasks, deadline attainment, number of useful generation/test iterations, and final kernel correctness and performance? Does any policy improve utilization by doing more low-value or slow work?
-
What LLM and prompt are used for each scheduling decision, and what is the decision latency and token cost? How many batches can one Scheduling Agent control? Which safeguards prevent erroneous admission or deletion decisions?
Response to Reviewer nxxz
Thank you for the review. We have revised the paper accordingly.
Q1.1 Can you run a controlled replay or a randomized A/B experiment with the same arrival sequence and capacity? Section 4.3 is a controlled replay: 25 scenarios, each with the same real load, capacity, and submission sequence, where each of the four methods decides for itself when batches start. We did not run an online randomized A/B experiment, and the production results in Section 4.4 are only observations.
Q1.2 Which component brings the gain? The four methods build on each other: first-come-first-served, then threshold-based feedback control, then reuse of historical curves, and finally semantic analysis of the Skill and the Workflow. The median batch scores are 0.39, 0.55, 0.49, and 0.64, and the Scheduling Agent scores highest in 20 scenarios. So the Scheduling Agent clearly helps system scheduling, and threshold-based feedback control also helps to some extent.
Q2.1 What does the Skill text and Git diff add to prediction? In each group of Tables 1 and 2, only one piece of Skill or Workflow semantics changes, and we compare against Baseline 3, which keeps version retrieval but does not read the diff or the script. The median Pearson similarity is 0.95 against 0.14 for Workflow changes and 0.90 against 0.60 for Skill changes. This shows that by analyzing the Git diff, the Scheduling Agent predicts the future GPU usage distribution reasonably well.
Q2.2 If a deployment always uses one fixed Skill, how does the method help? When the Skill is fixed, reading the Skill diff is of no use, but the rest of the method does not depend on it. First, a fixed Skill does not mean fixed GPU demand: with the same Skill, different Workflows launch different numbers of agents per task, and GPU usage can differ a lot. The parallelism is written in a Python script, which has to be read to be known. Table 2 is this case: the Skill and everything else stay the same and only the Workflow changes, and the median Pearson similarity is 0.95 against 0.14, because the historical curve does not contain the added agents. Second, when a batch starts and how many tasks each model API serves at once are decided from the current pool capacity, the queue, and the state of the model services, which also does not depend on whether the Skill changes.
Q3.1 Why is a normalized similarity enough for admission control? You are right that normalization discards information about the total. So we replaced the Wasserstein similarity with Pearson and cosine similarity, and added the volume ratio, the ratio of predicted to actual total GPU usage.
Q3.2 What are the absolute errors in submission count, peak demand, and demand per hardware pool? On the volume ratio, the Scheduling Agent is between 0.89 and 1.11 and Baseline 3 is between 0.09 and 1.29, as shown in Tables 1 and 2. We do not report the error in peak size, peak time, or per hardware pool.
Q4.1 What is the effect on completed tasks, deadline attainment, and final kernel correctness and performance? Appendix F gives a real case: when the queue backs up and the code an Agent submits is not tested promptly, the number of iterations drops and the final results get worse. This is easy to understand, since kernel generation keeps optimizing on top of the current kernel, so far fewer iterations hurt the final quality. Besides, completed tasks, correctness, and performance depend mostly on the model, the Skill, and similar factors, while our focus is system scheduling. It is enough to show that queue backlog hurts quality, and that Section 4.3 shows scheduling can reduce queueing.
Q4.2 Could a policy raise utilization just by doing more low-value work? The batch score looks at whether a queue builds up while the batch runs, not at utilization. Submitting more tasks only adds to the queue, so it cannot raise the score.
Q5.1 Which LLM and prompt are used for each scheduling decision? A model from the DeepSeek-Flash series. A simplified version of the prompts is in Appendix G.
Q5.2 What are the decision latency and the token cost? The token usage is about 0.07% of that of the kernel-generation agents. The latency of each round is low, and negligible compared with a kernel-generation task that runs for dozens of hours.
Q5.3 How many batches can one Scheduling Agent control? In the real deployment, a separate Scheduling Agent is started for the prediction of each batch. Once tasks have started, one Scheduling Agent keeps analyzing the batches waiting in the queue and decides whether each should be submitted. This is sufficient for our setting.