acceptodds
Under review as a conference paper at ICLR 2027

AOS-Bench: Can LLM Agents Manage Dynamic Workloads?

Abstract

Coding agents are increasingly asked to handle multiple requests on the same computer, yet current benchmarks evaluate tasks one at a time. We introduce AOS-Bench, a benchmark that evaluates a persistent agent system as a workload manager: tasks arrive over time and must be delivered under shared CPU and memory limits. Across five model–harness pairs, isolated task success poorly predicts workload delivery. On medium workloads, systems differ by 13 percentage points in isolated success but 29 points in stream success; on heavy workloads, two systems with identical isolated baselines deliver 72.2% and 49.7% of the same tasks. Execution-policy interventions trace much of this gap to task intake: serial dispatch leaves 73% of heavy-workload tasks unclaimed. Yet more elaborate management does not necessarily help—a fixed concurrency of two matches explicit management guidance on medium workloads, while no tested policy uses more than 30% of the provisioned CPU capacity. These results identify workload management as a distinct challenge for agents and establish AOS-Bench as a way to measure it.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Agent, Scheduling

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_DN8nOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents AOS-Bench, a benchmark that evaluates how a persistent agent system manages tasks arriving over time under a shared CPU and memory budget. It uses source benchmarks such as Terminal-Bench and SkillsBench to construct workloads. A workload includes a set of tasks that are to be executed under a shared CPU and memory budget. The paper uses AOS-Bench to evaluate five model-harness pairs.

Critical strengths
  • The paper presents AOS-Bench to evaluate how well a persistent agent system can manage a stream of tasks. Dynamic workloads create challenges for agent systems to efficiently leverage limited resources to handle multiple tasks.
  • The evaluation includes five model-harness pairs. The results cover metrics such as task success rate and rewarded-task throughput ratio.
Critical weaknesses
  • The description of the benchmark is vague. It is unclear how the workload is generated and whether it matches realistic scenarios.
Questions for authors
  1. Does AOS-Bench consider distributed scenarios? Does the setup assume the shared CPU and memory resources on the same machine?

  2. AOS-Bench constructs workloads using a set of source benchmarks, such as Terminal-Bench, SkillsBench, TerminalWorld, etc. How do you draw tasks from these source benchmarks to construct a workload in AOS-Bench? Do you randomly draw tasks from these source benchmarks?

  3. In Section 2.2, the paper mentions that the source benchmarks are "supplemented by synthetic tasks". What are the synthetic tasks?

  4. Can you clarify how the release time is set in the evaluation? Are the tasks separated by a fixed duration? Do you consider using traces from realistic scenarios?

  5. Does AOS-Bench match realistic scenarios where an agent handles a stream of tasks? Can the way that the workloads are constructed from the source benchmarks and the release times represent real-world dynamic workloads?

1 comment
AuthorsOpenReview ↗

Official Response to Reviewer DN8N

[Q1] Does AOS-Bench consider distributed scenarios?

[A1] The benchmark targets one container on one machine with 4 CPUs, 8 GiB, one filesystem and one dispatcher. Multi-machine placement lies outside this work, as Section 1 states.

[Q2] Are workloads sampled at random from the source benchmarks?

[A2] No. Tasks are profiled by duration and CPU use into L1/L2/L3 bands and assembled under a low-overlap rule so that instructions, reference solutions and verifiers do not collide on shared paths. Four gates apply: a source audit, image build and launch, reference solutions reaching full score in the merged image, and an overlap check against admitted workloads. Sections 2.2 and Appendix A describe this procedure.

[Q3] What are the synthetic tasks?

[A3] Three families are developed for AOS-Bench: a compression-Pareto task, a C project-repair loop (also present as a repeated instance) and a prime-sieve task. Each has an instruction, environment, reference solution and executable verifier, and each passes the same gates. Appendix A lists them.

[Q4] How are release times set, and are production traces used?

[A4] Release times form a designed per-workload schedule in stream.toml: ten tasks over approximately 20 minutes on medium, twenty over approximately 60 minutes on heavy, with non-uniform gaps. The dispatcher hides future arrivals, so the agent sees only released tasks. These are designed schedules, not production traces.

[Q5] Does the setting represent realistic dynamic workloads?

[A5] The unit tasks come from real-world benchmarks — Terminal-Bench, SkillsBench, TerminalWorld, WildClawBench, CompileBench, CompileAgent, ParEval and the One Billion Row Challenge, supplemented by three synthetic task families. They are therefore genuine developer workloads rather than synthetic placeholders. It is the release process that is designed: the workload assembles these tasks into a shared container under a fixed 4 CPU / 8 GiB budget with a non-uniform arrival schedule. The overall release strategy is closer to a stress test than to a replay of a production trace, in that it concentrates heterogeneous tasks in time under tight shared resources. Such bursts are nevertheless a challenge that a persistent agent may face in real deployments, and intake failure under concurrency is exactly the failure mode the benchmark isolates. We do not claim that the arrival distribution matches production traffic; priorities, deadlines, dependencies, multi-machine placement and trace-derived arrivals are outside the current design. The design isolates intake and execution organisation under a controlled budget.

Reviewer_g82jOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper presents AOS-Bench, a benchmark that examines whether LLM agents can manage an ongoing workload on a single shared machine. Rather than giving the agent one task and waiting for it to finish, the benchmark introduces tasks over time through an external dispatcher. The tasks come from existing command-line benchmarks, while the agent is responsible for deciding which tasks to accept, when to start them, and how to handle new arrivals while other work is still in progress. The authors evaluate five model–harness pairs in both isolated-task and streaming settings to examine how well individual task-solving ability translates to workload management. The results suggest that these are distinct capabilities: an agent that performs well on standalone tasks may struggle when it must divide its attention across a stream of requests. A recurring failure is that agents become occupied with the current task and stop picking up new ones, effectively processing the workload sequentially despite opportunities to overlap execution.

Critical strengths
  • Timely problem. People are starting to hand agents several jobs at once, and current benchmarks only test one task at a time. This paper is one of the first to treat a whole workload as the thing being evaluated, and I think that framing will be useful to the community.
  • Clean experimental design. Running the same tasks alone, as a stream, and all at once (burst) makes the comparison easy to understand. Verification happens only after the agent stops, and the agent’s own claims of “done” don’t count toward the score, which avoids an obvious source of inflated results.
Critical weaknesses
  • The evaluation range is very limited. My main concern is that the content of this paper is too thin. It only shows some findings of an experiment of different model harnesses on a compound benchmark. And the evaluation is limited too. Everything runs on one machine configuration: 4 CPUs and 8 GB of memory. That setting is never varied, so we have no idea whether the findings hold on a larger machine, a tighter one, or with many more tasks. Even within this one configuration, resources barely seem to matter, since the CPU sits mostly idle. The “shared resource constraints” in the title aren’t really put under pressure. Beyond the hardware, there are only two workload sizes (10 and 20 tasks) and a small number of workloads. The policy experiments use just one model, and the burst-arrival condition is tested once. Tasks also have no priorities, deadlines or dependencies, and the dispatcher hands them out in fixed order. The agent therefore never faces the harder decisions a real workload manager would, like what to run first or what to drop. As it stands, the results describe one narrow point in a large design space. I’d want to see at least a few different resource settings, larger workloads, and tasks that actually compete for resources.
  • The paper lacks high-level insight into what the results mean. There are a lot of measurements, but not much interpretation. After reading it, I’m not sure what someone building agents should take away. A few examples: • The main failure is agents forgetting to keep pulling tasks from the queue. Is that a limitation of the models, or of the interface, where the agent has to keep checking an MCP tool? Would a simple reminder fix it? The paper doesn’t say. • A fixed rule of “run two tasks at a time” does as well as the agent’s own management on medium workloads. That suggests a trivial heuristic might be enough. It also raises the question of whether the benchmark measures anything beyond “does the agent keep taking work.” • The operating-system framing in the introduction promises a lot: scheduling, memory management, a natural-language OS. The results are never connected back to that vision. • There’s very little discussion of why some models collapse and others don’t. A section that pulls the findings together into a few clear lessons would make the paper much stronger.
  • The comparisons aren’t fully apples-to-apples. The systems differ in harness capabilities. For example, Codex has no way to delegate to sub-agents, and web tools are enabled for some systems but not others. So the headline gap between Opus and GPT can’t be cleanly attributed to how well the model manages a workload. The isolated and stream settings also differ in time budget and prompt, not just in scheduling.
  • Scoring choices and overall polish. Some trials and tasks are excluded after the fact, and the exclusion rules are described differently in different sections. Runs that used too much memory are dropped, even though overusing shared memory is itself a workload-management failure. Several numbers don’t line up between the main text and the appendices, and the appendix says parts of the configuration still need to be confirmed. Overall it reads like a preliminary version.
Questions for authors
  • Would the conclusions change on a larger machine, or a more constrained one?
  • Does a simple push notification or reminder about waiting tasks make intake collapse go away?
  • How does a basic non-LLM dispatcher that starts a fresh solver for every task compare?
  • What do you see as the main lesson for people designing agent systems?
3 comments
AuthorsOpenReview ↗

Official Response to Reviewer g82j [1/2]

[Q1] Would the conclusions change on a larger or a more constrained machine?

[A1] We evaluate all systems on one fixed budget: one container on one machine with 4 CPUs and 8 GiB. Fixing a single modest container is the standard evaluation unit in executable agent benchmarks. SWE-bench, Terminal-Bench, CORE-Bench and TerminalWorld normally score one objective in one resettable container, while OSWorld and WebArena extend the same containerised evaluation to desktop and browser interaction (Appendix L); a workload on one machine under a shared budget is our extension of that setting to several objectives at once. The budget is declared identically in all 26 workload specifications and is held constant across all five systems, both tiers, and the individual, stream and burst conditions, including the individual reference that defines Delta. Holding the hardware fixed is what makes the comparison controlled: differences in delivery reflect how a system organises work, not how much machine it receives. We therefore do not claim that the observed ordering is invariant to machine size. Sweeping CPU and memory budgets, or repeating the study on larger and on more constrained machines, introduces a separate experimental axis; We leave such a sweep, together with distributed placement, to future work.

[Q2] Does a simple reminder about waiting tasks remove intake collapse?

[A2] We have not run a direct experiment on a push-notification or reminder mechanism, and we therefore cannot answer this question from a dedicated ablation. Our evidence is indirect but constrains the answer. Emphasising intake in the prompt raises claim coverage on both tiers (by 5.3 and 10.2 points) but improves success only on medium (a 5.2-point loss on heavy), which suggests that the bottleneck is not merely the absence of a reminder. Section 4.2 shows that the coordinator holds the only dispatcher tools, so foreground execution blocks claiming: median main-agent shares of execution calls are 36% and 19% (medium and heavy) for Opus, 21% and 11% for Kimi, 84% and 37% for GLM, and 100% and 100% for DeepSeek. Within a workload, a ten-point increase in that share is associated with 3.9 points lower reward (95% CI 2.9 to 4.8) and 5.2 points lower coverage; tasks arriving during a foreground step wait twice as long to be claimed. A harness-level rule that reserves execution for sub-agents does not establish an improvement in reward over the unmodified agent. A controlled reminder mechanism — a dispatcher-side notification delivered to the coordinator on each arrival — is a natural design for a subsequent ablation, which would isolate the interface contribution from the model's own admission control. These are observational associations; no causal claim is made.

[Q3] How does a basic non-LLM dispatcher that starts a fresh solver per task compare?

[A3] This is the fixed-policy arm (Section 4.1, Table 3): a short scheduler launching one solver per task with the same model, tools, budget and prompt. Serial FIFO leaves 73.3% of heavy tasks unclaimed, whereas fixed concurrency of two comes within four points of the best agent-managed policy on both tiers. A simple heuristic is therefore a strong baseline, as the revised text states.

[Q4] What is the main lesson for agent-system designers?

[A4] The conclusion draws five lessons from the measurements. We state them separately:

(i) Keep admission control alive: a coordinator that stops claiming released work loses delivery even when it solves the tasks it holds.

(ii) Separate scheduling from execution: the main agent holds the dispatcher tools, so a foreground execution step blocks intake, whereas the fixed-concurrency driver avoids this by construction.

(iii) A constant concurrency floor is a strong baseline and should be enforced by the harness rather than requested in a prompt.

(iv) Report completion only after checking the artifact, since self-reported completion is poorly calibrated under workload pressure.

(v) Current models are optimized primarily for the role of the executor: their training targets single-task solving rather than the coordination of concurrent work. Optimizing models specifically for the managerial role is a natural direction, so that admission control and delegation become learned capabilities rather than prompt-level requests.

The operating-system analogy is tied to the measurements: the bottleneck is admission control and a non-blocking scheduler, not process scheduling or memory pressure. Beyond intake, the benchmark measures who executes the work, resource under-use, calibration of self-reports and stability across identical repeats.

AuthorsOpenReview ↗

Official Response to Reviewer g82j [2/2]

[Q5] Are the comparisons apples-to-apples, and how narrow is the evaluation?

[A5] The unit of comparison is a model-harness pair. Kimi invokes its web tool in 4 of 45 native trials; Codex has no delegation tool and is marked not applicable rather than zero. Approximately 11% of individual runs reach the 5400 s cap, and granting them full credit makes every Delta more negative. The stream budget is tighter per task (900 s on medium, 720 s on heavy); unfinished work accounts for up to 22.9 points of Delta, yet Opus leaves none unfinished. Execution time excludes setup, the preamble contains no scheduling algorithm, and task instructions are byte-identical; a prompt-only effect cannot be bounded without a further ablation. The Limitations record measurement gaps, statistical limits and evaluation scope, including the coarse heavy intervals, incomplete cells and fixed-order release.

The rebuttal addresses only a few of my concerns and does not resolve the issues I raised about the evaluation. I will therefore maintain my original rating.

Reviewer_ifswOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Coding and computer-use agent benchmarks run one instruction in one container against one verifier, so they cannot say what happens when requests keep arriving at a machine that is already busy. This paper defines a workload as a set of requests with release times sharing a fixed budget (4 CPUs, 8 GiB) and recasts the agent as a workload manager that decides when to claim new work, how to delegate it, and what may run concurrently. AOS-Bench merges tasks from Terminal-Bench, TerminalWorld, CompileBench and others into a single Docker image and compares three settings: individual (each task alone), stream (an external dispatcher releases tasks at their arrival times and the agent claims them over MCP), and burst (everything at t=0), with verification run only after all workers stop. Five model-harness pairs, two tiers, medium and heavy. The headline is that isolated ability does not predict delivery: GPT-5.6-sol and Claude Opus 5 share a 72.4% heavy individual baseline but deliver 49.7% and 72.2%, and the paper traces the gap to task intake rather than task solving.

Critical strengths
  • Moving the unit of evaluation from task to workload buys a separation no single-task benchmark can show: identical 72.4% heavy individual baselines delivering 72.2% against 49.7%, and Kimi-K3 completing 30, 30 and 5 tasks across three identical repeats. The paper names the mechanism, intake collapse, where a system stops claiming released tasks and degenerates to one at a time, and supports it with claim latency, peak concurrency, intake tool-call share and terminal-state decomposition, having ruled out API errors and output caps.
  • The appendices document the study's own confounds far above the norm for benchmark papers. H.3 admits amortized container setup "favors stream execution in the throughput comparison", §3.2 admits repeated instances on a shared filesystem bias stream SR upward, Appendix G discloses a reachable path for an agent to write reward.txt. Most of the material that defeats the headline claims comes from the authors themselves, so the defects are at least fixable by recomputation rather than new runs.
Critical weaknesses
  • The sign of the headline result is partly manufactured by the exclusion rule, and the main text describes a different rule than the one that was used. §3.1 says seven construction-defective tasks and three broken graders are dropped from both settings. Appendix E says Table 2 actually keeps only tasks whose individual baseline is nonzero for that system, 11 to 15 of 53 heavy tasks each. That is selection on the dependent variable, and it removes exactly the tasks where stream benefits from sibling artifacts in the shared filesystem. Appendix E concedes that under the restriction "no positive value" remains. Each system is scored on a different set, which makes the Abstract's "the same tasks" false, and no unrestricted ΔSR\Delta\mathrm{SR} appears anywhere in the paper.
  • The manuscript does not agree with its own tables, on the numbers that carry the claims:
    • Eq. (2) reproduces no row of Table 2. Heavy Kimi recomputes to 2.38 against the reported 2.87, and all ten rows are high by 2–21%. Dropping OOM trials from SR but not ET cannot explain it: GPT has 0.0% OOM at both tiers and is still off by 4%.
    • The Abstract's "no tested policy uses more than 30% of the provisioned CPU capacity" is refuted by Table 3's own heavy native row, 282.7/(4×217.8)=32.4%282.7/(4\times217.8) = 32.4\%.
    • Appendix D cites Table 2 for GLM retention of 78%. Table 2 gives 49.4/68.4=72.2%49.4/68.4 = 72.2\%, and §3.2 independently says 67%–72%.
    • The inventory is 27 streams / 53 tasks / 250 slots in Figure 3 against 26 workloads / 122 distinct tasks / 280 occurrences in Appendix A.
  • ΔSR\Delta\mathrm{SR} is not a scheduling contrast. Individual runs get a raw task brief, no task-pool tools and a 5400 s agent-phase cap; stream gets a 2095-byte mission prompt, amortized setup and a 9000–21600 s per-request cap. §5 and H.5 say so, while the Abstract still narrates the gap as isolated ability versus delivery ability. The statistics cannot carry what is left either: five heavy clusters the paper itself calls "too few for a calibrated interval", no intervals in Table 2, every policy arm but serial FIFO null, all of it on GLM-5.3 alone.
Questions for authors
  1. Which exclusion rule generated Table 2? The fixed set in §3.1, "Seven tasks with evidenced construction defects and three with broken graders are excluded from both settings ... retained scoring sets contain 95 and 98 task occurrences", or the per-system rule in Appendix E, "Table 2 restricts the paired comparison to tasks whose individual baseline is nonzero," dropping 5–7 tasks per system on medium and 11–15 on heavy? Please also reconcile §2.3 and the Table 2 caption, which state seven excluded tasks, with §3.1's seven plus three.
  2. Please report ΔSR\Delta\mathrm{SR} for all five systems at both tiers without the zero-baseline exclusion, on a task set identical across systems. Does the universal negative sign survive? If not, which claims in the Abstract and Introduction change?
  3. Given that 11–15 different heavy tasks are dropped per system, on what task set are the Abstract's "72.2% and 49.7% of the same tasks" and §3.2's "on the identical task set" computed? How does H.7's note that GPT's individual baseline covers "31 of 32 medium task occurrences" affect that pairing?
  4. Appendix E states a heavy nonzero-baseline range of −3.1-3.1 to −28.0-28.0 while Table 2's heavy Δ\Delta column runs −0.2-0.2 to −22.6-22.6, and a medium range of −3.1-3.1 to −30.2-30.2 against Table 2's −4.6-4.6 to −30.1-30.1. Which set is correct, and what explains the identical −3.1-3.1 lower bound at both tiers?
1 comment
AuthorsOpenReview ↗

Official Response to Reviewer ifsw

[Q1] Which exclusion rule generated Table 2, and why did Sections 2.3 and 3.1 disagree with the caption?

[A1] We agree that the submitted manuscript was inconsistent on this point. Table 2 applies one fixed exclusion set to every system: ten tasks that produced a false zero (a zero reward written by a grader that never ran) and one task whose LLM-judge verifier cannot execute in the sandbox. Both settings drop the same eleven tasks, retaining 87 of 100 medium and 79 of 100 heavy task slots. The per-system nonzero-baseline restriction is absent; zero-baseline tasks are retained and reported separately. Sections 2.3 and 3.1, the caption and the appendix state the same rule. Individual success is matched to each stream trial's scored tasks by task identifier and instruction hash, averaged within workloads and then equally across workloads.

[Q2] Does the negative sign of Delta survive without the zero-baseline exclusion, on a task set identical across systems?

[A2] Yes. Unrestricted Delta (medium; heavy, with 95% workload-clustered confidence intervals): Opus -1.8 [-10.7, 6.1] and -13.0 [-25.1, -4.8]; GLM -21.1 [-32.0, -12.7] and -15.7 [-26.4, -3.5]; DeepSeek -25.1 [-29.4, -20.5] and -22.7 [-33.3, -8.3]; Kimi -6.3 [-17.7, 5.6] and -19.3 [-25.1, -14.0]; GPT -14.5 [-22.6, -6.7] and -18.1 [-28.5, -8.0]. All ten estimates are negative and eight intervals exclude zero. Medium Opus and Kimi are not individually significant, and the heavy tier rests on five clusters; Section 3.2 states both caveats, and the Limitations note the coarse heavy intervals.

[Q3] On what task set were the "same tasks" figures computed, and how does GPT's incomplete medium baseline affect the pairing?

[A3] Those figures belong to the submitted version and no longer appear. "Same tasks" denotes a task-matched individual reference: for each stream trial, the individual baseline is computed over exactly that trial's scored tasks, matched by task identifier and instruction hash. The caption discloses that GPT-5.6-sol medium covers 27 trials over nine of ten workloads, owing to an image setup failure, and that Opus medium covers 29 trials. Cross-system comparisons involving GPT medium therefore use nine workloads.

[Q4] Which values are correct, the appendix ranges or Table 2, and why was the lower bound identical at both tiers?

[A4] The appendix passage describing the per-system restriction is absent, as is the restriction itself; zero-baseline counts are reported instead (four to six per system on medium, six to eight on heavy). Both the mismatch and the repeated lower bound follow from that inconsistent selection rather than from the scored task set.

[Q5] Can Eq. (2) reproduce Table 2; is the 30% CPU claim consistent with Table 3; and what is the correct inventory?

[A5] All ten rows reproduce to rounding. The heavy DeepSeek value of 1.57 is correct under Q_str/Q_ind, which sums rewards and times over workloads before dividing (1.567); multiplying the displayed SR and ET columns gives 1.522 because workloads contain 13 to 19 scored tasks. Eq. (2) uses an approximation sign and states the equality condition. The 30% statement refers to the per-trial median CPU of the four GLM-5.3 policies while tasks wait; the substantially higher peaks are reported, and comparable telemetry is unavailable for the other systems. The inventory comprises 26 workloads, 280 task slots and 58 distinct tasks. The earlier count of 60 arose from an exporter treating version numbers as repeat-instance suffixes; the figure, appendix, release note and exporter are consistent.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.