- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents AOS-Bench, a benchmark that evaluates how a persistent agent system manages tasks arriving over time under a shared CPU and memory budget. It uses source benchmarks such as Terminal-Bench and SkillsBench to construct workloads. A workload includes a set of tasks that are to be executed under a shared CPU and memory budget. The paper uses AOS-Bench to evaluate five model-harness pairs.
Critical strengths
- The paper presents AOS-Bench to evaluate how well a persistent agent system can manage a stream of tasks. Dynamic workloads create challenges for agent systems to efficiently leverage limited resources to handle multiple tasks.
- The evaluation includes five model-harness pairs. The results cover metrics such as task success rate and rewarded-task throughput ratio.
Critical weaknesses
- The description of the benchmark is vague. It is unclear how the workload is generated and whether it matches realistic scenarios.
Questions for authors
-
Does AOS-Bench consider distributed scenarios? Does the setup assume the shared CPU and memory resources on the same machine?
-
AOS-Bench constructs workloads using a set of source benchmarks, such as Terminal-Bench, SkillsBench, TerminalWorld, etc. How do you draw tasks from these source benchmarks to construct a workload in AOS-Bench? Do you randomly draw tasks from these source benchmarks?
-
In Section 2.2, the paper mentions that the source benchmarks are "supplemented by synthetic tasks". What are the synthetic tasks?
-
Can you clarify how the release time is set in the evaluation? Are the tasks separated by a fixed duration? Do you consider using traces from realistic scenarios?
-
Does AOS-Bench match realistic scenarios where an agent handles a stream of tasks? Can the way that the workloads are constructed from the source benchmarks and the release times represent real-world dynamic workloads?
Official Response to Reviewer DN8N
[Q1] Does AOS-Bench consider distributed scenarios?
[A1] The benchmark targets one container on one machine with 4 CPUs, 8 GiB, one filesystem and one dispatcher. Multi-machine placement lies outside this work, as Section 1 states.
[Q2] Are workloads sampled at random from the source benchmarks?
[A2] No. Tasks are profiled by duration and CPU use into L1/L2/L3 bands and assembled under a low-overlap rule so that instructions, reference solutions and verifiers do not collide on shared paths. Four gates apply: a source audit, image build and launch, reference solutions reaching full score in the merged image, and an overlap check against admitted workloads. Sections 2.2 and Appendix A describe this procedure.
[Q3] What are the synthetic tasks?
[A3] Three families are developed for AOS-Bench: a compression-Pareto task, a C project-repair loop (also present as a repeated instance) and a prime-sieve task. Each has an instruction, environment, reference solution and executable verifier, and each passes the same gates. Appendix A lists them.
[Q4] How are release times set, and are production traces used?
[A4] Release times form a designed per-workload schedule in stream.toml: ten tasks over approximately 20 minutes on medium, twenty over approximately 60 minutes on heavy, with non-uniform gaps. The dispatcher hides future arrivals, so the agent sees only released tasks. These are designed schedules, not production traces.
[Q5] Does the setting represent realistic dynamic workloads?
[A5] The unit tasks come from real-world benchmarks — Terminal-Bench, SkillsBench, TerminalWorld, WildClawBench, CompileBench, CompileAgent, ParEval and the One Billion Row Challenge, supplemented by three synthetic task families. They are therefore genuine developer workloads rather than synthetic placeholders. It is the release process that is designed: the workload assembles these tasks into a shared container under a fixed 4 CPU / 8 GiB budget with a non-uniform arrival schedule. The overall release strategy is closer to a stress test than to a replay of a production trace, in that it concentrates heterogeneous tasks in time under tight shared resources. Such bursts are nevertheless a challenge that a persistent agent may face in real deployments, and intake failure under concurrency is exactly the failure mode the benchmark isolates. We do not claim that the arrival distribution matches production traffic; priorities, deadlines, dependencies, multi-machine placement and trace-derived arrivals are outside the current design. The design isolates intake and execution organisation under a controlled budget.