- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a novel benchmark to evaluate general-purpose terminal computer-use agents. The benchmark includes 120 tasks spanning five families: Office & Productivity, Web & Information, System & Software Operations, Scientific & Engineering, and Multimedia & Design. The tasks are also representative of both everyday digital tasks and professional scientific tasks that were manually curated by PhD-level experts. Their evaluation spans a large variety of model-agent pairs and proposes different metrics investigating both task-solving skills and reliability.
Critical strengths
- S1: Timely approach
Terminal tools are gradually becoming a general interface for agents to operate beyond developer tools. Current benchmarks only focus on those. This work successfully fills this gap. - S2: Wide evaluation, with metrics that separate occasional from reliable success. The evaluation presented covers several agent scaffolds, models, agent-model-effort configurations on multiple trials each. The metrics also allow to disentangle occasional success from reliable task completion.
- S3 (Minor): Clear writing.
The paper is pleasant to read and well structured.
Critical weaknesses
- W1: The interface comparison is missing
Section 1 highlights how GUI might not be the most suitable type of interface for LLM-based agents. In contrast, they propose CLI environments as "naturally aligned with the strength of language models". It's not clear to me if this fact has been confirmed by the literature. I believe it would be great for the motivations of this paper to compare also against GUI environment (for tasks that allow it) to give evidence for such statement. - W2: Solvability of task is not established
Some tasks (like task 104-arxiv-llm-workbook) are not solved by any agents. However, they might depend on external sources changing over time (for 104 it depends on aXiv's results). Since the gold answer is static, it is unclear whether an agent that successfully completes the task, could still be marked as failing.
Questions for authors
- Q1: How do task that rely on live sources (like 104) remain solvable?
- Q2: Can the author clarify their statement reported in W1, is there accepted evidence confirming it?