acceptodds
Under review as a conference paper at ICLR 2027

TUA-Bench: A Benchmark for General-Purpose Terminal-Use Agents

Abstract

As large language models and harness frameworks continue to advance, agents operating in terminals are increasingly capable of performing a broader range of general computer-use tasks beyond coding. However, existing benchmarks do not adequately evaluate general-purpose terminal computer-use agents (TUAs): general computer-use benchmarks primarily target graphical user interfaces (GUIs), whereas terminal-based benchmarks largely emphasize technical and programming-centric workflows historically native to the shell. We introduce TUA-Bench, a general-purpose benchmark for terminal-use agents. TUA-Bench includes 120 real-world tasks across five task families, covering routine digital activities—including document editing, email management, and live-web information seeking—as well as scientific and engineering workflows co-designed with PhD-level domain experts that require specialized software. This breadth distinguishes TUA-Bench from prior shell-focused or domain-specific benchmarks. Each task is manually designed, runs in a real terminal with a deterministic setup script, and is evaluated by an execution-based scoring protocol. We find that the strongest frontier agent, Claude Code with Claude Opus 5.5 max reasoning effort, achieves 74.5% overall performance, with substantial gaps across both tracks. By providing a broad and realistic evaluation of terminal-use capabilities, TUA-Bench aims to accelerate the transition from narrow, task-specific assistants to general-purpose agents capable of operating reliably across diverse digital environments.

Primary area:
datasets and benchmarks
Keywords:
Terminal-use agent, Computer-use agent, general-purpose

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_yRK2OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a novel benchmark to evaluate general-purpose terminal computer-use agents. The benchmark includes 120 tasks spanning five families: Office & Productivity, Web & Information, System & Software Operations, Scientific & Engineering, and Multimedia & Design. The tasks are also representative of both everyday digital tasks and professional scientific tasks that were manually curated by PhD-level experts. Their evaluation spans a large variety of model-agent pairs and proposes different metrics investigating both task-solving skills and reliability.

Critical strengths
  • S1: Timely approach
    Terminal tools are gradually becoming a general interface for agents to operate beyond developer tools. Current benchmarks only focus on those. This work successfully fills this gap.
  • S2: Wide evaluation, with metrics that separate occasional from reliable success. The evaluation presented covers several agent scaffolds, models, agent-model-effort configurations on multiple trials each. The metrics also allow to disentangle occasional success from reliable task completion.
  • S3 (Minor): Clear writing.
    The paper is pleasant to read and well structured.
Critical weaknesses
  • W1: The interface comparison is missing
    Section 1 highlights how GUI might not be the most suitable type of interface for LLM-based agents. In contrast, they propose CLI environments as "naturally aligned with the strength of language models". It's not clear to me if this fact has been confirmed by the literature. I believe it would be great for the motivations of this paper to compare also against GUI environment (for tasks that allow it) to give evidence for such statement.
  • W2: Solvability of task is not established
    Some tasks (like task 104-arxiv-llm-workbook) are not solved by any agents. However, they might depend on external sources changing over time (for 104 it depends on aXiv's results). Since the gold answer is static, it is unclear whether an agent that successfully completes the task, could still be marked as failing.
Questions for authors
  • Q1: How do task that rely on live sources (like 104) remain solvable?
  • Q2: Can the author clarify their statement reported in W1, is there accepted evidence confirming it?
Reviewer_w1cNOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The motivation for this paper is the idea that GUI benchmarks measure visual grounding while terminal benchmarks measure programming. The goal is therefore to create a terminal-only benchmark. It adapts 100 tasks from OSWorld for the terminal and adds 20 expert created. The evals consist of 41 agents with 5 trials each. They find that an Opus 5.5 model with a claude code harness performs best.

Critical strengths

The biggest strength is the breadth of evaluation covering 41 configurations including the very latest models at the point of submission. Some of the new expert workflows are genuinely interesting.

Critical weaknesses

One critical issue is the lack of clarity around the metrics. The pass@1 rate for 120 tasks is not a multiple of 1/120 so it must be the average across the five runs, which should make it exactly the same as the “success rate” column, but they don’t match.

Success rate is defined in the paper as “per-task pass rate averaged over five independent runs”. And some values are outright confusing. Opus 4.8 max has higher average success rate than the pass@5, which doesn’t seem possible.

And success rate differs significantly from pass@1 and is strictly higher, so it reads more like some sort of partial reward metric.

Another major clarification issue is for the task validity. There are no oracle solutions given, so whether these tasks are actually solvable at all remains unproven. From figure 8 it looks like a number of tasks have 0 for all models shown. Verifiers are also not described.

Some cases of overstatement. The benchmark is described as manually designed but 100 of the 120 tasks are adapted from OSWorld. Terminal-Bench 2.0 is listed as having 0 scientific subjects while it actually has scicomp category with DNA, protein assembly etc.

Questions for authors
  1. What was the performance per track?
  2. How was manual verification done? I dont find documentation of the process, criteria, counts etc.
  3. These are described as terminal tasks, but there are references to tasks in Chrome. How are GUI apps operated from the terminal envs?
  4. Does every task have an oracle solution that passes verifier?
  5. Having the solvability on a subset of the models filter the tasks can lead to potential bias vs models that played no part in task selection. Was anything done to either measure this bias or counteract it?
  6. The abstract mentions gaps in long horizon planning, execution monitoring and error recovery yet I don’t see any trajectory analysis.
Reviewer_2PzVOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper differentiates itself from general CUA benchmarks and CLI benchmarks by reformulating tasks that require GUIs into terminal tasks. In my opinion, the most significant contribution is moving the first step toward the goal of designing a new interface for agents to use. As the coding capabilities are increasing rapidly, more and more agents are working on personal everyday online tasks. TUA-Bench includes areas like document editing, email management, web search, and scientific and engineering workflows. Claude Opus 5.5 achieves 74.5% overall performance.

According to Figure 2, this benchmark contains 38.3% office tasks and 14.2% science and engineering tasks. The experiments are comprehensive, and the analysis is detailed. Concerns and limitations include benchmark saturation and limited support for all cua-related tasks. This paper also lacks technical details about how to transform a GUI task into a CLI task. Lines 248-257 only provide a very vague description of how to transform OSWorld tasks into terminal tasks.

Critical strengths
  1. This paper is working in an interesting direction. Designing the correct interface for modern AI agents is one goal our community is working toward. This paper transforms traditional GUI tasks into terminal tasks and provides comprehensive experiments and analysis. This helps distinguish this benchmark from existing coding benchmarks and personal agent benchmarks.

  2. This paper provides detailed analysis and task-level results. Table 4 and Figure 8 are informative and provide detailed results of each model and each task. This helps the community to do further case studies and work on maximizing the performance on this benchmark. Table 5 provides concrete examples of scientific tasks, which are quite interesting.

  3. This paper provides comprehensive experiments across different models and harnesses. Models include GPT-5.5, Opus 5.5, Gemini 3.1 Pro, DeepSeek-V4 Pro, and harnesses include Terminus-2, Mini-SWE-Agent, OpenHands, Codex, and Claude Code. The combination of different models and harnesses provides information for the community on which harnesses and models to use when handling CLI-based personal agent tasks. Besides, Figure 6 provides a nice illustration of Pareto improvement.

Critical weaknesses
  1. This paper does not provide enough details on how the GUI tasks are transformed into terminal tasks. Lines 1404-1410 and lines 1419-1425 include the same prompt. "You can do this optionally using Chrome if needed. Open the relevant page and, when finished, leave that exact page selected as the active tab in the foreground window." How can this be achieved without using the GUI? Similarly for Task 052, "Given the file ˋ/app/dog_with_background.pngˋ, could you help me open the Vignette filter window, optionally using GIMP if needed? Once the window is open, please quit the application." How can this action be achieved without using the GUI? If this is achieved by a virtual display like Xvfb, what is the difference between TUI-Bench and a GUI-related CUA benchmark?

  2. Task quality concerns. "Task • Office & Productivity - Presentations - 082-brighten-slide-image", "Task • System & Software Operations - Software Development - 092-fix-tetris-bug", "Task • Office & Productivity - Documents - 097-slides-to-document", and "Task • Web & Information - Public Reference - 105-search-cell-b6" contain "Given the file {path}"; what is the meaning of "{path}"? This raises concerns about data quality. If {path} is not filled with the exact paths, these four tasks are not solvable by agents. "Task • Multimedia & Design - Video & Audio Editing - 112-capture-video-frame" requires "Given the file ~/Desktop/Interstellar Movie - Official Trailer.mp4, could you help me snap a photo of a scene from the video? You can optionally use VLC Media Player if needed, and a headless ˋvlcˋ or ˋcvlcˋ workflow is perfectly acceptable in this terminal-first Harbor environment. Please save the final output file exactly at ~/Desktop/interstellar.png, as only the file at that exact path will count for the score." This task does not require the specific information about which frame to capture. According to Figure 8, 112-capture-video-frame does not have a green grid, which supports that this task may not be solvable.

  3. Metrics definition. According to Table 4, success rate is the per-task success rate averaged over five independent runs. Pass@k is the fraction of tasks solved within K attempts, and All-5 is the fraction solved on every attempt. In Table 4, line 3, Claude Code + Claude Opus 4.8: Pass@5 is 64.2%, and the pass rate is 65.8±0.75; how is this possible?

Questions for authors
  1. This paper does not provide enough detail on how the GUI tasks are transformed into terminal tasks. Lines 1404-1410 and lines 1419-1425 include the same prompt. "You can do this optionally using Chrome if needed. Open the relevant page and, when finished, leave that exact page selected as the active tab in the foreground window." How can this be achieved without using the GUI? Similarly for Task 052, "Given the file ˋ/app/dog_with_background.pngˋ, could you help me open the Vignette filter window, optionally using GIMP if needed? Once the window is open, please quit the application." How can this action be achieved without using the GUI? If this is achieved by a virtual display like Xvfb, what is the difference between TUI-Bench and a GUI-related CUA benchmark? Question: Please answer the questions above, and provide more technical details about how the transformation is performed.

  2. Task quality concerns. "Task • Office & Productivity - Presentations - 082-brighten-slide-image", "Task • System & Software Operations - Software Development - 092-fix-tetris-bug", "Task • Office & Productivity - Documents - 097-slides-to-document", and "Task • Web & Information - Public Reference - 105-search-cell-b6" contain "Given the file {path}"; what is the meaning of "{path}"? This raises concerns about data quality. If {path} is not filled with the exact paths, these four tasks are not solvable by agents. "Task • Multimedia & Design - Video & Audio Editing - 112-capture-video-frame" requires "Given the file ˋ~/Desktop/Interstellar Movie - Official Trailer.mp4ˋ, could you help me snap a photo of a scene from the video? You can optionally use VLC Media Player if needed, and a headless ˋvlcˋ or ˋcvlcˋ workflow is perfectly acceptable in this terminal-first Harbor environment. Please save the final output file exactly at ~/Desktop/interstellar.png, as only the file at that exact path will count for the score." This task does not require the specific information about which frame to capture. According to Figure 8, 112-capture-video-frame does not have a green grid, which supports that this task may not be solvable. Question: Please answer the questions above about the data quality.

  3. Metrics definition. According to Table 4, success rate is the per-task success rate averaged over five independent runs. Pass@k is the fraction of tasks solved within K attempts, and All-5 is the fraction solved on every attempt. In Table 4, line 3, Claude Code + Claude Opus 4.8: Pass@5 is 64.2%, and the pass rate is 65.8±0.75; how is this possible? Question: Please provide further illustrations of how the metrics are calculated.

Reviewer_pnz9OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces TUA-Bench, a benchmark for general-purpose terminal-use agents, with 120 tasks spanning everyday and professional workflows. The paper evaluates a broad range of frontier models and agent frameworks. Overall, the benchmark addresses a useful and underexplored setting, though I have concerns about task-set novelty and some empirical conclusions.

Critical strengths

1.Useful and well-motivated evaluation setting. The paper identifies a meaningful gap between GUI-based general computer-use benchmarks and programming-centric terminal benchmarks. Evaluating office productivity, web/information tasks, multimedia tasks, and professional workflows through a terminal interface is a useful direction.

2.Broad model and agent evaluation. The paper evaluates a relatively comprehensive set of frontier models and multiple agent scaffolds. The repeated trials and reporting of Success Rate, Pass@1, Pass@5, and All-5 are also useful, particularly because reliability across repeated executions is important for computer-use agents.

3.Execution-grounded evaluation. Evaluating the final environment/artifact state rather than relying on trajectory-level heuristics is appropriate for this setting and makes the benchmark practically meaningful.

4.The professional task subset is valuable. The 20 expert-designed scientific and engineering tasks extend the benchmark beyond standard software-engineering workloads and provide some genuinely new task settings.

Critical weaknesses

1.Limited task-set novelty and unclear additional evaluation signal beyond OSWorld. 100 of the 120 tasks are directly adapted from OSWorld, reusing the underlying task goals and artifacts. While converting these tasks from GUI to terminal interaction requires nontrivial effort, this substantially limits the novelty of the benchmark construction. This concern could be alleviated by demonstrating that the two benchmarks measure meaningfully different capabilities. For example, if task difficulty is highly correlated between OSWorld and TUA-Bench, the additional evaluation value may be limited. In contrast, analyzing tasks where GUI agents succeed but terminal agents fail (and vice versa), and explaining the causes of these differences, would provide much stronger evidence for the novelty and value of TUA-Bench.

2.Category-level conclusions need stronger statistical support. Some top-level categories contain only 16–22 tasks, while the 20 subcategories contain even fewer. Such results are useful descriptively, but stronger conclusions about which capabilities or categories are harder require uncertainty estimates. Repeating each task five times does not increase the number of independent tasks. Task-level confidence intervals or bootstrap analyses would make these claims more convincing.

3.Several empirical conclusions are useful but relatively expected. The paper finds that stronger underlying models matter substantially, higher reasoning effort improves performance with diminishing returns, and different agent scaffolds produce relatively modest differences when controlling for the model. These are reasonable observations, but they provide limited new insight. The model-vs-scaffold conclusion is also not fully disentangled because the newest models are evaluated only with their native agents. A more detailed failure analysis could provide more novel diagnostic insights.

4.The time-budget experiment is more useful for protocol calibration than as a scientific finding. The observed trend (more execution time leads to fewer timeouts and higher success) is largely mechanical and therefore not particularly informative by itself. However, this experiment could be valuable for determining an appropriate evaluation timeout: how much time should an agent receive before the benchmark primarily measures capability rather than premature termination? Framing and analyzing the experiment from this perspective, for example through completion-time distributions, would make it substantially more informative.

Questions for authors

1.For the 100 tasks adapted from OSWorld, have the authors analyzed the relationship between OSWorld task difficulty and TUA-Bench task difficulty? In particular, which tasks become substantially easier or harder after moving from GUI to terminal interaction, and why?

2.Can the authors provide task-level confidence intervals or statistical significance analyses for the per-category comparisons in Figure 5? How robust are the reported category-level differences to task sampling?

3.Do the task verifiers expose meaningful partial-credit or subgoal-level signals? If so, could these be reported to provide a more granular measure of agent capability than binary task success?

4.For the timeout experiment, what is the empirical distribution of completion times among successful trajectories? Is there direct evidence that trajectories terminated under smaller budgets were generally progressing toward successful solutions?

5.Could the authors provide a more systematic failure taxonomy? Understanding whether failures arise from planning, tool use, environment interaction, artifact manipulation, verification, or recovery would make the benchmark considerably more informative.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.