- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces TUI-Bench, a benchmark of 492 tasks across 77 terminal applications and 19 categories. Agents interact through keyboard actions and text-grid observations, while task-specific verifiers evaluate the final application state and persistent artifacts. The paper evaluates a broad set of agent configurations and analyzes their failures. The best reported average success rate is 22.9%, suggesting substantial difficulty under the benchmark's interaction protocol.
Critical strengths
- The benchmark is relatively comprehensive and self-contained, covering a wide range of terminal applications with packaged environments and executable verifiers. These tasks and evaluation infrastructure could be a useful contribution to the community.
- The evaluation covers many models and provides analyses of interaction failures, including repeated ineffective actions and response truncation. These results provide useful evidence about the limitations of language agents under the specified interface.
Critical weaknesses
- The contribution relative to existing terminal benchmarks could be clearer justified. In particular, what important tasks or interaction requirements are missing from Terminal-Bench and TUA-Bench, and how does TUI-Bench address them? The application catalog is broad, but does not include coding-agent interfaces such as Codex or Claude Code, or terminal games. The authors should explain the intended scope and establish why the selected applications and tasks represent an important gap in existing evaluations.
- The reliability and fidelity of the evaluation are not sufficiently established by the reported results. Appendix A.6 describes positive and negative verifier fixtures and a human audit, which are valuable checks. However, agreement on expert ratings does not establish human completion rates or verifier accuracy. Furthermore, observations omit color and styling, which may carry useful information in applications such as htop/iotop. Without quantitative human results and an assessment of the information removed by the interface, it is difficult to determine how much of the low success rate reflects agent limitations rather than the evaluation protocol.
Questions for authors
- Task solvability and verifier reliability: What fraction of the final tasks did the experts successfully complete under the same observations and budgets as the models, according to the final verifiers? Were the positive fixtures produced by successful keyboard-action trajectories or constructed directly? Please report agreement between verifier decisions and independent human judgments, including valid alternative solutions and invalid near-success states.
- Observation fidelity and budgets: Do any tasks depend on color or styling that is absent from the text-grid observation? A comparison using color-preserving observations would help establish whether this omission materially changes task difficulty. Separately, how sensitive are the reported results to the response-token limits responsible for truncation?
- Real-time interaction: Do monitoring applications such as htop continue updating while the model is generating its next action? If so, can process ordering, selection, or other task-relevant state change between observation and action? Please explain how
wait_idle, refresh timing, and final-state verification handle continuously updating interfaces, and whether differences in model latency affect task difficulty.
3 comments
Response to Reviewer tPGq (PART 2/2)
Q2. Observation fidelity and output budget.
We thank you for the careful questions on observation fidelity and output budget. We address color and the output cap in turn.
1. Does any task depend on color? We checked all 492 instructions and found no task that requires color or styling to solve. Verdicts check machine-decidable conditions such as application state, files, session state, and Git state, and the 223 terminal-surface checks are token-level. Of these 492 instructions, only 12 mention color and none uses it for solving; wherever a visual attribute could be mistaken for task-relevant information, the instruction asks the agent to judge by visible characters alone. The htop selection is verified from the final application state; the verifier does not read color.
2. Does color-preserving observation change task difficulty? We ran a matched comparison: on the same model and the same 100-task stratified subset, we augment the character grid with a compact color hint; Table 1 reports the results. The hint is anchored to the entry frame and reused.
Table 1. Three-pass Avg@3 by observation modality (%)
| Observation / action | State-backed, 72 tasks | Disk-backed, 28 tasks | Overall (scored) |
|---|---|---|---|
| Character grid / keyboard | 9.3 | 28.6 | 15.5 |
| Character grid + color hint / keyboard | 13.4 | 30.3 | 18.3 |
Note: Overall is computed over scored tasks; the state and disk columns use their respective scored tasks.
The gain is small and concentrated on state-backed tasks: 15.5% to 18.3% overall, 9.3% to 13.4% state-backed, disk-backed unchanged. The color hint gives a limited improvement; state-backed success remains at 13.4%, so the gap is far from closed. The task-level answer is direct: the verifier checks application state, files, and lifecycle; color is outside the verdict, and every task is solvable from the character grid by design.
3. How sensitive are the results to the output cap? Our evaluation uses a 32,768-token output cap throughout; repair and transport retries are capped at two, and the episode budget is 150 steps, 2,048 keyboard input units, and 5,400 seconds. Long reasoning can exhaust a single response, so we label episodes that hit the cap and compare them against the thinking and non-thinking twins reported in the appendix. At 32,768, cap hits are most prominent in MiniMax-M3, while other thinking families show them to varying degrees. We therefore do not attribute the main results to any single model's truncation behavior; task difficulty and state reconstruction remain the more likely causes, and our conclusions are stated for the fixed 32,768-token protocol without extrapolation to other caps.
Q3. Continuously updating interfaces and latency.
We thank you for the question on real-time interaction. We address continuous refresh, state changes, and latency in turn.
1. Continuous refresh and state change. Yes, they do. Monitoring programs such as htop keep refreshing while the model generates its next action. The model sees the interface at request time, and between observation and action, process ordering, the current selection, and other task-relevant state can indeed change. This is a real property of TUI interaction, and the protocol is designed to handle it.
2. wait_idle and final-state verification. The action space provides wait_idle: the model can ask the runtime to wait for the screen to settle before taking a fresh observation, at the cost of one step, and the model decides when to use it. After each action, the runtime also waits for the interface to settle before the next step. Scoring uses only the final state: once an evaluation ends, the final state is captured, and monitoring tasks use the final screen content rather than any intermediate frame.
3. Does model latency affect task difficulty? Across 20 generative configurations, none solved the monitoring tasks; including the non-generative baseline yields only isolated successes. Notably, faster and slower configurations exhibit consistent failure patterns: repeated ineffective actions and incorrect terminal states. This reveals a distinction between response speed and sustained interface control: faster responses may narrow the window for state changes but cannot replace continuous state tracking. These results thus highlight the central challenge of dynamic TUI operation: reliably maintaining and updating interface state as the screen changes.
We thank you again for pressing on evaluation reliability and fidelity. If these responses address your concerns. We would sincerely appreciate it if you could consider raising the current score in support of accepting this work.
Dear Reviewer tPGq,
Thank you very much for your time and effort in reviewing our paper. We sincerely appreciate your valuable comments and constructive suggestions, which have helped us improve the quality of our work.
We would like to kindly follow up on our previous rebuttal and ask whether our responses have adequately addressed your concerns. If you have any remaining questions or concerns, we would be more than happy to provide further clarifications.
Thank you again for your time, consideration, and valuable feedback.
Best regards, The Authors
Response to Reviewer tPGq (PART 1/2)
We thank you for recognizing that the benchmark is comprehensive and self-contained: packaged environments and executable verifiers across 77 applications are a valuable resource, and the failure analysis provides useful evidence. We have carefully adopted your suggestions on solvability, observation fidelity, and real-time interaction. We address the four points in order.
W1. Positioning against terminal benchmarks and scope.
We thank you for the question on positioning and scope. The key contribution of TUI-Bench is establishing native TUI operation as a distinct evaluation paradigm, rather than simply expanding the collection of terminal tasks. Its novelty lies in three layers: the interaction paradigm, the evaluation protocol, and the application scope.
1. A distinct interaction paradigm. Existing terminal benchmarks such as Terminal-Bench and TUA-Bench primarily evaluate task outcomes through CLI workflows, where agents can issue commands, execute scripts, or manipulate files directly. Native TUIs require a fundamentally different form of interaction: text is the interface itself, not merely command output. Characters encode spatial layouts and interactive controls, while the meaning of each keystroke depends on hidden application state. Agents must continuously interpret the evolving interface and act through the keyboard, a capability that conventional CLI evaluations do not isolate.
2. A complete evaluation protocol. TUI-Bench makes this capability directly measurable by starting inside running applications, restricting interaction to character-grid observations and keyboard actions, and jointly verifying application state, data preservation, and lifecycle. This prevents agents from bypassing the interface and ensures that success reflects actual TUI operation. Our results further show that models perform substantially better when the interface offers a place to type a command, while struggling with continuous stateful interaction. This reveals a capability gap that existing outcome-oriented terminal benchmarks cannot independently measure.
3. A well-defined application scope. TUI-Bench focuses on native, stateful terminal applications with concrete operational goals, spanning 77 TUI tools and 19 application categories. Coding-agent interfaces introduce additional model-driven behavior, while terminal games emphasize game-specific objectives. Our scope deliberately isolates general-purpose TUI operation from these additional factors.
Overall, TUI-Bench introduces a systematic and reproducible evaluation paradigm for understanding and controlling stateful interfaces rendered entirely as text, complementing existing CLI and GUI benchmarks.
W2/Q1. Solvability, human completion, and verifier reliability.
We thank you for the careful questions. Task solvability and verifier reliability rest on four independent forms of evidence.
Executable gate: verifier reliability. The positive fixtures are constructed directly from each task's expected-state specification, not produced by any keyboard trajectory. Before freezing, each task must pass its positive fixture twice and reject all three negative fixtures; all 492 tasks and 1,476 negatives satisfy this.
Expert rating agreement. Before freezing, application experts attempted all 492 tasks under the same observation and budget as the models and rated them; inter-rater agreement was Cohen's kappa = 0.81, and flagged tasks were revised.
Human completion rate. Our audit records indicate that at least one expert solved 96.3%, or 474/492; all rating experts solved 81.7%. For all tasks that were never solved by any model, experts also solved 94.6%.
Verifier-human agreement. We reviewed 450 completed episodes one by one; agreement was 98.4%, or 443/450, with Cohen's kappa = 0.91; of the seven disagreements, four were cases where the verifier rejected a valid alternative and three were boundary cases admitted on preservation or lifecycle, and none came from an unsolvable task.
Beyond this evidence, the best model in the original 21-configuration campaign solves 22.9%, which points to model capability as the main source of low success. Quality assurance is a shared challenge for real software-interaction benchmarks, and heterogeneous stateful applications especially benefit from multiple forms of evidence. Our quality pipeline already combines executable gates, expert ratings, human completion rates, and verifier agreement; we will continue to expand independent expert audits, formalize the review process, and systematically test valid alternative solutions.