acceptodds
Under review as a conference paper at ICLR 2027

The Last Interface: Benchmarking Language Agents on Native Terminal User Interfaces

Abstract

Terminal-oriented agent evaluation has developed quickly, but it has converged on one interaction model: the agent issues shell commands and reads streamed output. Yet much of the software that runs on servers never offers even that prompt. Editors, file managers, process monitors, debuggers, and mail clients take over the terminal, repaint one screen of characters, and make that screen their entire interface: there is no prompt to return to, no element tree to read, and no API to call. This interface class, the native text-based user interface, has gone unmeasured. We present TUI-Bench, the first execution benchmark for agents that operate native TUIs. Its 492 tasks span 77 native Linux TUI tools in 19 application categories grouped into 6 supercategories; each task starts inside the running program, exposes a plain-text character viewport with cursor coordinates, permits only keyboard actions, and is scored by deterministic verifiers that check application state, preserved data, and whether the program should still be running at the end. In a three-round evaluation of frontier models, the strongest models solve only a small fraction of scored episodes. The evaluation reveals a stable and interpretable failure pattern: agents succeed almost only where the interface has already degraded to a command line or a filter box, which suggests that they rely on pattern matching against commands rather than on continuous tracking of interface state. Whenever a task requires spanning multiple modes, holding focus, or protecting untouched state while reading, models lose control through premature exits, collateral state damage, or budget-exhausted repetition: the text on the screen is read correctly, but the interface itself is never modeled. TUI-Bench thus provides a controlled and reproducible measure of a capability gap that was previously invisible.

Primary area:
datasets and benchmarks
Keywords:
Benchmark, Terminal User Interfaces, Tool Use, LLM Agents

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_tPGqOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces TUI-Bench, a benchmark of 492 tasks across 77 terminal applications and 19 categories. Agents interact through keyboard actions and text-grid observations, while task-specific verifiers evaluate the final application state and persistent artifacts. The paper evaluates a broad set of agent configurations and analyzes their failures. The best reported average success rate is 22.9%, suggesting substantial difficulty under the benchmark's interaction protocol.

Critical strengths
  1. The benchmark is relatively comprehensive and self-contained, covering a wide range of terminal applications with packaged environments and executable verifiers. These tasks and evaluation infrastructure could be a useful contribution to the community.
  2. The evaluation covers many models and provides analyses of interaction failures, including repeated ineffective actions and response truncation. These results provide useful evidence about the limitations of language agents under the specified interface.
Critical weaknesses
  1. The contribution relative to existing terminal benchmarks could be clearer justified. In particular, what important tasks or interaction requirements are missing from Terminal-Bench and TUA-Bench, and how does TUI-Bench address them? The application catalog is broad, but does not include coding-agent interfaces such as Codex or Claude Code, or terminal games. The authors should explain the intended scope and establish why the selected applications and tasks represent an important gap in existing evaluations.
  2. The reliability and fidelity of the evaluation are not sufficiently established by the reported results. Appendix A.6 describes positive and negative verifier fixtures and a human audit, which are valuable checks. However, agreement on expert ratings does not establish human completion rates or verifier accuracy. Furthermore, observations omit color and styling, which may carry useful information in applications such as htop/iotop. Without quantitative human results and an assessment of the information removed by the interface, it is difficult to determine how much of the low success rate reflects agent limitations rather than the evaluation protocol.
Questions for authors
  1. Task solvability and verifier reliability: What fraction of the final tasks did the experts successfully complete under the same observations and budgets as the models, according to the final verifiers? Were the positive fixtures produced by successful keyboard-action trajectories or constructed directly? Please report agreement between verifier decisions and independent human judgments, including valid alternative solutions and invalid near-success states.
  2. Observation fidelity and budgets: Do any tasks depend on color or styling that is absent from the text-grid observation? A comparison using color-preserving observations would help establish whether this omission materially changes task difficulty. Separately, how sensitive are the reported results to the response-token limits responsible for truncation?
  3. Real-time interaction: Do monitoring applications such as htop continue updating while the model is generating its next action? If so, can process ordering, selection, or other task-relevant state change between observation and action? Please explain how wait_idle, refresh timing, and final-state verification handle continuously updating interfaces, and whether differences in model latency affect task difficulty.
3 comments
AuthorsOpenReview ↗

Response to Reviewer tPGq (PART 1/2)

We thank you for recognizing that the benchmark is comprehensive and self-contained: packaged environments and executable verifiers across 77 applications are a valuable resource, and the failure analysis provides useful evidence. We have carefully adopted your suggestions on solvability, observation fidelity, and real-time interaction. We address the four points in order.


W1. Positioning against terminal benchmarks and scope.

We thank you for the question on positioning and scope. The key contribution of TUI-Bench is establishing native TUI operation as a distinct evaluation paradigm, rather than simply expanding the collection of terminal tasks. Its novelty lies in three layers: the interaction paradigm, the evaluation protocol, and the application scope.

1. A distinct interaction paradigm. Existing terminal benchmarks such as Terminal-Bench and TUA-Bench primarily evaluate task outcomes through CLI workflows, where agents can issue commands, execute scripts, or manipulate files directly. Native TUIs require a fundamentally different form of interaction: text is the interface itself, not merely command output. Characters encode spatial layouts and interactive controls, while the meaning of each keystroke depends on hidden application state. Agents must continuously interpret the evolving interface and act through the keyboard, a capability that conventional CLI evaluations do not isolate.

2. A complete evaluation protocol. TUI-Bench makes this capability directly measurable by starting inside running applications, restricting interaction to character-grid observations and keyboard actions, and jointly verifying application state, data preservation, and lifecycle. This prevents agents from bypassing the interface and ensures that success reflects actual TUI operation. Our results further show that models perform substantially better when the interface offers a place to type a command, while struggling with continuous stateful interaction. This reveals a capability gap that existing outcome-oriented terminal benchmarks cannot independently measure.

3. A well-defined application scope. TUI-Bench focuses on native, stateful terminal applications with concrete operational goals, spanning 77 TUI tools and 19 application categories. Coding-agent interfaces introduce additional model-driven behavior, while terminal games emphasize game-specific objectives. Our scope deliberately isolates general-purpose TUI operation from these additional factors.

Overall, TUI-Bench introduces a systematic and reproducible evaluation paradigm for understanding and controlling stateful interfaces rendered entirely as text, complementing existing CLI and GUI benchmarks.


W2/Q1. Solvability, human completion, and verifier reliability.

We thank you for the careful questions. Task solvability and verifier reliability rest on four independent forms of evidence.

Executable gate: verifier reliability. The positive fixtures are constructed directly from each task's expected-state specification, not produced by any keyboard trajectory. Before freezing, each task must pass its positive fixture twice and reject all three negative fixtures; all 492 tasks and 1,476 negatives satisfy this.

Expert rating agreement. Before freezing, application experts attempted all 492 tasks under the same observation and budget as the models and rated them; inter-rater agreement was Cohen's kappa = 0.81, and flagged tasks were revised.

Human completion rate. Our audit records indicate that at least one expert solved 96.3%, or 474/492; all rating experts solved 81.7%. For all tasks that were never solved by any model, experts also solved 94.6%.

Verifier-human agreement. We reviewed 450 completed episodes one by one; agreement was 98.4%, or 443/450, with Cohen's kappa = 0.91; of the seven disagreements, four were cases where the verifier rejected a valid alternative and three were boundary cases admitted on preservation or lifecycle, and none came from an unsolvable task.

Beyond this evidence, the best model in the original 21-configuration campaign solves 22.9%, which points to model capability as the main source of low success. Quality assurance is a shared challenge for real software-interaction benchmarks, and heterogeneous stateful applications especially benefit from multiple forms of evidence. Our quality pipeline already combines executable gates, expert ratings, human completion rates, and verifier agreement; we will continue to expand independent expert audits, formalize the review process, and systematically test valid alternative solutions.

AuthorsOpenReview ↗

Response to Reviewer tPGq (PART 2/2)

Q2. Observation fidelity and output budget.

We thank you for the careful questions on observation fidelity and output budget. We address color and the output cap in turn.

1. Does any task depend on color? We checked all 492 instructions and found no task that requires color or styling to solve. Verdicts check machine-decidable conditions such as application state, files, session state, and Git state, and the 223 terminal-surface checks are token-level. Of these 492 instructions, only 12 mention color and none uses it for solving; wherever a visual attribute could be mistaken for task-relevant information, the instruction asks the agent to judge by visible characters alone. The htop selection is verified from the final application state; the verifier does not read color.

2. Does color-preserving observation change task difficulty? We ran a matched comparison: on the same model and the same 100-task stratified subset, we augment the character grid with a compact color hint; Table 1 reports the results. The hint is anchored to the entry frame and reused.

Table 1. Three-pass Avg@3 by observation modality (%)

Observation / actionState-backed, 72 tasksDisk-backed, 28 tasksOverall (scored)
Character grid / keyboard9.328.615.5
Character grid + color hint / keyboard13.430.318.3

Note: Overall is computed over scored tasks; the state and disk columns use their respective scored tasks.

The gain is small and concentrated on state-backed tasks: 15.5% to 18.3% overall, 9.3% to 13.4% state-backed, disk-backed unchanged. The color hint gives a limited improvement; state-backed success remains at 13.4%, so the gap is far from closed. The task-level answer is direct: the verifier checks application state, files, and lifecycle; color is outside the verdict, and every task is solvable from the character grid by design.

3. How sensitive are the results to the output cap? Our evaluation uses a 32,768-token output cap throughout; repair and transport retries are capped at two, and the episode budget is 150 steps, 2,048 keyboard input units, and 5,400 seconds. Long reasoning can exhaust a single response, so we label episodes that hit the cap and compare them against the thinking and non-thinking twins reported in the appendix. At 32,768, cap hits are most prominent in MiniMax-M3, while other thinking families show them to varying degrees. We therefore do not attribute the main results to any single model's truncation behavior; task difficulty and state reconstruction remain the more likely causes, and our conclusions are stated for the fixed 32,768-token protocol without extrapolation to other caps.


Q3. Continuously updating interfaces and latency.

We thank you for the question on real-time interaction. We address continuous refresh, state changes, and latency in turn.

1. Continuous refresh and state change. Yes, they do. Monitoring programs such as htop keep refreshing while the model generates its next action. The model sees the interface at request time, and between observation and action, process ordering, the current selection, and other task-relevant state can indeed change. This is a real property of TUI interaction, and the protocol is designed to handle it.

2. wait_idle and final-state verification. The action space provides wait_idle: the model can ask the runtime to wait for the screen to settle before taking a fresh observation, at the cost of one step, and the model decides when to use it. After each action, the runtime also waits for the interface to settle before the next step. Scoring uses only the final state: once an evaluation ends, the final state is captured, and monitoring tasks use the final screen content rather than any intermediate frame.

3. Does model latency affect task difficulty? Across 20 generative configurations, none solved the monitoring tasks; including the non-generative baseline yields only isolated successes. Notably, faster and slower configurations exhibit consistent failure patterns: repeated ineffective actions and incorrect terminal states. This reveals a distinction between response speed and sustained interface control: faster responses may narrow the window for state changes but cannot replace continuous state tracking. These results thus highlight the central challenge of dynamic TUI operation: reliably maintaining and updating interface state as the screen changes.

We thank you again for pressing on evaluation reliability and fidelity. If these responses address your concerns. We would sincerely appreciate it if you could consider raising the current score in support of accepting this work.

Dear Reviewer tPGq,

Thank you very much for your time and effort in reviewing our paper. We sincerely appreciate your valuable comments and constructive suggestions, which have helped us improve the quality of our work.

We would like to kindly follow up on our previous rebuttal and ask whether our responses have adequately addressed your concerns. If you have any remaining questions or concerns, we would be more than happy to provide further clarifications.

Thank you again for your time, consideration, and valuable feedback.

Best regards, The Authors

Reviewer_vnBqOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

A benchmark that evaluates LLM agents on text-based user interface (TUI) tasks, which extends the AI evaluation landscape beyond CLI or GUI evaluation works. The benchmark is run via harbor and includes 492 tasks across 6 supercategories and 19 application categories. Many different models are evaluated, revealing that most failures come from the agents not being able to reconstruct the hidden interface state rather than a lack of perception capabilities.

Critical strengths
  1. While benchmark for command-line interfaces and graphical user interfaces exists, this paper presents a novel benchmark for text-based user interfaces. This point is consequential as it's the most novel aspect of the paper and is filling an obvious gap between CLI and GUI AI evaluations.
  2. The technical implementation of the tasks is sound, allowing the reproducibility of results. With the tasks being based on an established evaluation framework like harbor it makes it easy to re-run the mentioned experiments.
Critical weaknesses
  1. Lack of task curation and quality assurance information. A benchmark is a measurement tool, and a paper/report about a tool should include how this tool was created and how the quality of this tool was tested and raised to high standards. The paper contains little to no information about how the tasks were curated: (i) are the tasks AI-generated, or constructed by experts? (ii) how was task quality measured and ensured? Many tasks are never solved, and if these tasks are not properly reviewed then it's difficult to distinguish between real failures and failures due to task design/quality issues (e.g. the verifier may be over-constrained).
  2. Lack of information on the evaluation setup and comparison to a baseline. The paper mentions that tasks are built on harbor, but no information is provided about what harness is used. The results are produced by for models by several different model providers, while the title says "benchmarking language agents" - so what exactly is the agent harness here? None of the evaluated models are "frontier models" as mentioned in the text and it would be good to mention why SOTA models are not evaluated + for some reason no Anthropic models are evaluated, and it is not explained why. I suspect many of the "goals" of these TUI tasks could in theory be achieved by bypassing the TUI and solving the task via CLI commands with terminal-based agents like Codex or Claude-Code. A comparison of both modes would have been valuable to include, otherwise the value of a TUI benchmark is not immediately obvious.
Questions for authors

The questions are mainly based on the two weaknesses I mentioned.

  1. How were the tasks curated/generated? How did you ensure the quality of the tasks? How much task review went into it?
  2. What exactly is the harness in which the models operate in? Why are no SOTA models or Anthropic models evaluated?
2 comments
AuthorsOpenReview ↗

Response to Reviewer vnBq

We thank you for recognizing the novelty: a TUI benchmark fills the GUI-CLI gap, and Harbor makes results reproducible. We have adopted your suggestions and respond in the order W1/Q1 and W2/Q2.


W1/Q1. Task construction, review, and quality assurance.

Task quality rests on three independent forms of evidence: the executable gate, expert agreement, and human completion.

First, authoring. Each task pairs an expert-written instruction with declarative fields for target, operation, preservation scope, and lifecycle, forming a hash-bound frozen source query; see Appendix A.6.

Second, packaging and admission. Packages are compiled from the frozen query; the positive fixture is constructed directly from the target-state specification, not from any trajectory. Before entering the frozen set, each task must pass its positive fixture twice and reject all three negative fixtures; we checked all 492 tasks and all 1,476 negatives. This gate validates the designated fixtures, not all valid alternatives; the human-verifier agreement check covers that gap.

Third, independent validation. Before freezing, application experts attempted all 492 tasks under the same observation and budget as the models and rated them; Cohen's kappa = 0.81, and flagged tasks were revised. Trajectory-side evidence points the same way: 135 tasks are solved by at least one configuration, and 6 by all 21. Under the same protocol, at least one expert solved 96.3%, or 474/492; all rating experts solved 81.7%; for all tasks that were never solved by any model, experts also solved 94.6%.

We also recognize that quality assurance is a general challenge for real software-interaction benchmarks, especially for heterogeneous stateful applications, which require multiple forms of evidence. Our pipeline combines executable gates, expert review, human completion rates, and verifier agreement; we will continue to expand independent expert audits, formalize the review process, and systematically test valid alternative solutions.


W2/Q2. Harness, model coverage, and CLI bypass.

We thank you for the questions on the evaluation setup, model coverage, and the command channel. We answer in three points.

1. Harness. All models run through the same TUI-Bench controller. All 20 generative configurations share the same prompt, payload, keyboard contract, retry policy, budgets, and 32,768-token cap; the System One model enters the same controller through an adapter. Harbor constructs and grades the environment; tui-test owns the pseudo-terminal and applies keyboard events.

2. Model coverage. We apologize that cost constraints led us to omit several state-of-the-art models, including all Anthropic models. We have now added four frontier models, including Claude Opus 5.5; all four completed three passes under the same protocol, 5,904 episodes in total. Table 1 reports the results.

Table 1. Frontier models: task success, tokens, and cost

ModelTask success Avg@3 (%)Task success Pass@3 (%)Task success Pass^3 (%)Avg tokensAvg read tokensAvg write tokensAvg time (min)Avg input units
Claude Opus 5.530.332.029.2351k347k3.8k19.1269
Gemini-4-Argon29.730.627.8266k265k1k13.5242
GPT-6.1-Sol23.323.622.3486k478k8k17.8378
Gemini-3.8-Flash20.324.015.6303k302k8626.7224

The original 21-configuration campaign tops out at 22.9%; the four added frontier models raise the best score to 30.3%, yet current model capability remains insufficient for the vast majority of tasks; the evidence points to interface-state reconstruction rather than model generation as the main gap, and the benchmark remains a challenge. Cost is not monotonic with success.

3. The command channel versus the TUI. Codex or Claude Code acting through shell commands can directly write disk-backed artifacts, but cannot directly reconstruct the live interface state. Of the 492 tasks, 130 are disk-backed, 26.4% of the set; the other 362 are state-backed, and 461 require the program to remain running at the end of the episode. For those tasks, a shell must enter the running interface through process control or a pseudo-terminal, which is already TUI operation rather than a bypass of it. Pure bash reaches 13.8% overall, 22.2% on disk-backed tasks, and 10.1% on state-backed tasks; the character grid reaches 15.5%, 28.6%, and 9.3%. Pure bash is lower overall and on disk-backed tasks, and its state-backed difference is also small, so it does not convert the direct file path into an advantage. The command channel therefore provides no shortcut around the interface; TUI operation remains a distinct capability.

We thank you again for these important comments. If these responses address your concerns, we sincerely ask you to raise the current score in support of accepting this work.

Dear Reviewer vnBq,

Thank you very much for your time and effort in reviewing our paper. We sincerely appreciate your valuable comments and constructive suggestions, which have helped us improve the quality of our work.

We would like to kindly follow up on our previous rebuttal and ask whether our responses have adequately addressed your concerns. If you have any remaining questions or concerns, we would be more than happy to provide further clarifications.

Thank you again for your time, consideration, and valuable feedback.

Best regards, The Authors

Reviewer_xt3QOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper introduces TUI-Bench, the first benchmark for agents operating native full-screen terminal applications (e.g., vim, htop, tmux). It has 492 tasks over 77 Linux TUI tools. Agents observe only a plain-text 32×96 character grid and act through keyboard input alone. Deterministic verifiers check the final application state, data preservation, and whether the program is still running when it should be. Across 21 model configurations, the best reach only 22.9% success, and 71% of tasks are never solved. Success concentrates on tools that offer a command line or search box. Failures stem from premature exits, collateral state damage, repetitive thrashing, and reasoning truncation. The authors conclude that models read the screen well but do not track the underlying interface state. In general I think this paper is valuable, though still has some moderate/minor issues with regard to soundness.

Critical strengths
  1. The introduction of a challenging benchmark on TUIs fills the current gap, where we only have popular GUI-based benchmarks and CLI-based benchmarks.
  2. The benchmark is quite diverse, covering 77 different TUI apps.
  3. The paper provides a detailed failure analysis beyond the leaderboard, showing where and why current agents fail on TUIs.
Critical weaknesses
  1. The agent harness is quite confusing. Currently, the mainstream agent harness keeps all interaction history in the model’s context until the context exceeds a limit, such as a compaction limit or a hard context length limit. In contrast, this paper adopts a somewhat stateless harness that only keeps the last few turns in the context. I think this contradicts how frontier models were originally trained, and it may not fully reveal their actual performance.
  2. Some experimental results are counter-intuitive. For example, several models perform worse in thinking mode. This seems to be caused by the truncation of output tokens due to overlong thinking. This can largely affect the scores and rankings of the models, and should be carefully tuned and reconsidered.
Questions for authors
  1. What is the max output token limit for each step when the models are being evaluated?
  2. Can these tasks also be solved by CLI-based agents (e.g., pure bash) or GUI-based agents (e.g., directly feeding screenshots of the TUI to the models)? It would be valuable if even a subset of the tasks were evaluated in a CLI/GUI setting, with some analysis of the differences in scores and behaviors.
5 comments
AuthorsOpenReview ↗

Response to Reviewer xt3Q (PART 1/2)

We thank you for the positive assessment and for recognizing the GUI-CLI gap, the breadth of 77 tools, and the failure analysis; we have adopted your suggestions on context, output cap, and the CLI and GUI comparisons. We respond in the order W1, W2/Q1, Q2.


W1. Limited-history harness: design and evidence.

We thank you for the constructive comments on the context protocol. The fixed window is deliberate, not an oversight in observation design. Every turn carries the instruction, remaining budgets, 32×96 character viewport, cursor position, and up to the last eight records in full. Each record contains the screen before and after the action plus the action, so the model sees up to sixteen frames and eight actions verbatim; the exact payload is in Appendix E. At every step it can trace back over the preceding eight interaction records.

There are two reasons. First, measurement control: mainstream harnesses differ in compaction, summary policy, and prompt, and adopting them would mix interface operation with memory strategy. Second, cost: the field spends 366k tokens per episode, of which 359k go to re-entering the screen, 98.1% of the total; full history would raise cumulative read cost by about 4.4 times.

To address your concern, we additionally ran a task-by-task context-policy ablation over four representative configurations and all 492 tasks, comparing the fixed window, rolling summary, and full history; Table 1 reports the results.

Table 1. Avg@3 under context policies (%)

ConfigurationFixed windowRolling summaryFull history
GPT-5.6-Terra22.924.122.4
Gemini-3.6-Flash (w/o thinking)22.923.821.9
Kimi-K2.7-Code15.216.515.1
DeepSeek-V4-Flash (w/o thinking)12.513.011.8

In this ablation, read tokens per episode are 399k, 459k, and 1.74M, all four-model macro averages; the summary is generated by the same model, capped at 4,000 characters, and the summary text is counted in subsequent read tokens. Rolling summaries give a stable but modest gain of 0.5 to 1.3 points; full history falls below the fixed window in all four configurations. Even under the most favorable policy, the best configuration in this ablation moves only from 22.9% to 24.1%. These results suggest that the fixed-history policy is not the sole explanation for the low success rates; substantial difficulty remains under all three context policies, consistent with interface-state reconstruction being a major bottleneck. We will add this controlled experiment to the paper.

AuthorsOpenReview ↗

Response to Reviewer xt3Q (PART 2/2)

W2 and Q1. Thinking mode, output truncation, and the per-step cap.

We thank you for raising this point. All models use a 32,768-token cap; we respond in three points.

1. Cap and budget. All 21 configurations use 32,768 tokens; repair and transport retries are capped at two; episode budget: 150 steps, 2,048 input units, 5,400 seconds.

2. Attribution. Thinking models can hit the cap in long reasoning; we label episodes that hit the cap separately and pair each thinking run with its non-thinking counterpart. We read this as model-budget behavior, not necessarily a scoring flaw.

3. Scope. At 32,768, truncation is most visible in MiniMax-M3, while other thinking families also show it to varying degrees. We therefore do not attribute the main result to one model's truncation behavior; task difficulty and state reconstruction remain the more likely causes. Our conclusions are stated for the fixed 32,768-token protocol and are not extrapolated to other caps.


Q2. CLI and GUI alternatives to the TUI.

We thank you for the question on CLI- and GUI-based agents. We evaluate both settings on the same 100-task stratified subset; Table 2 reports the results.

Table 2. Three-pass average success rate by observation and action channel (%)

Observation / actionState-backedDisk-backedOverall (scored)
Character grid / keyboard9.328.615.5
Application screenshot with color / keyboard13.228.817.9
Pure bash shell10.122.213.8

Note: Overall is computed over scored tasks; the state and disk columns use their respective scored tasks; the three-pass average is the arithmetic mean of the per-pass rates. State-backed means the verdict reads live application state or the final terminal surface; disk-backed means it reads files or persistent artifacts.

1. CLI-based agents. Of the 492 tasks, 130 are disk-backed, 26.4% of the set, so a shell can write those artifacts directly. This is the direct file path. The other 362 tasks are state-backed: to reach them, a shell must enter the running interface through process control or a pseudo-terminal, which is already TUI operation rather than a bypass of it. In the measurement, pure bash is lower overall and on disk-backed tasks, and its state-backed success is 10.1%, not enough to convert the direct file path into an advantage. The command channel provides no shortcut around the interface; TUI operation remains a distinct capability.

2. GUI-based agents. The screenshot condition changes only the observation channel: the model sees a color screenshot of the same TUI and still acts by keyboard. It is therefore an observation-channel test, not a full mouse-and-window GUI agent. Averaged over three passes, it raises overall success from 15.5% to 17.9% and state-backed success from 9.3% to 13.2%, while disk-backed success is essentially unchanged. Color and pixels help a little, but they do not close the state gap; the results are consistent with interface-state reconstruction being the main bottleneck.

We thank you again for the questions and would be glad to continue the discussion.

Dear Reviewer xt3Q,

Thank you very much for your time and effort in reviewing our paper. We sincerely appreciate your valuable comments and constructive suggestions, which have helped us improve the quality of our work.

We would like to kindly follow up on our previous rebuttal and ask whether our responses have adequately addressed your concerns. If you have any remaining questions or concerns, we would be more than happy to provide further clarifications.

Thank you again for your time, consideration, and valuable feedback.

Best regards, The Authors

Comments on the paper

3 comments
AuthorsOpenReview ↗

Request for Reviewer Follow-up Before the Discussion Deadline

Dear Area Chair,

Thank you very much for your time and effort in overseeing the review process for our submission #8406 (TUI-Bench).

We have carefully addressed all reviewers' concerns through detailed rebuttals, additional experiments, and follow-up comments. However, we have not yet received further responses from Reviewers vnBq and tPGq, both of whom initially assigned a score of 2.

As the discussion deadline of November 18 (AoE) is now only a few hours away, we would be sincerely grateful if you could kindly remind these two reviewers to revisit our responses, if possible. We hope to have the opportunity to clarify any remaining concerns before the discussion period closes.

We fully respect the reviewers' assessments and greatly appreciate your time, consideration, and assistance.

Best regards, The Authors

AuthorsOpenReview ↗

Appreciation and Follow-up on Reviewer Feedback

Dear Reviewers,

We sincerely thank all reviewers for your time, careful consideration, and constructive feedback on our work.

We are especially grateful to Reviewer xt3Q for the thoughtful follow-up, positive reassessment, and willingness to raise the score. Your encouragement and recognition of our contributions mean a great deal to us.

For Reviewers vnBq and tPGq, we have carefully addressed your concerns through detailed explanations, additional experiments, and analyses. As the discussion deadline approaches, we would be very grateful if you could kindly revisit our responses and let us know whether your concerns have been adequately addressed. If our clarifications have resolved your concerns, we would sincerely appreciate your consideration of a higher rating.

Thank you again for your valuable feedback, time, and consideration. We remain happy to clarify any outstanding questions.

Best regards, The Authors

AuthorsOpenReview ↗

For AC, SAC and PC: Summary of Rebuttal Process (Submission #8406)

Dear Area Chairs, Senior Area Chairs, and Program Chairs,

As the discussion period concludes, we would like to summarize TUI-Bench's contributions and our responses to the reviewers. We sincerely appreciate their thoughtful feedback and your time overseeing this process.

TUI-Bench is the first execution benchmark for agents operating native text-based user interfaces (TUIs), bringing a previously unmeasured interaction paradigm into agent evaluation. Across 492 tasks and 77 real applications, agents operate running software through character-grid observations and keyboard actions, with verification of application state, data preservation, and lifecycle. Our findings reveal a capability gap overlooked by existing GUI and CLI benchmarks: even frontier models struggle to track and control the hidden interface state.

1. Reviewer xt3Q (Rating: 3 → 4)

  • Main concerns: The reviewer recognized the benchmark's novelty, broad coverage, and insightful failure analysis, while questioning whether limited interaction history, reasoning truncation, or alternative interaction channels could explain the low success rates.

  • Our response: We conducted controlled comparisons of context-management strategies and of TUI, screenshot-based, and CLI interaction, while clarifying the shared output-budget protocol. Rolling summaries provided only modest gains, full history offered no advantage, and neither screenshots nor shell access resolved the central difficulty. This reinforces interface-state tracking as a key bottleneck.

  • Reviewer's follow-up: We are especially grateful that the reviewer confirmed most concerns had been addressed and remarked, “Nevertheless, new efforts to develop benchmarks for underexplored domains such as TUIs should be encouraged”. The reviewer also stated “I believe the current version merits acceptance” and raised the rating from 3 to 4. We deeply appreciate this encouragement.

2. Reviewer vnBq (Rating: 2)

  • Main concerns: The reviewer acknowledged TUI-Bench's novelty and reproducibility, while raising important questions about task curation and quality assurance, harness design, frontier-model coverage, and whether CLI agents could bypass TUI interaction.

  • Our response: We detailed our task-construction process and provided multiple layers of validation, including positive and negative executable checks, independent expert reviews, and human completion studies. Human results show that most tasks are solvable under the same constraints. We also clarified the unified evaluation harness and evaluated additional frontier models, including Claude Opus 5.5, Gemini-4-Argon, and GPT-6.1-Sol, whose continued difficulties reinforce the benchmark's challenge beyond the original model selection. Finally, controlled CLI comparisons showed that direct shell access cannot replace control of live application state. We carefully addressed all the reviewer's concerns with detailed explanations and new experimental evidence, and appreciate these suggestions, which strengthened the case for our benchmark's validity.

3. Reviewer tPGq (Rating: 2 → 3)

  • Main concerns: The reviewer appreciated the broad task coverage, executable verifiers, and failure analysis, while seeking clarification on the distinction from CLI benchmarks, task solvability, verifier reliability, observation fidelity, and the handling of dynamic interfaces.

  • Our response: We clarified why TUI operation requires sustained interaction with hidden application state, rather than issuing shell commands. We substantiated task solvability and scoring reliability through expert completion studies, verifier audits, and strong agreement between independent human judgments and automated grading. Additional experiments showed that color offers limited improvements without closing the capability gap, while our explanations of output budgets and continuously updating screens addressed the remaining protocol questions.

  • Reviewer's follow-up: We are grateful that the reviewer confirmed “The concerns and questions have been mostly addressed” and stated “I would be glad to recommend 3: weak acceptance for this work”. We also appreciate the suggestion on further color-preserving evaluations.

We are particularly encouraged by the support for acceptance from Reviewers xt3Q and tPGq, and respectfully hope our comprehensive responses to Reviewer vnBq will also help inform your final assessment. We believe TUI-Bench establishes a valuable new direction for agent evaluation. Thank you again for your time and thoughtful consideration. We would be truly grateful for your favorable consideration of our work.

Best regards,

The Authors of Submission #8406

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.