acceptodds
Under review as a conference paper at ICLR 2027

A Decade in a Day: Synthesizing Agent-Native Interfaces from Human-Oriented Software Evolution

Abstract

Large language model agents are emerging as an important new class of users of desktop applications. However, existing desktop interfaces were designed for human end users or application developers, leaving agents to interpret visual controls or discover and compose low-level software operations. This mismatch can hinder even capable agents, yet it remains unclear what interface properties make desktop software easier for agents to use. To study this question, we treat software evolution as a natural source of interface variation and introduce DeskIF-Bench, a benchmark of 255 artifact-verified tasks across six desktop applications. By holding the task, agent, and verifier fixed while varying the software release, we measure how interface changes affect agent performance. Across two models and six applications, the newest release consistently outperforms the oldest. Trajectory analysis identifies three relevant interface properties: discoverability, granularity, and observability. Guided by these properties, we develop ANIS, a framework that synthesizes agent-native interfaces from the capabilities of installed applications. On an expanded evaluation of 368 tasks across eight applications, atomic interfaces improve success over native SDK use by 3.8 and 3.5 percentage points with DeepSeek V4 Flash and Claude Haiku 4.5, respectively, while reducing online token usage by approximately 73%. Goal-level commands further raise success to 74.5% and 67.4%, respectively.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Large Language Model Agents, Agent-Native Interfaces, Tool Use, Desktop Software, Programmatic Interfaces, Agent Evaluation, Benchmarking

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_8R6ZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces

  1. a new benchmark, DeskIFBench, that asks agents to perform 255 tasks across 6 GUI applications (blender, etc.). Task completion is verified based on generated/manipulated artifact state
  2. ANIS, a method/agent scaffold that generates a new CLI interface for GUI applications and allows agents to achieve higher scores on DeskIFBench
Critical strengths
  • Improving usability for agents and humans is a pressing concern in software development
  • Using software release history as a natural experiment for interface quality is a neat idea
  • The DeskIFBench benchmark paired with the release history is a good contribution for studying the evolution of interfaces
  • Being able to automatically generate good interfaces for agents is very useful
Critical weaknesses
  • I find the link between "human-oriented software evolution" to agent-native interfaces generally weakly supported:
    • generally, humans for all of these applications only use the GUI, which is not varied/explored in this paper. So it feels like rather the claim would be that the human-written developer-API evolution could also inform the interface for agents (which is somewhat less surprising)
    • the 3 criteria in section 4.3.1, 4.3.2, etc., are mostly supported by examples, not by clearly ablated experiments. These experiments would probably be much clearer on ANIS, rather than derived from an evolution of a framework that was optimized for humans that mostly use the GUI. As it stands, I do not find this very clearly motivated/ablated
  • The ANIS setup could benefit from stronger baselines and clearer ablations for its components.
    • the performance increase seems relative low (and does not generalize to all apps)
      • furthermore, the central Table 2 only shows results for one LM, it is not supported that gains generalize across (stronger) models
    • CLI-anything seems like a weak baseline for other ways to construct dedicated agent APIs
Questions for authors

Metrics

  • why not report pass@1 with variance error bars? It seems like pass@3 just inflates the metric. Not sure if I follow the motivation for using pass@3 here, it seems like it would reduce the sensitivity of the benchmark
  • is tokens/task for 3 attempts or just 1?

Benchmark interpretation

  • Is there a more quantitative support fort the conclusions presented in 4.3.1, 4.3.2 (basically a way of more proving that the derived conditions are indeed the basis of building successful interfaces)

ANIS

  • How much budget did CLI-anything get compared to ANIS? Also it seems like CLI-anything has a :refine command, was that used?
  • Are there any other (stronger?) baselines that ANIS could be compared to? For example, could this be compared to MCP or similar interfaces to the programs?
  • It seems like the success gains for ANIS vs native are relatively small, but costs drop very significantly (mostly unrelated to goals). What is the reason for the lower cost? Could you provide turn counts, trajectory analyses etc., that show why costs are so different? Could this also be slightly conflated with the pass@3 metric (e.g., maybe your pass@1 success rate actually gets higher with ANIS, and the dropped costs are because failed runs are typically more expensive, so higher pass@1 success rate leads to lower cost even if pass@3 stays the same?)
  • For results from table 2, would be great to see the findings generalize across different LMs. Seeing reductions in tokens per task only for one model doesn't mean this transfers
Reviewer_8RPHOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the mismatch between existing desktop software interfaces and the needs of LLM agents. The authors identify discoverability, granularity, and observability as key interface properties that affect agents' task success and execution efficiency. To address this problem, they propose ANIS (Agent-Native Interface Synthesizer), a framework that automatically constructs agent-friendly software interfaces through runtime capability discovery, atomic command synthesis and verification, and the extraction of goal-level commands from historical execution traces. Experiments on eight desktop applications show that ANIS improves overall task success rates while substantially reducing token consumption compared with native SDK interfaces.

Critical strengths
  1. A valuable research perspective supported by empirical evidence. The paper identifies software interface design as an important factor influencing agent performance. Through controlled cross-version comparisons and trajectory analyses across six desktop applications, the authors identify discoverability and granularity as interface properties associated with reduced exploration overhead and shorter action sequences. Concrete examples, including higher-level operations in QGIS and document-level operations in Inkscape, make these findings interpretable and practically relevant. This perspective broadens the scope of agent research and establishes interface design as a promising complementary direction for improving agent capabilities.

  2. A substantial system contribution with broad empirical evaluation. The paper translates its findings into ANIS, a framework that combines runtime capability discovery, artifact-verified atomic commands, and goal-level command synthesis. Evaluations across eight desktop applications and two backbone models show improved overall task success rates and substantially reduced token consumption compared with native SDK interfaces. The artifact-level verification protocol, observation-command ablation, and held-out evaluation of goal-level commands provide additional empirical support. Although the individual contributions of discoverability and granularity remain insufficiently isolated through controlled ablations, the combination of a concrete system, diverse application coverage, and multiple forms of empirical analysis makes this a useful contribution to agent-interface research.

Critical weaknesses
  1. Limited methodological novelty and insufficiently controlled comparisons with existing approaches. ANIS combines runtime capability discovery, LLM-based command synthesis, artifact-level verification, and the extraction of reusable command sequences from agent trajectories. While this integration is useful, the paper does not sufficiently establish the methodological novelty of the framework relative to existing approaches to agent-oriented interface generation, such as Doc2Agent and CLI-Anything. In particular, CLI-Anything exposes substantially fewer commands, making the comparison difficult to interpret as evidence of ANIS's methodological superiority. Comparisons with alternative approaches that provide comparable functional coverage would help establish the specific contribution of ANIS. This limitation weakens the case for the paper's technical originality.

  2. Insufficient causal evidence for the proposed interface properties. The claims about discoverability and granularity are primarily supported by cross-version comparisons, trajectory analyses, and illustrative examples. Software versions may differ in several aspects beyond interface discoverability and granularity, making it difficult to attribute the observed performance gains specifically to these properties. Unlike observability, which is examined through a direct ablation, discoverability and granularity are not independently manipulated in controlled experiments. Furthermore, the held-out evaluation of goal-level commands shows a modest improvement of 2.7 percentage points in task success, leaving the generalizability of trajectory-derived commands insufficiently established. These limitations reduce confidence in the proposed explanations and in the broader applicability of the findings.

Questions for authors
  1. Fair comparison with existing interface-generation methods. The paper reports substantially higher success rates for ANIS than for CLI-Anything, while also noting that ANIS exposes over 20 times as many commands across the applications shared by the two systems, and that most CLI-Anything failures are attributed to missing required commands. Did the authors conduct any experiments that control for functional coverage, or analyze the performance difference when the two methods provide comparable command coverage? If not, how do the authors distinguish the gains attributable to broader functional coverage from those attributable to ANIS's interface synthesis and verification mechanisms?

  2. Independent contributions of discoverability and granularity. The claims regarding discoverability and granularity are primarily supported by cross-version comparisons, trajectory analyses, and illustrative examples. Since software versions may differ in several aspects, including functionality, interface descriptions, and underlying behavior, did the authors conduct controlled experiments that independently vary interface description quality or operation granularity within the same software version? If not, how do the authors establish that the observed performance improvements can be attributed to these two properties rather than to other concurrent changes?

  3. Generalization of goal-level commands. Did the authors check for overlap in task templates or execution sequences between the training and test sets in the held-out evaluation? Since goal-level commands are intended to capture reusable subgoals, some overlap in low-level operations may be legitimate. It would therefore be helpful to know whether the evaluation excludes near-duplicate tasks and whether the authors assessed generalization to semantically distinct tasks that share reusable operations with the training set.

Reviewer_oXEqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies agent performance across historical software releases and proposes ANIS, which converts application capabilities into discoverable commands with ways to inspect the resulting state. The benchmark construction is a useful contribution.

Critical strengths

Checking saved artifacts, verifying task feasibility across releases, and evaluating two applications held out during method development are meaningful strengths. The substantial reduction in online token use is also encouraging.

Critical weaknesses

My main concern is that the evidence does not cleanly establish the proposed explanation for the gains. Changing a software release changes several things at once, and the exploration and operation-count trends are mixed. The historical study therefore does not isolate which interface properties are responsible. Similarly, the large gap over CLI-Anything is largely attributed to missing commands, leaving capability coverage difficult to separate from better interface design.

The accuracy gains over direct SDK use are modest and lack uncertainty estimates. For goal-level commands, the main evaluation includes tasks whose trajectories helped build the commands. The separate held-out experiment addresses this concern, but the gain is only three additional successes out of 110 tasks. ANIS also underperforms direct SDK use on Blender and FreeCAD. Finally, the online savings omit the generation cost, which takes 17.1–36.4 hours per application.

Questions for authors

Could the authors provide paired uncertainty estimates, a comparison with matched capability coverage, and the generation cost needed to assess when the savings pay off? I lean toward rejection because the current evidence is not strong enough to separate the proposed design principles from coverage and preprocessing advantages, despite the useful benchmark and efficiency results.

Reviewer_T7g8OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces two main contributions:

DeskIF-Bench (Desktop Interface Benchmark): A benchmark of 255 artifact-verified tasks across six desktop applications and five domains. Its main distinguishing feature is evaluating the same tasks and agents across four historical releases of each application, allowing the authors to study how software evolution affects agent performance. The benchmark combines 102 adapted tasks from existing benchmarks with 153 newly synthesized tasks. The newest versions consistently outperform the oldest across both evaluated models. Through trajectory analysis, the authors identify three relevant interface properties: discoverability, granularity, and observability.

ANIS (Agent-Native Interface Synthesizer): A framework for generating agent-friendly command-line interfaces from existing desktop applications. It has two phases:

-- Atomic command synthesis: At build time, ANIS discovers application capabilities through runtime introspection, registered operations, and menus, then converts them into commands validated against the application's persistent artifact state. Commands are organized for efficient discovery and paired with read operations where available.

-- Goal-level command synthesis: At runtime, ANIS identifies recurring command sequences, converts them into reusable higher-level operations, and validates their behavior through replay. This phase is simulated using previously solved task trajectories in the experiments.

On an expanded benchmark of 368 tasks across eight applications, ANIS improves success from 68.5% with the native SDK to 74.5% with DeepSeek V4 Flash, while reducing tokens per task by approximately 75%. Similar improvements are observed with Claude Haiku 4.5.

Critical strengths
  • Novel evaluation perspective: Studying historical software releases provides a distinctive and informative way to investigate how interface design affects agents. The controlled cross-version methodology is arguably the paper's strongest novel contribution.

  • Strong motivation: The paper follows a coherent progression from an empirical study of interface evolution to a system designed around its findings. This makes the proposed system's design choices intuitive and easy to follow.

  • Careful and extensive evaluation: The authors control important variables across software releases and evaluate both open-weight and proprietary models. The experiments cover diverse applications, include two applications held out from method development, and provide useful ablations and a held-out evaluation of goal-level commands.

  • Promising results: The approximately 73–75% reduction in online token usage is particularly impressive, alongside consistent aggregate success improvements over native SDKs. The observation-command ablation also provides useful evidence for the importance of inspecting artifact state.

  • Clarity: The paper is generally well-written, with clear explanations and helpful examples illustrating the interface design principles.

Critical weaknesses
  • The three design principles are not equally supported by the initial study. Discoverability and granularity are illustrated through concrete cross-version changes, but observability is primarily motivated by persistent limitations and failed trajectories rather than improvements across releases. Furthermore, the principles are not independently isolated as explanations for the observed performance gains. The analysis could better justify why these three properties, specifically, constitute the proposed framework.

  • Limited comparisons with related interface-generation methods. Beyond native SDKs, the only evaluated baseline is CLI-Anything. Other relevant non-training approaches, such as AXIS and OSWorld-MCP's tool-generation pipeline, deserve closer comparison or a clearer explanation of their methodological differences. Moreover, ANIS exposes over 20 times as many commands as CLI-Anything, making it difficult to separate improvements in capability coverage from improvements in interface design.

  • The performance regressions on FreeCAD and Blender are insufficiently explained. ANIS underperforms native SDKs on both applications, particularly FreeCAD. The explanation based on "implicit object relationships" is plausible but too broad, since such dependencies exist in many applications. The paper would benefit from concrete failure examples and a clearer breakdown of missing capabilities, command-validation failures, and agent execution errors.

  • The practical cost of interface synthesis is underexplored. Generating an interface requires 17–36 hours per application, but the paper does not adequately quantify the associated generation cost or its amortization against online token savings. This matters given the relatively modest aggregate success improvement over native SDKs and the smaller gain from goal-level commands on held-out tasks (+2.7 percentage points).

Questions for authors
  • Could you provide a concrete end-to-end example of a benchmark task, including its reference artifact, verifier, and how the same artifact-level outcome is validated across software releases? This would help clarify the benchmark's interface-independent verification.

  • Could you provide representative failure cases for FreeCAD and Blender, showing precisely which application dependencies prevent command synthesis or validation? What changes to ANIS would be necessary to recover those missing capabilities?

  • Could you better isolate the contributions of the three design principles, particularly by testing the effect of command organization independently of capability coverage? A coverage-matched comparison with CLI-Anything, if feasible, would also help clarify the source of ANIS's improvements.

2 comments
AuthorsOpenReview ↗

Reply to Weakness1

We thank the reviewer for the careful reading and for recognizing the cross-version methodology as a main contribution of this paper. We respond to each point below; all revisions will be incorporated into the paper.

Weakness 1: Why these three properties; uneven support in the study; no independent isolation

Why these three properties. Recent studies of software engineering agents consistently divide agent trajectories into three phases: exploring and localizing what to change, writing the change, and verifying and repairing [Agentless, FSE 2025; Bai et al., 2026]. In Appendix B, we likewise attribute context tokens to exploration, writing, and verification & repair, and these phases cover all tokens.

Our three properties correspond to these three phases:

Agent phaseInterface propertyWhat the property captures
Exploration & localizationDiscoverabilityThe cost of locating the required operation and understanding its parameters
WritingGranularityThe length of the operation sequence needed to reach the goal
Verification & repairObservabilityThe ability to read back artifact state and determine whether the goal is met

We will state this correspondence explicitly in the revision to make the three principles more systematic.

Support in the study. We agree that the cross-version study supports the three properties unevenly. Its role is to generate hypotheses; the ablations below test them.

  • Discoverability and granularity change across releases, and success rate and cost improve accordingly. Since each release changes many things at once, this trend suggests, but does not establish, their effect.
  • Observability does not change: all four releases lack read access (e.g., QGIS's setVertexMarkerAppearance has no read function in any release), so cross-version data cannot reveal its effect. The study only shows a recurring symptom — 581 trajectories claim completion, yet the saved file fails verification — which motivates the property and is exactly why a controlled ablation is needed.

Independent isolation. We test each property by changing only that property within the same release. The main text already isolates observability and granularity (Section 5.3; DeepSeek V4 Flash, 368 tasks):

PropertyOnly changeSuccess rateLocation
ObservabilityDisable read commands72.3% → 60.3%Appendix D
GranularityAdd goal-level commands72.3% → 74.5%Table 2, Appendix E

The revision will add a discoverability experiment: keeping the command set and implementations unchanged, we only replace the object-grouped catalog with an alphabetically sorted flat list, retaining command summaries. Both conditions were run concurrently with the same model, prompts, and timeouts; runs interrupted by service outages were rerun under a uniform criterion.

ApplicationTasksGrouped by object (default)Flat (no grouping)
QGIS5477.8%74.1%
Writer5689.3%85.7%
Blender3545.7%45.7%
GIMP3775.7%75.7%
Calc7663.2%63.2%
Impress4866.7%58.3%
Inkscape3183.9%83.9%
FreeCAD3177.4%67.7%
Total36872.3%69.3%

Removing grouping lowers the success rate by 3.0 percentage points. With grouping, the agent first selects an object category and then searches for commands within it; search results are also labeled with their category. The flat condition places all commands in a single list; in applications such as Writer (1,053 commands) and QGIS (320 commands), the agent easily misses the target command and repeatedly tries similarly named but wrong commands.

For example, the QGIS dangling-endpoint detection task requires first reading line coordinates. In the flat condition, the agent did not find the read command gdal-ogrinfojson; it instead inspected related write commands with different purposes, such as vector-feature-write and vector-layer-create, then spent more than 40 steps trying to read the file through the shell, and eventually timed out without producing a result. In the grouped condition, the agent searched for "info", found gdal-ogrinfojson labeled under the gdal category, then completed the detection and passed verification.

References. Xia et al., "Demystifying LLM-Based Software Engineering Agents" (Agentless), FSE 2025. Bai et al., "How Do AI Agents Spend Your Money? Analyzing and Predicting Token Consumption in Agentic Coding Tasks," arXiv:2604.22750, 2026.

(continued in the next comment)

AuthorsOpenReview ↗

Reply to weakness2 and weakness3

Weakness 2: Limited comparison with related methods; coverage vs. design is hard to separate

Coverage. Capability coverage is itself part of interface quality. GUI vs. CLI [Zhou et al., 2026] finds that CLI-Anything's original skills satisfy only 37.6% of verification checkpoints; closing the gap relies on verifier information and human approval, and the authors list "reaching this coverage automatically without verifier information" as an open problem. ANIS never accesses tasks or verifiers; it discovers capabilities automatically through runtime introspection, registered operations, and menus (Section 5.1).

Design. On the 4 tasks where CLI-Anything had sufficient commands but still failed, ANIS completed all of them. For example, in the FreeCAD base-plate lengthening task, CLI-Anything did not read the original dimensions and guessed the plate thickness and boss radius wrongly when rebuilding; ANIS first read the dimensions, modified only the length, and verified the result. Conversely, compared with the native SDK, which has the highest coverage, ANIS still achieves a higher success rate (68.5% → 72.3% / 74.5%) with about 73% fewer tokens (Section 5.3), indicating that the advantage also comes from interface design.

Why we did not run experimental comparisons with AXIS and OSWorld-MCP. Both work differently from our setting. We will add a comparison table along the following dimensions to the related work. If the reviewer considers it necessary, we would also be glad to reproduce OSWorld-MCP's tool-generation pipeline on a subset of applications and include it in the comparison:

  • AXIS: a UI agent explores the software interface, summarizes skills from its action trajectories, and then translates them into APIs; it is evaluated only on Microsoft Word. ANIS does not rely on GUI exploration and synthesizes commands directly from the software's programming interfaces.
  • OSWorld-MCP: tool generation takes target tasks as input — it first generates solution code for specific tasks, then wraps it into tools, which are manually reviewed by at least two developers. Applied to our benchmark, this would amount to generating tools from the test tasks themselves, with human involvement.
  • ANIS: takes no tasks as input and is fully automatic; each command is admitted only after being verified by saving and reopening the file, and editing commands are paired with read commands.

References. Zhou, X., Zhang, S., Zhao, Y., Wei, J., Song, T., Cohan, A., and Zhao, C. "GUI vs. CLI: Execution Bottlenecks in Screen-Only and Skill-Mediated Computer-Use Agents." arXiv:2606.24551, 2026.

Weakness 3: Performance regressions on FreeCAD and Blender

We will replace the original broad explanation with the per-application failure analysis below and add it to the appendix.

FreeCAD: 7 of 31 tasks failed; 6 lacked a required command, and 1 was an agent execution error. Command validation requires actually executing the operation in the application, and some commands must act on specific objects; for example, the extrusion (Pad) command requires an existing closed sketch. Validation currently starts from a blank document that lacks these objects, so such commands cannot execute, are misjudged as unavailable, and are excluded from the generated interface. This can be addressed by automatically creating the required sketches or solids before validating the corresponding commands.

Blender: 19 of 35 tasks failed; 15 lacked a required command, and 4 were agent execution errors. The first 15 tasks require joining multiple parts, but capability validation had no objects to join, so the join command was excluded. The agent fell back on an alternative operation, which produced a 6.5% dimensional deviation and failed the verifier. Of the remaining 4 tasks, 2 failed because of poorly chosen positions or sizes, 1 confused coordinate systems and kept retrying until it timed out, and 1 did not inspect the modified scene and missed that the starting point was blocked.

  • Richer validation scenes. Before validation, automatically create the objects a command requires, e.g., a closed sketch for extrusion and multiple parts for joining, so that fewer commands are missed because their preconditions are not met.
  • More usage examples for error-prone cases. On top of the existing command descriptions, add concrete examples covering the positions, sizes, and coordinate conversions involved in the failure cases, to further help the agent use commands correctly.

(continued in the next comment)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.