acceptodds
Under review as a conference paper at ICLR 2027

Built for Humans, Not yet for Agents: Turning Apps into Agent-Native CLIs at scale

Abstract

Applications (Apps) expose rich functionalities, such as document editing and image processing, through graphical user interfaces (GUIs). However, reliably interacting with these GUIs remains challenging for large language model (LLM) agents. Command-line interfaces (CLIs) offer an alternative, but generating com- mands for GUI functionalities requires preserving their implicit context and be- havioral constraints. We present APP2CLI, a framework that generates agent- native commands for existing GUI functionalities in apps. APP2CLI traces GUI “trigger-handler-effect” paths to extract behavioral semantics and represent them in a structured semantic specification. Skill guidance directs the construction of CLI templates through public programmatic interfaces. Cooperating agents im- plement templates as standalone, parameterized commands. We evaluate APP2CLI on 1,467 target GUI functionalities across 11 application units from 9 apps. APP2CLI achieves a 60.05% agent-native commands gen- eration success rate , producing semantically correct commands for 881 target functionalities, . The CLI executions also achieve 98.01% line-coverage overlap with the corresponding GUI executions. Removing the semantic specification or skill-guided template construction reduces the generation success rate to 1.77% and 0.14%, respectively. Code is available at https://zenodo.org/records/22970644.

Primary area:
foundation or frontier models, including LLMs
Keywords:
GUI, CLI, LLM Agents

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_1nhUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

APP2CLI turns GUI functions into commands with explicit parameters. It follows GUI trigger–handler–effect paths to work out what each function should do, uses skills to build templates around public APIs, and has agents implement and check the commands. On 1,467 target functions from nine apps, the paper reports 881 successful commands (60.05%); 41 targets match CLI-Anything's existing commands. The evaluation also includes ablations, 98.01% average GUI-to-CLI line overlap in the semantic-core code of six comparable apps, and 114 passing replays out of 124 eligible LibreOffice targets using unchanged upstream assertions. The main idea is to carry the GUI's context and behavior into the generated interface. The evaluation focuses on individual functions.

Critical strengths
  1. Making the GUI's hidden context explicit is a useful idea. The specification records what must hold before a command runs, what should change, and what should stay untouched. The sorting and merging examples show why the selected objects, active column, and content policy matter. Getting these details right is central to making the commands useful.

  2. The work covers 11 application units, and I like the decision to check commands against existing application tests. STOR keeps the upstream fixtures and assertions, with 114 replays passing. This gives some support for the commands' actual behavior, though it only covers a subset of them.

Critical weaknesses
  1. I couldn't tell how thoroughly the 881 commands were checked. Section 3.3 defines PASS, but says too little about how many inputs were tested, how varied they were, or how the checks catch things that the extracted specification missed. Appendix E.1 gives examples of independent observations, but doesn't show how widely they were used. For STOR, only 124 of the 376 already-supported LibreOffice targets from fully supported source-test cases were eligible, and 114 passed. It's unclear how the ten failures affect the main count. The paper does acknowledge that these checks don't prove full equivalence. Even within that stated scope, I need more detail to judge the claim that 881 commands are semantically correct and reusable.

  2. The 21.5× comparison is hard to interpret as a gain from the method itself. APP2CLI is built for the chosen fine-grained targets. CLI-Anything is checked by statically matching its released commands, while BASELINE starts from documentation units that don't line up with the benchmark intents. The ratio tells us about coverage of this target set; it doesn't tell us how the methods would compare if asked to build commands for the same targets. CLI-Anything already maps GUI actions to backend APIs, so checking its existing commands leaves that question open. The ablations are helpful, but don't fix this mismatch. I'd like to see a comparison with the same targets, source information, and a similar total inference budget.

Questions for authors
  1. Clarification: How were the test inputs and oracles for the 881 passing commands built? Which checks were independent of the generated specification and construction feedback? Please give the number of input/state settings tested and any held-out parameter variations. I'm missing these details when judging the correctness and reuse claims.

  2. Evidence needed: For the ten STOR failures, which commands behaved incorrectly, and which failures can be traced to unstable tests or limits of the observations? What evidence supports that split? Are these commands included in the 881 passes, and would confirmed errors change the count? These cases would let us check whether the main validator agrees with the upstream assertions.

  3. Evidence needed: To separate the method's gains from differences in coverage and scoring, have you compared against a baseline builder or a simpler source-aware generator given the same targets, similar information, and a similar total inference budget? Also, can the static matching combine existing commands?

1 comment
AuthorsOpenReview ↗

Response to Reviewer 1nhU

1. How the 881 commands were checked (R-1nhU-Weak1-881Checks / R-1nhU-Ques1-InputsOracles)

A target passes when its command executes successfully, produces the specified effects and preserves the non-target invariants (Section 3.3). All 881 commands were executed and checked dynamically. Inputs and initial states are taken from the source tests or from historical GUI executions. The oracles reuse the developers' assertions in the source tests for LibreOffice (453 targets), and compare the final state with a GUI reference run or a reference value for the other applications (428 targets). Evidence independent of the generated specification comes from STOR, source-line coverage (98.01% of the GUI-executed lines within the semantic-core units of six comparable applications), which indicates that the commands run the application's own handlers, and the validation on unseen parameters and states. Details of the oracles and coverage are in Appendix E.6 and E.3.

2. The ten STOR failures (R-1nhU-Ques2-TenSTORFailures)

STOR needs tests that exercise the semantic GUI operation, so that their assertions can serve as an independent oracle, and among our applications only LibreOffice provides such tests at scale. Before any replay, we checked whether the GUI step in each test can be replaced by the command, which holds for 124 of the 376 targets. Six of the ten failures are Undo defects in our commands: Undo granularity differs from the GUI when deleting multiple sheets (2), and the test's Undo does not revert four Calc statistics functions (4). In the other four, two assertions read GUI state that a CLI run does not produce (Impress find & replace, Writer change tracking), and two source tests are unstable (Writer index and page-number format). Appendix E.7 (Table 10) describes each failure.

3. Unseen parameters and states (R-1nhU-Ques1-InputsOracles)

To test the generalization of the commands, our held-out validation keeps each command unchanged and uses new parameters (V1), a new initial state (V2), or repeated or interleaved invocations (V3, A→A or A→B→A). We prepared 137 variants on 51 targets (V1 42, V2 51, V3 44). We ran 10 variants on 5 targets: 7 pass, and in all 10 the command carries out the main operation correctly. Details of the protocol and results are in Appendix E.5. We will extend this validation to the remaining variants later.

4. Comparison with CLI-Anything (R-1nhU-Weak2-21.5x)

We agree that 21.5× (881/41) is a coverage ratio on this target set and does not by itself compare the methods on the same targets. The targets were fixed before any command was validated and were not selected by what CLI-Anything supports. By source-code analysis, only 3 of its 564 released commands (0.53%) always pass the GUI operation directly to the application. On the 18 LibreOffice targets we ran dynamically, APP2CLI passes all 18, while CLI-Anything passes 8 by the final artifact and none with evidence that LibreOffice produced the effect. Details of the static analysis and of each dynamic run are in Appendix F.

5. Source-aware baseline (R-1nhU-Ques3-SourceAwareBaseline)

We added this baseline (ACGBaseline). Codex receives the same model, source code, documentation and task input as APP2CLI, but no expected effects or APP2CLI artifacts. It passes 541 of the 1,467 targets (36.88%) versus 881 (60.05%) for APP2CLI, and APP2CLI uses less than a tenth of the tokens and about a seventh of the time per target. Of its randomly sampled 50 failures, 30 miss hidden GUI context: 10 misuse a parameter or miss a precondition (P), 18 leave part of the intended effect undone (E), and 2 change content that should stay unchanged (N). For example, it leaves cell comments behind when sorting a Calc range, and it does not disable header sharing in Writer. The other 20 fail for other reasons, e.g., invoking a same-named command with different semantics, or an interface that cannot perform the GUI operation. Details of the setup, results and cost are in Sections 3.2, 4.1 and 4.3 and Appendix E (Table 9).

Reviewer_8vwfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents APP2CLI, a framework that turns GUI functions in existing desktop apps into standalone command-line tools that AI agents can call directly with explicit arguments. It does this by reading each app's source code to work out exactly what a GUI action does. The evaluation results look promising. APP2CLI generates sematically correct commands for 60.05%, where CLI-anything covers 2.8% and another baseline covers 0%. Furthermore, the CLI executions achieve 98.01% line-coverage overlap with the corresponding GUI executions.

Critical strengths
  • Reasonable problem framing. As documentation and API signatures leave out hidden context, using the app's source code as the reference for what each GUI action does is a convincing answer to a real problem:
  • A good validation idea. STOR re-runs the app's own GUI tests with the GUI action swapped for the generated command, keeping the original test checks. This is a good way to check equivalence against tests the authors didn't write.
Critical weaknesses
  • The pipeline seems to be incremental. Phase2 and 3 follow a standard agentic patterns. Also, the relation to classical GUI analysis and specification-mining work is not discussed.
  • The benchmark and baseline comparison may favor APP2CLI. CLI-Anything provides higher-level commands and was matched statically against fine-grained intents.
  • Limited applicability. Each command opens the file, acts, and saves, so it does not keep session state. That makes undo, live sessions, and chaining multiple steps harder. Also, there's no discussion of keeping commands working regarding the maintenance.
Questions for authors

Q1:CLI-Anything provides coarser, task-level commands and was matched statically against fine-grained intents. How much of the gap comes from this difference in granularity? Can you compare on a task-level benchmark, or on targets that both systems are designed to cover?

Q2: How much manual effort does a new app need, and how are commands kept working as apps change versions?

1 comment
AuthorsOpenReview ↗

Response to Reviewer 8vwf

1. What APP2CLI contributes (R-8vwf-Weak1-Incremental)

For each GUI functionality, Phase I formalizes the context that the GUI supplies implicitly into a structured semantic specification: parameters and their roles, preconditions (P), expected effects (E) and non-target invariants (N). This specification then guides Phases II and III. Public interfaces often do not expose this context (see the Calc Function Wizard example in Appendix E.1). Without the specification, semantically correct commands drop from 881 to 26, and without the skill guidance the success rate drops from 60.05% to 0.14% (Table 3), so both components are necessary. A new source-aware Codex baseline (ACGBaseline) with the same model, source code and documentation passes 541 of the 1,467 targets versus 881 for APP2CLI, and 30 of its randomly sampled 50 failures miss this context, most often part of the intended effect (E) (Section 4.1, Appendix E, Table 9). APP2CLI combines GUI analysis, which locates the handler but not its required effect, with specification mining, which describes required properties but not the GUI functionality. Trajectory-based agent learning trains agents to operate the GUI. APP2CLI instead turns each GUI operation into a command that agents invoke directly (related work and references in Section 5).

2. Comparison with CLI-Anything (R-8vwf-Weak2-BaselineComparison / R-8vwf-Ques1-Granularity)

We examined how CLI-Anything differs from APP2CLI by combining source-code analysis of its released commands with dynamic execution.

  • What the released commands cover. Of the 564 released commands (commit 810c18b0), 19 map to our targets and cover 41 of them. The others address GUI actions outside our targets or are not single GUI functionalities (queries, CLI session management, format conversion).
  • How they are implemented. Many released commands never call the application. They edit a JSON copy of the document kept by CLI-Anything, which reaches the application only at export, or they perform the operation in Python or with third-party tools, so they are not agent-native. Only 3 of the 564 released commands (0.53%) always pass the GUI operation directly to the application.
  • Dynamic run. We also ran the released commands on LibreOffice, whose oracles come from the developers' own GUI tests, starting with 18 of its 29 statically matched targets. Using only CLI-Anything's documentation and help output, the LLM agent chooses the commands to call, may combine several of them, and may revise them once. Every process is traced. APP2CLI passes all 18, while CLI-Anything passes 8 by the final artifact and none with evidence that LibreOffice produced the effect. In seven of the eight, CLI-Anything edits its JSON copy of the document and outputs the ODF file without starting LibreOffice, and for the remaining one our trace does not show where the effect is produced. For 7 other targets, its own import process loses state that the target depends on.

Details of the static analysis and of each dynamic run are in Appendix F. We will continue to run the remaining targets later.

3. Manual effort, session state and maintenance (R-8vwf-Weak3-Applicability / R-8vwf-Ques2-ManualEffort)

We clarify how the commands are executed and correct a misleading phrase in the paper.

  • Manual effort. The "manually specified inventory" (Sections 2.1 and 3.1, Appendix D.2) was inaccurate and has been corrected. Once the runtime environment is configured, the only manual input is a trigger vocabulary for each GUI framework, which lists the framework constructs that register GUI actions, such as menu items, and bind them to event handlers (e.g., QAction and onTriggered in Qt, Appendix C, Table 4). A single Qt vocabulary serves Krita, OBS Studio, FreeCAD, MuseScore and Shotcut. Extraction, specification, generation and validation are automatic.
  • Session state. The commands are not limited to an open–act–save workflow. The LibreOffice connector can attach to a running process, and STOR runs the commands inside a live GUI session, where we validated Undo/Redo behavior and found six commands that do not preserve the GUI's Undo behavior, as detailed in Appendix E.7. The validation on unseen parameters and states also covers repeated (A→A) and interleaved (A→B→A) invocations.
  • Maintenance. When the interfaces and behavior a command depends on remain compatible, existing commands are reused. Otherwise, the affected functionalities are regenerated and revalidated with the same pipeline easily, without hand-editing commands.
Reviewer_y1WzOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents APP2CLI, a framework for converting GUI functionalities in existing desktop applications into standalone, parameterized CLI commands for LLM agents. It first traces GUI trigger–handler–effect paths in application source code to extract semantic specifications, then uses skill guidance and public APIs to construct CLI templates, and finally employs cooperating agents to implement and validate the commands. Across 1,467 selected GUI functionalities from nine applications, APP2CLI produces 881 commands that pass its semantic checks (60.05%). The paper additionally reports ablations, GUI-to-CLI source-line coverage overlap, and supplementary validation against existing LibreOffice test assertions.

Critical strengths
  1. The paper targets a relevant problem: exposing GUI functionality to agents without brittle screen interactions. Explicit parameters, effects, and state invariants are a useful design choice.
  2. The implementation spans nine applications, with functionality-level results, component ablations, and supplementary validation against existing LibreOffice tests.
Critical weaknesses
  1. Limited novelty and weak baselines. Source inspection, API discovery, coding, and iterative validation are already within the capabilities of general-purpose agents such as Codex. The paper lacks a source-aware, budget-matched Codex baseline; its documentation-only baseline also uses far fewer tokens (2.79M vs. 76.26M), making the contribution of the proposed workflow unclear.
  2. Insufficient semantic validation and reusability evidence. The main correctness checks rely on specifications extracted by the framework itself, potentially missing GUI behavior. Independent test replay covers only 124 eligible LibreOffice targets (114 pass). There is little evidence that commands remain correct across unseen inputs and application states, or stable under repeated and interleaved invocations.
  3. Limited generalization and downstream evaluation. The benchmark uses selected functionality families of uneven complexity (e.g., five GIMP transformations and 43 Shotcut video-mode choices). The paper does not demonstrate that the commands generalize to broader application functions or improve end-to-end agent task performance.
Questions for authors
  1. How does APP2CLI compare with direct Codex given the same source code, tools, instructions, and token budget?
  2. Can the authors validate commands on unseen parameters and states, including repeated invocations, using independent behavioral checks?
  3. How representative are the selected functionalities, and do the generated commands improve downstream agent task success?
1 comment
AuthorsOpenReview ↗

Response to Reviewer y1Wz

1. What APP2CLI contributes (R-y1Wz-Weak1-NoveltyBaselines)

For each GUI functionality, Phase I formalizes the context that the GUI supplies implicitly into a structured semantic specification: parameters and their roles, preconditions (P), expected effects (E) and non-target invariants (N). This specification then guides Phases II and III. Public interfaces often do not expose this context (see the Calc Function Wizard example in Appendix E.1). Without the specification, semantically correct commands drop from 881 to 26, and without the skill guidance the success rate drops from 60.05% to 0.14% (Table 3), so both components are necessary. APP2CLI combines GUI analysis, which locates the handler but not its required effect, with specification mining, which describes required properties but not the GUI functionality. Trajectory-based agent learning trains agents to operate the GUI. APP2CLI instead turns each GUI operation into a command that agents invoke directly (related work and references in Section 5).

2. Source-aware Codex baseline (R-y1Wz-Weak1-NoveltyBaselines / R-y1Wz-Ques1-DirectCodex)

We added this baseline (ACGBaseline). Codex receives the same model, source code, documentation and task input as APP2CLI, but no expected effects or APP2CLI artifacts. It passes 541 of the 1,467 targets (36.88%) versus 881 (60.05%) for APP2CLI, and APP2CLI uses less than a tenth of the tokens and about a seventh of the time per target. Of its randomly sampled 50 failures, 30 miss hidden GUI context: 10 misuse a parameter or miss a precondition (P), 18 leave part of the intended effect undone (E), and 2 change content that should stay unchanged (N). For example, it leaves cell comments behind when sorting a Calc range, and it does not disable header sharing in Writer. The other 20 fail for other reasons, e.g., invoking a same-named command with different semantics, or an interface that cannot perform the GUI operation. Details of the setup, results and cost are in Sections 3.2, 4.1 and 4.3 and Appendix E (Table 9).

3. How the 881 commands were checked (R-y1Wz-Weak2-SemanticValidation)

A target passes when its command executes successfully, produces the specified effects and preserves the non-target invariants (Section 3.3). All 881 commands were executed and checked dynamically. The oracles reuse the developers' assertions in the source tests for LibreOffice (453 targets), and compare the final state with a GUI reference run or a reference value for the other applications (428 targets). Evidence independent of the generated specification comes from STOR, source-line coverage (98.01% of the GUI-executed lines within the semantic-core units of six comparable applications), which indicates that the commands run the application's own handlers, and the validation on unseen parameters and states. Details of the oracles and coverage are in Appendix E.6 and E.3.

4. STOR failures (R-y1Wz-Weak2-SemanticValidation)

STOR needs tests that exercise the semantic GUI operation, so that their assertions can serve as an independent oracle, and among our applications only LibreOffice provides such tests at scale. Before any replay, we checked whether the GUI step in each test can be replaced by the command, which holds for 124 of the 376 targets. Six of the ten failures are Undo defects in our commands. In the other four, two assertions read GUI state that a CLI run does not produce, and two source tests are unstable. Appendix E.7 (Table 10) describes each failure.

5. Unseen parameters and states (R-y1Wz-Ques2-UnseenParamsStates)

To test the generalization of the commands, our held-out validation keeps each command unchanged and uses new parameters (V1), a new initial state (V2), or repeated or interleaved invocations (V3, A→A or A→B→A). We prepared 137 variants on 51 targets (V1 42, V2 51, V3 44). We ran 10 variants on 5 targets: 7 pass, and in all 10 the command carries out the main operation correctly. Details of the protocol and results are in Appendix E.5. We will extend this validation to the remaining variants later.

6. Scope and downstream use (R-y1Wz-Weak3-Generalization / R-y1Wz-Ques3-Representative)

APP2CLI provides infrastructure commands for apps, so we evaluate whether each command performs its GUI functionality correctly, rather than end-to-end task success, which also depends on how LLM agents use the given commands. We selected representative functionality families rather than every functionality, so the number of targets per application depends on the family sizes. The new Appendix G shows targets with up to 39 GUI steps and 44 inputs, and APP2CLI's success rate stays between 72% and 89% across their terciles.

Comments on the paper

1 comment

We thank all three reviewers. We conducted the following additional experiments and analyses. All changes in the revised manuscript are marked in red.

  • A source-aware Codex baseline (ACGBaseline). Given the same model, source code and documentation, a general-purpose coding agent (Codex) passes 541 of the 1,467 targets (36.88%), compared with 881 (60.05%) for APP2CLI (Sections 3.2, 4.1 and 4.3, Appendix E, Table 9).
  • New analyses that identify which checks are independent of the generated specification and the construction feedback, and a root-cause analysis of each of the ten STOR failures, which reveals six real Undo defects (Appendix E).
  • Dynamic execution of CLI-Anything's released commands on 18 of the 29 LibreOffice targets they statically match, with every process traced, and a classification of all 564 released commands (Appendix F).
  • Validation on unseen parameters and states, including repeated and interleaved invocations. We report results on 10 variants and will continue with the remaining ones (Appendix E).
  • Complexity of the targets (Appendix G) and two related-work paragraphs (Section 5).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.