Built for Humans, Not yet for Agents: Turning Apps into Agent-Native CLIs at scale
Abstract
Applications (Apps) expose rich functionalities, such as document editing and image processing, through graphical user interfaces (GUIs). However, reliably interacting with these GUIs remains challenging for large language model (LLM) agents. Command-line interfaces (CLIs) offer an alternative, but generating com- mands for GUI functionalities requires preserving their implicit context and be- havioral constraints. We present APP2CLI, a framework that generates agent- native commands for existing GUI functionalities in apps. APP2CLI traces GUI “trigger-handler-effect” paths to extract behavioral semantics and represent them in a structured semantic specification. Skill guidance directs the construction of CLI templates through public programmatic interfaces. Cooperating agents im- plement templates as standalone, parameterized commands. We evaluate APP2CLI on 1,467 target GUI functionalities across 11 application units from 9 apps. APP2CLI achieves a 60.05% agent-native commands gen- eration success rate , producing semantically correct commands for 881 target functionalities, . The CLI executions also achieve 98.01% line-coverage overlap with the corresponding GUI executions. Removing the semantic specification or skill-guided template construction reduces the generation success rate to 1.77% and 0.14%, respectively. Code is available at https://zenodo.org/records/22970644.
est. 32% chance this paper gets accepted at ICLR 2027.
What do you think this paper will get?
All positions stay anonymous.