- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
APP2CLI turns GUI functions into commands with explicit parameters. It follows GUI trigger–handler–effect paths to work out what each function should do, uses skills to build templates around public APIs, and has agents implement and check the commands. On 1,467 target functions from nine apps, the paper reports 881 successful commands (60.05%); 41 targets match CLI-Anything's existing commands. The evaluation also includes ablations, 98.01% average GUI-to-CLI line overlap in the semantic-core code of six comparable apps, and 114 passing replays out of 124 eligible LibreOffice targets using unchanged upstream assertions. The main idea is to carry the GUI's context and behavior into the generated interface. The evaluation focuses on individual functions.
Critical strengths
-
Making the GUI's hidden context explicit is a useful idea. The specification records what must hold before a command runs, what should change, and what should stay untouched. The sorting and merging examples show why the selected objects, active column, and content policy matter. Getting these details right is central to making the commands useful.
-
The work covers 11 application units, and I like the decision to check commands against existing application tests. STOR keeps the upstream fixtures and assertions, with 114 replays passing. This gives some support for the commands' actual behavior, though it only covers a subset of them.
Critical weaknesses
-
I couldn't tell how thoroughly the 881 commands were checked. Section 3.3 defines PASS, but says too little about how many inputs were tested, how varied they were, or how the checks catch things that the extracted specification missed. Appendix E.1 gives examples of independent observations, but doesn't show how widely they were used. For STOR, only 124 of the 376 already-supported LibreOffice targets from fully supported source-test cases were eligible, and 114 passed. It's unclear how the ten failures affect the main count. The paper does acknowledge that these checks don't prove full equivalence. Even within that stated scope, I need more detail to judge the claim that 881 commands are semantically correct and reusable.
-
The 21.5× comparison is hard to interpret as a gain from the method itself. APP2CLI is built for the chosen fine-grained targets. CLI-Anything is checked by statically matching its released commands, while BASELINE starts from documentation units that don't line up with the benchmark intents. The ratio tells us about coverage of this target set; it doesn't tell us how the methods would compare if asked to build commands for the same targets. CLI-Anything already maps GUI actions to backend APIs, so checking its existing commands leaves that question open. The ablations are helpful, but don't fix this mismatch. I'd like to see a comparison with the same targets, source information, and a similar total inference budget.
Questions for authors
-
Clarification: How were the test inputs and oracles for the 881 passing commands built? Which checks were independent of the generated specification and construction feedback? Please give the number of input/state settings tested and any held-out parameter variations. I'm missing these details when judging the correctness and reuse claims.
-
Evidence needed: For the ten STOR failures, which commands behaved incorrectly, and which failures can be traced to unstable tests or limits of the observations? What evidence supports that split? Are these commands included in the 881 passes, and would confirmed errors change the count? These cases would let us check whether the main validator agrees with the upstream assertions.
-
Evidence needed: To separate the method's gains from differences in coverage and scoring, have you compared against a baseline builder or a simpler source-aware generator given the same targets, similar information, and a similar total inference budget? Also, can the static matching combine existing commands?
Response to Reviewer 1nhU
1. How the 881 commands were checked (R-1nhU-Weak1-881Checks / R-1nhU-Ques1-InputsOracles)
A target passes when its command executes successfully, produces the specified effects and preserves the non-target invariants (Section 3.3). All 881 commands were executed and checked dynamically. Inputs and initial states are taken from the source tests or from historical GUI executions. The oracles reuse the developers' assertions in the source tests for LibreOffice (453 targets), and compare the final state with a GUI reference run or a reference value for the other applications (428 targets). Evidence independent of the generated specification comes from STOR, source-line coverage (98.01% of the GUI-executed lines within the semantic-core units of six comparable applications), which indicates that the commands run the application's own handlers, and the validation on unseen parameters and states. Details of the oracles and coverage are in Appendix E.6 and E.3.
2. The ten STOR failures (R-1nhU-Ques2-TenSTORFailures)
STOR needs tests that exercise the semantic GUI operation, so that their assertions can serve as an independent oracle, and among our applications only LibreOffice provides such tests at scale. Before any replay, we checked whether the GUI step in each test can be replaced by the command, which holds for 124 of the 376 targets. Six of the ten failures are Undo defects in our commands: Undo granularity differs from the GUI when deleting multiple sheets (2), and the test's Undo does not revert four Calc statistics functions (4). In the other four, two assertions read GUI state that a CLI run does not produce (Impress find & replace, Writer change tracking), and two source tests are unstable (Writer index and page-number format). Appendix E.7 (Table 10) describes each failure.
3. Unseen parameters and states (R-1nhU-Ques1-InputsOracles)
To test the generalization of the commands, our held-out validation keeps each command unchanged and uses new parameters (V1), a new initial state (V2), or repeated or interleaved invocations (V3, A→A or A→B→A). We prepared 137 variants on 51 targets (V1 42, V2 51, V3 44). We ran 10 variants on 5 targets: 7 pass, and in all 10 the command carries out the main operation correctly. Details of the protocol and results are in Appendix E.5. We will extend this validation to the remaining variants later.
4. Comparison with CLI-Anything (R-1nhU-Weak2-21.5x)
We agree that 21.5× (881/41) is a coverage ratio on this target set and does not by itself compare the methods on the same targets. The targets were fixed before any command was validated and were not selected by what CLI-Anything supports. By source-code analysis, only 3 of its 564 released commands (0.53%) always pass the GUI operation directly to the application. On the 18 LibreOffice targets we ran dynamically, APP2CLI passes all 18, while CLI-Anything passes 8 by the final artifact and none with evidence that LibreOffice produced the effect. Details of the static analysis and of each dynamic run are in Appendix F.
5. Source-aware baseline (R-1nhU-Ques3-SourceAwareBaseline)
We added this baseline (ACGBaseline). Codex receives the same model, source code, documentation and task input as APP2CLI, but no expected effects or APP2CLI artifacts. It passes 541 of the 1,467 targets (36.88%) versus 881 (60.05%) for APP2CLI, and APP2CLI uses less than a tenth of the tokens and about a seventh of the time per target. Of its randomly sampled 50 failures, 30 miss hidden GUI context: 10 misuse a parameter or miss a precondition (P), 18 leave part of the intended effect undone (E), and 2 change content that should stay unchanged (N). For example, it leaves cell comments behind when sorting a Calc range, and it does not disable header sharing in Writer. The other 20 fail for other reasons, e.g., invoking a same-named command with different semantics, or an interface that cannot perform the GUI operation. Details of the setup, results and cost are in Sections 3.2, 4.1 and 4.3 and Appendix E (Table 9).