- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces
- a new benchmark, DeskIFBench, that asks agents to perform 255 tasks across 6 GUI applications (blender, etc.). Task completion is verified based on generated/manipulated artifact state
- ANIS, a method/agent scaffold that generates a new CLI interface for GUI applications and allows agents to achieve higher scores on DeskIFBench
Critical strengths
- Improving usability for agents and humans is a pressing concern in software development
- Using software release history as a natural experiment for interface quality is a neat idea
- The DeskIFBench benchmark paired with the release history is a good contribution for studying the evolution of interfaces
- Being able to automatically generate good interfaces for agents is very useful
Critical weaknesses
- I find the link between "human-oriented software evolution" to agent-native interfaces generally weakly supported:
- generally, humans for all of these applications only use the GUI, which is not varied/explored in this paper. So it feels like rather the claim would be that the human-written developer-API evolution could also inform the interface for agents (which is somewhat less surprising)
- the 3 criteria in section 4.3.1, 4.3.2, etc., are mostly supported by examples, not by clearly ablated experiments. These experiments would probably be much clearer on ANIS, rather than derived from an evolution of a framework that was optimized for humans that mostly use the GUI. As it stands, I do not find this very clearly motivated/ablated
- The ANIS setup could benefit from stronger baselines and clearer ablations for its components.
- the performance increase seems relative low (and does not generalize to all apps)
- furthermore, the central Table 2 only shows results for one LM, it is not supported that gains generalize across (stronger) models
- CLI-anything seems like a weak baseline for other ways to construct dedicated agent APIs
- the performance increase seems relative low (and does not generalize to all apps)
Questions for authors
Metrics
- why not report pass@1 with variance error bars? It seems like pass@3 just inflates the metric. Not sure if I follow the motivation for using pass@3 here, it seems like it would reduce the sensitivity of the benchmark
- is tokens/task for 3 attempts or just 1?
Benchmark interpretation
- Is there a more quantitative support fort the conclusions presented in 4.3.1, 4.3.2 (basically a way of more proving that the derived conditions are indeed the basis of building successful interfaces)
ANIS
- How much budget did CLI-anything get compared to ANIS? Also it seems like CLI-anything has a
:refinecommand, was that used? - Are there any other (stronger?) baselines that ANIS could be compared to? For example, could this be compared to MCP or similar interfaces to the programs?
- It seems like the success gains for ANIS vs native are relatively small, but costs drop very significantly (mostly unrelated to goals). What is the reason for the lower cost? Could you provide turn counts, trajectory analyses etc., that show why costs are so different? Could this also be slightly conflated with the pass@3 metric (e.g., maybe your pass@1 success rate actually gets higher with ANIS, and the dropped costs are because failed runs are typically more expensive, so higher pass@1 success rate leads to lower cost even if pass@3 stays the same?)
- For results from table 2, would be great to see the findings generalize across different LMs. Seeing reductions in tokens per task only for one model doesn't mean this transfers
Reply to Weakness1
We thank the reviewer for the careful reading and for recognizing the cross-version methodology as a main contribution of this paper. We respond to each point below; all revisions will be incorporated into the paper.
Weakness 1: Why these three properties; uneven support in the study; no independent isolation
Why these three properties. Recent studies of software engineering agents consistently divide agent trajectories into three phases: exploring and localizing what to change, writing the change, and verifying and repairing [Agentless, FSE 2025; Bai et al., 2026]. In Appendix B, we likewise attribute context tokens to exploration, writing, and verification & repair, and these phases cover all tokens.
Our three properties correspond to these three phases:
We will state this correspondence explicitly in the revision to make the three principles more systematic.
Support in the study. We agree that the cross-version study supports the three properties unevenly. Its role is to generate hypotheses; the ablations below test them.
setVertexMarkerAppearancehas no read function in any release), so cross-version data cannot reveal its effect. The study only shows a recurring symptom — 581 trajectories claim completion, yet the saved file fails verification — which motivates the property and is exactly why a controlled ablation is needed.Independent isolation. We test each property by changing only that property within the same release. The main text already isolates observability and granularity (Section 5.3; DeepSeek V4 Flash, 368 tasks):
The revision will add a discoverability experiment: keeping the command set and implementations unchanged, we only replace the object-grouped catalog with an alphabetically sorted flat list, retaining command summaries. Both conditions were run concurrently with the same model, prompts, and timeouts; runs interrupted by service outages were rerun under a uniform criterion.
Removing grouping lowers the success rate by 3.0 percentage points. With grouping, the agent first selects an object category and then searches for commands within it; search results are also labeled with their category. The flat condition places all commands in a single list; in applications such as Writer (1,053 commands) and QGIS (320 commands), the agent easily misses the target command and repeatedly tries similarly named but wrong commands.
For example, the QGIS dangling-endpoint detection task requires first reading line coordinates. In the flat condition, the agent did not find the read command
gdal-ogrinfojson; it instead inspected related write commands with different purposes, such asvector-feature-writeandvector-layer-create, then spent more than 40 steps trying to read the file through the shell, and eventually timed out without producing a result. In the grouped condition, the agent searched for "info", foundgdal-ogrinfojsonlabeled under the gdal category, then completed the detection and passed verification.References. Xia et al., "Demystifying LLM-Based Software Engineering Agents" (Agentless), FSE 2025. Bai et al., "How Do AI Agents Spend Your Money? Analyzing and Predicting Token Consumption in Agentic Coding Tasks," arXiv:2604.22750, 2026.
(continued in the next comment)