- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper proposes measurement and optimization of the efficiency of compiling (converting) agent rollouts into repeatable programs for computer-use on recurring tasks. It specifically measures the cost of successful and failed compilation, estimates the break-even point (i.e. "payback count"), and proposes an online algorithm to decide whether to compile an agent run that gets at most the cost of running the baseline agent for every instance of a task family. Experiments on a number of task families in AndroidWorld and OSWorld show token reduction between 17.3% to 24.9% against baseline ReAct and AutoRPA, with .
Critical strengths
- The overall paradigm of compiling agent experience into repeatable programs is highly valuable, and this work contributes to the growing body of work in this paradigm.
- The paper is precise in defining the costs and failure probabilities involved in various parts of an agent-to-program compilation algorithm, so that a cost analysis can be done rigorously.
Critical weaknesses
One main weakness is that this paper tries to solve a problem that either 1) has a known trivial solution in all contemporary CUA products, or 2) diminishes in importance as models get better at multimodal understanding and code generation. The stated problem (line 063) is that "compiling early can waste tokens if tasks do not repeat, while waiting can lose savings on tasks that repeat often." However, in practice, all generally usable products already allow the user to define routines (i.e. workflows or recurring tasks), so the user already tells you whether a task is repetitive. If the task is repetitive, then it is definitely worth the effort to compile agent runs into programs if you believe that models will get better at understanding long multi-modal trajectories and code generation. In other words, as models get better, your successful verifications will approach your compilation attempts, then probability of a successful program approaches 1, so there will be some number of runs at which the cost of compilation will be amortized low enough to be lower than the cost of always running the base agent loop. If some users keep prompting the agent with similar task instances without creating routines, then we need to decide whether it's worth the compilation cost. But in this case, the necessary condition is to first detect that the many task instances actually belong to a family of parameterized tasks, and this necessary step is not addressed in this paper. If you can't detect recurring tasks, there's no point making decision about whether to compile an agent run.
Second main weakness is that the method (under stated assumptions) can only guarantee getting at most the cost of running the baseline agent. Then this theoretical guarantee is worthless; I should just run the baseline agent. Sure, in your experiments you get better than that upper bound, but there's no point making the calculations if the upper bound is worse than the baseline.
Thirdly, one key choice inside the cost model in Section 2 is that "GUI drift stops further program reuse." When the environment changes, an agent can take over and try to finish the task, and then the experience can be given to a coding model to fix the program, so the new program can be reused. This is not hard to implement. So, this choice in the cost model will not reflect real-world agent-to-program methods in general for very long, which diminishes the value of this work.
Fourthly, the experiments were done on seven task families (4 AndroidWorld, 2 OSWorld, one WebArena). Each task family contains multiple task instances, but nevertheless, seven task families is way too small and below the standard usually seen in the field of CUA, where people evaluate on the entire AndroidWorld and OSWorld, etc. I know that OSWorld does not by default provide parameterized tasks that you can use to test the recurring task scenario, but still I expect more effort to create more parameterized task families from the base tasks.
Minor:
- The introduction says that "computer-use agents consume more tokens than general agents because of the screenshot inputs, making them more expensive to run". That's not really the main problem of CUA. The main problem is that most contemporary CUA products plan step-by-step from scratch for every instance of every task, even if they've seen the task a million times. Also, some read the entire OS-level accessibility tree or HTML, so it's not just a screenshot problem.
- Introduction says "We focus on the program method, which has the highest token efficiency". Not clear what is meant by "program method", whether it refers to constructing a full program for the entire task, or still using an agent loop but calling short snippets of code.
- Section 5, Sepsis and BPI 2019 process logs appear suddenly and I have no idea what is the purpose.
- Section 2, is defined as the probability that compilation succeeds but is not used anywhere I could find.
Questions for authors
None