acceptodds
Under review as a conference paper at ICLR 2027

When to Compile a Computer-Use Agent? Measuring Payback and Making Compilation Decisions for Token Efficiency

Abstract

Compiling GUI procedures that agents execute repeatedly into programs can reduce their token costs. However, measuring payback and deciding when to compile have two challenges. First, compilation costs are uncertain because attempts can require repair and still fail to produce a usable program. Second, future reuse is unknown because tasks may stop arriving or GUI drift may stop the program from working. To address these challenges, we propose PACE (Payback-Aware Compilation from Experience), a system with a measurement protocol and an online compilation algorithm. The measurement protocol records successful and failed compilation costs, and compares agent and program execution costs on matched task inputs to estimate per-use savings and payback counts. Using these measurements, the online algorithm compares estimated future savings with compilation costs, including failed attempts, based on past task arrivals and compilation outcomes. It checks execution and compilation charges against a cumulative budget determined by observed task arrivals before allowing either action. Under stated action-cost assumptions, total cost after each arrival is at most times the cost of running every task with the agent. For successful compilation attempts, estimated payback counts excluding source agent runs are – uses. In simulations using recorded task arrivals, PACE reduces token costs by compared with ReAct, with the AutoRPA adaptation, and with the ToolPro adaptation on average ().

Primary area:
foundation or frontier models, including LLMs
Keywords:
computer-use agent, large language model

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_qcLjOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes measurement and optimization of the efficiency of compiling (converting) agent rollouts into repeatable programs for computer-use on recurring tasks. It specifically measures the cost of successful and failed compilation, estimates the break-even point (i.e. "payback count"), and proposes an online algorithm to decide whether to compile an agent run that gets at most (1+ϵ)×(1 + \epsilon) \times the cost of running the baseline agent for every instance of a task family. Experiments on a number of task families in AndroidWorld and OSWorld show token reduction between 17.3% to 24.9% against baseline ReAct and AutoRPA, with ϵ=0.25\epsilon = 0.25.

Critical strengths
  1. The overall paradigm of compiling agent experience into repeatable programs is highly valuable, and this work contributes to the growing body of work in this paradigm.
  2. The paper is precise in defining the costs and failure probabilities involved in various parts of an agent-to-program compilation algorithm, so that a cost analysis can be done rigorously.
Critical weaknesses

One main weakness is that this paper tries to solve a problem that either 1) has a known trivial solution in all contemporary CUA products, or 2) diminishes in importance as models get better at multimodal understanding and code generation. The stated problem (line 063) is that "compiling early can waste tokens if tasks do not repeat, while waiting can lose savings on tasks that repeat often." However, in practice, all generally usable products already allow the user to define routines (i.e. workflows or recurring tasks), so the user already tells you whether a task is repetitive. If the task is repetitive, then it is definitely worth the effort to compile agent runs into programs if you believe that models will get better at understanding long multi-modal trajectories and code generation. In other words, as models get better, your gg successful verifications will approach your jj compilation attempts, then probability of a successful program approaches 1, so there will be some number of runs at which the cost of compilation will be amortized low enough to be lower than the cost of always running the base agent loop. If some users keep prompting the agent with similar task instances without creating routines, then we need to decide whether it's worth the compilation cost. But in this case, the necessary condition is to first detect that the many task instances actually belong to a family of parameterized tasks, and this necessary step is not addressed in this paper. If you can't detect recurring tasks, there's no point making decision about whether to compile an agent run.

Second main weakness is that the method (under stated assumptions) can only guarantee getting at most (1+ϵ)×(1 + \epsilon) \times the cost of running the baseline agent. Then this theoretical guarantee is worthless; I should just run the baseline agent. Sure, in your experiments you get better than that upper bound, but there's no point making the calculations if the upper bound is worse than the baseline.

Thirdly, one key choice inside the cost model in Section 2 is that "GUI drift stops further program reuse." When the environment changes, an agent can take over and try to finish the task, and then the experience can be given to a coding model to fix the program, so the new program can be reused. This is not hard to implement. So, this choice in the cost model will not reflect real-world agent-to-program methods in general for very long, which diminishes the value of this work.

Fourthly, the experiments were done on seven task families (4 AndroidWorld, 2 OSWorld, one WebArena). Each task family contains multiple task instances, but nevertheless, seven task families is way too small and below the standard usually seen in the field of CUA, where people evaluate on the entire AndroidWorld and OSWorld, etc. I know that OSWorld does not by default provide parameterized tasks that you can use to test the recurring task scenario, but still I expect more effort to create more parameterized task families from the base tasks.

Minor:

  1. The introduction says that "computer-use agents consume more tokens than general agents because of the screenshot inputs, making them more expensive to run". That's not really the main problem of CUA. The main problem is that most contemporary CUA products plan step-by-step from scratch for every instance of every task, even if they've seen the task a million times. Also, some read the entire OS-level accessibility tree or HTML, so it's not just a screenshot problem.
  2. Introduction says "We focus on the program method, which has the highest token efficiency". Not clear what is meant by "program method", whether it refers to constructing a full program for the entire task, or still using an agent loop but calling short snippets of code.
  3. Section 5, Sepsis and BPI 2019 process logs appear suddenly and I have no idea what is the purpose.
  4. Section 2, pp is defined as the probability that compilation succeeds but is not used anywhere I could find.
Questions for authors

None

Reviewer_rL69OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies when a computer-use agent should compile a repeatedly executed GUI procedure into a parameterized program, given that compilation attempts can fail after costly repair and that future reuse is unknown.

Measurement study. For seven task families (four AndroidWorld tasks, two LibreOffice tasks, one WebArena Reddit task) and three small models, the authors compile a program from three agent traces, record the cost of every successful and failed attempt, verify each program on its source tasks, and then run it on thirty fresh inputs alongside the agent on the same inputs. Eleven of twenty attempts produce a working program. Once a program exists it is almost free to run compared with the agent, but recouping the compilation cost takes between about 1 - 16 uses, failed attempts can cost several times a successful one, and simulated interruptions such as notifications break most programs, with a sizable share of those failures producing no detectable error.

PACE, an online rule for deciding when to compile. After each task, a family with enough traces compiles if an estimate of future reuse times the per-use saving exceeds an estimate of compilation cost that includes expected failed attempts. A separate budget check caps total spending at (1+ε) times what running the agent on every task would have cost, and a short proof shows the cap holds whenever the cost of each action is known in advance.

Evaluation of PACE in simulation, using the measured cost profiles on synthetic task streams and on arrival orders taken from two process-mining logs and two Wikipedia edit streams. With ε = 0.25, PACE spends about 17% less than always running the agent on the recorded streams, and less than two decision rules adapted from AutoRPA and ToolPro. Ablations show that the savings test and the separate accounting of failed attempts drive the average savings, while the budget mainly limits the worst case. Alternative compilation rules under the same budget come within one percent of PACE. A small live test on a calendar web app, twelve paired tasks, shows the program-with-fallback path succeeding more often and at lower cost than the agent, though the cost-difference interval includes zero.

Critical strengths

Prior compile agent traces to code work reports payback counts (4 to 17 uses) measured on different system parts, or none at all (lines 050-053). This paper charges failed compilation attempts explicitly, including recorded lower bounds when usage records are missing, repeats compilations to see whether outcomes persist, replays the agent on the same 30 bindings the program ran on, and measures program failure and detectability under nine GUI perturbations. The results are interesting - failed attempts can cost several times a successful one; payback of up to ~16 uses before counting failures; interruptions break most programs and a substantial fraction of failures go undetected.

Critical weaknesses

The algorithm is a small contribution, and the paper's own appendix shows that the parts specific to PACE do not matter. The cost guarantee Kt≤(1+ϵ)AtK_t \le (1+\epsilon)A_t (Theorem 4.1) is a one-line induction: before each action the policy reserves the largest charge that action could incur, and if the reservation does not fit it skips the action and falls back to the agent. That only works when the charge of every action is known ahead of time. Historical means do not give such bounds (Sec. 4.3, Sec. 7), and the live test in Appendix D already violates the assumption when a binding reservation exceeds the allowance and a task is denied before any service occurs.

What distinguishes PACE from other budgeted rule is its pair of estimates in Eq. 4: the reuse count H^=min⁡ ⁣(Ndd+5, 1h)\hat H = \min\!\big(\tfrac{Nd}{d+5},\,\tfrac{1}{h}\big) and the compilation cost B^=C+(1p^−1),Cfail\hat B = C + \big(\tfrac{1}{\hat p}-1\big),C^{\mathrm{fail}}. Appendix C.6 compares these against a rule that uses min⁡(N,1/h)\min(N, 1/h) under the same budget, and that rule is marginally cheaper (paired cost ratio to PACE in [0.9997, 0.9999][0.9997,\,0.9999]). The results do not identify the exact reuse-count estimate as a separate source of improvement. The budget, for its part, changes mean cost on the recorded logs by 0.0013%. The abstract and the second contribution describe something more: an algorithm that "compares estimated future savings with compilation costs, including failed attempts, based on past task arrivals and compilation outcomes," together with a proven cumulative bound. The estimation from experience is the part shown not to matter, and the bound is the part shown not to hold outside the simulator. To be the contribution the paper describes, PACE would need either an estimator that beats break-even on streams with real arrival structure, or a bound that survives attempt costs varying by an order of magnitude within a family.

None of the Section 5 results come from running anything on a GUI. They come from a simulation in which every quantity is set by the authors: the chance that a compilation succeeds is a number drawn at random, the policy is handed the family label of each task and told when drift has happened, and the task streams are hospital and purchase-order logs and Wikipedia edits relabeled as "families." The simulation is built from the same assumptions PACE is built from, so it cannot tell you whether those assumptions hold. Only a run on real GUI tasks can. On top of that, the authors say they used these same simulation grids while developing PACE, so the method's design choices were made while watching results on the exact conditions it is then reported on.

The methods PACE is compared against in the main tables are not the AutoRPA and ToolPro systems. They are decision rules the authors wrote and labeled with those names. The ToolPro adaptation compiles only if the saving from a single task exceeds the full compilation cost. Since compiling costs between 137K and 3424K tokens while a single task saves at most 650K, that condition is never satisfied, and the rule never compiles. In practice it is just "always use the agent," which is why its row matches ReAct exactly in every table. The AutoRPA adaptation errs in the other direction. It compiles each family once, the moment three traces are available, with no retry and no budget, so it pays for compilation even on families that never come back and on models that rarely produce a working program. The two named baselines are therefore the trivial extremes: one never compiles, the other always compiles once. Any rule with some judgment about when to compile beats both; beating them shows nothing about the quality of that judgment.

Questions for authors

Given App. C.6, what do the authors consider the contribution of the Eq. 4 estimators over "Count + budget"? Would they reframe the method as budgeted compilation with any monotone reuse estimate?

Is there any condition in which the ToolPro adaptation compiles? If not, why is it reported as a baseline?

Can the online algorithm be run on a real stream of GUI tasks (e.g., the four AndroidWorld families with sampled parameters, a few hundred arrivals, one model) with real compilation and drift outcomes, rather than assigned verification probabilities?

How sensitive are Table 3 results to which cost profile and verification probability each log family receives? A per-family breakdown of where the BPI 2019 / Wiki-tools savings come from would help.

How should a deployment set the max(C, C_fail) reservations when attempt costs vary by 10x within a family (Table 1; App. A.5), and how does the bound degrade when reservations are exceeded as in App. D?

Reviewer_ULZyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies when a CUA should compile a repeated GUI procedure into a program to save tokens. The paper proposed PACE which measures compilation costs and reuse savings, and combines a compilation rule with a cumulative spending limit.The paper evaluates the method on 7 task families, with 3 models and simulated task streams.

Critical strengths
  1. The focus and motivation of this paper is clear. It studies an important cost that is often overlooked, including unsuccessful compilation and repair attempts. And it also separates compilation-only payback from payback including source agent runs.
  2. The proposed budget mechanism is simple and practical. It gives a clear worst-case guarantee relative to the agent-only cost. The separation between compilation proposals and budget checks is clear and useful.
Critical weaknesses
  1. The evidence for the online algorithm comes almost entirely from simulation with strong assumptions, such as non-GUI arrival logs, perfectly detected failures and GUI drift reported by the environment. It is therefore unclear whether the reported savings would hold in practice. This is my biggest concern and the main reason for my current score.
  2. The additional benefit of the proposed compilation rule is not clearly established. The baselines are weak: the ToolPro adaptation never compiles and is identical to ReAct. There is no comparison with standard rent-or-buy algorithms.
Questions for authors

I'm happy to adjust my score if the authors can effectively address the following questions:

  1. Would it be possible to evaluate the full online algorithm on a real stream of GUI tasks, even a small one? This would show whether the simulated savings still hold in practice.
  2. Could you provide a comparison of PACE with standard rent-or-buy algorithms under the same budget, such as the break-even rule or learning-augmented ski rental?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.