acceptodds
Withdrawn from ICLR 2027

Actions Are Not Text: Direct Executable-Set Decoding for GUI Agents

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

GUI agents ultimately execute typed environment commands, yet many policies first serialize each action into text and then parse it back. We ask whether this autoregressive round trip is necessary when the current legal action set is finite and explicit. DirectAct retains the pretrained backbone but replaces action-string generation with a small head that scores complete executable tuples, emitting no action tokens. In an ecological comparison across Qwen2.5-VL-7B, MAI-UI-8B, and Llama-3.1-8B, an unadapted frozen JSON interface emits 70.53–94.78 action tokens and takes 29.70–42.40 times the loaded action-model-path latency of DirectAct. We then test whether this practical gap survives a strong trained comparator by aligning training data, input, parameter scale, and legal support with greedy exact-set LoRA generation. On 1,218 prospective states from 88 officially successful AndroidWorld reference workflows, DirectAct satisfies a predeclared two-percentage-point action-weighted non-inferiority criterion on Qwen and MAI; Llama provides descriptive transfer. Across the three backbones, DirectAct is 4.51–5.13 faster even though the generator emits only 6.20–6.68 tokens. These results identify an interface boundary: language models may construct policy state, while finite environment actions can be selected natively rather than serialized autoregressively.

Primary area:
foundation or frontier models, including LLMs
Keywords:
GUI agents, action representation, efficient inference, structured prediction, executable action selection, autoregressive decoding

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_gMXiOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

DirectAct replaces autoregressive action-string generation with a small head that scores complete executable action tuples from a supplied candidate set using a frozen backbone. The main comparison holds training data, structured inputs, candidate sets, and approximate trainable parameter counts constant against a compact constrained LoRA generator. On 1,218 states from 88 prospectively collected successful workflows, DirectAct meets a predeclared two-percentage-point non-inferiority criterion on Qwen and MAI while reducing measured action inference latency by 4.51-5.13 times across three backbones.

Critical strengths
  1. The controlled compact-generator comparison is persuasive for the paper's conditional interface claim. The generator already emits only six to seven tokens and selects from the same executable set, so the result is more informative than a comparison only against verbose JSON. The exhaustive sequence-scoring analysis further separates greedy decoding quality from the underlying generator's candidate scores.

  2. The prospective collection protocol, checkpoints fixed before the primary collection, and workflow-clustered non-inferiority analysis improve the credibility of the offline comparison. The paper distinguishes the exploratory Llama result, ecological JSON timing, and separate closed-loop variant rather than presenting them all as the same controlled experiment.

Critical weaknesses
  1. The demonstrated benefit is conditional on an unusually structured and relatively small supplied action set, and is not yet evidence of retained closed-loop GUI task success. The test states come from successful reference workflows in seven task families also represented in training. Candidate generation already supplies control/value constraints, and its coverage under policy errors or unforeseen values is not quantified. Appendix H uses only five expense tasks and a different conditional-value checkpoint, so it does not close this gap for the main DirectAct model. This limits deployment/generalization claims, although it does not invalidate the matched offline result.

  2. Pooled action-weighted non-inferiority hides potentially important heterogeneity. Appendix F reports sizeable family-specific regressions, including SMS reply, and a negative MAI difference for actions without values. A user-facing workflow can fail from a small number of such errors. The current evidence therefore supports average action accuracy on this collection more strongly than uniform replacement of the generator across workflow types.

Questions for authors
  1. Please report workflow-macro differences and uncertainty, alongside the existing action-weighted result, and explain the SMS/no-value regressions. Are they concentrated in a small number of workflows or systematic action types?

  2. What are candidate-set coverage and failure modes on states reached by the main checkpoint itself, including incorrect actions? Can the main-checkpoint closed-loop result be separated from the different Appendix H variant?

  3. How much of the accuracy/latency result remains with a simpler flat candidate scorer or one-token candidate-index output? This would help distinguish the benefit of direct selection itself from the proposed factorized tuple head.

Reviewer_2VgYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

DirectAct studies GUI action selection when a finite set of complete executable actions is already available. It freezes a pretrained backbone and trains a lightweight factorized head to score action tuples, avoiding autoregressive action-token generation. On 1,218 states from 88 successful AndroidWorld reference workflows, DirectAct meets the predeclared action-accuracy non-inferiority criterion against compact, trie-constrained LoRA generation on Qwen and MAI. Across three backbones, the reported action-inference speedup is 4.51–5.13×. A separate comparison against an unadapted JSON interface yields larger speedups.

Critical strengths
  1. The main comparison controls several important factors. DirectAct and exact-set LoRA use the same training examples, inputs, candidate order, and executable sets, with approximately matched trainable parameter counts. Exhaustive sequence scoring further shows that greedy decoding contributes substantially to the generator’s accuracy deficit.
  2. The latency measurements include relevant controls. Timing includes full input contextualization and uses interleaved measurements with CUDA synchronization. Removing the Python trie callback preserves a substantial speed gap. The prospective Qwen/MAI evaluation and workflow-level bootstrap also strengthen the offline accuracy assessment.
Critical weaknesses
  1. The executable-set assumption externalizes substantial action-construction work. Appendix E constructs complete candidates using goal-derived field–value requirements and executor-provided actions. This is reasonable for structured tasks, but leaves argument construction and candidate coverage outside the learned policy. The paper does not sufficiently establish how broadly this interface can be obtained without already solving a substantial part of the task.
  2. A simpler candidate-ID baseline is missing. The supplied executable sets contain only 16.83 actions on average and at most 31, yet the generator emits a four-field sequence of approximately six to seven tokens. A trained selector that outputs a single local candidate-ID token could remove most sequential decoding. Without this comparison, the incremental benefit of the proposed factorized head remains unclear.
  3. Closed-loop effectiveness is not established for the main method. The primary evaluation measures reference-action agreement on successful workflows, with all seven task families represented in training. It does not test recovery from the policy’s own errors. The live experiment uses a different checkpoint and method variant against an unadapted baseline, so it does not resolve this gap.
  4. The reported speedup covers only the downstream selection stage. Timing excludes upstream state acquisition and candidate construction. Since DirectAct depends on a complete executable set, the cost and reliability of producing that set matter to its practical value. The reported inference ratios therefore do not establish a comparable reduction in end-to-end task latency.
  5. The methodological contribution remains modest. Direct scoring is a natural alternative once complete actions are explicitly enumerated. The paper acknowledges related non-autoregressive decision approaches, including Jev, but does not experimentally establish an advantage over an alternative direct-selection design. The current evidence demonstrates an advantage over the evaluated serialization interfaces more clearly than a distinctive benefit of DirectAct itself.
Questions for authors
  1. How much task-specific logic is required to construct the executable candidates, and how does candidate coverage behave outside the successful reference trajectories?
  2. How does DirectAct compare with a trained single-token candidate-ID selector using the same executable set?
  3. Can the main DirectAct and matched LoRA checkpoints be compared closed-loop, reporting both task success and latency including candidate construction?
Reviewer_gQkoOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

DirectAct uses a small action head over a frozen pretrained backbone to select executable GUI actions directly, reporting faster inference with comparable action accuracy on states collected from AndroidWorld workflows.

Critical strengths
  • The paper offers a timely empirical study of direct GUI action selection without autoregressive text generation, following the recent release of Jev.
Critical weaknesses
  • The architectural novelty is limited given prior works with specialized GUI prediction heads in META-GUI and GUI-Actor.
  • The models like MAI-UI is designed for screenshot-based interaction, so evaluating it with accessibility-tree text alone may underestimate its performance.
  • Offline accuracy on AndroidWorld reference states does not establish closed-loop task success on AndroidWorld. Also the work is missing some key benchmarks in this field, e.g., Screenspot-Pro, MobileWorld, and key baselines as comparison.
Questions for authors
  • How does DirectAct compare with similar methods such as META-GUI in accuracy and inference latency?
  • What is DirectAct's closed-loop task success rate on AndroidWorld, and how does it compare with the baselines?
  • What is the total task latency on AndroidWorld?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.