acceptodds
Under review as a conference paper at ICLR 2027

LocalLSTC: A Long Short-Term Control Architecture for Locally Deployed GUI Agents

Abstract

Modern GUI-agent frameworks achieve strong desktop task performance with frontier API models, yet persistent control information often remains implicit in growing interaction trajectories. At each step, the planner reconstructs the active task stage, accumulated evidence, and runtime feedback before deciding the next action. This dependence becomes more pronounced under weaker local reasoning backbones. Across four representative state-of-the-art frameworks, replacing GPT-5 with Qwen3.5-9B reduces average OSWorld SR-100 from 60.9% to 35.2%. Trajectory annotation further identifies at least one control failure in 94.7% of failed trajectories. To address this problem, we introduce LocalLSTC, a training-free architecture that organizes control by temporal scope, maintaining persistent cross-step state to guide short-term execution commitments. The framework comprises two complementary mechanisms. Long-to-Short Planning forms each commitment from persistent state, while Short-to-Long Control integrates execution outcomes back into that state for progress assessment, recovery, and termination. With Qwen3.8-27B, LocalLSTC reaches 73.7% SR-100 on OSWorld and 68.5% on WindowsAgentArena, achieving the strongest local OSWorld result and a new state of the art on WindowsAgentArena. Ablations further support contributions from both mechanisms. Together, these results show that temporal organization of control information can reduce cross-step control failures and complement backbone scaling in locally deployed GUI agents.

Primary area:
foundation or frontier models, including LLMs
Keywords:
GUI agents, AI agents, computer-use, runtime control, training-free

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_hGUUOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

LocalLSTC addresses long-horizon GUI-agent control by maintaining an explicit persistent state for the active subgoal, accumulated evidence, and runtime feedback. Long-to-Short Planning uses this state to form a bounded short-term execution commitment; Short-to-Long Control abstracts execution outcomes and updates the state for progress assessment, recovery, and termination. The authors evaluate on OSWorld and WindowsAgentArena, report gains with local backbones, annotate cross-step control failures, and ablate L2S, S2L, and runtime components. The paper reports 73.7% SR-100 on OSWorld and 68.5% on WindowsAgentArena with Qwen3.8-27B.

Critical strengths
  1. The paper addresses practical challenges in long-horizon desktop automation. It targets stage tracking, progress assessment, action repetition, recovery, and premature termination, particularly for locally deployed models.

  2. The evaluation combines benchmark results with mechanistic analysis. It includes OSWorld and WindowsAgentArena results, fixed-backbone L2S/S2L ablations, runtime diagnostics, and partial human validation of trajectory annotations.

Critical weaknesses
  1. The paper’s novelty boundary remains unclear. Its candidate contribution is the closed-loop combination of persistent task state, bounded multi-action commitments, and evidence-driven routing/recovery. Related mechanisms appear in TSR, ATMem, HiSA, VLAA-GUI, CES, CoAct-1, and ActionEngine. Direct comparisons and ablations are needed to establish the combination’s incremental value.
  2. Evidence for comparative advantage and failure attribution is limited.
Questions for authors
  1. How does LocalLSTC’s state/commitment/routing loop differ from TSR, ATMem, and CoAct-1? What incremental gain does the existing CoAct-1 comparison show for state updates and routing?
  2. Which ablations support each claimed contribution? Since removing state-conditioned routing also disables recovery/stall handling and final verification, how can their effects be separated? Which components are inherited from HiSA?
  3. Does the 22-percentage-point reduction in ANY incidence (63.7% baseline average vs. 41.7% for LocalLSTC) persist with human-adjudicated or uncertainty-aware labels? Please quantify uncertainty in this difference.
1 comment

We thank the reviewer for the detailed comments on novelty, comparative evidence, and failure attribution. We combine the closely related weaknesses and questions below.

Weakness 1 and Questions 1–2

LocalLSTC builds on existing mechanisms for Step Abstraction, Context Refinement, and hybrid GUI/Bash/API interaction. As described in Section 3.2.3, it extends HiSA's Step Abstraction from GUI execution to a unified, subgoal-aligned evidence interface covering GUI, Bash, and software APIs, and replaces HiSA's fixed-period Context Refinement with execution-state-triggered refinement. Its main contribution is the closed-loop organization of persistent task state, bounded short-term action commitments, and execution-evidence-driven control.

Specifically, L2S uses the persistent subgoal, accumulated evidence, and runtime feedback to generate and select a bounded action commitment. After execution, S2L abstracts the outcomes, updates the persistent state, and uses state-conditioned rules to determine subsequent progress assessment, refinement, recovery, and termination. The updated state then informs the next L2S planning step. This coupling allows planning and execution feedback to interact continuously throughout a task.

Existing approaches address related aspects of this process. TSR maintains an instruction summary, subgoal progress, and a transition-aware verifier, updating task state through visual observations before and after actions. ATMem maintains structured memory of task-relevant information, including its workflow roles and execution status, and uses STR-GRPO to train the model to access this memory selectively. HiSA provides Step Abstraction and Context Refinement. CoAct-1 coordinates GUI and coding agents through subtask delegation, with execution results summarized and returned to an orchestrator. We will expand the discussion of these approaches, together with VLAA-GUI, CES, and ActionEngine, to make the relationship between existing techniques and our proposed organization more explicit.

Under the same Qwen3.5-9B planning backbone, LocalLSTC achieves 45.4% mean SR-100, compared with 34.2% for CoAct-1 and an average of 36.0% across the four baseline frameworks. This is an improvement of 11.2 percentage points over CoAct-1 and 9.4 points over the baseline average. Evidence for the proposed mechanisms comes from the grouped and component ablations below. Each reported drop is computed from the corresponding two-run mean.

MechanismSupporting ablationSR drop (points)
Complete L2Sw/o L2S8.2
Persistent subgoalw/o persistent subgoal4.8
Candidate-based planningw/o candidate proposals3.4
Bounded action commitmentw/o multi-action list3.8
Complete S2Lw/o S2L6.8
Execution evidence abstractionw/o Step Abstraction5.8
Routing-dependent controlw/o state-conditioned routing7.7
Termination verificationw/o Final Verification1.8
Stall and loop handlingw/o stall and loop handling4.1

No current ablation isolates the routing decision rule alone. State-conditioned routing determines when to invoke recovery, stall handling, Final Verification, and Context Refinement, and these operations are coupled to routing within the proposed control mechanism. Retaining them while removing the routing rule would require an alternative scheduler and would therefore evaluate a different control design. The routing ablation measures the integrated routing-dependent package, while the grouped L2S and S2L ablations provide the primary evidence for the complete mechanisms and the remaining component ablations examine their constituent operations. We will clarify these dependencies and the scope of each ablation in the revision.

Weakness 2 and Question 3

We conducted paired task-level bootstrap analyses to quantify uncertainty in the reported ANY incidence difference. For the human validation subset, two annotators independently labeled the trajectories. Since no adjudicated consensus labels were collected, we report the results for each annotator and their average separately.

Annotation sourceBaseline avg. (%)LocalLSTC (%)Difference (points)95% CI
GPT annotations63.1938.3324.86[20.07, 29.65]
Human annotator 150.6944.446.25[-8.33, 20.83]
Human annotator 252.0852.78-0.69[-16.67, 15.97]
Average of human annotations51.3948.612.78[-11.11, 17.01]

On the whole set, the GPT annotations give an ANY-incidence difference of 24.86 points. On the human-validation subset, the corresponding differences are 6.25 points for Annotator 1 and -0.69 points for Annotator 2, with an average difference of 2.78 points. The table reports paired-bootstrap 95% confidence intervals for each estimate.

Reviewer_ad7UOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

LocalLSTC combines persistent subgoals, execution evidence, and recovery feedback in a training-free control loop for local GUI agents. Results are strong across two desktop benchmarks and several backbones, but attribution to the specific control mechanisms remains incomplete.

Critical strengths
  1. Results span two operating systems, several local backbones, and different step budgets. Gains also hold with a fixed small backbone, so model scaling alone does not explain them. This supports the system's practical value and is the main reason for acceptance.
  2. Several components improve success while reducing subsequent interaction. This suggests that better control can offset its own inference cost, making the contribution more useful than a success-rate improvement alone.
Critical weaknesses
  1. The added control organization is not sufficiently isolated. HiSA already provides execution abstraction and context refinement [1]. Online feedback from explicit subgoal status also has precedent [2] LocalLSTC's integration differs, but the routing ablation in Table 11 also disables recovery, loop handling, and Final Verification, and makes refinement periodic. This supports the package, but does not identify the routing design's own contribution. The resulting uncertainty about what the new organization adds beyond existing components is my main reason for leaning weak reject.
  2. The smaller component gains are difficult to assess for robustness. Section 4.1 specifies single runs outside the main benchmark unless stated otherwise. Table 3 does not identify the runs behind each row or report paired uncertainty. This weakens the fine-grained component conclusions, without invalidating the larger full-system gains.

[1] https://aclanthology.org/2026.findings-acl.581.pdf

[2] https://arxiv.org/pdf/2603.19685

Questions for authors
  1. If the intended claim extends to state-conditioned routing itself, which existing results isolate that contribution? Clarifying this boundary would affect my assessment of the added mechanism.
  2. Which runs and how many repetitions underlie Full and each ablation in Table 3? Are paired task outcomes or uncertainty analyses already available for inspection? This would affect my assessment of the smaller gains.
1 comment

We thank the reviewer for recognizing the practical value of the full-system results. We address the two weaknesses together with their corresponding questions.

Weakness 1 and Question 1

No current ablation isolates the routing decision rule alone. State-conditioned routing in LocalLSTC maps execution feedback to downstream operations including recovery, stall handling, Final Verification, and Context Refinement, so the routing decisions and these operations form a coupled mechanism. Retaining the downstream operations while removing their routing rule would require introducing an alternative scheduler. We therefore scope the current claim to the integrated routing-dependent control package. Its ablation measures the routing decisions together with the operations they invoke.

As described in Section 3.2.3, LocalLSTC extends HiSA's Step Abstraction from GUI execution to a unified, subgoal-aligned evidence interface covering GUI, Bash, and software APIs, and replaces HiSA's fixed-period Context Refinement with execution-state-triggered refinement. These extensions form part of a closed loop connecting persistent task state, bounded commitments, execution evidence, and state-conditioned post-execution control. Grouped L2S/S2L ablations evaluate the complete mechanisms, while persistent-subgoal, candidate-proposal, and multi-action-list ablations examine individual L2S components. We will define this contribution boundary and describe routing as coupled with recovery, stall handling, Final Verification, and refinement.

Weakness 2 and Question 2

We have completed a second independent run for every grouped and component ablation. Full and all ablations therefore now have two runs. For the paired analysis, we average each task's two Full outcomes and its two outcomes under the corresponding ablation, then bootstrap the paired task-level differences.

ConfigurationMean SR-100 ±\pm SDTokens (k)Prompt (k)Completion (k)StepsTime (s)Paired-bootstrap 95% CI
Full45.4 ±\pm 2.1253.0238.614.325.8399.5--
w/o L2S37.2 ±\pm 0.1169.1159.010.125.6313.7[5.79, 16.63]
w/o S2L38.6 ±\pm 2.1197.9187.210.727.0381.6[3.82, 17.51]
w/o persistent subgoal40.6 ±\pm 4.0244.7230.114.625.0470.9[1.26, 13.59]
w/o candidate proposals42.0 ±\pm 1.5240.2229.111.126.9386.3[-1.05, 13.11]
w/o Step Abstraction39.6 ±\pm 0.5241.2228.612.727.8415.7[2.39, 14.66]
w/o multi-action list41.6 ±\pm 0.4225.0211.114.026.1407.8[1.21, 11.84]
w/o state-conditioned routing37.7 ±\pm 2.8208.3196.012.323.0389.7[3.65, 17.05]
w/o Final Verification43.6 ±\pm 2.1188.1177.910.319.8344.3[-2.04, 10.96]
w/o stall and loop handling41.3 ±\pm 2.1289.2273.715.528.3498.3[0.92, 12.58]

The two-run means remain below Full for every ablation. Removing complete L2S and S2L lowers mean SR-100 by 8.2 and 6.8 points, respectively. Among the individual components, the mean reductions are 4.8 points for the persistent subgoal, 3.4 for candidate proposals, 5.8 for Step Abstraction, 3.8 for the multi-action list, 7.7 for state-conditioned routing, 1.8 for Final Verification, and 4.1 for stall and loop handling.

We will include these updated uncertainty estimates in the revision.

Reviewer_gNtcOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes LocalLSTC, a training-free agent framework for desktop GUI tasks with small local models. It keeps an explicit subgoal and evidence state across steps, and adds outcome checking, stall handling and a verifier before termination. The authors report state-of-the-art local results on OSWorld and WindowsAgentArena. The problem is real, but the gains are confounded by a stronger backbone, extra tools, benchmark-specific skills and changes to the evaluation harness, so the central claim is not supported.

Critical strengths
  • Real and practical problem. Local GUI agents are a legitimate need (privacy, cost, latency). The authors re-run Agent S3, GTA1, CoAct-1 and HiSA under Qwen3.5-9B with native settings, which tells us more than another API-only leaderboard row.
  • Current open backbones, two OSes. Six models across Qwen3.5/3.6/3.8 and Gemma-4, OSWorld-Verified plus WindowsAgentArena.
  • Ablations report cost, not only SR.
Critical weaknesses
  • Runtime uses hidden evaluator config and post-hoc fixes, undisclosed. Before Final Verification, main.py reads task_config["config"] and the evaluator's postconfig window name to reopen office files after Bash edits. On Windows it force-saves LibreOffice documents via API right before scoring. Neither appears in the paper; no baseline runner has them.

  • Benchmark environment and evaluator were modified. update_server_main.sh installs openpyxl, pandas, python-docx, odfpy, xlsxwriter and others into the OSWorld guest, which directly helps the Bash channel on office tasks. WAA's Clock timer evaluator was rewritten and app-launch setup changed. The copied numbers in Table 1 came from unmodified benchmarks.

  • Skills are an OSWorld answer key. Against the official task configs, 22 of OSWorld's 26 infeasible tasks have a dedicated skill rule (Xenothian language, Python4, $sourceDir, GIMP CMYK/SVG/"Blue" theme, Bluetooth, battery), worth up to ~6 SR points. Other rules encode evaluator conventions (Sheet2, GUI pivot tables).

  • Headline SOTA is mostly the backbone. 73.7%/68.5% come from Qwen3.8-27B, yet no baseline framework is run on Qwen3.8-27B. Table 1's "local" rows are UI-TARS, OpenCUA, Qwen2.5-VL generation models. Going 9B→27B adds 26 points; the method adds 12.4 points over a baseline average at 9B.

  • Loses at matched scale. LocalLSTC-9B, plus a separate 7B grounder, gets 44.5 SR-50 and 34.8 SR-15. EvoCUA-8B reaches 46.1 SR-50; AutoGLM-OS-9B reaches 47.3 / 46.3 as a single model, built on the same ComputerRL API stack LocalLSTC borrows. The "training-free alternative" pitch does not hold at equal size.

  • The key ablation is missing: action channels. Table 2 shows Bash is 54% of actions in OS and APIs 13% in Calc. We need GUI-only LocalLSTC, and a plain ReAct planner with the same grounder, Bash, API and verifier. "w/o L2S" changes five things at once.

  • Annotation results contradict the mechanism story. Table 8: LocalLSTC has worse STAGE (18.2%), LOOP (28.5%) and RECOVER (33.6%) than Agent S3 (10.8/11.7/17.3), despite failing fewer tasks. Almost all of the ANY drop comes from STOP, i.e. Final Verification. Persistent subgoals do not visibly fix stage maintenance.

  • Component ablations sit inside noise. All ablations are single runs, while full-model SD is 2.7 points. Drops of 2.5 (stall/loop), 2.9 (Final Verification) and 3.7 (routing) are not distinguishable from run variance.

Questions for authors

See Weaknesses. For a training-free framework paper, this is test-set leakage as a possible integrity concern.

  • How were the skills written? Explain why 22 infeasible tasks have task-specific rules (e.g., “Xenothian”, Python4, $sourceDir). Also explain why two rules declare infeasible tasks that are feasible in the current OSWorld-Verified (a1af9f1c, 5ac2891a).

  • Report OSWorld and WAA results with all app skills removed, without reading task_config or the evaluator’s postconfig, without the pre-evaluation force-save, and on unmodified benchmark images.

2 comments

We thank the reviewer for the detailed assessment. We respond to each weakness separately and incorporate the related questions where applicable. All results reported in the responses below are from reruns conducted in accordance with the requirements in Question 2.

Weakness 1 and Question 2

The Office-file reopening and save operations were introduced to synchronize files modified through Bash with the state displayed in the application. To address the concern, we have removed access to task-specific configuration and evaluator postconfig metadata from the agent runtime and disabled the pre-evaluation force-save operation. We also reran OSWorld using the official benchmark environment and evaluator, with all application skills and the additional guest packages discussed in Weakness 2 removed. LocalLSTC with Qwen3.8-27B achieves 71.5% SR-100, 70.5% SR-50, and 60.1% SR-15. The mean SR-100 result is 2.2 percentage points below the previously reported 73.7%.

Weakness 2 and Question 2

During development, the agent repeatedly installed Python packages such as openpyxl, pandas, and python-docx within task trajectories when processing Office documents. These repeated installations introduced substantial runtime overhead, so we pre-installed the packages to make debugging and experimentation more efficient. For the updated evaluation, we removed the additional guest packages and restored the official OSWorld image and evaluator. The updated OSWorld results are reported under Weakness 1.

The WAA task configuration notes that Clock is updated on its first launch. Following the update, Clock displayed timer durations in UI Automation text variants that the original evaluator did not recognize because it expected a complete hours-minutes-seconds string. We extended the timer getter to recognize these formats and retained the requirement that the timer be running. The task instruction and target timer state remained unchanged. The other server changes addressed runtime stability. We have now rerun WAA using the official benchmark environment and evaluator, with the same runtime restrictions described under Weakness 1, including removal of application skills, access to evaluator metadata, and the pre-evaluation force-save operation. Under these conditions, LocalLSTC with Qwen3.8-27B achieves 65.6% SR-100, 64.3% SR-50, and 51.3% SR-15. The SR-100 result is 2.9 percentage points below the previously reported 68.5% while retaining the state-of-the-art advantage on WAA.

Weakness 3 and Questions 1–2

The application skills were developed from HiSA's Pattern Induction mechanism, which extracts lessons from completed task trajectories and reuses them when solving similar tasks. HiSA stores these lessons in a Qdrant vector database for subsequent retrieval. After running HiSA on the benchmark, we exported the accumulated experience entries from its Qdrant database to patterns/*.json for inspection. The Xenothian language, Python4, and $sourceDir examples highlighted by the reviewer were actually present in the HiSA patterns. These task-specific observations were generalized, and the corresponding specific rules were not directly included in LocalLSTC's application skills. We did not select patterns based on the OSWorld infeasible-task list or manually match rules to individual task configurations.

Regarding the infeasible tasks, our original experiments used the OSWorld-Verified release dated July 28, 2025. The benchmark has since been updated, and the current evaluation includes 25 infeasible tasks. In the current repository, task a1af9f1c is absent from the infeasible-task index, although its configuration still uses the infeasible evaluator. Task 5ac2891a is now treated as feasible.

To further assess the impact of application skills on infeasible tasks, we conducted the additional evaluation described under Weakness 1 with all application skills removed. In the updated run, LocalLSTC correctly handles 20/25 infeasible tasks, compared with 22/25 in the original evaluation, a decrease of only two tasks. This suggests that LocalLSTC already possesses some understanding of application capabilities and limitations, enabling it to identify many infeasible tasks independently of the application skills.

Weakness 4

Table 1 serves two complementary purposes. The Qwen3.8-27B results report the system-level performance achieved by LocalLSTC with a stronger local planner. To measure the contribution of control organization under a fixed planning backbone, we also evaluated Agent S3, GTA1, CoAct-1, HiSA, and LocalLSTC using Qwen3.5-9B while keeping the evaluation setting aligned as closely as the frameworks' native implementations allow. Under this setting, LocalLSTC achieves 45.4% SR-100, compared with an average of 36.0% across the four baseline frameworks, giving a 9.4-point difference. We will make these two purposes clearer in the paper.

Weakness 5

EvoCUA-8B and AutoGLM-OS-9B are GUI-specialized agent policies whose training involves computer-use interaction data and learning procedures. LocalLSTC uses an off-the-shelf Qwen3.5-9B planner with its model parameters unchanged by GUI-specific supervised fine-tuning or reinforcement learning. These systems provide informative comparison points across different training and execution designs. Their differences extend beyond parameter count. We will clarify the scope of the comparison with trained GUI-agent systems.

Weakness 6

GUI, Bash, and software-API channels are inherited execution capabilities, with precedents in CoAct-1 and ComputerRL. Our contribution focuses on the control mechanisms built upon these capabilities.

To isolate the effect of the action channels and provide the ReAct comparison, we evaluated GUI-only LocalLSTC and a PromptAgent ReAct-style baseline using the same Qwen3.5-9B backbone and grounder. The results are shown below.

MethodSR-100SR-50SR-15Tokens (k)Prompt (k)Completion (k)StepsTime (s)
Full LocalLSTC45.443.034.3253.0238.614.325.8399.5
GUI-only LocalLSTC43.538.919.0251.3233.817.429.1519.0
ReAct-style baseline35.930.425.8345.9336.39.541.5454.0

GUI-only LocalLSTC remains strong at 43.5% SR-100, only 1.9 points below Full, showing that the framework retains most of its long-horizon success without Bash or software APIs. However, it requires more planner steps and longer runtime than Full (29.1 vs. 25.8 steps and 519.0 vs. 399.5 seconds), while the ReAct-style baseline requires even more planner steps (41.5). Full outperforms ReAct by 9.5 SR-100 points while using fewer tokens, steps, and runtime.

Final Verification is part of our proposed framework, and its contribution is evaluated separately through the w/o Final Verification ablation. This ablation achieves 43.6% mean SR-100, which is 1.8 points below Full. The w/o-L2S ablation retains the same backbone, grounder, action channels, and S2L components. It achieves 37.2% SR-100, which is 8.2 points below Full. These results support the contribution of our control mechanisms under a shared execution stack.

Weakness 7: Failure annotations and mechanism interpretation

The annotation results show that the largest reduction in control-failure incidence comes from termination-related errors. Agent S3 has an ANY incidence of 67.8% and a STOP incidence of 64.7%, compared with 38.3% and 25.6% for LocalLSTC. When STOP is excluded, the union of STAGE, ASSESS, LOOP, and RECOVER is 38.6% for Agent S3 and 37.8% for LocalLSTC. These annotations therefore provide direct descriptive evidence for improved termination control, while the non-STOP union is similar between the two systems. Evidence for the persistent-subgoal mechanism comes from the fixed-backbone component ablation. The w/o persistent-subgoal result is 40.6% mean SR-100, which is 4.8 points below Full. We will distinguish the conclusions supported by the trajectory annotations from those supported by the controlled ablations.

Weakness 8: Uncertainty in component ablations

We completed a second independent run for every component ablation. We also updated the paired task-level bootstrap analysis by averaging the two Full outcomes and the two ablation outcomes for each task before resampling the paired differences.

ConfigurationMean SR-100 ±\pm SDDrop from FullPaired-bootstrap 95% CI
Full45.4 ±\pm 2.1----
w/o state-conditioned routing37.7 ±\pm 2.87.7[3.65, 17.05]
w/o Final Verification43.6 ±\pm 2.11.8[-2.04, 10.96]
w/o stall and loop handling41.3 ±\pm 2.14.1[0.92, 12.58]

We will include these repeated-run results and uncertainty estimates in the revision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.