- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper defines fully test-time adaptation for GUI agents: no ground truth, one attempt per task, no pre-deployment training, and proposes SOLO uses auxiliary models to extract learning signal from each episode, updating a LoRA adapter via top-K self-distillation anchored by a sliding window. On three benchmarks with two agents, SOLO improves over the frozen baseline by 3–6 points and exceeds two memory baselines on the web streams.
Critical strengths
-
The paper defines fully test-time adaptation for GUI agents via three deployment-realistic constraints: no ground truth, one attempt per task, and no pre-deployment training phase. This setting is both practically grounded and differentiated from existing paradigms, filling a gap that prior work on agent improvement has not explicitly addressed.
-
Unlike prior in-setting methods that store learned workflows or experiences in the prompt, SOLO distills them into a lightweight LoRA adapter. This weight-space approach offers better generalization.
-
Across two agents and three benchmarks, SOLO improves over the frozen baseline by 3–6 points on every stream, and surpasses both in-setting memory baselines on the web.
Critical weaknesses
-
SOLO continually updates a LoRA adapter via a sliding window, how to ensure that catastrophic forgetting does not occur during the change of tasks. For instance, the adapter's previously learned episodes are not overwritten by subsequent tasks?
-
Table 4 reveals the judge achieves a precision of only 0.51 on VisualWebArena and 0.66 on WebArena, meaning that half of the episodes are failures. These mislabeled episodes directly enter the sliding window and drive adapter updates, injecting a substantial amount of noisy gradient signal. The paper does not address how to avoid or filter this noisy data.
-
The paper omits any discussion of deployment-time computational costs. For each task, SOLO requires call to a closed-source model for judging, and up to two additional calls (proposer and verifier) when an episode is judged as a failure. On top of this inference overhead, each admission triggers a gradient update over the sliding window. These costs are not quantified, nor are they compared against the baselines, which are training-free and operate entirely in the prompt.
-
In ablation auxiliary model dependence study are restricted to Qwen3-VL-8B on the two web streams, omitting MobileWorld Yet the main results in Tab. 2 show that SOLO's gains on MobileWorld with UI-TARS-7B are marginal. Without ablation evidence on MobileWorld, it remains unclear whether each component contributes comparably or whether the method's behavior differs qualitatively in the mobile domain.
-
Table 4 shows that gpt-5 yields a lower success rate than gpt-5-mini. If higher-precision filtering degrades performance, what exactly is the judge selecting for that benefits the adapter? The mechanism by which judge precision interacts with the quality of the learning signal is not analyzed.
-
Figure 2 shows that the improvement primarily in rounds 2 and 3, when the same instances reappear and the adapter can exploit previously learned patterns. If deployment scenarios where task recurrence is low or absent, the method's practical benefit may be diminished.
Questions for authors
-
Please provide additonal experiments to address the proposed concerns.
-
For the catastrophic forgetting concern, can the author consider incorporating some previously validated high-quality episodes during the adapter update process? I believe this could further enhance the performance of SOLO.
If my concerns are addressed, I will consider improve my score.
2 comments
Thank you for the follow-up and for updating your score. We are glad that our rebuttal and revision addressed your concerns, and we appreciate the time and care you put into reviewing our paper. Your feedback has helped us improve it.
Thanks for the detailed response. The rebuttal has addressed my main concerns. I will raise my score.