acceptodds
Under review as a conference paper at ICLR 2027

GUIHub: Personalizing GUI Agents via Capability-Driven LoRA Composition

Abstract

Graphical user interface (GUI) agents must adapt to task distributions that vary across users. Independently trained low-rank adaptation (LoRA) adapters offer specialized capabilities, but reusing them for a target user with only a few examples remains challenging. We present GUIHub, an asynchronous framework for sharing and composing LoRA adapters. Our key idea is to connect user demand with measured adapter capabilities through shared GUI tasks. GUIHub builds a shared capability space from adapter behavior on these tasks. Semantic relevance between local support examples and the shared tasks specifies user demand in this space, guiding adapter selection and weighting without local action labels. The resulting composition is reused across requests, while new adapters join the same space without retraining existing adapters. We evaluate personalization, generalization, mixed-task composition, and Hub expansion on mobile GUI trajectories from FedMABench. GUIHub improves offline Step Success Rate by up to 5.61 pp over the strongest baseline on in-hub and held-out clients. With prebuilt Hub assets, GUIHub achieves a 2.08× speedup over the fastest baseline in recorded setup-plus-execution time on Qwen3.5-4B. Expanding the Hub from 8 to 32 adapters improves Step Success Rate by 6.16–7.61 pp across all three backbones on the same 32 target clients.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
GUI Agent, Low-rank Adaptation

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_3AumOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

GUIHub studies post-hoc reuse of independently trained GUI LoRA adapters. Adapters are profiled on a labeled global support set (GSS) of 768 episodes, and their centered behavioral responses are projected into a frozen eight-dimensional representation. An unlabeled local support set (LSS) specifies target demand: relevance distributions over the 64 nearest GSS episodes are computed separately for each support example and then averaged. Demand–capability scores determine the selected adapters and their weights. The default composition contains two adapters, using an available own/source adapter as an anchor or selecting the two highest-scoring candidates when no anchor is available. Their weighted output contributions are scaled by 1.5 on Qwen and 1.0 on Gemma, and the composition is reused across requests.

Evaluation uses teacher-forced next-action prediction on AndroidControl-derived mobile trajectories, with 24 in-hub and eight held-out synthetic semantic clients. IID/OOD and expansion experiments cover three backbones; mixed-client experiments cover two Qwen backbones; detailed ablations, diagnostics, and timing use Qwen3.5-4B. A single-client route uses ten selected LSS episodes, while support preparation accesses 40 distinct local instruction/screenshot inputs (Appendix A.2).

Critical strengths
  1. The routing diagnostics provide direct evidence about candidate selection. All 24 adapters are evaluated on each of 32 targets using common singleton execution settings, including scale 1.5. Behavioral scores achieve a mean within-target Spearman correlation of 0.513, compared with 0.103 for the specified semantic scorer. Selected-set regret is 1.09 pp [0.60, 1.64] versus 3.07 pp [1.93, 4.33] (Table 15). These measurements do not use composition weights, so they support selection quality independently of how the selected adapters are combined. Figure 5(b) shows that regret reductions are uneven across targets, with many near zero and a smaller group contributing larger reductions. This qualifies the average selection benefit; it does not measure target-level gains in composed-model Step SR.

  2. The framework addresses a useful reuse setting and has broad, well-documented offline evidence. GUIHub has the highest reported IID/OOD Step SR among the evaluated methods on all three backbones (Table 1). Paired-episode intervals exclude zero for all three IID comparisons with LoraRetriever and both Qwen OOD comparisons; Gemma OOD remains uncertain (Table 16). The expansion study evaluates seven pool sizes from 8 to 32 on the same targets, and Table 9 also tracks a fixed initially covered cohort. The paper clearly documents information access, prediction reuse, non-monotonic individual outcomes, and timing exclusions. Figures 1–2 make the profiling/adaptation/serving distinction clear, while Figures 6–7 describe heterogeneity in the constructed client distributions.

Critical weaknesses
  1. The advantage over directly using one's own adapter is not yet cleanly attributed, and the measured increment from combining two adapters is modest. Table 12 reports 64.98 IID Step SR for GUIHub versus 46.90 for Own-LoRA, but Appendix D.1 does not explicitly specify Own-LoRA's additional scale or its correspondence to GUIHub's execution path. The 63.43 IID score for behavioral top-1 in Table 14 changes adapter selection as well, so its 16.53 pp difference from Own-LoRA cannot estimate the effect of gamma. Figure 5(a), which pools different target–adapter pairs, does not resolve this ambiguity. The main baselines also use different composition operators, including factor fusion, so an explicit gamma is not sufficient to compare effective update magnitudes across methods.

Table 14 does provide a useful controlled adapter-count comparison. Within the anchor-free group at the same scale, moving from one to two adapters changes IID/OOD/mixed Step SR by approximately +0.62/−0.24/+0.35 pp. Thus, one selected adapter already achieves most of the accuracy in these conditions. These comparisons do not isolate the own-anchor/helper combination used in the main IID and mixed experiments, and paired intervals for the adapter-count differences are not reported. I would prioritize a common-path Own-LoRA control and uncertainty estimates for the existing singleton-versus-pair results. Cross-client selection is itself a useful form of adapter sharing; the issue is how much additional benefit can be attributed to simultaneous composition and how the larger Own-LoRA gap should be interpreted.

  1. The semantic ablation identifies an advantage of the implemented behavioral scorer, but does not fully separate the sources of the composed-model gain. Equation (11) uniformly averages semantic similarities over the LSS and an adapter source client's probes. GUIHub instead aggregates measured behavioral responses using target-dependent relevance over the shared GSS. Downstream selection, routing temperature, composition scale, and evaluation settings are fixed. The 1.02/5.18/3.86 pp IID/OOD/mixed differences therefore compare two complete scoring procedures, including their aggregation and score distributions.

This limitation should not erase the positive selection evidence: Table 15's rank-based diagnostics cannot be explained by a change in softmax weight concentration alone. Calibration could nevertheless affect the magnitude of the composed-model accuracy difference. The M = 64 versus 768 experiment makes truncation alone an unlikely explanation in the tested behavioral configuration; it does not test uniform versus soft relevance or all interactions with a different scorer. Reporting within-target score dispersion and selected-weight entropy, followed by an equal-weight composition check on the existing selected sets or a development-set calibration check, would provide a focused way to investigate this remaining ambiguity.

Additional Concerns

(a) The projection is an explicitly disclosed compression trade-off. Direct matching exceeds the projected variant by 0.21/0.76/0.26 pp in Table 13, without paired intervals for these small differences. Appendix D.2 already explains descriptor compression, shared storage, and the absence of measured indexing speedups. These results support a compact representation with similar observed accuracy; they do not show an accuracy benefit from the routing projection. Actual indexing or memory measurements at larger pool sizes would strengthen the deployment case. Direct matching retains the SVD used in GSS construction.

(b) Efficiency is established for the recorded workload with prebuilt assets. Figure 4 and Tables 10–11 show that GUIHub's advantage over LoRAHub comes from setup, while its recorded online mean is slightly higher (4.354 versus 4.102 s/action). Initial GSS selection requires 24 reference adapters to evaluate 1,200 candidate episodes; a new adapter is profiled on 768 selected probes. The costs of these offline operations are excluded, so their share of total cost and the amortization point remain unknown. Table 11 already reports target setup, including constructing and applying compositions. The useful missing evidence is offline construction/admission cost and its amortization over stated workloads, with the unpaired timing conditions kept explicit.

(c) LoGo's mixed-client evaluation does not replay the mixed request sequence. Appendix B.1 discloses reuse of single-client predictions, whereas Appendix A.4 describes routing state carried across episodes. This does not establish performance under a newly ordered mixed workload; the direction and size of any difference are unknown. A replay would strengthen the LoGo comparison. This concern does not by itself invalidate the comparison with LoraRetriever, whose prediction reuse is justified under identical inputs and settings.

(d) Expansion and transfer have specific experimental limits. The capability basis is fitted using 24 IID reference adapters before serving-pool expansion begins at eight, and admission is ordered by ascending mean GSS response. Figures 3 and 8 therefore show performance under a pre-established representation and a particular quality-ordered schedule. The curves are not uniformly monotonic, and Figure 8 shows both Qwen OOD groups starting below LoraRetriever at Hub8. These observations qualify subgroup performance without contradicting Section 4.5's pooled All32 Step SR claim. Table 9 also shows positive gains for targets whose own adapters are present throughout, supporting the paper's narrower claim that expansion gains are not solely due to own-adapter arrival. The evaluation still uses synthetic clients, a common training recipe, and teacher forcing; its intervals do not cover training variability or closed-loop recovery.

Questions for authors
  1. Please specify Own-LoRA's scale and execution path and how gamma was selected. Can Own-LoRA be evaluated in the same execution path as Table 14 at gamma = 1.5, together with a GUIHub gamma = 1.0 control, stating which routes and weights are held fixed? Can paired IID/OOD intervals be reported for the existing one-versus-two-adapter comparison? These would clarify the large Own-LoRA gap and the incremental benefit of composition without requiring a full new factorial study.

  2. Can you report within-target score dispersion and selected-weight entropy for the two scorers, and test either equal-weight composition on their existing selected sets or a comparable calibration procedure using development data? The goal is to distinguish the ranking advantage already supported by Table 15 from any additional effect of score scale on composition weights. A more aggregation-aligned semantic control would be helpful if its adapter–probe scoring rule is specified precisely.

The following would strengthen secondary comparisons and deployment interpretation.

  1. Section 2 identifies LoRAuter as combining validation performance and task embeddings for per-query adapter routing. What prevents a comparison under compatible information and execution budgets, or can such a comparison be supplied? Explaining adaptation barriers would clarify the scope; an empirical comparison would provide additional performance evidence.

  2. Can the excluded initial profiling and new-adapter admission costs be measured or estimated with a clearly stated method, and amortized over explicit numbers of targets and requests? Please distinguish one-time construction from per-adapter admission and the target setup already reported in Table 11.

  3. Can LoGo's mixed workloads be replayed with explicit routing-state initialization and episode order, or can a smaller representative replay establish whether prediction reuse materially changes the comparison?

Suggestions For Future Work

  1. Evaluate closed-loop execution to measure error accumulation and recovery with a fixed target composition.
  2. Test real user partitions and adapters with heterogeneous quality, training budgets, and compatible ranks.
  3. Study alternative arrival orders and a representation initialized from only the initial pool, together with admission or eviction rules addressing the individual regressions in Table 9.
7 comments
AuthorsOpenReview ↗

Response to Reviewer 3Aum (1/5)

(W1 & Q1) "Table 12 reports 64.98 IID Step SR for GUIHub versus 46.90 for Own-LoRA, but Appendix D.1 does not explicitly specify Own-LoRA's additional scale or its correspondence to GUIHub's execution path. … Please specify Own-LoRA's scale and execution path and how gamma was selected. Can Own-LoRA be evaluated in the same execution path as Table 14 at gamma = 1.5, together with a GUIHub gamma = 1.0 control …?"

Thank you for the careful and constructive review. The focused controls you proposed were feasible, and we have run all of them. Table, figure, and appendix numbers refer to the revised PDF; where a number changed, we give both.

  • Own-LoRA's scale and how γ was selected. In the submission, Own-LoRA is the target's own adapter loaded as trained, with its native LoRA scaling and no additional scale, which is how a user would normally deploy an adapter of their own. The composition scale γ is part of GUIHub's composition rule (Section 3.4) and was selected on development data (γ = 1.5 for Qwen). We thank you for pointing out that Appendix D.1 did not state this. As you note, the gap in the submission's Table 12 therefore reflects both adapter selection and output scale, so we ran the control you requested to separate the two. Section 4.2 now refers to this controlled comparison instead of reporting the gap, and Figure 10 in Appendix D.1 replaces the submission's Table 12.

  • Controlled comparison across γ. For this control we apply the same γ to Own-LoRA. Own-LoRA runs in the same inference pipeline as GUIHub’s single-adapter execution, with weight one; GUIHub keeps the selected adapters and relative weights of the main experiments. Only γ varies; test episodes, decoder, teacher-forced history, and evaluator are shared.

    γ\gammaIID: GUIHub / OwnOOD: GUIHub / Own†Mixed: GUIHub / Own‡
    1.0048.91 / 46.8454.45 / 48.6449.66 / 47.98
    1.2559.31 / 58.7862.86 / 59.7760.13 / 57.64
    1.5064.98 / 64.4167.00 / 66.0565.62 / 63.44
    1.7566.10 / 63.0267.19 / 64.8166.75 / 63.09
    2.0061.99 / 58.7863.53 / 61.1063.72 / 58.88

    † For held-out targets, Own-LoRA is trained on the target’s own labeled data and is not in GUIHub’s pool. ‡ For mixed targets, Own-LoRA is one own adapter of the target's source clients, chosen once by the highest behavioral score computed from the target's unlabeled LSS (Appendix D.1).

  • GUIHub's point estimate is higher in all 15 cells. At γ = 1.5, the paired 95% intervals are +0.57 pp [−0.41, +1.52] on IID and +0.95 pp [−0.95, +2.85] on OOD; at γ = 1 and γ = 2, the IID and OOD intervals exclude zero. The sweep also shows that γ = 1.5 is not GUIHub's test optimum (66.10% IID at γ = 1.75), consistent with γ having been fixed on development data.

  • What this shows about GUIHub's value. A growing Hub always contains users who have not trained an adapter of their own, and users whose tasks span several clients. This is where GUIHub's value lies:

    1. Held-out users: using only adapters already in the Hub, GUIHub matches or exceeds an adapter trained on the user's own data. With a few unlabeled examples and no adapter or label from the target, GUIHub's Step SR is higher at every scale, by 0.95–5.80 pp; the intervals exclude zero at γ = 1 (+5.80 pp [+4.06, +7.58]) and γ = 2 (+2.43 pp [+0.46, +4.41]). Users without an adapter thus get the accuracy of one without training it.
    2. Cross-client users: GUIHub exceeds a single own adapter. When a workload spans several clients, no single own adapter covers it. GUIHub exceeds the selected single own adapter by 2.18 pp at γ = 1.5 and by 1.68–4.84 pp across scales.
    3. In-hub users: GUIHub is above the own adapter at every scale. Its Step SR is higher by 0.53–3.21 pp, with intervals excluding zero at γ = 1 (+2.07 pp [+1.17, +2.99]) and γ = 2 (+3.21 pp [+2.11, +4.32]).
AuthorsOpenReview ↗

Response to Reviewer 3Aum (2/5)

(W1 & Q1, continued) "Within the anchor-free group at the same scale, moving from one to two adapters changes IID/OOD/mixed Step SR by approximately +0.62/−0.24/+0.35 pp. … These comparisons do not isolate the own-anchor/helper combination used in the main IID and mixed experiments, and paired intervals for the adapter-count differences are not reported."

  • Paired intervals for one versus two adapters. Appendix D.3 now reports them. Under anchor-free selection at the same scale, two adapters minus one gives +0.62 pp [−0.19, +1.46] on IID, −0.24 pp [−1.65, +1.13] on OOD, and +0.35 pp [−0.66, +1.29] on mixed targets. We agree that one behaviorally selected adapter already attains most of the accuracy, and we do not claim a stable additional gain from simultaneous composition.
  • The own-anchor/helper combination. GUIHub's in-hub composition is the own adapter plus one helper, so the controlled comparison in our previous comment isolates exactly this combination: adding the helper raises IID Step SR at every scale, by up to +3.21 pp at γ = 2 ([+2.11, +4.32]) and by +2.07 pp at γ = 1 ([+1.17, +2.99]).
  • Where the benefit comes from. As you note, cross-client selection is itself a useful form of adapter sharing, and our evidence indicates that this is where GUIHub's benefit comes from: without training an adapter, a held-out user obtains Step SR 0.95–5.80 pp above that of a supervised own adapter across scales, and a cross-client user obtains 1.68–4.84 pp above a single own adapter.

(W2) "Equation (11) uniformly averages semantic similarities over the LSS and an adapter source client's probes. GUIHub instead aggregates measured behavioral responses using target-dependent relevance over the shared GSS. … The 1.02/5.18/3.86 pp IID/OOD/mixed differences therefore compare two complete scoring procedures, including their aggregation and score distributions."

  • Aggregation-aligned control. To isolate the evidence, we added the control you suggested, with its adapter–probe rule stated exactly (Appendix D.4, Table 16). It keeps GUIHub's relevance vector ρu\rho_u and the same aggregation over the shared GSS, and replaces each measured response RijR_{ij} with an indicator of whether probe jj comes from adapter ii's source client:
  • Identical aggregation. The behavioral side is the Direct score ∑jρu(j)Rij\sum_j \rho_u(j) R_{ij}, i.e., without the low-rank projection. The two scorers then differ only in whether an adapter is credited for its measured behavior or for the provenance of the probes. On the 768 singleton evaluations:

    TargetsEvidenceSpearman with singleton Step SRSelected-set regret (pp)
    All 32Measured behavior0.532 [0.453, 0.607]1.10 [0.64, 1.61]
    All 32Probe provenance0.150 [0.029, 0.273]2.98 [1.73, 4.38]
    OOD 8Measured behavior0.554 [0.388, 0.682]1.34 [0.46, 2.36]
    OOD 8Probe provenance0.008 [−0.178, 0.239]6.08 [2.85, 9.38]
  • Result. Across all 32 targets, the differences are −0.382 [−0.496, −0.270] in Spearman correlation and +1.88 pp [0.72, 3.24] in regret, and the selected sets differ for 30 of 32 targets. (For reference, the projected behavioral score in Table 15 has Spearman 0.513.) For the eight held-out targets, probe provenance is essentially uncorrelated with actual accuracy (Spearman 0.008), whereas measured behavior keeps 0.554 and reduces regret from 6.08 to 1.34 pp. With the aggregation held fixed, measured behavior is what drives selection quality.

AuthorsOpenReview ↗

Response to Reviewer 3Aum (3/5)

(W2 & Q2) "Can you report within-target score dispersion and selected-weight entropy for the two scorers, and test either equal-weight composition on their existing selected sets or a comparable calibration procedure using development data?"

We report the statistics you requested and ran the equal-weight check (Appendix D.4, Table 17).

  • Dispersion and entropy. Within-target score dispersion is similar for the behavioral and semantic scorers (standard deviation 0.033 vs. 0.039; IQR 0.046 vs. 0.042). Both give near-uniform weights over the selected pair (normalized entropy 0.95 vs. 0.90; mean maximum weight 0.61 vs. 0.65); the semantic weights are slightly more concentrated.

  • Equal-weight composition. Each scorer keeps its selected pair, and the final weights are fixed to 0.5/0.5 (Step SR, %):

    Selected setWeightsIIDOOD
    BehavioralSoftmax (GUIHub)64.9867.00
    BehavioralEqual64.8567.10
    SemanticSoftmax63.9661.82
    SemanticEqual63.3262.10
  • Result. With equal weights, the behavioral selected sets remain ahead by +1.53 pp [+0.68, +2.38] on IID and +4.99 pp [+3.33, +6.66] on OOD. Softmax weighting itself changes behavioral Step SR by +0.13 pp [−0.39, +0.65] and −0.10 pp [−0.71, +0.51]. For the semantic scorer it gives +0.65 pp [+0.10, +1.20] on IID and −0.29 pp [−1.06, +0.49] on OOD; the difference between the two scorers' weighting effects has intervals that include zero (−0.52 pp [−1.29, +0.24] on IID; +0.19 pp [−0.79, +1.14] on OOD).

  • Conclusion. The behavioral advantage remains when both scorers use the same weights, so it comes from selecting better adapters, not from score scale. Consistent with your reading of Table 15, we claim a ranking advantage and do not claim a stable additional benefit from softmax weighting.

(a) "These results support a compact representation with similar observed accuracy; they do not show an accuracy benefit from the routing projection."

We agree. Paired 95% intervals for Direct matching minus the projected score are +0.21 pp [−0.40, +0.81] on IID, +0.76 pp [−0.24, +1.78] on OOD, and +0.26 pp [−0.27, +0.84] on Mixed (All); all include zero (Appendix D.2).

  • A compression choice. The projection is a compression choice rather than an accuracy component: it gives every adapter, including newly admitted ones, a fixed KKdimensional descriptor in a shared space (96:1 compression of the response profile) at similar observed accuracy.
Reviewer_d58XOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper presents the following idea for using a set of adaptors in a GUI-based application. You have a set of LoRA adaptors, and a global support set G which has a set of GUI rollouts. Then a client comes in with a local support set L. The goal is to choose a set of adaptors by matching L with G to choose a relevance score.

The way this is done is to create a (size of adaptor set) X (size of G) response matrix that scores how well each adaptor goes on each rollout in G. This score combines whether each action taken by the adaptor is correct and whether the adapter matches the reference. Each adaptor is represented by a capability vector computed using a SVD of the reference responses and the response matrix.

To select a set of adaptors, we embed the example episodes and the episodes from G, and use a softmax over a top-K cosine similarity between each example and every reference. Then we map this vector into the same space as the capability vector; a dot product and a normalization gives us a final routing score.

Then when we are pushing data through the adapted agent, at each linear layer we sum the adaptors weighted output contributions.

Critical strengths

This is a solid paper. It is technically sound, and the core contribution that appears novel is the idea that you can’t compare tasks using simple semantic similarity. Instead, you need to compare using behavioral similarity. To do this, the authors actually check how well each adaptor behaves on the reference tasks in G—because we know how well each adaptor actually behaves on the reference tasks, if we have an idea of how important the reference tasks are to the user, we have a super high quality signal as to how well the adaptors will work.

Experiments show the efficacy of the method. The authors have carefully compared their method against a significant number of state of the art methods for selecting adaptors. The evaluation metric seems to be: predict one step in the future, using the prior ground-truth state. In this case, GUIHub shows almost uniformly superior performance to other similar methods (albeit not for use with GUI adaptation) when they are applied to this domain.

Critical weaknesses

The whole paper is predicated on the idea that semantic similarity is not good enough. In fact, in Section 4.7 and Figure 5, the authors explicitly criticize semantic similarity. But still, the proposed method is still relying on cosine similarity as a fundamental link to check similarity between G and L. So semantic similarity is still fundamental to the proposed method.

The paper fundamentally presents GUIHub as being special-designed for GUI-based adaptation, but most of the main ideas here (profiling using the GSS-LSS computation, SVD capability projection, and the LoRA composition) has nothing to do with GUI-based adaptation. Yes, the authors use screenshot embeddings and bounding boxes, but other than that nothing is specific to GUI-based adaptation. The authors proposals could be used for any case where the goal is to choose from a library of adaptors.

The fact that the authors have designated it as GUI-specific does not mean that it is actually GUI-specific. One of the core aspects, for example, of GUI-based interaction is the fact that it happens online… in most applications, I’d imagine that you have a user whom you don’t know anything about and you discover it on-the-fly. The proposed method is offline in the sense that it requires the user to apply an LSS that has a set of representative tasks before it can serve anything, so I’m not too sure how applicable to real-life GUI adaptation it is. Given this, the experiments do not attempt to mimic learning via real-time interaction that would happen in the real world.

That does not mean that the proposed ideas are uninteresting, but if what we’re left with is something that really is for general-purpose adaptation (that just happens to be tested on GUI problems), and in that case a comparison with LoRAHub and LoraRetriever on a wide variety of cross-domain tasks would have been appropriate.

Another concern I have: I understand that especially if this paper is coming from an academic lab you cannot fine-tune a 500B parameter model using LoRA. That’s fine. But I still strongly feel that testing with a state-of-the-art model from OpenAI or Anthropic (un-modified, of course) on a small, representative sample of 50-100 episodes or trying a benchmark for which zero-shot numbers from a frontier model are available would really strengthen the paper. You do not have to beat it or even come close (a 4B parameter model has obvious cost advantages and you can run it locally on a smartphone so there are no privacy concerns and you don’t need a network), but it gives the reader an idea of what such a model would do on this task, so that the reader can judge whether the savings are worth it.

Questions for authors

I don't have any specific questions. I feel that I understood the paper. But I'd welcome an argument that rebuts any of my points.

4 comments
AuthorsOpenReview ↗

Response to Reviewer d58X (1/2)

Thank you for the thoughtful review and the concrete suggestions. We ran both experiments you proposed, a cross-domain comparison and a frontier-model reference. Table and appendix numbers refer to the revised PDF.

(W1) "The proposed method is still relying on cosine similarity as a fundamental link to check similarity between G and L. So semantic similarity is still fundamental to the proposed method."

We agree, and its role is specific. GUIHub separates two questions (Section 3.3).

  • Which shared tasks matter to this user? This is answered semantically. The relevance vector ρu\rho_u puts weight on the shared episodes whose instruction and first screenshot resemble the user's local examples.
  • Which adapter performs well on those tasks? This is answered only by measured behavior. The response matrix RR records how each adapter actually acts on each shared episode.

Our claim is therefore not that semantic similarity is useless. It is that semantic proximity between a user's tasks and an adapter's training tasks is an unreliable proxy for what that adapter can do.

(W2) "Most of the main ideas here … has nothing to do with GUI-based adaptation. … The authors' proposals could be used for any case where the goal is to choose from a library of adaptors."

We agree, and the introduction now says so: "While this mechanism is general, we instantiate it for GUI agents through action-level scoring of multi-step episodes and label-free demand estimation from instructions and screenshots."

  • Our question. We study whether post-hoc adapter reuse can serve GUI agents and what it takes to work there; LoRAHub and LoraRetriever were evaluated on text tasks.
  • Why GUI agents. GUI agents are a natural fit for this setting. Interaction data is spread across users, each user's apps and task mix differ, and the data is private and unlabeled, so independently trained adapters accumulate while no single party can pool the underlying data.

(W3) "The proposed method is offline in the sense that it requires the user to apply an LSS that has a set of representative tasks before it can serve anything … the experiments do not attempt to mimic learning via real-time interaction."

  • No cold-start block. A new user does not have to supply anything before being served: the base model serves from the first request. GUIHub becomes active once a few of the user's requests have been collected with consent; these are only instructions and initial screenshots, without action labels.
  • Cheap and small. Building the composition takes about 0.3 s per target (Table 11: 9.6 s for 32 targets); it is reused across later requests and can be rebuilt as the user's task mix drifts. Reducing the LSS from ten to five episodes changes Step SR by at most 0.33 pp (Table 14).
  • A first step toward deployment. In a new closed-loop study on AndroidWorld (Appendix D.7), the adapters are trained only on offline data; GUIHub builds its composition from ten unlabeled instruction–screenshot pairs observed in the target environment and then acts on its own predictions. It is best or tied-best on 11 of 15 tasks, compared with 8 for the base model and 6 for LoraRetriever; with five trials per task, this is preliminary. We have not evaluated a live deployment in which support examples arrive over time, and the revised conclusion lists it as future work.
AuthorsOpenReview ↗

Response to Reviewer d58X (2/2)

(W4) "If what we're left with is something that really is for general-purpose adaptation …, a comparison with LoRAHub and LoraRetriever on a wide variety of cross-domain tasks would have been appropriate."

Following your suggestion, we ran GUIHub on LoraRetriever's mixed-task text benchmark: 48 Flan-v2 tasks with one LoRA adapter each on Llama-2-7B, evaluated on 2,395 test examples (new Appendix F, Table 24).

  • Setup. GUIHub builds one two-adapter composition per task from ten unlabeled examples, with its configuration carried over from the GUI setting without retuning; LoRAHub builds one composition per task from five labeled examples; LoraRetriever routes each query to three adapters, as in its original protocol. We report two pools together: in-pool, where the task's own adapter is available, and held-out, where it is removed. Scores are normalized to the task-aligned adapter (= 100) and averaged over ten task groups.

    PoolLoraRetrieverGUIHubLoRAHub (5-shot)
    In-pool95.3597.6685.68
    Held-out85.2283.4883.73
  • In-pool. GUIHub has the highest normalized average and is best or tied-best on 7 of the 12 metric groups, with the largest gains over LoraRetriever on reading comprehension with commonsense (+10.0) and commonsense (+7.0). It also exceeds LoRAHub on 10 of the 12 groups.

  • Held-out. The three methods are close. GUIHub is higher than or equal to LoraRetriever on 7 of the 12 groups (+7.0 on coreference and paraphrase) but lower on structure-to-text generation (ROUGE −6.3 to −10.1) and NLI (−11.5).

(W5) "Testing with a state-of-the-art model from OpenAI or Anthropic (un-modified, of course) on a small, representative sample of 50-100 episodes … would really strengthen the paper."

Following your suggestion, we evaluated an unmodified frontier model (Astra, accessed through its API) on a stratified sample of 96 test episodes: three per client, 540 action steps (new Appendix E, Table 22).

  • Protocol. It receives the same inputs and ground-truth histories as our models and is scored by the same evaluator. Because it is not fine-tuned on our action format, its prompt additionally specifies the action space and coordinate convention.

    MethodTypeGroundStep SRIID Step SROOD Step SR
    Astra (frontier model)81.8580.3670.3771.9565.38
    GUIHub (Qwen3.5-4B)82.7864.3567.5966.5970.77
  • Overall. The frontier model reaches 70.37% Step SR and GUIHub, with a 4B backbone, 67.59%; GUIHub’s action-type accuracy is slightly higher (82.78% vs. 81.85%). An analysis of the incorrect steps (Appendix E, Table 23) shows that the two fail in different places; most errors of both models are wrong action types (98 and 93 steps).

  • Grounding. The frontier model localizes targets better: it is correct on 264 of the 328 reference tap steps, versus 213 for GUIHub.

  • Scrolling and termination. GUIHub handles scrolling better: it is correct on 61 of the 80 reference swipe steps, versus 40, and the frontier model taps an element instead of swiping 30 times. GUIHub also ends episodes early more often (37 vs. 19 steps).

I am impressed with the authors' responses to all reviewer comments. In response to my own comments, the authors added a comparison with Astra, and the proposed method fares quite well. I don't think this paper is earth-shattering, but it's well executed and I'm happy to change my rating to "accept".

AuthorsOpenReview ↗

Thanks for your reply

Thank you for the reply and for raising your rating to accept. Your suggestions have made the scope and value of our work much clearer. We sincerely appreciate your dedicated effort in reviewing our manuscript!

Reviewer_zZE7OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper targets the GUI agent domain and establishes a LoRA pool to dynamically match user needs, thereby achieving improved agent performance. Unlike existing approaches, the authors observe that routing LoRAs based solely on task semantics lacks sufficient relevance. To address this, they explicitly incorporate real human needs into the agent pipeline by analyzing user requirements to construct user profiles, which are then used to guide the routing process. Experimental results demonstrate that the proposed method outperforms prior work.

Critical strengths
  1. Combining behavioral profiling with unlabeled semantic needs is not only well-conceived but also backed by solid evidence. For instance, the authors ran a comparison experiment: when they swapped the "behavioral score" for a "semantic score," the model's success rate in unfamiliar scenarios dropped by 5.18 percentage points. Moreover, the data shows that the "behavioral score" has a correlation of 0.513 with the actual performance of individual adapters, while the "semantic score" manages only a meager 0.103. This set of comparisons is arguably the most convincing part of the entire paper, as it directly proves a simple truth: relying on surface-level semantics isn't reliable—what truly matters is what has actually been tested and used.
  2. The evaluation is comprehensive, and the experimental rigor is generally strong. The four RQs cover IID, OOD, mixed, and scalability settings across three backbones and two model families. Paired bootstrap confidence intervals are reported for all key gains, with most intervals excluding zero. The authors also provide an unusually detailed account of information access boundaries, which clearly specifying who had access to labels and who saw test queries. Additionally, the breakdown of efficiency metrics (Tables 10 and 11) is highly transparent.
Critical weaknesses
  1. I think the biggest weaknesses with this paper is that the authors evaluated their method solely on an offline test set, without conducting evaluations in real-world environments (such as AndroidWorld). As a result, the actual performance of the method in realistic settings remains unknown.
Questions for authors

Please provide a set of experimental results conducted in a real-world GUI environment.

1 comment
AuthorsOpenReview ↗

Response to Reviewer zZE7

(W1 & Q1) "The authors evaluated their method solely on an offline test set, without conducting evaluations in real-world environments (such as AndroidWorld)."

Thank you for the review. We agree: teacher-forced next-action prediction measures local action competence, not closed-loop task success. Following your suggestion, we added a closed-loop evaluation on AndroidWorld (new Appendix D.7, Table 21) and a stricter offline metric (new Appendix D.6). Section 4.3 now points to both, and the conclusion states that our main evaluation is offline and teacher-forced. Table and appendix numbers refer to the revised PDF.

  • Setup. Each method executes its own predicted actions in the emulator with Qwen3.5-9B, and success is judged by the AndroidWorld checker.
  • Tasks. We select 15 tasks that cover basic operations in everyday applications: adding and deleting expense and recipe records, deleting calendar events, managing notes and folders, handling SMS messages, launching an application, taking a photo, and operating the stopwatch, the timer, and the Bluetooth switch. We report all 15 tasks, with five trials per task and method.
  • Methods. All adapters are trained only on offline AndroidControl data; AndroidWorld observations are used only for routing at deployment. Before each task, GUIHub collects ten instruction–screenshot pairs through the official task initialization as its unlabeled support set. It selects two adapters and keeps this composition fixed for all episodes of the task. LoraRetriever retrieves three adapters at the start of each episode. Base uses no adapter. Each episode allows at most 30 action attempts and 600 s of decision time.

Successful trials out of five:

TaskGUIHubBaseLoraRetriever
ExpenseAddSingle202
ExpenseDeleteSingle520
RecipeAddSingleRecipe100
RecipeDeleteSingleRecipe541
RecipeDeleteSingleWithRecipeWithNoise253
SimpleCalendarDeleteEvents010
SimpleCalendarDeleteOneEvent545
MarkorCreateFolder031
MarkorDeleteNote010
SimpleSmsSendClipboardContent310
OpenAppTaskEval555
CameraTakePhoto555
ClockStopWatchRunning555
ClockTimerEntry433
SystemBluetoothTurnOff333
  • Result. GUIHub is best or tied-best on 11 of the 15 tasks, compared with 8 for the base model and 6 for LoraRetriever, and it is the sole best method on five. Relative to the base model, GUIHub succeeds more often on seven tasks and less often on four. The evidence is preliminary: there are five trials per task, the adapters receive no closed-loop or error-recovery training, and absolute success rates are not comparable with agents that use larger models or closed-loop training.
  • Stricter offline metric. Appendix D.6 (Table 20) also reports episode-level success, where an episode counts as successful only if every step is predicted correctly; GUIHub is highest or tied-highest in five of the six backbone–split settings.

The revised conclusion lists larger-scale closed-loop evaluation with recovery-aware adapter training as future work. Thank you for prompting these additions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.