- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper addresses sparse‑reward temporal credit assignment for long‑horizon mobile GUI agents. It proposes CompGraphRL: it leverages a frozen VLM to encode GUI states combining screenshot, prior action and interaction history, builds compressed semantic state DAGs, and generalizes GAE to graphs with separate node‑level and edge‑level advantage estimation. Evaluations run on MobileGym, with cross‑domain transfer tests on AndroidWorld and MobileWorld against sequence‑level (GSPO) and graph‑based RL baselines. Ablations and merge‑ratio analysis are also provided.
Critical strengths
- It properly targets two key GUI challenges: perceptually aliased states and functionally‑equivalent yet visually‑different screens. Moving past raw pixel‑based state matching is meaningful for GUI agent research.
- It includes backbone variants, cross‑benchmark generalization, ablations, and merge‑ratio diagnostics, showing clear improvements over compared baselines.
Critical weaknesses
- The main technical contribution is adapting existing GAE onto DAG structures. VLM‑based compression and merge‑budget controls are largely engineering tweaks. Prior graph‑credit‑assignment works already exist; incremental conceptual advances are modest for an ICLR submission.
- Performance hinges heavily on the frozen external VLM embedding quality and manually tuned target merge rate ρ_target. There is no adaptive scheme for choosing merge ratio, nor analysis on how bad VLM embeddings would break performance. Modified GiGPO/G2PO implementations also raise fairness concerns for comparison.
Questions for authors
- Why keep the VLM state encoder frozen? Have you tried jointly fine‑tuning it with policy? What results would we expect?
- ρ_target=0.15 is found via grid search. Can you propose an adaptive way to pick this hyperparameter? Would optimal values shift across different task suites?
- GiGPO and G2PO are modified to use image similarity in your setup. Could you provide results using their original logic for GUI settings?
- Your state graph is built offline after rollout collection. What are the main obstacles if you try incremental online graph construction?
4 comments
Response to Reviewer zLqi (Part 2 of 4)
Q1: Frozen VLM state encoder
Question. Why keep the VLM state encoder frozen? Have you tried jointly fine-tuning it with the policy? What results would we expect?
We first tried synchronously updating the VLM state encoder together with policy training. Empirically, this setting was very unstable: learning improved slowly at the beginning and only started to become better in later stages. We then considered whether this came from value-model initialization, and repeated the experiment using an already trained value model as the initialization. However, the training process remained unstable, with performance fluctuating across updates. Only after fixing the encoder/value parameters did we observe steady training improvement, and this trend held across multiple model sizes.
Our interpretation is that this stability is related to the objective used by our graph-GAE recursion. With uniform successor averaging, the value target is not intended to track the transition-frequency distribution induced by the current policy at every update. Instead, it is a branch-fair credit-propagation target for exploration-heavy GUI tasks, where low-frequency branches, including rare successful branches, should still participate in backward credit propagation. Therefore, the value model expected by this objective should not change with every policy update, which makes fixed parameters a better fit for this credit-propagation scheme.
For this reason, we use fixed VLM/state-value parameters when constructing and evaluating the compressed state graph. This keeps the semantic state space and graph value estimates stable while the policy is optimized, which was the first setting that produced reliable improvement in our experiments. In the revised manuscript, we will clarify this empirical observation and the connection to uniform successor averaging, rather than presenting freezing as the only possible design.
Q2: Adaptive merge budget and rho_target
Question. rho_target = 0.15 is found via grid search. Can you propose an adaptive way to pick this hyperparameter? Would optimal values shift across different task suites?
Yes. We agree that rho_target = 0.15 should be treated as a GUI-task empirical setting rather than a universal value. The optimal merge rate can shift across task suites depending on how often semantically equivalent states reappear, how noisy the visual observations are, and how well the embedding separates task stages.
The role of rho_target is to replace a poorly calibrated fixed embedding threshold with a more interpretable merge budget. In this sense, rho_target already functions as an adaptive threshold: for each batch of trajectories, candidate merges are ranked by similarity under temporal and acyclicity constraints, and the effective similarity cutoff is implicitly determined by the target merge rate and the current embedding distribution. In our GUI experiments, moderate merging improves value sharing and delayed-credit propagation; beyond the roughly 15% region, over-merge noise begins to dominate. We expect the best value to be lower in tasks with fewer repeated semantic states or weaker embeddings, and higher in tasks with many visually varied but functionally equivalent screens.
Response to Reviewer zLqi (Part 3 of 4)
Q3: Fairness of GiGPO/G2PO comparisons
Question. GiGPO and G2PO are modified to use image similarity in your setup. Could you provide results using their original logic for GUI settings?
We agree that this should be reported explicitly. The original GiGPO/G2PO logic relies on exact observation or environment-state matches. In mobile GUI settings, exact matches are extremely rare because screenshots change with system time, app content, scroll position, popups, cursor state, and hidden app state. Therefore, the original exact-match version forms very few useful merged nodes and tends to degenerate toward a trajectory-level sequence baseline. We will add original exact-match GiGPO/G2PO results in the appendix and clearly label the main-table versions as GUI-adapted baselines.
To further address fairness and isolate the source of improvement, we added a Qwen3.5-9B component-swap comparison under the same rollout, prompting, reward shaping, optimization budget, and evaluation protocol:
| Variant | Graph | Value | Advantage | MobileGym Succ. (%) | Delta vs. Full |
|---|---|---|---|---|---|
| GSPO | None / trajectory | Group reward | GSPO sequence-level | 23.8 ± 1.2 | -6.6 |
| G2PO | IAC graph | RTG | G2PO | 23.9 ± 2.4 | -6.5 |
| G2PO + Semantic Graph | VSC semantic graph | RTG | G2PO | 27.6 ± 0.8 | -2.8 |
| G2PO + Semantic Graph + Learned VF | VSC semantic graph | Learned VF | G2PO | 29.1 ± 1.2 | -1.3 |
| CompGraphRL w/ IAC Graph | IAC graph | Learned VF | Graph-GAE | 19.2 ± 1.5 | -11.2 |
| w/o Semantic Graph & VF | None / trajectory | RTG | Trajectory RTG | 22.6 ± 1.5 | -7.8 |
| w/o Semantic Graph | None / trajectory | Learned VF | Trajectory value-guided | 25.3 ± 0.8 | -5.1 |
| w/o Learned VF | VSC semantic graph | RTG | Graph-GAE | 25.0 ± 1.7 | -5.4 |
| CompGraphRL | VSC semantic graph | Learned VF | Graph-GAE | 30.4 ± 0.2 | 0.0 |
The component swaps give the following attribution. Replacing G2PO's IAC graph with the VSC semantic graph improves performance from 23.9% to 27.6%, showing that semantic grouping helps on its own. Adding the learned value function under the same semantic graph and G2PO advantage rule further improves performance from 27.6% to 29.1%, indicating that value prediction contributes a separate gain. Replacing G2PO's advantage rule with graph-GAE under the same graph and value setting improves performance from 29.1% to 30.4%, showing that graph-GAE adds another gain beyond semantic state information.
The IAC-graph swap also clarifies an important limitation. When CompGraphRL keeps the learned value function and graph-GAE but replaces the semantic graph with the IAC graph, performance drops to 19.2%, even below G2PO with the IAC graph (23.9%). This suggests that our value/graph-GAE pipeline is more sensitive to over-merging, and that graph-GAE is not automatically beneficial with an unsuitable graph.
Therefore, our revised conclusion is: semantic graph construction explains a substantial part of the improvement over G2PO, but the full CompGraphRL gain requires three aligned components--semantic graph construction, learned value prediction, and graph-GAE. In the revised manuscript, we will add this table to the Section 4.5 ablation study and use it to explicitly separate these component-level effects, thereby avoiding presenting Table 1 as evidence for full-system comparison alone.
Response to Reviewer zLqi (Part 4 of 4)
Q4: Offline graph construction and online extension
Question. Your state graph is built offline after rollout collection. What are the main obstacles if you try incremental online graph construction?
We view online or incremental graph construction as a valuable future direction. The current offline construction is mainly an implementation choice under the verl-style asynchronous training framework, where rollout generation and reward/value computation are separated rather than executed as a tightly coupled online loop. However, there are two practical obstacles in GUI environments:
-
Rewards and node values are delayed. For an intermediate GUI state, the value or process reward is often not reliable until the rollout outcome and subsequent trajectory are known. If we update nodes and edges online using only partial trajectories, we may assign high or low value to states before their downstream consequences are observed, which can inject noisy credit into later exploration.
-
GUI states are hard to reproduce exactly. Online graph expansion is most useful when the agent can return to a promising or uncertain node and resample alternative branches. In mobile GUI tasks, a state depends not only on the screenshot and action history, but also on app cache, account state, network responses, time, popups, background processes, and hidden UI state. Replaying the action sequence is often insufficient to recover the same state. Reliable online branching would require emulator- or VM-level snapshot and restore support.
For this reason, we view online graph construction in GUI tasks as a complex system-level problem that requires close coordination between the environment, snapshot/restore support, the rollout framework, and the value backfilling mechanism. We will add this discussion as an important direction for future research.
Planned revision
We will revise the manuscript in four ways in response to your comments. First, in the Introduction, we will sharpen the contribution statement to emphasize the coupled mechanism of GUI semantic state abstraction, controlled graph construction, learned value guidance, and graph-GAE credit recursion. Second, in the appendix, we will clarify the fixed VLM/state-value design and rho_target setting, including the instability observed under joint updating, the connection to uniform successor averaging, and why the best merge budget may vary across task suites. Third, in Section 4.5, we will add the component-swap comparison, report original exact-match GiGPO/G2PO results, and clearly distinguish GUI-adapted baselines from the original baseline logic. Fourth, in the Conclusion, we will add the online graph construction discussion as a future research direction, emphasizing that it requires coordinated environment, snapshot/restore, rollout-framework, and value-backfilling support in GUI tasks.
We hope these revisions address your concerns, and we would be grateful if you would consider updating your score if you find our response satisfactory.
Response to Reviewer zLqi (Part 1 of 4)
Thank you for the constructive review. We appreciate your recognition that CompGraphRL targets two central challenges in GUI-agent learning: perceptually aliased states and functionally equivalent but visually different screens, and that our experiments include backbone variants, cross-benchmark generalization, ablations, and merge-ratio diagnostics.
We agree that the paper should state its conceptual contribution more sharply. Our claim is not that the main novelty is simply adapting an existing GAE estimator to a DAG. The central difficulty is that GUI agents operate in environments where the state identities assumed by prior graph-credit-assignment methods are no longer available. In symbolic or text-based settings such as ALFWorld or WebShop, repeated rollouts can often share a stable discrete or textual state key. In realistic GUI tasks, however, exactly repeated observations are rare: screenshots may differ because of system time, scroll position, dynamic content, cursor state, popups, notifications, input fields, or hidden app state, while still corresponding to the same functional state. Conversely, visually similar screens may require different actions depending on the previous interaction history. Thus, applying graph credit assignment to GUI agents first requires solving the state-equivalence problem: which non-identical GUI observations should share delayed credit, and which visually similar observations must remain separated.
From this perspective, the VLM-based state encoder is not an optional external enhancement, but the state abstraction mechanism that makes graph construction well-defined in GUI environments. Exact observation matching would produce almost no useful merges, reducing the method to trajectory-level credit assignment. Pure visual similarity is also insufficient, because it cannot reliably distinguish visually similar but semantically different task stages, or connect visually different but functionally equivalent screens. CompGraphRL therefore uses VLM semantic embeddings to construct comparable GUI state representations, then applies controlled graph compression to form a semantic state graph on which delayed rewards can be shared.
The contribution is the coupled mechanism rather than any isolated engineering tweak. Semantic state abstraction supplies a workable notion of node equivalence for GUI rollouts. Controlled state graph construction then turns these semantic relations into a reliable compressed graph by selecting high-confidence merges, preserving temporal consistency, and avoiding structurally invalid credit paths. Graph-GAE finally propagates delayed credit through shared semantic nodes and action-conditioned edges, while reducing the variance of advantage estimates on the compressed graph.
Our existing analyses support this interpretation. Section 4.3 compares VSC, ISC, and TSE against human semantic state groupings, showing that VSC is closest to human-labeled GUI state equivalence rather than being an arbitrary embedding choice. The same representation differences also affect downstream RL performance: weaker embeddings lead to over-splitting, where equivalent states fail to share delayed reward, or over-merging, where distinct task stages contaminate each other's credit. The Section 4.4 merge-ratio diagnostics further show the expected trade-off: moderate merging improves value sharing and delayed-credit propagation, while excessive merging introduces noisy credit and hurts policy learning. This is why the merge budget is part of the method's GUI-specific state-graph construction, not merely a hand-tuned cosmetic control.
To further isolate the sources of improvement, we have added a detailed ablation study in Section 4.5 of the revised manuscript. These experiments compare variants that replace the VSC semantic graph with alternative state graphs, remove learned value guidance, remove graph-GAE, and combine the same semantic graph with G2PO-style credit assignment. This allows us to separate the effects of GUI semantic graph construction, learned value guidance, and graph-GAE / edge-credit recursion more explicitly.