acceptodds
Withdrawn from ICLR 2027

From Future to Present: Utility-Routed Future Distillation for Mobile Agent

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Mobile GUI agents rely on current screens and interaction history, yet anticipating the state a task should reach next offers additional guidance for action decisions. Although logged trajectories naturally provide future screens, these observations are unavailable at deployment and are not uniformly beneficial for action prediction. We propose Utility-Routed Future Distillation (URFD), which transfers useful future representations into a current-only GUI policy without additional annotations or future-image generation. Task- and Action-Conditioned Successor Initialization (TASI) combines dual-view action prediction with future-selection tasks to establish a future-aware initialization. URFD distills future-conditioned representations into a compact latent slot within a current-only policy. A frozen dual-view teacher quantifies future utility through the gain in demonstrated-action log-likelihood, jointly governing the interpolation of hidden-state targets and the weighting of policy distillation across current- and future-conditioned views. This utility-guided formulation mitigates interference from unhelpful future observations during distillation. The experimental results show that URFDimproves average action match by 2.26 percentage points across two offline benchmarks and online task success by 3.5 points on AndroidWorld. Comprehensive analyses examine the utility of future supervision, strategies for representation alignment, and their impact on action decisions. These results demonstrate that URFD enables models to approximate hidden representations of real future states and improve action prediction accuracy without observing real future states at deployment.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Mobile Agent

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_XN8MOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper mainly introduces the GUI state of the next frame that can be observed in the training trajectory into the model training process. The aim is to enable the model to utilize the future information learned during the training stage to assist in the current action decision-making even when it cannot see the actual future scene during deployment. Specifically, the authors proposed a two-stage training paradigm: Firstly, they constructed a specific data form based on the existing trajectory for SFT, enabling the model to simultaneously learn the action prediction under the current state and the tasks related to the next state; subsequently, they adopted a similar teacher-student dual-branch structure for distillation, leaving a CopyState slot in the model and having the hidden representation at that position learn the future representation corresponding to the real next frame. The main contributions of the paper are also quite clear: Firstly, it proposed TASI, which is initialized and trained through the successor-related data constructed by the authors; Secondly, it proposed URFD, which controls the distillation of future information at the two levels of hidden representation and action policy, so that the deployed model, which can only see the current state, can learn part of the future information. The ablation experiments are also quite solid, verifying the effectiveness of each component.

Critical strengths

One of the main advantages of this paper is that the overall method design is quite comprehensive. The authors first establish successor-aware initialization through TASI, and then use URFD to selectively distill the hidden representation and action policy based on future utility. The motivation between the two stages is relatively coherent. At the same time, the method utilizes the naturally existing information of the next frame in the existing GUI trajectory, without the need to generate future images additionally. Deployment does not rely on the actual future state. The experimental part is well done, apart from the conventional component ablation, the authors also systematically analyzed the effects of utility, alignment layer, slot size, and different action types, and simultaneously verified it on offline and online benchmarks. Overall, although the innovation of the method itself is not particularly strong, the completeness of method design, experimental verification, and analysis is relatively high.

Critical weaknesses

I consider this to be a well-structured and well-conducted study. The innovation of the method itself is not particularly strong, but from the overall method design to the ablation experiments and further analysis, the author has done a comprehensive job. My current evaluation is somewhere between "weak reject" and "weak accept"; if the author can provide clear explanations for several key issues in the rebuttal and supplement necessary control experiments, I would be more inclined to "weak accept". My main concern is that the current experiments have not fully demonstrated how much of the performance improvement comes from the proposed method itself. Although URFD has achieved good results compared to existing methods, compared to the most critical backbone Qwen3-VL-SFT baseline, the improvement is actually relatively limited, and the paper's explanation of the training configuration and training budget for this SFT baseline is not sufficient. At the same time, URFD introduces an additional 32-token CopyState slot during training and deployment, which is not completely consistent with the input format of the standard SFT. Therefore, the current results are still difficult to completely rule out the benefits brought by additional training budget or changes in input structure. Overall, I believe the main problem with this work is not that the method is ineffective, but that the current experiments have not sufficiently separated the contribution of the method itself from the impact of training and input settings. If these issues can be properly clarified, the entire paper is persuasive.

Questions for authors

Qwen3-VL-SFT is the most important counterpart to the backbone baseline in this paper, and it has already surpassed most of the existing methods in the table. The author is requested to further explain the specific training settings of this SFT baseline. Does it use the same data and optimization configuration as TASI/URFD? If the training budget is different, how much of the improvement of URFD compared to SFT might come from the additional training itself?

URFD uses an additional 32-token CopyState slot during training and deployment, while the ordinary baseline uses a single current screenshot and text history. Please ask the author if they have tried an input-matched Qwen3-VL-SFT, that is, using the same 32-token low-resolution copy of the current screenshot and text as the input, but only performing ordinary action SFT? This comparison can help determine whether the input structure of CopyState has already brought some benefits.

The symbol definitions in Eq. 10 of the main text seem to be in contradiction with the content stated in the paper.

Compared to Qwen3-VL-SFT, the overall improvement of URFD on AndroidWorld comes entirely from the Medium split, while Easy and Hard do not have any improvement. How does the author understand this phenomenon? Does this mean that the current one-step future supervision mainly improves local action decisions, and has limited help for longer horizon tasks?

Please ask the author if they can add a control group starting from the same TASI checkpoint, using the same CopyState input and the same number of training steps, but only performing action CE? The existing component ablation shows that each module has an effect, but this control group can more directly isolate the contribution of the second stage future distillation itself.

4 comments
AuthorsOpenReview ↗

Response to Reviewer XN8M (1/2)

We sincerely thank you for the positive assessment of our method design and the completeness of our experiments and analyses. To address yout central concern about how much of URFD's improvement comes from the method itself, we have added the control experiments and will respond to each point below.


W1/Q1: Configuration and training budget of the SFT baseline.

The SFT baseline shares the initialization, training data, and optimization configuration of URFD; the two differ only in the stage-2 input and training objective:

SFTURFD
InitializationTASI checkpointTASI checkpoint
TASI stage790,994 examples, 1 epoch790,994 examples, 1 epoch
Stage-2 dataFour-source mixture (AC, GO, OM, MF), 208,766 transitionsFour-source mixture (AC, GO, OM, MF), 208,766 transitions
Epochs / exposure3 epochs (each transition seen 3 times)3 epochs (each transition seen 3 times)
Updates4,893 (batch size 128)4,893 (batch size 128)
OptimizationPeak LR 1e-5, cosine schedulePeak LR 1e-5, cosine schedule
InputCurrent screenshot + text history (I,Ht,St)(I,H_t,S_t), no copy slotAdditional 32-token CopyState slot (I,Ht,St,St)(I,H_t,S_t,S_t)
ObjectiveAction CEAction CE + hidden-state alignment + policy distillation

Therefore, URFD's improvement over SFT does not come from additional training data or training steps. We will include these details in Section 4.1 and Appendix B.


Q2/Q5: Input-matched control trained with CE only from the same TASI checkpoint.

Since our SFT baseline is itself initialized from the TASI checkpoint, the two controls suggested in Q2 and Q5 correspond to the same experiment in our setting: starting from the same TASI checkpoint, using the same input as URFD (the current screenshot plus a 32-token low-resolution CopyState slot), the same data, and the same 4,893 updates, and training with action cross-entropy only. The results (action match, %) are as follows:

MethodInputAndroidControlGUI-OdysseyAverage
SFT(I,Ht,St)(I,H_t,S_t)76.7680.0678.41
SFT (CopyState)(I,Ht,St,St)(I,H_t,S_t,S_t)76.9280.2978.61
URFD(I,Ht,St,St)(I,H_t,S_t,S_t)77.9683.3880.67

The CopyState input structure alone contributes only 0.20 points on average. With identical input, data, and number of updates, URFD still outperforms this control by 2.06 points (3.09 points on GUI-Odyssey), indicating that the improvement stems mainly from the stage-2 future distillation, not the additional CopyState input.


Q3: Sign in Eq. (10).

We thanks for pointing this out. The two terms in Eq. (10) were inadvertently swapped. The utility is intended to measure the token-averaged increase in the log-likelihood of the demonstrated action when the real successor screen is visible, and the correct expression is

ui=1∣Ai∣∑j∈Ai[log⁡qi,jF(ai,j)−log⁡qi,jC(ai,j)]u_i = \frac{1}{|\mathcal{A_i}|}\sum_{j\in\mathcal{A_i}}[\log q^F_{i,j}(a_{i,j})-\log q^C_{i,j}(a_{i,j})]

A positive uiu_i indicates that the successor is helpful, in which case the routing weight wiw_i increases and the student is aligned more closely with the FutureState teacher, consistent with the main text and Appendix B. For example, with qC(a)=0.50q^C(a)=0.50 and qF(a)=0.60q^F(a)=0.60, we obtain u=+0.182u=+0.182 and w=1w=1 under the routing interval [0,0.01][0,0.01]; with qF(a)=0.45q^F(a)=0.45, we obtain u=−0.105u=-0.105 and w=0w=0, so the student remains anchored to the CopyState teacher. The implementation and all experiments use this correct expression, and we will correct Eq. (10) and the corresponding formula in Figure 1 in the revision.

(Continued in Part 2/2.)

AuthorsOpenReview ↗

Response to Reviewer XN8M (2/2)

Q4: Why the AndroidWorld gain concentrates on Medium tasks.

We thanks for this insightful question. We analyzed this phenomenon (pass@3) using the official AndroidWorld task metadata (optimal number of steps and task tags):

DifficultyTasksSFTURFDURFD wins / lossesMedian optimal stepsTasks requiring cross-step state*
Easy6150504 / 448%
Medium3617216 / 2619%
Hard19441 / 11363%

URFD distills future-state information and mainly improves navigation decisions, i.e., where to tap and where to go next (in the offline analysis, Scroll and Press show the largest gains, while Click contributes the most to the overall improvement; see Figure 5 and Appendix D). This mechanism interacts differently with the main bottleneck of each difficulty level:

  • Easy. The paths are short (median of 4 steps), and both models already solve 50 of the 61 tasks. A small improvement in per-step accuracy rarely changes the task outcome, especially under pass@3; the 8 tasks with changed outcomes are evenly split between wins and losses.
  • Hard. The paths are long (median of 13 steps), and 63% of the tasks require maintaining state across steps (e.g., passing information across apps, memorization, or repeated operations); both models fail on 14 of the 19 tasks. The bottleneck lies in long-horizon state management.
  • Medium. The paths are of moderate length with fewer cross-step state requirements, so success depends mainly on whether the model reaches the expected pages. Here, improvements in individual steps translate most readily into task success.

We would also like to note that long-horizon tasks involve two different kinds of difficulty. The first kind requires multi-step navigation to reach the target page, where every step involves deciding where to go next, and URFD is particularly effective on such tasks. Examples include FilesMoveFile, SimpleCalendarAddOneEventInTwoWeeks, and the Hard task SaveCopyOfReceiptTaskEval, which URFD solves but SFT does not; all of them require the agent to move through a sequence of new pages. The second kind requires repeating the same workflow while keeping track of completed progress (e.g., ExpenseAddMultiple and RetroSavePlaylist); here, the difficulty lies in retaining historical information rather than anticipating the next step, which is beyond the scope of future-state supervision. In essence, the former tests whether the model can anticipate the future states that its actions lead to, whereas the latter tests whether it can retain historical information over a long horizon; URFD is designed for the former. This is also reflected in the trajectories of the AndroidWorld tasks with changed outcomes: among the 11 tasks that SFT fails but URFD solves, SFT reaches the target page in only 4 tasks, whereas among the 7 tasks that URFD fails, URFD has already reached the target page in 6 tasks. In other words, SFT mostly fails at navigation, while URFD mostly fails after the target page has been reached. We therefore believe that URFD helps navigation-dominated long-horizon tasks but provides limited benefit for long-horizon tasks that rely on memory of past progress; combining URFD with memory mechanisms is a direction for our future work.


We hope that the added control experiments and analyses address your concern about the contribution of the method. Once again, we sincerely thank you for the time and effort devoted to reviewing our paper and for the valuable suggestions. If our responses have resolved your concerns, we would be very grateful if you could consider raising your score. If you have any further questions, we would be more than happy to engage in further discussion.

AuthorsOpenReview ↗

Summary of rebuttal to Reviewer XN8M

Dear Reviewer XN8M,

Thank you again for your thoughtful feedback. As the discussion period is nearing its end, we would be grateful if you could review our response.

We showed that the SFT baseline uses the same TASI initialization, data, and update budget as URFD, and added the control you suggested in Q2/Q5 (same TASI checkpoint and CopyState input, CE only), which URFD still outperforms by 2.06 points. We also clarified the sign typo in Eq. (10) and explained, using AndroidWorld task metadata, why the gains concentrate on Medium tasks.

We would greatly appreciate your feedback. If our responses have addressed your concerns, we would be grateful if you could kindly consider raising your score; if not, we would be glad to discuss further. Thank you again for your time and valuable comments.

Reviewer_XN8MOpenReview ↗

Response

Thank you to the author for providing a detailed Rebuttal response, and for supplementing the training configuration description and key control experiments to address the questions I raised previously.

However, after further examination of the training paradigm of the paper, I still have the following doubts about its core mechanism.

The paper emphasizes that the Student can approximate future-conditioned hidden representations without observing the actual future state. However, TASI primarily relies on successor selection among visible candidates and action prediction conditioned on real future screenshots, while URFD transfers the resulting representations through Hidden-State Alignment and Policy Distillation.

Although this approach improves action prediction, it remains unclear whether the Student genuinely learns representations containing predictive information about future states, or merely acquires better action-discriminative features.

In particular, the FutureState Teacher's hidden states may encode both future-state information and action-related discriminative features. Therefore, improvements in downstream Action Accuracy and Hidden Alignment Loss alone do not sufficiently establish that the Student has learned meaningful future-state representations.

Since learning future-conditioned representations is a central claim of the paper, I believe a more direct Embedding-Level Evaluation would be valuable.

Overall, I acknowledge the effectiveness of the additional experiments and the improvements in action prediction. However, I remain concerned that the proposed mechanism of future-state representation learning has not been directly validated. Therefore, I am currently inclined to maintain my original score.

Reviewer_ra3WOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a two-stage method for mobile GUI action prediction using logged successor screenshots as training supervision. Task- and Action-Conditioned Successor Initialization (TASI) combines action prediction under current-copy and real-successor inputs with two successor-selection tasks. Utility-Routed Future Distillation (URFD) then trains a student that receives only the current screen, using a frozen dual-view teacher to route hidden-state alignment and policy distillation according to the demonstrated action's likelihood difference. The extra visual slot contains 32 tokens at deployment. On Qwen3-VL-8B, action match improves over the reported SFT control from 76.76% to 77.96% on AndroidControl-High and from 80.06% to 83.38% on GUI-Odyssey. AndroidWorld pass@3 increases from 61.2% to 64.7% over 116 tasks. The paper also examines future utility, alignment depth and distance, visual-slot budget, and action-specific effects.

Critical strengths

A practical use of naturally available successor supervision with a clear deployment interface

Sections 3.1–3.3 distinguish the real-successor teacher input from the deployable copied-current-screen input (Eqs. 2–3 and 7–9). TASI constructs successor-selection targets from logged transitions, and deployment retains only the current-input student. Section 4.1 specifies a compact 32-token additional visual slot. This addresses the concrete training/deployment information gap without requiring generated future images or additional reasoning annotations. The compact deployed representation makes the approach relevant to existing trajectory-trained GUI agents.

The selective-transfer motivation is supported by targeted analysis and ablations

On 1,000 non-terminal examples per dataset, the future-visible TASI teacher improves action match by 13.7 and 8.8 points, while Figure 2 separately measures corrections and harms. Section 4.4.1 reports that a 0.01 utility threshold retains 88.8% of corrections and rejects 96.0% of harms. Table 3 reports a 0.60-point average action-match reduction when utility routing is removed, and Figure 3 compares several shallow- versus deep-emphasized alignment schedules. The paper investigates why uniform transfer may be suboptimal and connects that observation to specific design choices. This is more informative than reporting only the performance of the complete recipe, although the utility sign must be reconciled as described below.

Evaluation covers action accuracy, complete recorded trajectories, and live interaction

Table 1 reports action-type accuracy, action match, trajectory success, and prefix progress on two datasets. Relative to SFT, GUI-Odyssey action match increases by 3.32 points and recorded-trajectory success by 5.65 points; AndroidControl-High also improves on all four reported metrics. Table 2 additionally evaluates AndroidWorld pass@3, and Figure 5 identifies a decline for Stop actions rather than presenting transfer as uniformly beneficial. The combination provides evidence that the reported improvement extends beyond one action-level measure. The explicit action-specific limitation helps identify where the present method is useful and where it remains weak.

Critical weaknesses

The printed utility has the opposite sign to the intended routing behavior

provides a utility gain for action prediction

Source: Section 3.3, Decision utility of the observed successor; Continuous Copy-to-Future routing; PDF p. 5; manuscript lines 231–245; quotation at 238; Eq. (10) at 235.

Concern. Eq. (10) defines utility as the mean of log q^C(a) minus log q^F(a). When the future view increases the demonstrated action's probability, this quantity is negative. Yet the accompanying text treats positive utility as beneficial, Eq. (11) routes larger values toward the future target, and Section 4.1 uses the interval [0, 0.01]. Thus, following the printed equations would anchor beneficial examples to the current-copy teacher and route sufficiently detrimental examples toward the future teacher. Figure 1 depicts the same ratio, so this is a definite inconsistency in the main specification. It could be a manuscript error; the main paper does not establish which sign was implemented.

Suggested clarification or experiment. Please state the exact implemented utility expression and reconcile Eq. (10), Figure 1, the prose, and the routing thresholds. A short numerical sanity check showing a beneficial example and its resulting routing weight would resolve the convention. If the implementation uses mean(log q^F − log q^C), correcting the manuscript and confirming that Figure 2 and Table 3 use that expression should suffice; if the printed expression was implemented, the routing analysis and affected results need to be recomputed.

The main comparison does not fully establish a matched current-only training control

copy-state slot with 32 visual tokens

Source: Section 4.1, Baselines and implementation; PDF p. 6; manuscript lines 301–306; quotation at 303.

Concern. The paper specifies one TASI epoch followed by three URFD epochs and an extra copied-screen slot, whereas it does not state the SFT control's training schedule or total update count in the main body. Table 3 removes individual losses or initialization components but does not include a control that simultaneously removes future-derived supervision while retaining the copied slot and a comparable student training budget. Figure 4 does vary slot size, but compares DualView references with Align+CE students and therefore does not resolve this particular attribution issue. The gap is missing experimental information, rather than proof that SFT received less training.

Suggested clarification or experiment. Please report the SFT initialization, data mixture, number of updates, and transition exposure, and add a current-only CE control with the same 32-token copied slot and matched student update budget. A TASI-initialized student continued with CE alone for the same three epochs would additionally isolate the incremental benefit of URFD. These targeted controls would clarify how much of the improvement comes from future-derived training rather than the additional slot or training schedule.

The relationship between AndroidWorld training trajectories and evaluation tasks is unclear

open-source AndroidWorld trajectories

Source: Section 4.1, Datasets and evaluation; PDF p. 6; manuscript lines 291–297; quotation at 292.

Concern. The training mixture includes AndroidWorld trajectories from MobileForge and OpenMobile, while Table 2 evaluates AndroidWorld. The main body does not explain whether training and evaluation share task templates, instantiated goals, or environment initializations. Sharing a benchmark or a template is not by itself evidence of leakage, but without this provenance it is difficult to distinguish improved performance on familiar task families from generalization to unseen tasks or instances. This also affects interpretation of the online comparison if the SFT control's data mixture is not explicitly confirmed.

Suggested clarification or experiment. Please provide a compact source-by-source table giving training counts, task/template overlap, and any instance-level deduplication rules, and confirm that SFT uses the same training mixture. Where evaluated instances overlap, report a disjoint-instance subset; if unseen-template generalization is intended, also report the available disjoint-template subset. Otherwise, describe the result explicitly as adaptation within the evaluated task families.

The small online gain needs paired uncertainty information

75/116 tasks

Source: Section 4.2.2 and Table 2, Online benchmark results; PDF p. 7; manuscript lines 347–365; quotation at 349.

Concern. URFD’s 64.7% versus SFT’s 61.2% pass@3 corresponds to a net increase of four successful tasks among 116. The paper already identifies Medium tasks as the source of the aggregate improvement, while Easy and Hard rates are unchanged. However, the main body does not report paired task outcomes or variation across repeated evaluation batches, so the stability of the 3.5-point difference remains unclear. Paired outcomes are needed because equal aggregate rates can conceal task-level changes.

Suggested clarification or experiment. Please report paired per-task success outcomes and wins/losses under a shared evaluation protocol, together with a paired confidence interval or task-level bootstrap. If feasible, repeat the three-attempt evaluation with matched environment seeds to measure run variation; pass@1 from the same attempts would be a useful additional deployment measure. A brief analysis of the changed outcomes would help explain the practical source of the gain.

Questions for authors
  1. Which sign of the utility was implemented in the experiments, and can the authors show a numerical example reconciling Eq. (10), Figure 1, and the positive routing interval?

  2. What are the SFT control's data mixture, copied-slot configuration, and total update budget, and how does a matched current-only control compare?

  3. Which AndroidWorld task templates and instantiated evaluation goals are represented in the MobileForge/OpenMobile training data, and what performance remains on disjoint instances?

  4. What are the paired per-task wins and losses behind the net four-task AndroidWorld improvement, and how variable is this difference across repeated evaluations?

3 comments
AuthorsOpenReview ↗

Response to Reviewer ra3W (1/2)

We sincerely thank you for the very thorough review and the concrete suggestions, which have helped us improve the paper. We address each point below.


W1/Q1: Sign of the utility.

Thank you for catching this typo: the two terms in Eq. (10) were inadvertently swapped. The exact expression used in our implementation is

ui=1∣Ai∣∑j∈Ai[log⁡qi,jF(ai,j)−log⁡qi,jC(ai,j)]u_i = \frac{1}{|\mathcal{A_i}|}\sum_{j\in\mathcal{A_i}}[\log q^F_{i,j}(a_{i,j})-\log q^C_{i,j}(a_{i,j})]

i.e., the token-averaged increase in the log-likelihood of the demonstrated action when the real successor screen is visible. All of our implementation and experiments use this expression, so the analyses and results in Figure 2 and Table 3 do not need to be recomputed. Following the reviewer's suggestion, we provide a numerical sanity check with the routing interval [τstart,τfull]=[0,0.01][\tau_{\text{start}},\tau_{\text{full}}]=[0,0.01]:

CaseqC(a)q^C(a)qF(a)q^F(a)uuwwRouting
Successor helps0.500.60+0.1821.0Target the FutureState teacher
Successor helps slightly0.5000.503+0.0060.6Interpolate between the two teachers
Successor hurts0.500.45−0.1050Remain with the CopyState teacher

We will correct Eq. (10) and the corresponding formula in Figure 1 in the revision, so that the formula, figure, text, and routing interval are fully consistent.


W2/Q2: Matched current-only training control.

The training configurations of the SFT baseline and URFD are compared below; the two differ only in the stage-2 input schema and training objective:

SFTURFD
InitializationTASI checkpointTASI checkpoint
TASI stage790,994 examples, 1 epoch790,994 examples, 1 epoch
Stage-2 dataFour-source mixture (AC, GO, OM, MF), 208,766 transitionsFour-source mixture (AC, GO, OM, MF), 208,766 transitions
Epochs / exposure3 epochs (each transition seen 3 times)3 epochs (each transition seen 3 times)
Updates4,893 (batch size 128)4,893 (batch size 128)
OptimizationPeak LR 1e-5, cosine schedulePeak LR 1e-5, cosine schedule
InputCurrent screenshot + text history (I,Ht,St)(I,H_t,S_t), no copy slotAdditional 32-token CopyState slot (I,Ht,St,St)(I,H_t,S_t,S_t)
ObjectiveAction CEAction CE + hidden-state alignment + policy distillation

The results of the control suggested by the reviewer, in which a TASI-initialized student with the 32-token copy slot is continued with CE only for the same 3 epochs, are as follows (action match, %):

MethodInputAndroidControlGUI-OdysseyAverage
SFT(I,Ht,St)(I,H_t,S_t)76.7680.0678.41
SFT (CopyState)(I,Ht,St,St)(I,H_t,S_t,S_t)76.9280.2978.61
URFD(I,Ht,St,St)(I,H_t,S_t,S_t)77.9683.3880.67

The copy slot alone contributes only 0.20 points on average. With identical input, data, and number of updates, URFD still outperforms this control by 2.06 points, indicating that the gain stems from future distillation itself, not from the copy slot or the training budget. We will add the full SFT configuration and this control to Section 4.1 and Appendix B.


W3/Q3: Relationship between the AndroidWorld training data and the evaluation tasks.

The training counts by source are listed below; SFT and URFD use the same training mixture.

SourceDeduplicated goals in source poolSamples in source poolTraining episodesTraining steps
MobileForge1,69877,0839659,591
OpenMobile2,78427,3607998,772
Total4,482104,4431,76418,363

Deduplication and filtering rules: when converting the source trajectories, we removed 133,966 negative steps, 17,218 cross-split duplicates, and 10,314 steps without a successor frame. In addition, we removed tasks that could be instantiated from AndroidWorld task templates and required the cosine similarity between any retained training task and any evaluation task to be below 0.5.

(Continued in Part 2/2.)

AuthorsOpenReview ↗

Response to Reviewer ra3W (2/2)

Regarding the reviewer's two questions:

  • Any task or template overlap between training and evaluation? None. To verify this, we used AndroidWorld's own task classes to convert the templates of all 116 evaluation tasks into regular expressions (with wildcards at parameter positions) and matched them against all 1,764 training tasks; no training task matched any template (0/1,764). As a similarity reference, we instantiated each AndroidWorld evaluation template with 10 random seeds: two instances of the same template have a mean token-level Jaccard similarity of 0.663, whereas the nearest training task to each evaluation task has a mean similarity of only 0.417, and the mean similarity to all training tasks is 0.075.
  • What performance remains on disjoint instances? Since all 116 evaluation tasks are disjoint from the training data at both the template and the instance level, the results in Table 2 already reflect the performance on disjoint tasks, i.e., generalization to unseen task templates within the AndroidWorld environment.

W4/Q4: Paired outcomes and repeated evaluations.

Both models are evaluated under the same protocol (the same 116 task instances, step limit, and success criterion). In the evaluation reported in Table 2 (pass@3), URFD and SFT both succeed on 64 tasks and both fail on 34; URFD succeeds where SFT fails on 11 tasks, and the reverse holds on 7, yielding a net gain of 4 tasks. A task-level paired bootstrap (10,000 resamples) gives a success-rate difference of +3.45 points with a 95% confidence interval of [−3.4, +10.3]. The 18 tasks with changed outcomes are:

Tasks
URFD succeeds, SFT fails (11)SportsTrackerLongestDistanceActivity, TasksHighPriorityTasks, CameraTakeVideo, SimpleCalendarNextEvent, TurnOffWifiAndTurnOnBluetooth, SportsTrackerActivityDuration, SaveCopyOfReceiptTaskEval, AudioRecorderRecordAudioWithFileName, FilesMoveFile, SimpleCalendarAddOneEventInTwoWeeks, VlcCreatePlaylist
SFT succeeds, URFD fails (7)CameraTakePhoto, SportsTrackerActivitiesCountForWeek, MarkorDeleteNewestNote, BrowserMaze, RecipeDeleteMultipleRecipesWithNoise, ExpenseAddMultiple, RetroSavePlaylist

Among the changed outcomes, URFD benefits most on tasks that require navigating to a specific page before reading information (information-retrieval tasks: 4 wins vs. 1 loss, e.g., SimpleCalendarNextEvent, TasksHighPriorityTasks, and the SportsTracker tasks), which is attributed to its anticipation of the next state. On the longest multi-item entry or deletion tasks (e.g., ExpenseAddMultiple and RetroSavePlaylist), long-horizon state tracking remains the main difficulty.

URFD is designed to let the model anticipate the page that an action will lead to and to use this expectation to inform the current decision. To examine this, we inspected the trajectories of the 18 tasks with changed outcomes and recorded whether the failing model had reached the target page required to complete the task :

Failure caseTasksFailing model reached the target page
SFT fails, URFD succeeds114 (SFT)
URFD fails, SFT succeeds76 (URFD)

Most SFT failures occur before the target page is reached (only 4/11 reached it), i.e., they are navigation errors. In contrast, when URFD fails, it has almost always reached the target page (6/7), and the failure lies in the subsequent operations, such as entering or deleting multiple items. This is consistent with the training objective of URFD, which targets navigation decisions.

Following the reviewer's suggestion, we repeated the pass@3 evaluation three times. The number of solved tasks (out of 116) in each run is:

MethodRun 1Run 2Run 3Mean success rate (%)
Qwen3-VL-SFT71687260.63
URFD75737664.37
Difference (URFD − SFT)+4+5+4+3.74

URFD solved more tasks than SFT in all three repeat runs (by 4, 5, and 4 tasks, respectively), improving the mean success rate by 3.74 points. From the same attempts, pass@1 is 56.03% for SFT and 59.48% for URFD. The three repetitions consistently point in the same direction, and we therefore regard the online results as complementary evidence that for the effectiveness of our method.


Once again, we sincerely thank you for the time and effort devoted to reviewing our paper and for the valuable suggestions. If our responses have resolved your concerns, we would be very grateful if you could consider raising your score. If you have any further questions, we would be more than happy to engage in further discussion.

AuthorsOpenReview ↗

Summary of rebuttal to Reviewer ra3W

Dear Reviewer ra3W,

Thank you again for your thoughtful feedback. As the discussion period is nearing its end, we would be grateful if you could review our response.

We stated the exact utility expression with a numerical sanity check, reported the full SFT configuration, and added the matched copy-slot CE control you suggested, which URFD outperforms by 2.06 points. We also showed that the AndroidWorld training data contain no evaluation templates, and provided paired per-task outcomes, a confidence interval, a trajectory analysis, three repeated evaluations, and pass@1.

We would greatly appreciate your feedback. If our responses have addressed your concerns, we would be grateful if you could kindly consider raising your score; if not, we would be glad to discuss further. Thank you again for your time and valuable comments.

Reviewer_D6ZYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes transferring information from logged successor screenshots into a mobile GUI policy that uses only current observations at deployment. TASI first trains a shared model on two action-prediction views and task-/action-conditioned successor selection. URFD then distills a frozen teacher into a student with a small copied-screen slot, using demonstrated-action likelihood gains to route both hidden-state alignment and policy distillation. On Qwen3-VL-8B, the reported gains over current-only SFT are 1.20 and 3.32 percentage points in action match on AndroidControl and GUI-Odyssey, and 3.5 points in AndroidWorld pass@3.

Critical strengths
  1. Successor screenshots are already available in logged trajectories, and the deployed student adds only a small visual slot without generating future images. Routing supervision according to its usefulness for the demonstrated action is a sensible way to avoid treating every successor as equally informative.

  2. The improvements on both offline datasets are accompanied by an online evaluation and useful analyses of action categories, layer alignment, and slot size. The action-wise results also expose a stopping regression rather than suggesting that every behavior improves.

Critical weaknesses
  1. Equation (10) defines utility as the mean of log q^C(a) − log q^F(a). This is negative when the future view improves demonstrated-action likelihood, whereas Eq. (11) and the accompanying text route positive utility toward the future teacher. Appendix B instead describes a reduction in negative log-likelihood, which has the opposite sign. The routing evidence also needs clarification: Table 4 reports 83.52% match for uniform future distillation with MSE without clearly identifying the dataset and configuration. Its relationship to the GUI-Odyssey results of 83.38% for full URFD and 82.66% without routing in Table 7 is therefore unclear.

  2. TASI expands the training set to 790,994 examples, followed by three URFD epochs, but the SFT baseline’s corresponding optimization budget is not specified. Component gains are only about 0.5–0.8 points, without repeated-run uncertainty. Online, the 3.5-point gain corresponds to four additional successes among 116 tasks, while training includes AndroidWorld-derived trajectories. The submission does not explain the separation between those training tasks/instances and evaluation, or provide paired task outcomes and uncertainty. These omissions limit how confidently the gains can be attributed to utility routing and generalized beyond the reported runs.

Questions for authors
  1. Which sign of utility is actually used? Please also identify the configurations behind Table 4 and the no-routing row of Table 7, including any differences in initialization or training losses.
  2. What are the SFT baseline’s training budget and input configuration? Does the improvement persist against a CopyState control with matched optimization exposure, and how variable are the routing gains across runs?
  3. How are the OpenMobile/MobileForge training trajectories separated from AndroidWorld evaluation tasks and instances?
3 comments
AuthorsOpenReview ↗

Response to Reviewer D6ZY (1/2)

We sincerely thank you for the careful review and the constructive comments. We address each point below.


W1/Q1: Sign of the utility.

Thank you for catching this typo: the two terms in Eq. (10) were inadvertently swapped. The implementation and all experiments (including Figure 2, Table 3, and Table 7) use

ui=1∣Ai∣∑j∈Ai[log⁡qi,jF(ai,j)−log⁡qi,jC(ai,j)]u_i = \frac{1}{|\mathcal{A_i}|}\sum_{j\in\mathcal{A_i}}[\log q^F_{i,j}(a_{i,j})-\log q^C_{i,j}(a_{i,j})]

i.e., the token-averaged increase in the log-likelihood of the demonstrated action when the real successor screen is visible. This is consistent with the description in Appendix B (a reduction in negative log-likelihood). A positive uiu_i indicates that the successor helps predict the demonstrated action, which agrees with the routing direction of Eq. (11) and the routing interval [0,0.01][0, 0.01]. For example, with qC(a)=0.50q^C(a)=0.50 and qF(a)=0.60q^F(a)=0.60, we obtain u=+0.182u=+0.182 and w=1w=1, so the student is aligned with the FutureState teacher; with qF(a)=0.45q^F(a)=0.45, we obtain u=−0.105u=-0.105 and w=0w=0, so the student remains anchored to the CopyState teacher. We will correct Eq. (10) and the corresponding formula in Figure 1 in the revision.


W1/Q1: Configurations behind Table 4 and Table 7.

Table 4 reports a separate, earlier experiment conducted on GUI-Odyssey to compare alignment distances. All four settings use utility routing: after the full TASI stage, the second stage is trained for 5 epochs on the GUI-Odyssey training data only and evaluated on the GUI-Odyssey test set. The three configurations are summarized below.

InitializationStage-2 training dataEpochsUtility routingLossesAlignment distance
Table 4TASIGUI-Odyssey only5OnCE + align + KL4 variants
Table 1 (URFD)TASIFour-source mixture3OnCE + align + KLMSE
Table 7 (w/o routing)TASIFour-source mixture3Off (wi≡1w_i\equiv1)CE + align + KLMSE

The w/o-routing variant sets wi=1w_i=1 for all examples, so that both hidden-state alignment and policy distillation directly target the FutureState teacher; all other settings are identical to those of full URFD. Because the training data differ, the absolute numbers in Table 4 are not directly comparable to those in Tables 1 and 7. The effect of utility routing is given by the controlled comparison between Table 1 and Table 7 (83.38 vs. 82.66 on GUI-Odyssey). We are concerned that the phrase "under uniform Future distillation" in Section 4.4.3 may have caused a misunderstanding; we will revise it to "under the same utility-routed future distillation" and specify the GUI-Odyssey-only setting in the caption of Table 4.


W2/Q2: Training budget of the SFT baseline and an input-matched control.

The SFT baseline in Table 1 shares the initialization, training data, and training budget of URFD. Both models first go through the same TASI stage (790,994 examples, 1 epoch) and are then trained for 3 epochs on the same four-source mixture (AndroidControl, GUI-Odyssey, OpenMobile, and MobileForge; 208,766 transitions in total) with a batch size of 128, i.e., 4,893 updates, using a peak learning rate of 1e-5 and a cosine schedule. They differ only in the stage-2 input and training objective: SFT uses the current-only input with action cross-entropy, whereas URFD uses the CopyState input and adds utility-routed hidden-state alignment and policy distillation.

To further rule out the effect of the input structure, we added the matched control suggested by the reviewer, SFT (CopyState). Starting from the same TASI checkpoint, it uses exactly the same input as URFD (the current screenshot plus a 32-token CopyState slot), the same data, and the same 4,893 updates, but is trained with action cross-entropy only. The results (action match, %) are as follows:

MethodInputAndroidControlGUI-OdysseyAverage
SFT(I,Ht,St)(I,H_t,S_t)76.7680.0678.41
SFT (CopyState)(I,Ht,St,St)(I,H_t,S_t,S_t)76.9280.2978.61
URFD(I,Ht,St,St)(I,H_t,S_t,S_t)77.9683.3880.67

The CopyState slot alone contributes only 0.20 points on average. With identical input, data, and number of updates, URFD still outperforms this control by 2.06 points, indicating that the gain stems mainly from future distillation itself, not the additional image patches. We will add the full SFT configuration and this control to Section 4.1 and Appendix B.

(Continued in Part 2/2.)

AuthorsOpenReview ↗

Response to Reviewer D6ZY (2/2)

W2/Q2: Variability of the component gains across runs.

We repeated each ablation configuration five times and report the mean and standard deviation of the average action match over the two offline benchmarks (%):

MethodAvg. action matchΔ
URFD80.65 ± 0.11—
w/o utility routing80.12 ± 0.14−0.53
w/o successor initialization80.15 ± 0.13−0.50
w/o hidden alignment80.15 ± 0.13−0.50
w/o policy KL80.03 ± 0.11−0.62
w/o copy-state CE79.91 ± 0.10−0.74

The means over repeated runs are consistent with Table 3, and the drop caused by removing each component (0.50–0.74 points) is several times larger than the standard deviation across runs (0.10–0.14), indicating that these drops reflect the genuine contribution of each component rather than random variation at test time. Paired bootstrap tests further confirm that all differences are significant at p < 0.01.


W2/Q3: Separation between the AndroidWorld-derived training data and the evaluation tasks.

The AndroidWorld-related training data come from the open-source trajectories of MobileForge (965 episodes, 9,591 steps) and OpenMobile (799 episodes, 8,772 steps), accounting for 8.8% of the 208,766 training transitions; SFT and URFD use exactly the same data. When constructing the training set, we removed cross-split duplicates and tasks that could be instantiated from AndroidWorld task templates, and required the cosine similarity between any retained training task and any evaluation task to be below 0.5. We further verified the separation with two checks:

  • Template matching. Using AndroidWorld's own task classes, we converted the templates of all 116 evaluation tasks into regular expressions (with wildcards at parameter positions) and matched them against all 1,764 deduplicated training goals from the two sources. No training tasks matched any template (0/1,764); hence, the training data contain no task that can be instantiated from an evaluation template.
  • Similarity reference. For each AndroidWorld evaluation template, we instantiated the goal with 10 random seeds; two instances of the same template have a mean token-level Jaccard similarity of 0.663. In contrast, the nearest training task to each evaluation task has a mean similarity of only 0.417, and the mean similarity to all training tasks is 0.075. Even the most similar training task is therefore clearly farther away than another instance of the same template.

Consequently, all 116 evaluation tasks are disjoint from the training data at both the template and the instance level, and the results in Table 2 are free from train–test contamination. Since SFT and URFD are trained on the same data, the difference between them reflects the training objective.


W2: Paired task outcomes and uncertainty on AndroidWorld.

Both models are evaluated under the same protocol (the same 116 task instances, step limit, and success criterion). The paired per-task outcomes of the evaluation reported in Table 2 (pass@3) are shown below; the net gain of four tasks results from 11 wins and 7 losses.

SFT successSFT failure
URFD success6411
URFD failure734

To measure the variation across evaluation rounds, we repeated the pass@3 evaluation three times with the same environment seeds. The number of solved tasks (out of 116) in each run is:

MethodRun 1Run 2Run 3Mean success rate (%)
Qwen3-VL-SFT71687260.63
URFD75737664.37
Difference (URFD − SFT)+4+5+4+3.74

URFD solved more tasks than SFT in all three repeat runs (by 4, 5, and 4 tasks, respectively), improving the mean success rate by 3.74 points.


Once again, we sincerely thank you for the time and effort devoted to reviewing our paper and for the valuable suggestions. If our responses have resolved your concerns, we would be very grateful if you could consider raising your score. If you have any further questions, we would be more than happy to engage in further discussion.

AuthorsOpenReview ↗

Summary of rebuttal to Reviewer D6ZY

Dear Reviewer D6ZY,

Thank you again for your thoughtful feedback. As the discussion period is nearing its end, we would be grateful if you could review our response.

We clarified the sign typo in Eq. (10) and the configurations behind Tables 4 and 7, and added an input- and budget-matched SFT control, which URFD outperforms by 2.06 points. We also reported repeated runs for the ablations, verified that the AndroidWorld training data do not overlap with the evaluation tasks, and showed that URFD outperforms SFT in all three repeated AndroidWorld evaluations.

We would greatly appreciate your feedback. If our responses have addressed your concerns, we would be grateful if you could kindly consider raising your score; if not, we would be glad to discuss further. Thank you again for your time and valuable comments.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.