acceptodds
Under review as a conference paper at ICLR 2027

Trace2World: Reconstructing App Worlds from Real Interaction Traces for Scalable GUI Agent Training

55 views

Abstract

Scaling GUI reinforcement learning requires environments that preserve real-app behavior. Yet coupling behavior inference with coding can turn partial traces into runnable mocks with incorrect state transitions. We introduce Trace2World, which separates global behavior reconstruction from implementation. It reconciles traces into a shared, source-linked app model, then compiles an executable environment. Independent replay of source workflows localizes mismatches for bounded implementation repair. Validated worlds then support learner-adaptive task and environment augmentation within source-supported behavior. From 1,570 traces, Trace2World constructs 50 environments. On 200 workflows across ten apps, 91.1% are reproduced end to end, versus 59.8% for the strongest matched task-local baseline. Source-grounded validation retains 94.0% of simulated training gains on real devices, versus 54.7% under self-validation. Learner guidance raises benchmark success from 56.9% to 61.1%. Training in Trace2World further improves the model by 20.2 points on our benchmark, 5.4–16.2 points across four external GUI benchmarks, and 16.3 points on 100 real-device tasks. Reliable reconstruction separates what behavior source evidence supports from whether the implementation preserves it. The code and resources are released.

Primary area:
datasets and benchmarks
Keywords:
reinforcement learning, autonomous agents, GUI agents, environment generation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_n3VZOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Trace2World reconstructs executable GUI environments from real application interaction traces. The method separates semantic reconstruction from implementation: it first reconciles traces into a shared, source-linked application model, then compiles that model into an executable environment. Independent replay identifies discrepancies, while a bounded repair process modifies the implementation without changing the underlying semantic model. The validated environments subsequently support learner-guided task generation and reinforcement learning.

The paper constructs 50 application environments from 1,570 traces and reports 91.1% workflow fidelity on 200 workflows, compared with 59.8% for the strongest matched baseline. It also evaluates downstream policy performance, external benchmark transfer, and paired simulator–device outcomes.

Critical strengths

S1: The paper identifies an important distinction between executable environments and faithful reconstructions. Separating semantic reconstruction, implementation, source-based validation, and training rewards is a well-motivated response to this problem.

S2: The evaluation is extensive and includes meaningful checks on the automated reviewer. The fidelity study examines reachability, surface, state, effects, terminal outcomes, and data rather than relying solely on task completion.

S3: The reconstructed environments demonstrate practical training value. The paper reports gains with a fixed policy backbone, improvements on four external GUI benchmarks, and transfer to real devices.

Critical weaknesses

W1: Task-conditioned interaction restrictions complicate the interpretation of environment fidelity and training difficulty. Under Appendices A.2 and D.2, a visible component outside the current task binding or implemented scope returns NOT INTERACTIVE. Rejecting unsupported behavior is reasonable, but rejecting an implemented, otherwise legal action merely because it is unrelated to the current task differs from how a real application behaves. Such restrictions may reduce the exploration space and make off-task actions easier to identify. The real-device results establish that the environments are useful, but do not quantify the contribution of this restriction. The paper should distinguish unsupported actions from supported but task-unbound actions and evaluate the effect of allowing the latter.

W2: Generalization evidence is limited. Fidelity is evaluated on construction-exposed workflows, and “unseen” applications are held out from policy training but still provide construction traces. Real-device evaluation covers only policy-training applications. Testing workflows withheld from construction or real-device transfer on policy-unseen applications would strengthen the generalization claims.

Questions for authors
  1. How much would fidelity change if cross-trace semantic reconciliation or semantic freezing were removed, while keeping compilation, independent validation, and repair unchanged?

  2. What happens when implemented, legal controls remain interactive even if they are not bound to the current task? How does this affect success rates, invalid-action rates, and training gains?

  3. Are there results on real workflows withheld from environment construction, or paired real-device results on policy-unseen applications?

Reviewer_96sTOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies how to reconstruct reusable, resettable environments for GUI-agent training from partial real-app interaction traces. Trace2World reconciles traces into a shared app model, and then compiles an executable browser environment. It also supports learner adaptive augmentation during reinforcement learning process.

Critical strengths
  1. The separated manner of behavior reconstruction and code/environment reconstruction helps to restrict the implementation process by coding agents, making it more controllable and produce environments more loyal to the source traces.
  2. The idea of learner-adaptive task and environment augmentation is sound, which could produce targted tasks for learner's weaknesses. It shows generalization abilities.
Critical weaknesses
  1. One of the paper's core contributions is the importance of performing prior semantic behavior reconstruction, so that later stages could utilize it as the frozen contract to make sure the implementation is well-controlled. But I don't see the experiment that demonstrates how much better it behaves. For example, I would like to see whether this reconstruction only could lead to better fidelity (or lower cost to achieve the same fidelity) of the built environments, as well as the downstream RL results, etc. Ablations for this should be considered.
  2. Another limitation is from the "scalable" perspective. The paper does show that Trace2World supports providing many tasks from an environment, while they could be limited to "what the traces support" because of the hard restriction from the pre-built semantic app model. This might be an advantage in achieving better fidelity, but to some extent this is a shortcoming for reinforcement learning, because it can't provide tasks that can't be supported by source traces. Besides, since only a part of functionalities are implemented, it may lowe the possibility of getting lost in unrelevant pages, which somehow could overrestimate the model's performance. This could be checked and validated.
  3. The writing of the Experiments section is a little bit "crowded". I admit the authors perform extensive experiments and results, but the current compilation is more like many indices to the appendix, lacking clear statements of the most important conclusions, findings and relevant detailed analysis.
Questions for authors
  1. How are the execution traces for Table 1 collected? Who operates the environments, what source-trace information is available to the executor, and how are executor failures distinguished from environment defects?
  2. Of the 768 learner-guided tasks, how many only change entities or parameters, and how many introduce workflow or dependency structures absent from the seed tasks?
  3. The Doubao seed models have good performance on GUI tasks. If the time and effort are allowed, I suggest also adding evaluations with it as another proprietary model.
Reviewer_F2n4OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

Trace2World reconstructs resettable browser-based app environments from real user-interaction traces for GUI-agent training. It reconciles multiple traces into a shared, evidence-linked behavioral model, compiles that model into an interactive environment, and checks fidelity by replaying source workflows. The system uses 1,570 real traces to build 50 app worlds. The paper reports 91.1% complete fidelity on 200 source workflows from ten apps, compared with 59.8% for its strongest end-to-end baseline, and studies policy transfer with paired simulator/real-device tasks. In the 100-task paired study, source-validated training preserves a larger fraction of simulation improvement on real devices than self-validated training.

Critical strengths

1.A consequential reconstruction objective. The authors correctly distinguish matching terminal screens from reproducing intermediate transitions, guards, confirmations, and state changes. The shared behavior model and source-replay validation provide a plausible way to prevent visually convincing but behaviorally incorrect training environments.

  1. Evidence beyond in-simulator success. The paired simulator/real-device comparison is a stronger validation design than relying on simulator rewards alone. In the reported comparison, source-validated training preserves 15.7 of 16.7 percentage points of simulated improvement on devices (94.0%), versus 9.7 of 17.7 points for self-validation (54.7%).
Critical weaknesses
  1. Source-workflow fidelity is not evidence of full dynamics fidelity. The 91.1% figure measures reconstruction on recorded source workflows; it does not directly measure previously unobserved state combinations, actions, or branches. The approach deliberately makes unsupported behavior inert, which may be defensible but can also leave important legitimate task behavior unavailable. This limits the scope of the paper's fidelity and coverage claims.

  2. The separate contributions of global reconciliation and validation remain partially confounded. The principal comparisons are between end-to-end reconstruction pipelines. Source validation has reasonably strong evidence, but the incremental benefit of globally reconciling multiple traces, independently of using the same source traces and validator, is less clear. The real-device study also uses policy-training apps, so it is not a direct test on entirely unseen real applications.

Questions for authors

Fidelity or coverage clarification: What fraction of requested actions or transitions in the generated training tasks are unsupported or made inert? Can the authors report any evaluation of transitions or state combinations not present in the source traces?

Ablation: With the same source traces, replay validator, and compute budget, how does a task-local reconstruction approach perform without cross-trace global reconciliation?

Real-device generalization: Can the paired-device results be reported with app-clustered confidence intervals or per-app breakdowns? Are any real-device tasks drawn from applications unseen by both the environment builder and the policy learner?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.